Release Notes - Apache Avro - Version 1.12.0

Sub-task

• [AVRO-3122]: TestAvroKeyOutputFormat and other avro-mapred tests fail with Java 17
• [AVRO-3308]: Include a curated list of resources
• [AVRO-3384]: Define C# Coding Style Guidelines
• [AVRO-3449]: Add an onboarding guide for contributors, committers and PMC
• [AVRO-3458]: Add test coverage for GenericRecord
• [AVRO-3488]: Fix Spelling Mistakes
• [AVRO-3490]: Fix IDE0016 Use throw expression
• [AVRO-3491]: Fix IDE0020 Use pattern matching to avoid 'is' check followed by a cast
• [AVRO-3497]: Fix IDE0075 Simplify conditional expression
• [AVRO-3499]: Fix IDE0079 Remove unnecessary suppression
• [AVRO-3538]: Improve the contributions page
• [AVRO-3700]: Publish Java SBOM artifacts with CycloneDX
• [AVRO-3813]: Use list of primitiv
• [AVRO-3826]: Commons test for C++ module
• [AVRO-3916]: Add nanos support for the Rust SDK
• [AVRO-3926]: [Rust] Allow UUID to serialize to Fixed[16]

Bug

• [AVRO-265]: Protocol namespace always written out in toJson
• [AVRO-1318]: Python schema should store fingerprints
• [AVRO-1463]: Undefined values cause warnings when unions with null serialized
• [AVRO-1517]: Unicode strings are accepted as bytes and fixed type by perl API
• [AVRO-1521]: Inconsistent behavior of Perl API with 'boolean' type
• [AVRO-1523]: Perl API: int/long type minimum value checks are off by one
• [AVRO-1737]: Unhashable type: 'RecordSchema'
• [AVRO-1830]: Avro-Perl DataFileReader chokes when avro.codec is absent
• [AVRO-2254]: Unions with 2 records declared downward fail
• [AVRO-2284]: Incorrect EnumSymbol initialization in TestReadingWritingDataInEvolvedSchemas.java
• [AVRO-2498]: UUID generation is not working avro 1.9 version
• [AVRO-2598]: C++ standard of library implies C++ standard of projects using Avro
• [AVRO-2722]: impl/DataFile.cc use of boost::mt19937 for DataFileWriteBase::makeSync is not thread safe
• [AVRO-2771]: Java 1.9.X doesn't allow having Error in a Record
• [AVRO-2862]: C# Primitive Schema losing metadata
• [AVRO-2883]: Avrogen (csharp) namespace mapping missing for references
• [AVRO-2885]: Providing a decimal number in an int field doesn't return an error
• [AVRO-2943]: Map comparison between Utf8 and String keys fails
• [AVRO-2987]: pkg-config has a broken Requires: section
• [AVRO-3003]: c# apache avro codegen - default value for enum types are not setting up properly
• [AVRO-3133]: EnumAdjust.resolve should compare unqualified name rather than full name
• [AVRO-3216]: Rust: failure reading multiple use of named schemas in file
• [AVRO-3232]: Rust deserializer: add missing matches to deserialize_any union and string/map
• [AVRO-3234]: Rust: Add new codec: zstandard
• [AVRO-3240]: Schema deserialization is not backwards compatible
• [AVRO-3259]: When opening an avro file which is encoded with anything besides none and deflate, it defaults to none and then returns garbage.
• [AVRO-3273]: [Java] avro-maven-plugin breaks on old versions of Maven
• [AVRO-3316]: [Rust] build breaks in docker build
• [AVRO-3322]: JavaScript: Buffer is not defined in browser environment
• [AVRO-3331]: Rust: Cannot extract Decimal value
• [AVRO-3350]: Validate that Default value is found in Enum
• [AVRO-3386]: [PHP] Build failing on github and travis
• [AVRO-3410]: [Rust] lint failure
• [AVRO-3433]: Rust: The canonical form should preserve schema references
• [AVRO-3448]: Rust: Encoding Panic with valid schema and input
• [AVRO-3452]: [rust] Derive Deserialize produces invalid Name struct
• [AVRO-3460]: [rust] Value::validate does not validate against Schema Refs
• [AVRO-3461]: [rust] Resolution Flow does not handle schema Refs
• [AVRO-3466]: Rust: serialize Schema to JSON loses inner namespace names
• [AVRO-3468]: Default values for logical types not supported
• [AVRO-3471]: Microseconds logical types are rounded to milliseconds
• [AVRO-3481]: Input and output variable type mismatch
• [AVRO-3482]: DataFileReader should reuse MAGIC data read from inputstream
• [AVRO-3486]: Protocol namespace not parsed correctly if protocol is defined by full name
• [AVRO-3495]: Rust: Record serialization is sensitive to order of fields in struct
• [AVRO-3511]: Rust: Fix the parsing of decimal logical type
• [AVRO-3516]: [rust] Avro Derive not working outside of repo context
• [AVRO-3529]: [Rust][branch-1.11] Cargo.toml is a virtual manifest, requires actual package
• [AVRO-3534]: Rust: Use dependency-review-action only for pull_request events
• [AVRO-3536]: Union type not inheriting type conversions
• [AVRO-3549]: [rust] Avro reader fails if it tries to read data compressed with codec that is not enabled in features
• [AVRO-3560]: avro ignores input after end of avsc json
• [AVRO-3568]: C# ToParsingForm normalizes logical type to "logical" rather than base type
• [AVRO-3581]: Usage of deprecated configuration properties in Velocity
• [AVRO-3585]: Unable to encode Value::String as Schema::UUID
• [AVRO-3587]: C: Fix possible heap-buffer-overflow in avro::DataFileReaderBase::readDataBlock()
• [AVRO-3595]: Release Notes missing for 1.11.1
• [AVRO-3597]: Recent changes in GenericDatumReader.java break compatibility
• [AVRO-3601]: C++ API header contains breaking include
• [AVRO-3612]: Report specific location of incompatibility in record schema
• [AVRO-3613]: Unions cannot have more than one logical type in C#
• [AVRO-3617]: [C++] Integer overflow risks with Validator::count_ and Validator::counters_
• [AVRO-3618]: [Java] TestBinaryDecoder should check consistency with directBinaryDecoder
• [AVRO-3619]: [Java] TestBinaryDecoder should check consistency with directBinaryDecoder
• [AVRO-3622]: Python compatibility check fails if record with and without namespace are compared
• [AVRO-3625]: [Rust] UnionSchema.is_nullable() should return true if any of the variants is Schema::Null
• [AVRO-3631]: Fix serialization of structs containing Fixed fields
• [AVRO-3632]: Union defaults are not handled as per the specification
• [AVRO-3642]: GenericSingleObjectReader::read_value fails on non-exhaustive read
• [AVRO-3645]: Fix deserialization of enum with unit () type
• [AVRO-3650]: Fix C++ Build on Manjaro
• [AVRO-3656]: Vulnerabilities from dependencies - jackson-databind & commons-text
• [AVRO-3657]: Computation of initial buffer size in OutputBuffer makes no sense
• [AVRO-3659]: Typo in python example
• [AVRO-3662]: [Ruby] Ruby 2.6 CI workflow fails since a while
• [AVRO-3663]: rust crate apache_avro_derive creates invalid schemas for raw identifiers
• [AVRO-3667]: [Python] Python 3.10 CI test fails since a while
• [AVRO-3669]: Missing py.typed file
• [AVRO-3674]: Value::Record containing enums fail to validate when using namespaces in Schema
• [AVRO-3683]: Rust Writer, Reader can't use Schemas with dependencies in other Schemas. i.e. The output of Schema::parse_list
• [AVRO-3687]: Rust enum missing default
• [AVRO-3688]: Schema resolution panics when a custom record field is included multiple times
• [AVRO-3698]: [Java] SpecificData.getClassName must replace reserved words
• [AVRO-3706]: AVDL nested imports cannot be resolved if path contains spaces
• [AVRO-3712]: C++ Build Failure on Manjaro
• [AVRO-3724]: C# JsonEncoder can't handle nested array of records
• [AVRO-3737]: [C] memcheck_test_avro_commons_schema is failing
• [AVRO-3738]: [Build][C#] The release build fails with .NET 7.0 target
• [AVRO-3747]: Make serde is_human_readable configurable
• [AVRO-3748]: issue with DataFileSeekableInput.SeekableInputStream.skip
• [AVRO-3749]: incorrect conflicting field when field name starts with symbols
• [AVRO-3751]: FastReaderBuilder in multithread lead to infinite loop also blocking other threads
• [AVRO-3755]: [Rust] Deserialization fails for reader schema with namespace
• [AVRO-3756]: Support writing types back to the user in memory without writing files to disk
• [AVRO-3767]: [Rust] Fix ref resolving in Union
• [AVRO-3772]: [Rust] Deserialize Errors for an Unknown Enum Symbol instead of Returning Default
• [AVRO-3773]: [Ruby] Decimal logical type fail to validate default
• [AVRO-3775]: [Ruby] decimal default is not converted to BigDecimal
• [AVRO-3780]: [Rust] Bug: decimal logical type usage through Fixed schema
• [AVRO-3782]: [Rust] Incorrect decimal resolving
• [AVRO-3785]: [Rust] Deserialization if reader schema has a namespace and a union with null and a record containing a reference type
• [AVRO-3786]: [Rust] Deserialization results in FindUnionVariant error if the writer and reader have the same symbol but at different positions
• [AVRO-3787]: [Rust] Deserialization fails to use default if an enum in a record in a union is given an unknown symbol
• [AVRO-3800]: profile section should be declared in the root package.
• [AVRO-3809]: Faulty validation of a type reference with implicit nested namespace
• [AVRO-3814]: [Rust] Schema resolution fails when extending a nested record with a union type
• [AVRO-3818]: Enclosing namespace should be inherited to the inner named types if they have no their own namespaces
• [AVRO-3820]: Don't allow invalid field names
• [AVRO-3821]: Rust: Record (de?)serialization is sensitive to order of fields in struct
• [AVRO-3823]: Show helpful error messages
• [AVRO-3824]: The instruction for building the website should be more precise
• [AVRO-3827]: Disallow duplicate field names
• [AVRO-3830]: Handle namespace properly if a name starts with dot
• [AVRO-3837]: Disallow invalid namespaces for the Rust binding
• [AVRO-3846]: Race condition can happen among serde tests
• [AVRO-3847]: Record field doesn't accept default value if field type is union and the type of default value is pre-defined name
• [AVRO-3849]: [Rust] "make readme" doesn't work
• [AVRO-3855]: [rust] lint/clippy fails in ubertool
• [AVRO-3858]: [Build] Add some config to ./build.sh sign
• [AVRO-3859]: [Build][C#] build.sh ...

The Apache Avro community is pleased to announce the release of Avro 1.11.3!

This release addresses 39 Avro JIRA.

All signed release artifacts, signatures and verification instructions can
be found here: https://avro.apache.org/releases.html

This is a minor release, specifically addressing known issues with the 1.11.2 release, but also contains version bumps and doc fixes[1].

Language SDK / Convenience artifacts

In addition, language-specific release artifacts are available:

• C#: https://www.nuget.org/packages/Apache.Avro/1.11.3
• Java: https://repo1.maven.org/maven2/org/apache/avro/avro/1.11.3/
• Javascript: https://www.npmjs.com/package/avro-js/v/1.11.3
• Perl: https://metacpan.org/release/Avro
• Python 3: https://pypi.org/project/avro/1.11.3
• Ruby: https://rubygems.org/gems/avro/versions/1.11.3
• Rust: https://crates.io/crates/apache-avro/0.16.0

Thanks to everyone for contributing!

The Apache Avro community is pleased to announce the release of Avro 1.11.2!

This release addresses 89 Avro JIRA.

Highlights

C#

• AVRO-3434: Support logical schemas in reflect reader and writer
• AVRO-3670: Add NET 7.0 support
• AVRO-3724: Fix C# JsonEncoder for nested array of records
• AVRO-3756: Add a method to return types instead of writing them to disk

C++

• AVRO-3601: C++ API header contains breaking include
• AVRO-3705: C++17 support

Java

• AVRO-2943: Add new GenericData String/Utf8 ARRAY comparison test
• AVRO-2943: improve GenericRecord MAP type comparison
• AVRO-3473: Use ServiceLoader to discover Conversion
• AVRO-3536: Inherit conversions for Union type
• AVRO-3597: Allow custom readers to override string creation
• AVRO-3560: Throw SchemaParseException on dangling content beyond end of schema
• AVRO-3602: Support Map(with non-String keys) and Set in ReflectDatumReader
• AVRO-3676: Produce valid toString() for UUID JSON
• AVRO-3698: SpecificData.getClassName must replace reserved words
• AVRO-3700: Publish Java SBOM artifacts with CycloneDX
• AVRO-3783: Read LONG length for bytes, only allow INT sizes
• AVRO-3706: accept space in folder name

Python

• AVRO-3761: Fix broken validation of nullable UUID field
• AVRO-3229: Raise on invalid enum default only if validation enabled
• AVRO-3622: Fix compatibility check for schemas having or missing namespace
• AVRO-3669: Add py.typed marker file (PEP561 compliance)
• AVRO-3672: Add CI testing for Python 3.11
• AVRO-3680: allow to disable name validation

Ruby

• AVRO-3775: Fix decoded default value of logical type
• AVRO-3697: Test against Ruby 3.2
• AVRO-3722: Eagerly initialize instance variables for better inline cache hits

Rust

• Many, many bug fixes and implementation progress in this experimental SDK.
• Rust CI builds and lints are passing, and has been released to crates.io as version 0.15.0

In addition:

• Upgrade dependencies to latest versions, including CVE fixes.
• Testing and build improvements.
• Performance fixes, other bug fixes, better documentation and more...

Language SDK / Convenience artifacts

• C#: https://www.nuget.org/packages/Apache.Avro/1.11.2
• Java: https://repo1.maven.org/maven2/org/apache/avro/avro/1.11.2/
• Javascript: https://www.npmjs.com/package/avro-js/v/1.11.2
• Perl: https://metacpan.org/release/Avro
• Python 3: https://pypi.org/project/avro/1.11.2
• Ruby: https://rubygems.org/gems/avro/versions/1.11.2
• Rust: https://crates.io/crates/apache-avro/0.15.0

Known issues

• AVRO-3789 [Java]: Problem when comparing empty MAP types.

Thanks to everyone for contributing!

The Apache Avro community is pleased to announce the release of Avro 1.11.1!

Most interesting

This release includes 256 Jira issues, including some interesting features:

Avro specification

• AVRO-3436 Clarify which names are allowed to be qualified with namespaces
• AVRO-3370 Inconsistent behaviour on types as invalid names
• AVRO-3275 Clarify how fullnames are created, with example
• AVRO-3257 IDL: add syntax to create optional fields
• AVRO-2019 Improve docs for logical type annotation

C++

• AVRO-2722 Use of boost::mt19937 is not thread safe

C#

• AVRO-3383 Many completed subtasks for modernizing C# coding style
• AVRO-3481 Input and output variable type mismatch
• AVRO-3475 Enforce time-millis and time-micros specification
• AVRO-3469 Build and test using .NET SDK 7.0
• AVRO-3468 Default values for logical types not supported
• AVRO-3467 Use oracle-actions to test with Early Access JDKs
• AVRO-3453 Avrogen Add Generated Code Attribute
• AVRO-3432 Add command line option to skip creation of directories
• AVRO-3411 Add Visual Studio Code Devcontainer support
• AVRO-3388 Implement extra codecs for C# as seperate nuget packages
• AVRO-3265 avrogen generates uncompilable code when namespace ends
  with ".Avro"
• AVRO-3219 Support nullable enum type fields

Java

• AVRO-3531 GenericDatumReader in multithread lead to infinite loop
• AVRO-3482 Reuse MAGIC in DataFileReader
• AVRO-3586 Make Avro Build Reproducible
• AVRO-3441 Automatically register LogicalTypeFactory classes
• AVRO-3375 Add union branch, array index and map key "path"
  information to serialization errors
• AVRO-3374 Fully qualified type reference "ns.int" loses namespace
• AVRO-3294 IDL parsing allows doc comments in strange places
• AVRO-3273 avro-maven-plugin breaks on old versions of Maven
• AVRO-3266 Output stream incompatible with MagicS3GuardCommitter
• AVRO-3243 Lock conflicts when using computeIfAbsent
• AVRO-3120 Support Next Java LTS (Java 17)
• AVRO-2498 UUID generation is not working

Javascript

• AVRO-3489 Replace istanbul with nyc for code coverage
• AVRO-3322 Buffer is not defined in browser environment
• AVRO-3084 Fix JavaScript interop test to work with other languages on CI

Perl

• AVRO-3263 Schema validation warning on invalid schema with a long field

Python

• AVRO-3542 Scale assignment optimization
• AVRO-3521 "Scale" property from decimal object
• AVRO-3380 Byte reading in avro.io does not assert bytes read
• AVRO-3229 validate the default value of an enum field
• AVRO-3218 Pass LogicalType to BytesDecimalSchema

Ruby

• AVRO-3277 Test against Ruby 3.1

Rust

• AVRO-3558 Add a demo crate that shows usage as WebAssembly
• AVRO-3526 Improve resolving Bytes and Fixed from string
• AVRO-3506 Implement Single Object Writer
• AVRO-3507 Implement Single Object Reader
• AVRO-3405 Add API for user-provided metadata to file
• AVRO-3339 Rename crate from avro-rs to apache-avro
• AVRO-3479 Derive Avro Schema macro

Website

• AVRO-2175 Website refactor
• AVRO-3450 Document IDL support in IDEs

Rust

This is the first release that provides the apache-avro crate at crates.io!

JIRA

A list of all JIRA tickets fixed in 1.11.1 could be found here

Language repositories

In addition, language-specific release artifacts are available:

• C#: https://www.nuget.org/packages/Apache.Avro/1.11.1
• Java: https://repo1.maven.org/maven2/org/apache/avro/avro/1.11.1/
• Javascript: https://www.npmjs.com/package/avro-js/v/1.11.1
• Perl: https://metacpan.org/release/Avro
• Python 3: https://pypi.org/project/avro/1.11.1
• Ruby: https://rubygems.org/gems/avro/versions/1.11.1
• Rust: https://crates.io/crates/apache-avro/0.14.0

Thanks to everyone for contributing!

The Apache Avro community is pleased to announce the release of Avro 1.11.0!

All signed release artifacts, signatures and verification instructions can
be found here: https://avro.apache.org/releases.html

This release includes 120 Jira issues, including some interesting features:

Specification: AVRO-3212 Support documentation tags for FIXED types
C#: AVRO-2961 Support dotnet framework 5.0
C#: AVRO-3225 Prevent memory errors when deserializing untrusted data
C++: AVRO-2923 Logical type corrections
Java: AVRO-2863 Support Avro core on android
Javascript: AVRO-3131 Drop support for node.js 10
Perl: AVRO-3190 Fix error when reading from EOF
Python: AVRO-2906 Improved performance validating deep record data
Python: AVRO-2914 Drop Python 2 support
Python: AVRO-3004 Drop Python 3.5 support
Ruby: AVRO-3108 Drop Ruby 2.5 support

For the first time, the 1.11.0 release includes experimental support for
Rust. Work is continuing on this donated SDK, but we have not versioned and
published official artifacts for this release.

Python: The avro package fully supports Python 3. We will no longer publish a
separate avro-python3 package

And of course upgraded dependencies to latest versions, CVE fixes and more:
https://issues.apache.org/jira/issues/?jql=project%3DAVRO%20AND%20fixVersion%3D1.11.0

The link to all fixed JIRA issues and a brief summary can be found at:
https://github.com/apache/avro/releases/tag/release-1.11.0

In addition, language-specific release artifacts are available:

• C#: https://www.nuget.org/packages/Apache.Avro/1.11.0
• Java: from Maven Central,
• Javascript: https://www.npmjs.com/package/avro-js/v/1.11.0
• Perl: https://metacpan.org/release/Avro
• Python 3: https://pypi.org/project/avro/1.11.0
• Ruby: https://rubygems.org/gems/avro/versions/1.11.0

Thanks to everyone for contributing!

The Apache Avro community is pleased to announce the release of Avro 1.10.2!

All signed release artifacts, signatures and verification instructions can
be found here: https://avro.apache.org/releases.html

This release includes 31 Jira issues, including some interesting features:

C#: AVRO-3005 Support for large strings
C++: AVRO-3031 Fix for reserved keywords in generated code
Java: AVRO-2471 Fix for timestamp-micros in generated code
Java: AVRO-3060 Support ZSTD level and bufferpool options
Ruby: AVRO-2998 Records with symbol keys validation
Ruby: AVRO-3023 Validate with Ruby 3

Migration notes:
Python: AVRO-2656 The standard avro package supports Python 3, and
the avro-python3 package is in the process of being deprecated. 

And of course upgraded dependencies to latest versions, CVE fixes and more:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.10.2

The link to all fixed JIRA issues and a brief summary can be found at:
https://github.com/apache/avro/releases/tag/release-1.10.2

In addition, language-specific release artifacts are available:

• C#: https://www.nuget.org/packages/Apache.Avro/1.10.2
• Java: from Maven Central,
• Javascript: https://www.npmjs.com/package/avro-js/v/1.10.2
• Perl: https://metacpan.org/release/Avro
• Python 3: https://pypi.org/project/avro/1.10.2/
• Ruby: https://rubygems.org/gems/avro/versions/1.10.2

Thanks to everyone for contributing!

The Apache Avro community is pleased to announce the release of Avro 1.10.1!

All signed release artifacts, signatures and verification instructions can be
found here: https://avro.apache.org/releases.html

This release includes 33 Jira issues, including some interesting features:

C#: AVRO-2750 Support for enum defaults
C++: AVRO-2891 Expose last sync offset written on DataFileWriter
Java: AVRO-2924 SpecificCompiler add 'LocalDateTime' logical type
Java: AVRO-2937 Expose some missing flags in SpecificCompilerTool
PHP: AVRO-2096 Fixes to missing functions
Ruby: AVRO-2907 Ruby schema.single_object_schema_fingerprint is reversed

Migration notes:
Java: AVRO-2817 Turn off validateDefaults when reading legacy Avro files
Python: AVRO-2656 avro-python package is now the preferred python3 library and
                  avro-python3 is prepared to be deprecated

And of course upgraded dependencies to latest versions, CVE fixes and more: https://issues.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.10.1

This release includes 189 Jira issues, including some interesting features:

C#: AVRO-2389 Add Avro serialization for POCO (Reflection)
Java: AVRO-2723 Automatically find defaults on POJO when using reflection
Java: AVRO-2438 Better support for URI and URL types
Perl: AVRO-1461 Distribute Perl module in CPAN
PHP: AVRO-2527 Update to PHP 7.x
Python: AVRO-2387 Type checking added to python
Ruby: AVRO-1740 Support fingerprinting
Ruby: AVRO-2535 Support enum defaults
Ruby: AVRO-2545 Support aliases

Migration notes:
Java: AVRO-2278 Throw an exception when getting a non-existent field from a record
Java: AVRO-2581 Maven plugin generates specific records with private fields
Java: AVRO-2335 Remove Joda Time library
Python: AVRO-2656 avro-python package is now the preferred python3 library and avro-python3 is prepared to be deprecated

And of course upgraded dependencies to latest versions, CVE fixes and more
https://issues.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.10.0