1

drum roll … 1.22.0 is out!

1.22.0 represents various major upstream dependency upgrades, sweeping improvements to ingestion performance, and unlocks another level of eagerly-awaited native experience for our users running on M1 processors (where previously LMDB and Kafka were missing support). The release also contains a number of important bugfixes.

;; project.clj, e.g.
[com.xtdb/xtdb-core "1.22.0"]

;; deps.edn, e.g.
{com.xtdb/xtdb-core {:mvn/version "1.22.0"}}

The upgrades to both RocksDB and LMDB require an ‘index version bump’. This means that you’ll need to clear your XTDB query indices, and re-index from the transaction log as per the docs. The new index version is 22 also (a happy coincidence!).

In green/blue production settings, we recommend doing this by starting a new cluster of XTDB nodes, waiting for them to catch up with the tx-log, switching over, and decommissioning the old nodes. If you’ve got any questions/concerns about this, please do get in touch via hello@xtdb.com - we’re happy to help out.

Ingestion Performance

For those who are interested in the main technical changes here, we have:

  • Pushed down in-transaction speculative writes to RocksDB’s WriteBatchWithIndex API, in the cases where RocksDB is used for the index-store #1762
  • Moved to a pipelined ingestion model where work is able to happen across multiple threads, and document requests can be batched up across transactions #1762
  • Made use of RocksDB’s “column families” facility for storing different parts of the index keyspace separately. In particular the 2 temporal indexes that are on the critical path for ingestion now have reduced seek overhead because there is less scanning over temporarily unsorted data #1773

In our own testing and evaluation this work has improved bulk ingestion by as much as 40% (e.g. a 10 hour import job is now 6 hours). The benefit is also experienced during re-indexing (e.g. when upgrading from 1.21.0 to 1.22.0). These changes also open up several new avenues for further ingestion performance work in future.

Apple M1 Support

@wotbrew joined the development team earlier in the year and has successfully conquered the remaining M1 issues. Maintaining end-to-end support will remain a priority for us looking ahead.

Elsewhere in this release

  • #1823 Switch to using LRU as the default cache, due to unresolved memory issues with the “Second Chance Cache” during bulk query workloads - if you have concerns about any query performance regressions after upgrading to this release then please get in touch
  • #1816 Fix our LMDB usage for better stability
  • #1815 Fix OOM whilst consuming JDBC tx-logs
  • #1808 Avoid overwhelming the document-stores with excessively sized requests
  • #1806 Bump to Kafka, motivated by M1 support also
  • #1782 Fix segfault issue with RocksDB during node close
  • #1766 Fix the sort order for available checkpoints

As ever, a big thanks to everyone contributing to this release by raising/fixing issues, and helping us with repros!

Cheers,

XT Team

1 Like