Tags: xtdb/xtdb
Tags
We're pleased to announce the release of XTDB 2.0.0-beta6 🚀 Coming up in the release notes: we have some *breaking configuration changes*, but largely this release has been driven by the real-world practical usage of our Design Partners (thanks folks!) - the sorts of changes that don't make for exciting release notes, but the ones you appreciate when you're actually working with the database day-to-day. 2.0.0-beta7, however, is a different story - we'll be bringing you the first database migration since we released beta1, in order to better support a wider variety of temporal data shapes. More details below! While you're here, one date for you diary: Jeremy will be giving a talk on ["Streamlining Regulatory Compliance and Reporting with XTDB: The Future of Bitemporal Data Management"](https://us06web.zoom.us/webinar/register/2517386708730/WN_QqygMHteRvmPo0O4aQDW5Q#/registration) - *19th February* at *15:00 UK/16:00 CET/10:00 ET*. Come join us for a deep dive into how XTDB can transform your data management strategy - making data reconciliation, reporting, and analysis simpler, more accurate, and cost-effective. Hope to see you there, of course - but if you can't make it, do still sign up, and we'll send you the recording. ## Breaking change: Kafka configuration As part of this release, we've simplified the log by merging the two log topics together. * If you're configuring using YAML: ```yaml txLog: !Kafka txTopic: "xtdb-tx-topic" filesTopic: "xtdb-files-topic" autoCreateTopics: true # becomes log: !Kafka topic: "xtdb-log-topic" autoCreateTopic: true ``` * If you're using the environment variables in the various Docker images, `XTDB_TX_TOPIC` -> `XTDB_LOG_TOPIC`; `XTDB_FILES_TOPIC` is removed. You can then safely remove the file topic. ## Breaking change: `BEGIN AT SYSTEM_TIME` We've streamlined the UX for specifying the 'basis' of SQL transactions. tl;dr: breaking change is `BEGIN AT SYSTEM_TIME TIMESTAMP '...'` -> `BEGIN READ WRITE WITH (SYSTEM_TIME = TIMESTAMP '...')` 'Basis' is an important concept in XTDB - we take great pride that our queries run on a completely immutable snapshot of the database, so that you can re-run queries later and know that you'll receive the same results. This isn't just within a single transaction, nor does it require creating expensive snapshots manually ahead-of-time, nor does it require separate nodes - at any point, you can ask the same XTDB node to run as if it were last Tuesday at 4pm, even if you didn't know you'd need to at the time. When a transaction starts, we fix a basis for all of the queries/operations that run within it. It has sensible defaults - you shouldn't need to specify it for most transactions - but you can manually set them if need be. There are two dimensions to the basis: firstly, `CLOCK_TIME` - this ensures that any references to wall-clock time within the queries can be repeated. This defaults to the wall-clock time on the node at the start of the transaction, and you can see the current value with `SHOW CLOCK_TIME`. As well as being used for `NOW()` et al, this is also used for the default valid-time for each referenced table. Secondly, `SNAPSHOT_TIME` - this influences which transactions are visible to the queries. This is a strict upper bound - no matter if you ask for a later system time in your query, you will not see any updates later than this time. `SNAPSHOT_TIME` defaults to the latest completed transaction on the node serving the query, and can be read with `SHOW SNAPSHOT_TIME`. In summary, by default, you'll see new transactions and scheduled updates come through as you'd expect, but when the time comes(!) you have control over exactly what's visible if need be. To set these values, simply start your transaction with `BEGIN READ ONLY WITH (SNAPSHOT_TIME = TIMESTAMP '...', CLOCK_TIME = TIMESTAMP '...')`. (h/t to Rich Hickey's 2012 "The Database as a Value" talk which was very inspirational to us in this area 🙏) ## Elsewhere * `EXPLAIN <SQL query>` now gets you the query plan for the query. It's not in the most end-user readable format at the moment, we can definitely make improvements here - but if you've got an ill-performant query it's a good start! * We've added support for a 'read-only' port - for use-cases where you know you'll only want to read data (e.g. Metabase or similar), you can expose the read-only port and lock down the normal read-write one. ```yaml server: port: 5432 readOnlyPort: 5433 ``` * We've relaxed the SQL timestamp literal syntax - you can now optionally omit the time-part if it's midnight. For example, `TIMESTAMP '2025-01-01Z'` gets you a TZ-aware timestamp. * In the unlikely case of an unrecoverable error that halts transaction processing (to prevent corruption of your data) XT will now save a crash log to your object store. We obviously hope nobody ever sees this one - but if you do you/we should have more diagnostic information to get back up and running. * Full list (44 cards), as always, in the [milestone](https://github.com/xtdb/xtdb/milestone/54?closed=1) ## Beta7 preview In this iteration, we're particularly looking at optimising data where the number of distinct entities is relatively low, but each entity has a significant number of updates through its history - think market pricing data or sensor readings. The proposed index structure makes a cleaner separation of current and historical data (both explicitly superseded and data that's still valid but not as recent, like old orders/trades), so we're expecting to see improved OLTP as-of-now query performance as well as being able to quickly identify the partitions required to serve a historical query. We're also naturally taking this opportunity to apply all of the disk format simplifications and improvements that we've had queued up for a while, like keeping more metadata about the data contained within each file and more table-level statistics, so that should speed things up as well! There'll likely be two options for migration: 1. (easier) If your log still contains all of your transactions - it's no longer mandatory in XT2, but we're aware some folks still do - just spin up a new node in the background and let it replay. 2. Otherwise, we'll be releasing a migration tool from beta6 -> beta7 - spin this up in the background as a one-shot task, let it run, then you'll be able to do the usual green/blue release for the nodes themselves. If you'd like early access to these changes, please do let us know! Beyond that, we're very much in the final stretch towards our General Availability (GA) release - hence the iteration on our underlying data structures before launch. Thank you to everyone who's worked with us throughout this beta phase - your feedback and guidance has been instrumental in steering XTDB v2 towards becoming what we hope is a very exciting and valuable open-source tool. ## Get in touch! As always, we'd love to hear your thoughts - you can get in touch either through GitHub, https://discuss.xtdb.com, or [email protected]. Cheers! James & the XT Team
We're pleased to announce the release of 2.0.0-beta5! 🚀
In amongst the usual array of bugfixes and minor improvements, 2.0.0-beta5 adds "template-friendly SQL", as well as a 4x reduction in per-transaction overhead for small transactions through the Postgres wire-server.
## Template-friendly SQL
Hands up if you've ever had to get the right number of commas or ANDs when generating a SQL SELECT/WHERE clause respectively 😆 🖐
SQL was originally intended to be a language written by hand, and was created long (_long_) before modern languages had decent support for string templating.
While SQL itself is incredibly powerful, adapting it to dynamic contexts in application code has often felt like trying to fit a square peg into a round hole.
From string concatenation bugs to unexpected syntax errors caused by stray commas, many developers have experienced the frustration of working with SQL in dynamic templates.
A common workaround has been to introduce tools or libraries to generate SQL programmatically, but these come with their own set of challenges.
Developers often find themselves unsure about the exact SQL being generated and sometimes need to cajole these tools into producing the desired query, adding extra complexity and reducing transparency.
With XTDB, we're introducing **template-friendly SQL**, a modern take on making SQL generation smoother and more forgiving in templated environments, particularly for languages and libraries that already offer strong support for string templating.
Whether you're using Kotlin, Python, JavaScript, or Clojure libraries like YeSQL and HugSQL which emphasize 'getting back to SQL', this feature integrates seamlessly into workflows focused on dynamic SQL generation.
Here's what's new:
- **Flexible Comma Placement**: Forget the days of meticulously managing commas in your SELECT and WHERE clauses.
Taking inspiration from Clojure's "commas as whitespace," trailing commas, multiple commas in a row, or even empty predicates are gracefully handled without causing syntax errors.
You can now safely generate SQL without needing additional libraries or worrying about “did I add that comma in the right place?”
```python
query = f"""
SELECT *
FROM my_table
WHERE
{"col1 = 'value1'," if condition1 else ''}
{"col2 = 'value2'," if condition2 else ''}
"""
# The extra commas are ignored, making templates more forgiving and human-readable.
```
- **Query Pipelining**: SQL's top-level structure is relatively constrained - it must only have one SELECT, FROM, WHERE, GROUP BY, and if you require multiple steps, you need to use subqueries.
Take this query to calculate a frequency distribution, for example:
```sql
SELECT order_count, COUNT(*) AS freq
FROM (SELECT customer, COUNT(*) AS order_count
FROM orders
GROUP BY customer) counts
GROUP BY order_count
ORDER BY order_count DESC
```
With XTDB, you can now:
* optionally move the SELECT to the place in the query where it logically runs
* run multiple aggregations in a pipeline
* elide the GROUP BY in this common case
```sql
FROM orders
SELECT customer, COUNT(*) AS order_count
SELECT order_count, COUNT(*) AS freq
ORDER BY order_count DESC
```
To clarify: as always, the standard SQL structure still works (so your existing tooling shouldn't be affected) - this is an opt-in feature.
These enhancements align with XTDB's philosophy of developer-first tooling, reducing friction and making your code cleaner and more robust.
Whether you're dynamically generating queries or simply writing complex SQL templates, template-friendly SQL lets you focus on the logic rather than the syntax.
We'd love to hear how this feature improves your workflow. Try it out and share your feedback with us!
## Transaction ingestion performance
We've also spent some time improving XTDB's ingestion performance through the Postgres wire-server.
Our recommendation has always been (and continues to be) to batch your inserts wherever possible - the right granularity will obviously depend on your exact use-case but in practice we've found batches of the order of magnitude of ~1k seem to be a sweet spot.
(This generally applies to whatever database you're using - XT is no different!)
Of course, though, there will be times when this isn't possible, and we'd like to make that case faster too.
So, since beta4, we've added in a micro-benchmark to measure this performance - ingesting 100k small documents at various batch sizes.
On my desktop, we've seen the following improvements:
- batches of 10: 24s -> 6s
- batches of 100: 8s -> 1.8s
- batches of 1000: 6s -> 1.2s
How? Well:
- we found a case in Apache Arrow where one of the vectors had a O(n²) operation in its copy implemention
- we've spent some time optimising our low-level Postgres wire implementation - down to the buffers, bytes and sockets
- we're now able to recognise another class of INSERT queries that don't need to read the database to make their writes.
Again, do give it a spin, and let us know what you find on your hardware!
## Elsewhere
Two breaking changes to tell you about:
* If you are using XT in your Clojure process, we've moved our time reader-macros off the `#time` namespace to not clash with Henry's [](time-literals) library.
You can either depend on that library specifically, or replace `#time/` with `#xt/`.
* The Postgres wire-server and FlightSQL server now start on an unused port by default (rather than 5432/9832), to avoid clashing with other processes you may have running.
To restore the old behaviour, you'll need to explicitly specify the port in your configuration.
(The official Docker images remain on the original ports.)
As always, we'd love to hear your thoughts - you can get in touch either through GitHub, https://discuss.xtdb.com, or [email protected].
Cheers!
James & the XT Team
Hey folks, here's beta3 🚀 This one's mostly a SQL tooling support release - we've been improving XTDB's interoperability with common PostgreSQL tools, but also improving the XT2 operational story too. - On the operational side, we've been working on Kubernetes (k8s) deployments - there's a new [Azure-specific guide](https://docs.xtdb.com/guides/starting-with-azure.html) but this should be relatively applicable to wherever you get your k8s from. - We've added simple `/healthz` endpoints for you to point your healthchecks at - add `healthz: {port: 8080}` / `:healthz {:port 8080}` to your YAML/EDN config respectively. - Under the hood, we've made significant changes to the buffer pool to allow us to more accurately account for the memory/disk we're using - these should be available as monitoring metrics shortly. - We have initial support for [Elixir](https://docs.xtdb.com/clients/elixir.html), using the standard 'postgrex' library. - We've added Postgres's dollar-quoted strings: `$$string where you don't need to escape any 'quotes'$$` There are a couple of *breaking changes* to inform you of: - If you were using `#xt.time` Java 8 time literals, these are now just `#time`, in keeping with the time-literals library. - Configuring metrics: we now include Prometheus metrics as standard under the new `healthz` module - see #3838 for the migration. - We now disallow inserting/updating any attributes with a `_`/`:xt/` prefix by default (except as explicitly defined in the XTDB APIs) For full details, see the [milestone](https://github.com/xtdb/xtdb/milestone/51?closed=1) Enjoy! XT Team
temporarily use `match` so that it tags `latest` even on pre-releases
Here's 1.24.4! 🚀 This is a bugfix release containing the following: * #3169 removed usage of Clojure private var removed in 1.12 * #3113 can now run multiple XTDB processes that use RocksDB on Windows * Environment variables to disable cryptographic libraries can now also be supplied as JVM options Cheers! XT Team
PreviousNext