Traditional relational databases are pretty great, but when your use-case is well-served by them depends a great deal not only on the use-case, but also on what you consider to be ‘well served’, and what you consider to be a traditional relational database.
Here are a couple of non-scalability reasons to think about why “traditional” might not be the right approach (or, at least, to question the definition of “traditional”):
Databases tend to be extremely sensitive to cache hit rate for performance. This is sometimes good: high locality workloads can lead to extremely good performance. It’s also a sharp edge. Failing over often means losing the cache, which can take minutes or even hours to rehydrate on a new machine, causing latency issues and possibly metastable failure issues upstream. You can mitigate this by keeping failover candidates caches warm (like Aurora), but is that still “traditional”? Changes in the size or composition of your working set can cause sudden changes in performance, perhaps over multiple orders of magnitude. The size and composition of your working set may not be under your control, and may even be under the control of attackers. You can mitigate this by being able to scale cache in-place, but is that still “traditional”? You can also mitigate it by buying a lot of memory, which is an OK solution but not always the most economical (you’ve just built an ad-hoc in-memory database).
Databases have historically been large and complex attack surfaces, and handle a whole lot of ad-hoc data. Because of this, it’s a good idea to keep them patched and up-to-date, and ideally keep relatively current with new major versions. But patching and upgrading relational databases without risk requires a lot of expertise if you’re doing it yourself. You can mitigate this by picking a database service that manages upgrades for you, but is that still “traditional”? How hard upgrading is tends to be proportional to how many database features you depend on, and proportional to how painful a long period of downtime can be.
Same story with durability. Can you lose data? Should you use a service that provides PITR and backup management and so on. What’s your ransomware strategy? What’s your logical corruption (“oops, I forgot the WHERE on a DELETE”) strategy? And so on.
Good system designers have a lot to consider in the data space, and would do well to ignore thought-terminating ideas like “traditional”. It may well be that the needs of your business are well met by a simple solution, or that you can accept some operational debt now to pay in future. Or, it may well be that something on the managed service spectrum offers you a ton of operational value. At very least consider availability, security, durability, predictability of performance, and (yes) scalability when you pick.
I used to work on the CosmosDB team. The core engine is a really excellent multi-tenant partitioned document store. The conceptual model at that layer isn’t that different from Cassandra or Dynamo, but that’s true for basically everything in this space. If you’re doing a distributed database, you have to have a means of partitioning so you can split data across nodes and route to it, so the underlying kv store ends up looking like that. The NewSQL systems work the same way under the hood.
The differentiator for Cosmos is that the underlying engine is flexible enough where you can layer lots of semantics on top of it, so instead of trying to have separate products for each different niche, it exposes an API to the same storage layer. I often joked that the best way to run Mongo in production is to point your app at CosmosDB’s Mongo API. I’m no longer with Microsoft, but I still think that’s probably true.
It’s a mature database. I don’t think it’s ever lost customer data. The biggest operational problems we hit were during the hardware crunch during covid when, like every other service in Azure, there were regions we just couldn’t get capacity in and had to tell customers they couldn’t replicate data there.
The NewSQL systems work the same way under the hood.
How does that work with Spanner and cockroach? I’m curious, as I’ve never used one, but do they suffer from slow reads and writes, because so much ends up crossing an invisible partition?
I often joked that the best way to run Mongo in production is to point your app at CosmosDB’s Mongo API.
I noticed that, when doing the research, that it does have more hands off scaling than mongo, but there was lots of people complaining online about the API’s not being equivalent but I guess that’s to be expected and is why they recommend the ‘no-sql’ api.
TBH, the thing most off-putting about Cosmos is the marketing doesn’t seem to do a good job of explaining the pros and cons of using it, and just presents it as the only logical choice for storing data. There are things saying its a drop in replacement for SQL database, and it doesn’t have a schema, so how could that be the case? Those things bother me. To me, that raised a bunch of red flags, and made me want to dig in and find out more.
This article is the end result of that, and I think getting more in the weeds makes it clear when you might want to use it.
A query is managed by a coordinator node. It dispatches to each storage node that has data it needs, and in good systems it pushes as much of the filtering to the storage node as possible so you’re not reading pages or transferring data that you don’t need. Part of what the query planner does is take your filtering conditions and finding the smallest set of partitions it needs to check. As soon as you are using a distributed database like this you never get to treat your partitions as invisible, and choosing your partition key so that partitions aren’t horribly unbalanced in size and traffic and your major queries don’t have to access large numbers of partitions is one of the biggest issues in distributed schema design.
The full relational layer lets you produce more complicated access patterns from your query, and opens lots of optimization opportunities/requirements in the query planner, but it’s still basically the same thing as Dynamo or Cassandra.
lots of people complaining online about the API’s not being equivalent
This is true. We wrote an actual managed Cassandra service to bridge the gap for that database because there were enough large customers that were moving to Cosmos but had stuff that wasn’t supported by the Cosmos Cassandra API yet.
The quality of the marketing probably makes more sense when you realize that Cosmos is driven by very, very large customers, including much of Azure itself. They aren’t reading the marketing copy. They have a phone number of an email address. The engineering team is focused on 1) don’t lose data, 2) don’t go down, 3) build what’s needed to get big customer workloads onto Cosmos.
Truthfully, you shouldn’t be trying to use one of these databases unless you know underlying inevitable topology, which puts them in a terrible marketing position. It’s not complicated stuff, but it’s not super common knowledge. I wrote down what I know about it a couple years back: https://madhadron.com/topology_of_distributed_databases.pdf
The quality of the marketing probably makes more sense when you realize that Cosmos is driven by very, very large customers, including much of Azure itself. They aren’t reading the marketing copy. They have a phone number of an email address. The engineering team is focused on 1) don’t lose data, 2) don’t go down, 3) build what’s needed to get big customer workloads onto Cosmos.
We’ve been using CosmosDB for a new feature and it’s a database that tries to be one-size-fits-all with the multiple APIs. So we tried to use it instead of MongoDB Atlas and it worked with the same client libraries but it’s not really a MongoDB. The pricing aspect of CosmosDB is complex too, you have the vCore based (similar to Atlas) and the RU based, which may also have some provisioned throughput.
Really enjoyed getting starting using CosmosDB after good experiences with DynamoDB as it came with a bit more flexibility around indexing etc. However, our workload was quite bursty as it was recieving aggregated streaming data. The burstable performance was far too slow to scale in comparison with DynamoDB and constantly provisioning the required scale for the peak bursts made it prohibitively expensive. When we needed burst performance it took over a minute to scale up if I remember correctly.
We ended up ditching and rewriting on SQL Server which had significantly better performance and was significantly cheaper.
So close to being great, but just missed the mark for us.
The only thing wrong with this statement is the ‘this is a personal opinion’ part.
Well, that’s a big
if, isn’t it?Traditional relational databases are pretty great, but when your use-case is well-served by them depends a great deal not only on the use-case, but also on what you consider to be ‘well served’, and what you consider to be a traditional relational database.
Here are a couple of non-scalability reasons to think about why “traditional” might not be the right approach (or, at least, to question the definition of “traditional”):
Databases tend to be extremely sensitive to cache hit rate for performance. This is sometimes good: high locality workloads can lead to extremely good performance. It’s also a sharp edge. Failing over often means losing the cache, which can take minutes or even hours to rehydrate on a new machine, causing latency issues and possibly metastable failure issues upstream. You can mitigate this by keeping failover candidates caches warm (like Aurora), but is that still “traditional”? Changes in the size or composition of your working set can cause sudden changes in performance, perhaps over multiple orders of magnitude. The size and composition of your working set may not be under your control, and may even be under the control of attackers. You can mitigate this by being able to scale cache in-place, but is that still “traditional”? You can also mitigate it by buying a lot of memory, which is an OK solution but not always the most economical (you’ve just built an ad-hoc in-memory database).
Databases have historically been large and complex attack surfaces, and handle a whole lot of ad-hoc data. Because of this, it’s a good idea to keep them patched and up-to-date, and ideally keep relatively current with new major versions. But patching and upgrading relational databases without risk requires a lot of expertise if you’re doing it yourself. You can mitigate this by picking a database service that manages upgrades for you, but is that still “traditional”? How hard upgrading is tends to be proportional to how many database features you depend on, and proportional to how painful a long period of downtime can be.
Same story with durability. Can you lose data? Should you use a service that provides PITR and backup management and so on. What’s your ransomware strategy? What’s your logical corruption (“oops, I forgot the WHERE on a DELETE”) strategy? And so on.
Good system designers have a lot to consider in the data space, and would do well to ignore thought-terminating ideas like “traditional”. It may well be that the needs of your business are well met by a simple solution, or that you can accept some operational debt now to pay in future. Or, it may well be that something on the managed service spectrum offers you a ton of operational value. At very least consider availability, security, durability, predictability of performance, and (yes) scalability when you pick.
Honestly? No, not really. Everything you described is just part of having a databases, relational or not.
So I started working at Pulumi and I’ve been trying to wrap my head around Azure’s Cosmos DB. And I did fall a little bit down a rabbit hole.
Azure markets Cosmos DB as this magical database that can do everything, but it’s more like a pricier and faster DynamoDB.
Anyone on here using Azure and have used or experimented with it?
I’d love to hear your war stories!
I used to work on the CosmosDB team. The core engine is a really excellent multi-tenant partitioned document store. The conceptual model at that layer isn’t that different from Cassandra or Dynamo, but that’s true for basically everything in this space. If you’re doing a distributed database, you have to have a means of partitioning so you can split data across nodes and route to it, so the underlying kv store ends up looking like that. The NewSQL systems work the same way under the hood.
The differentiator for Cosmos is that the underlying engine is flexible enough where you can layer lots of semantics on top of it, so instead of trying to have separate products for each different niche, it exposes an API to the same storage layer. I often joked that the best way to run Mongo in production is to point your app at CosmosDB’s Mongo API. I’m no longer with Microsoft, but I still think that’s probably true.
It’s a mature database. I don’t think it’s ever lost customer data. The biggest operational problems we hit were during the hardware crunch during covid when, like every other service in Azure, there were regions we just couldn’t get capacity in and had to tell customers they couldn’t replicate data there.
Sounds like a cool job!
How does that work with Spanner and cockroach? I’m curious, as I’ve never used one, but do they suffer from slow reads and writes, because so much ends up crossing an invisible partition?
I noticed that, when doing the research, that it does have more hands off scaling than mongo, but there was lots of people complaining online about the API’s not being equivalent but I guess that’s to be expected and is why they recommend the ‘no-sql’ api.
TBH, the thing most off-putting about Cosmos is the marketing doesn’t seem to do a good job of explaining the pros and cons of using it, and just presents it as the only logical choice for storing data. There are things saying its a drop in replacement for SQL database, and it doesn’t have a schema, so how could that be the case? Those things bother me. To me, that raised a bunch of red flags, and made me want to dig in and find out more.
This article is the end result of that, and I think getting more in the weeds makes it clear when you might want to use it.
A query is managed by a coordinator node. It dispatches to each storage node that has data it needs, and in good systems it pushes as much of the filtering to the storage node as possible so you’re not reading pages or transferring data that you don’t need. Part of what the query planner does is take your filtering conditions and finding the smallest set of partitions it needs to check. As soon as you are using a distributed database like this you never get to treat your partitions as invisible, and choosing your partition key so that partitions aren’t horribly unbalanced in size and traffic and your major queries don’t have to access large numbers of partitions is one of the biggest issues in distributed schema design.
The full relational layer lets you produce more complicated access patterns from your query, and opens lots of optimization opportunities/requirements in the query planner, but it’s still basically the same thing as Dynamo or Cassandra.
This is true. We wrote an actual managed Cassandra service to bridge the gap for that database because there were enough large customers that were moving to Cosmos but had stuff that wasn’t supported by the Cosmos Cassandra API yet.
The quality of the marketing probably makes more sense when you realize that Cosmos is driven by very, very large customers, including much of Azure itself. They aren’t reading the marketing copy. They have a phone number of an email address. The engineering team is focused on 1) don’t lose data, 2) don’t go down, 3) build what’s needed to get big customer workloads onto Cosmos.
Truthfully, you shouldn’t be trying to use one of these databases unless you know underlying inevitable topology, which puts them in a terrible marketing position. It’s not complicated stuff, but it’s not super common knowledge. I wrote down what I know about it a couple years back: https://madhadron.com/topology_of_distributed_databases.pdf
Ah, this is the hidden piece I’m missing!
All of this is helpful. Thanks for sharing!
I am actually most excited about its graph API based on Gremlin. Has anyone experimented with that?
We’ve been using CosmosDB for a new feature and it’s a database that tries to be one-size-fits-all with the multiple APIs. So we tried to use it instead of MongoDB Atlas and it worked with the same client libraries but it’s not really a MongoDB. The pricing aspect of CosmosDB is complex too, you have the vCore based (similar to Atlas) and the RU based, which may also have some provisioned throughput.
Really enjoyed getting starting using CosmosDB after good experiences with DynamoDB as it came with a bit more flexibility around indexing etc. However, our workload was quite bursty as it was recieving aggregated streaming data. The burstable performance was far too slow to scale in comparison with DynamoDB and constantly provisioning the required scale for the peak bursts made it prohibitively expensive. When we needed burst performance it took over a minute to scale up if I remember correctly.
We ended up ditching and rewriting on SQL Server which had significantly better performance and was significantly cheaper.
So close to being great, but just missed the mark for us.
Wild! I wouldn’t expect greater write through-put with SQL Server.
[Comment removed by author]