It seems to me like the whole world would be a lot better off if we added support for determinism right on down to the hardware level. As mentioned here, to get any semblance of determinism, one has to jump through elaborate hoops at higher levels of the stack to get anywhere close to determinism.
That’s why the Antithesis approach is the most exciting one that I’ve seen so far, since it handles so much of the determinism under the hood.
The closest thing I’ve seen to implementing determinism in the software stack itself is Deterministic Process Groups, which allow for determinism in OS task scheduling and thus concurrency. Does anyone know of anything else like this?
It’s not clear to me that it’s desirable to solve the problem of determinism as far down the stack as possible.
For example, can deterministic hardware be as performant as non-deterministic hardware? Is solving it at the OS level better than at the programming language runtime level? The programming language level should have more information about what the programmer intends to do, and might therefore be in a better place to make the right decision with regards to scheduling/introducing interesting entropy? (A bit similar to how database people don’t want to rely on the OS scheduler or IO system, because at the database level they got more information and can make better scheduling and IO decisions.)
I agree that if you can make a specific simulator for your use case you probably will have a better time.
The issue is it’s hard. You need to make sure every library, third party service,etc is deterministic or you can mock it. If your foundationdb or tigerbeetle that took time to structure the code to make that easy, that’s awesome!
If you have a codebase that is older and you want to add DST to it, that’s where I view the hypervisor approach as the better option. You don’t have to change anything and you get DST “for free”.
If you want to cover as much software as possible (which is a desirable thing in this case), then you need to be as low in the stack as possible. FoundationDB built a simulator, but that simulator only makes sense for FoundationDB. A database only needs to care about itself, but if you have software that uses POSIX threads, networking, locking, atomic operations and filesystems the language has no way of handling that without completely restricting what resources from the OS you have available. And if you do that then it only works for that language.
Antithesis (from the same people as FoundationDB) tries to be generic because it works a lot lower in the stack so it works with all software (in theory, there are probably limitations).
The issue is, at each level it can’t be solved because of leaky non-determinism at some lower level. So I hear you about performance, but there is likely a way to have deterministic and non-deterministic modes that are functionally equivalent, but where the non-deterministic mode can be chosen for performance reasons.
It seems to me like the whole world would be a lot better off if we added support for determinism right on down to the hardware level. As mentioned here, to get any semblance of determinism, one has to jump through elaborate hoops at higher levels of the stack to get anywhere close to determinism.
That’s why the Antithesis approach is the most exciting one that I’ve seen so far, since it handles so much of the determinism under the hood.
The closest thing I’ve seen to implementing determinism in the software stack itself is Deterministic Process Groups, which allow for determinism in OS task scheduling and thus concurrency. Does anyone know of anything else like this?
It’s not clear to me that it’s desirable to solve the problem of determinism as far down the stack as possible.
For example, can deterministic hardware be as performant as non-deterministic hardware? Is solving it at the OS level better than at the programming language runtime level? The programming language level should have more information about what the programmer intends to do, and might therefore be in a better place to make the right decision with regards to scheduling/introducing interesting entropy? (A bit similar to how database people don’t want to rely on the OS scheduler or IO system, because at the database level they got more information and can make better scheduling and IO decisions.)
(Antithesis employee here)
I agree that if you can make a specific simulator for your use case you probably will have a better time. The issue is it’s hard. You need to make sure every library, third party service,etc is deterministic or you can mock it. If your foundationdb or tigerbeetle that took time to structure the code to make that easy, that’s awesome!
If you have a codebase that is older and you want to add DST to it, that’s where I view the hypervisor approach as the better option. You don’t have to change anything and you get DST “for free”.
If you want to cover as much software as possible (which is a desirable thing in this case), then you need to be as low in the stack as possible. FoundationDB built a simulator, but that simulator only makes sense for FoundationDB. A database only needs to care about itself, but if you have software that uses POSIX threads, networking, locking, atomic operations and filesystems the language has no way of handling that without completely restricting what resources from the OS you have available. And if you do that then it only works for that language.
Antithesis (from the same people as FoundationDB) tries to be generic because it works a lot lower in the stack so it works with all software (in theory, there are probably limitations).
The issue is, at each level it can’t be solved because of leaky non-determinism at some lower level. So I hear you about performance, but there is likely a way to have deterministic and non-deterministic modes that are functionally equivalent, but where the non-deterministic mode can be chosen for performance reasons.
Other than that, the lower the better to me.
For Rust, madsim supports deterministic simulation of tokio.