The title of this article really buries the lede: it’s about building a self-modifying unit test library with Janet macros, which is actually really cool. Running a test generates a new file, with expected values replaced with actual ones. Writing tests this way is similar to working in a REPL.
Are there any other languages with a test system like this?
Probably! Lots of languages have some kind of snapshot test library, but I don’t know of any framework that works exactly like Judge. I assume that is due to ignorance, though; I can’t claim to have any particular knowledge of the domain.
The .corrected file just-look-at-a-diff thing is taken straight from Jane Street’s expect test framework, which I think got the idea from Mercurial’s unified tests – see cram for a generic description. cram uses .err instead of .corrected, but whatever.
But ppx_expect doesn’t serialize values or work on arbitrary expressions. The way it works is by redirecting stdout for the duration of the test, and by inserting string literals into the source code.
This makes some sense in OCaml because there is not really a canonical way to serialize values – even though there is a de facto standard at Jane Street – so having the user produce strings is reasonable.
(The stdout capture is also extremely useful, because it allows you to use literal printf debugging deep in your application code, and see the output show up as you run your tests. It’s also very useful to have tests that produce ASCII tables or whatever. It’s good for a lot of reasons, but of course it’s easy to do stdout redirection yourself, and you can easily implement the ppx_expect approach on top of the “view a value” primitives.)
But apart from that difference, the workflow you see in the post matches the ppx_expect workflow very closely. Here are some other examples:
But I don’t know what the ergonomics of those are like; I don’t know how easy it is to write tests in this REPLy-style using those libraries. The only one I’ve actually tried is k9, and while it did support “inline snapshots,” at least at the time it didn’t write .corrected files. It will still show you failing tests, but… I really like being able to bring my own diffs to the table. Let me use emerge or delta or vimdiff or whatever else.
It is one of the best Janet-related articles for me. Thank you!
I was not sure about the prospect of the Judge, yet I think I will start next lib with it to see if I can wrap my head around it. And taste the ambrosia. And then maybe lure you into a Good Place org one day.
The title of this article really buries the lede: it’s about building a self-modifying unit test library with Janet macros, which is actually really cool. Running a test generates a new file, with expected values replaced with actual ones. Writing tests this way is similar to working in a REPL.
Are there any other languages with a test system like this?
(author here)
Probably! Lots of languages have some kind of snapshot test library, but I don’t know of any framework that works exactly like Judge. I assume that is due to ignorance, though; I can’t claim to have any particular knowledge of the domain.
The
.correctedfile just-look-at-a-diff thing is taken straight from Jane Street’s expect test framework, which I think got the idea from Mercurial’s unified tests – see cram for a generic description. cram uses.errinstead of.corrected, but whatever.But
ppx_expectdoesn’t serialize values or work on arbitrary expressions. The way it works is by redirecting stdout for the duration of the test, and by inserting string literals into the source code.This makes some sense in OCaml because there is not really a canonical way to serialize values – even though there is a de facto standard at Jane Street – so having the user produce strings is reasonable.
(The stdout capture is also extremely useful, because it allows you to use literal
printfdebugging deep in your application code, and see the output show up as you run your tests. It’s also very useful to have tests that produce ASCII tables or whatever. It’s good for a lot of reasons, but of course it’s easy to do stdout redirection yourself, and you can easily implement theppx_expectapproach on top of the “view a value” primitives.)But apart from that difference, the workflow you see in the post matches the
ppx_expectworkflow very closely. Here are some other examples:But I don’t know what the ergonomics of those are like; I don’t know how easy it is to write tests in this REPLy-style using those libraries. The only one I’ve actually tried is k9, and while it did support “inline snapshots,” at least at the time it didn’t write
.correctedfiles. It will still show you failing tests, but… I really like being able to bring my own diffs to the table. Let me useemergeordeltaorvimdiffor whatever else.The first place I heard of this approach was Jane Street’s expect test framework for OCaml.
This is super cool. As a Janet user, I will find an excuse to use this framework!
This is very cool. I think Haskell has a (somewhat rudimentary) version of this called ‘golden tests’ .
It is one of the best Janet-related articles for me. Thank you!
I was not sure about the prospect of the Judge, yet I think I will start next lib with it to see if I can wrap my head around it. And taste the ambrosia. And then maybe lure you into a Good Place org one day.