Python has had this for a long time in the form of the doctest module, which allows running code samples from docstrings as tests. But use of it tends to be somewhat rare these days and is often discouraged, for a reason that also affects any other variant of “literate test”: it’s very difficult to maintain a clear boundary between tests of the software, and tests of the documentation/examples.
Over time they tend to blur into each other, and so you start relying on them as part of the normal test suite of the software (why bother re-testing something that’s already covered in a doctest?), etc., and then you start realizing that while these kinds of tools are handy for a quick example or two they’re really bad at being proper test suites/runners and have all sorts of limitations that full-on test frameworks don’t. And then you rip it all out until the next time the cycle starts again.
Personally, if I’m going to do this, I prefer to use a real documentation framework like Sphinx, and use it to incidentally test code samples in the documentation. Which is about as close as I think it’s possible to get at drawing a clear line between “documentation tests” and testing the software itself, though still a little bit uncomfortably blurry to me.
Nice, I have something very similar in Elvish’s docs (here is an example in Markdown source and how it renders, note that the code after the prompt is syntax-highlighted). I call them “transcripts”, in keeping with calling shell programs “scripts”.
I’ve always wanted to use them as tests but haven’t got around to doing that since the outputs can depend on some environmental factors - as an obvious example ls’s output depends on the working directory and what it contains - so I need some syntax to skip certain parts of the output (or entire commands).
I wonder if tesh has to solve that too, or do you just avoid writing environment-dependent sessions?
Edit: I see that there is syntax for ignoring part of the output, nice!
This is a nice way to test software! By the way OCaml’s default build system dune has this feature built-in.
Similar: https://github.com/zombocom/rundoc/blob/main/README.md
Python has had this for a long time in the form of the
doctestmodule, which allows running code samples from docstrings as tests. But use of it tends to be somewhat rare these days and is often discouraged, for a reason that also affects any other variant of “literate test”: it’s very difficult to maintain a clear boundary between tests of the software, and tests of the documentation/examples.Over time they tend to blur into each other, and so you start relying on them as part of the normal test suite of the software (why bother re-testing something that’s already covered in a doctest?), etc., and then you start realizing that while these kinds of tools are handy for a quick example or two they’re really bad at being proper test suites/runners and have all sorts of limitations that full-on test frameworks don’t. And then you rip it all out until the next time the cycle starts again.
Personally, if I’m going to do this, I prefer to use a real documentation framework like Sphinx, and use it to incidentally test code samples in the documentation. Which is about as close as I think it’s possible to get at drawing a clear line between “documentation tests” and testing the software itself, though still a little bit uncomfortably blurry to me.
Nice, I have something very similar in Elvish’s docs (here is an example in Markdown source and how it renders, note that the code after the prompt is syntax-highlighted). I call them “transcripts”, in keeping with calling shell programs “scripts”.
I’ve always wanted to use them as tests but haven’t got around to doing that since the outputs can depend on some environmental factors - as an obvious example
ls’s output depends on the working directory and what it contains - so I need some syntax to skip certain parts of the output (or entire commands).I wonder if
teshhas to solve that too, or do you just avoid writing environment-dependent sessions?Edit: I see that there is syntax for ignoring part of the output, nice!