-
Notifications
You must be signed in to change notification settings - Fork 457
feature: add aliases for inline tests #11109
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feature: add aliases for inline tests #11109
Conversation
b5d5d4f to
0e1c216
Compare
0e1c216 to
9842240
Compare
This bug was noticed when working on ocaml#11109. The issue is that the cram rules create two copies of the anonymous cram action, however since the alias names are different (runtest vs cram test name) this ends up with having two different digests. This means that when building `@runtest` and `@mycram` we end up running the cram test `mycram.t` twice. This test demonstrates this behaviour. Signed-off-by: Ali Caglayan <[email protected]>
0a26c59 to
8a845fe
Compare
8a845fe to
6401184
Compare
|
@emillon I've updated the documentation but it could be better. Any ideas on improving it? |
6401184 to
ef3524f
Compare
This bug was noticed when working on ocaml#11109. The issue is that the cram rules create two copies of the anonymous cram action, however since the alias names are different (runtest vs cram test name) this ends up with having two different digests. This means that when building `@runtest` and `@mycram` we end up running the cram test `mycram.t` twice. This test demonstrates this behaviour. Signed-off-by: Ali Caglayan <[email protected]>
ef3524f to
ff7d3da
Compare
* test: cram tests are run twice when both aliases are built This bug was noticed when working on #11109. The issue is that the cram rules create two copies of the anonymous cram action, however since the alias names are different (runtest vs cram test name) this ends up with having two different digests. This means that when building `@runtest` and `@mycram` we end up running the cram test `mycram.t` twice. This test demonstrates this behaviour. Signed-off-by: Ali Caglayan <[email protected]> * fix: double run of cram tests We fix an issue where cram tests were being run twice due to anonymous actions being created for each alias associated to a given cram test. We fix this by distinguishing one of the aliases and only creating the action for that while allowing the other aliases to depend on that alias. This required changing the cram specification slightly, but has simplified the rule generation somewhat. Signed-off-by: Ali Caglayan <[email protected]> --------- Signed-off-by: Ali Caglayan <[email protected]>
* test: cram tests are run twice when both aliases are built This bug was noticed when working on ocaml#11109. The issue is that the cram rules create two copies of the anonymous cram action, however since the alias names are different (runtest vs cram test name) this ends up with having two different digests. This means that when building `@runtest` and `@mycram` we end up running the cram test `mycram.t` twice. This test demonstrates this behaviour. Signed-off-by: Ali Caglayan <[email protected]> * fix: double run of cram tests We fix an issue where cram tests were being run twice due to anonymous actions being created for each alias associated to a given cram test. We fix this by distinguishing one of the aliases and only creating the action for that while allowing the other aliases to depend on that alias. This required changing the cram specification slightly, but has simplified the rule generation somewhat. Signed-off-by: Ali Caglayan <[email protected]> --------- Signed-off-by: Ali Caglayan <[email protected]>
|
How about we add a |
|
@rgrinberg I'm okay with the idea, but isn't it a bit confusing to run a test as |
eb01971 to
9ab2826
Compare
|
I've pushed a commit changing it to |
|
Ah, we are getting complaints from The validation that we do in Is the culprit. Do you think its time to get rid of it and put something more sophisticated with a clear specification of what is valid/invalid? |
|
Forbidding |
I don't see how you some rather hard to debug cycles without this. You can reproduce this by:
|
|
OK so the problem we have here is that |
|
I'm going to mark this PR as draft while I think of a way to validate alias names and keep them flexible. |
9ab2826 to
9c32c77
Compare
9c32c77 to
b38d581
Compare
For a given library with inlines tests, we add an alias with the name of the library for running those tests prefixed with `runtest-`. Further work on separating out the partition actions would allow us to have an alias for each partition. For now, this allows us to run inline tests with the runtest command. Signed-off-by: Ali Caglayan <[email protected]>
b38d581 to
cf522f1
Compare
* test: cram tests are run twice when both aliases are built This bug was noticed when working on ocaml#11109. The issue is that the cram rules create two copies of the anonymous cram action, however since the alias names are different (runtest vs cram test name) this ends up with having two different digests. This means that when building `@runtest` and `@mycram` we end up running the cram test `mycram.t` twice. This test demonstrates this behaviour. Signed-off-by: Ali Caglayan <[email protected]> * fix: double run of cram tests We fix an issue where cram tests were being run twice due to anonymous actions being created for each alias associated to a given cram test. We fix this by distinguishing one of the aliases and only creating the action for that while allowing the other aliases to depend on that alias. This required changing the cram specification slightly, but has simplified the rule generation somewhat. Signed-off-by: Ali Caglayan <[email protected]> --------- Signed-off-by: Ali Caglayan <[email protected]>
For a given library with inlines tests, we add an alias with the name of the library for running those tests prefixed with `runtest-`. Further work on separating out the partition actions would allow us to have an alias for each partition. For now, this allows us to run inline tests with the runtest command. Signed-off-by: Ali Caglayan <[email protected]>
CHANGES: ### Fixed - Stop re-running cram tests after promotion when it's not necessary (ocaml/dune#11994, @rgrinberg) - fix: `$ dune subst` should not fail when adding the version field in opam files (ocaml/dune#11801, fixes ocaml/dune#11045, @btjorge) - Kill all processes in the process group after the main process has terminated; in particular this avoids background processes in cram tests to stick around after the test finished (ocaml/dune#11841, fixes ocaml/dune#11820, @Alizter, @Leonidas-from-XIV) ### Added - `(tests)` stanzas now generate aliases with the test name. To run `(test (name a))` you can do `dune build @runtest-a`. (ocaml/dune#11558, grants part of ocaml/dune#10239, @Alizter) - Inline test libraries now produce aliases `runtest-name_of_lib` allowing users to run specific inline tests as `dune build @runtest-name_of_lib`. (ocaml/dune#11109, partially fixes ocaml/dune#10239, @Alizter) - feature: `$ dune subst` use version from `dune-project` when no version control repository has been detected (ocaml/dune#11801, @btjorge) - Allow `dune exec` to run concurrently with another instance of dune in watch mode (ocaml/dune#11840, @gridbugs) - Introduce `%{os}`, `%{os_version}`, `%{os_distribution}`, and `%{os_family}` percent forms. These have the same values as their opam counterparts. (ocaml/dune#11863, @rgrinberg) - Introduce option `(implicit_transitive_deps false-if-hidden-includes-supported)` that is equivalent to `(implicit_transitive_deps false)` when `-H` is supported by the compiler (OCaml >= 5.2) and equivalent to `(implicit_transitive_deps true)` otherwise. (ocaml/dune#11866, fixes ocaml/dune#11212, @nojb) - Add `dune describe location` for printing the path to the executable that would be run (ocaml/dune#11905, @gridbugs) - `dune runtest` can now understand absolute paths as well as run tests in specific build contexts (ocaml/dune#11936, @Alizter). - Added 'empty' alias which contains no targets. (ocaml/dune#11556 ocaml/dune#11952 ocaml/dune#11955 ocaml/dune#11956, grants ocaml/dune#4161, @Alizter and @rgrinberg) - Allow `dune promote` to properly run while a watch mode server is running (ocaml/dune#12010, @ElectreAAS) - Add `--alias` and `--alias-rec` flags as an alternative to the `@` and `@@` syntax in the command line (ocaml/dune#12043, fixes ocaml/dune#5775, @rgrinberg) ### Changed - Format long lists in s-expressions to fill the line instead of formatting them in a vertical way (ocaml/dune#10892, fixes ocaml/dune#10860, @nojb) - Switch from MD5 to BLAKE3 for digesting targets and rules. BLAKE3 is both more performant and difficult to break than MD5 (ocaml/dune#11735, @rgrinberg, @Alizter) - Print a warning when `dune build` runs over RPC (ocaml/dune#11836, @gridbugs) - Stop emitting empty module group wrapper `.js` file in `melange.emit` (ocaml/dune#11987, fixes ocaml/dune#11986, @anmonteiro)
CHANGES: ### Fixed - Stop re-running cram tests after promotion when it's not necessary (ocaml/dune#11994, @rgrinberg) - fix: `$ dune subst` should not fail when adding the version field in opam files (ocaml/dune#11801, fixes ocaml/dune#11045, @btjorge) - Kill all processes in the process group after the main process has terminated; in particular this avoids background processes in cram tests to stick around after the test finished (ocaml/dune#11841, fixes ocaml/dune#11820, @Alizter, @Leonidas-from-XIV) ### Added - `(tests)` stanzas now generate aliases with the test name. To run `(test (name a))` you can do `dune build @runtest-a`. (ocaml/dune#11558, grants part of ocaml/dune#10239, @Alizter) - Inline test libraries now produce aliases `runtest-name_of_lib` allowing users to run specific inline tests as `dune build @runtest-name_of_lib`. (ocaml/dune#11109, partially fixes ocaml/dune#10239, @Alizter) - feature: `$ dune subst` use version from `dune-project` when no version control repository has been detected (ocaml/dune#11801, @btjorge) - Allow `dune exec` to run concurrently with another instance of dune in watch mode (ocaml/dune#11840, @gridbugs) - Introduce `%{os}`, `%{os_version}`, `%{os_distribution}`, and `%{os_family}` percent forms. These have the same values as their opam counterparts. (ocaml/dune#11863, @rgrinberg) - Introduce option `(implicit_transitive_deps false-if-hidden-includes-supported)` that is equivalent to `(implicit_transitive_deps false)` when `-H` is supported by the compiler (OCaml >= 5.2) and equivalent to `(implicit_transitive_deps true)` otherwise. (ocaml/dune#11866, fixes ocaml/dune#11212, @nojb) - Add `dune describe location` for printing the path to the executable that would be run (ocaml/dune#11905, @gridbugs) - `dune runtest` can now understand absolute paths as well as run tests in specific build contexts (ocaml/dune#11936, @Alizter). - Added 'empty' alias which contains no targets. (ocaml/dune#11556 ocaml/dune#11952 ocaml/dune#11955 ocaml/dune#11956, grants ocaml/dune#4161, @Alizter and @rgrinberg) - Allow `dune promote` to properly run while a watch mode server is running (ocaml/dune#12010, @ElectreAAS) - Add `--alias` and `--alias-rec` flags as an alternative to the `@@` and `@` syntax in the command line (ocaml/dune#12043, fixes ocaml/dune#5775, @rgrinberg) - Added a `(timeout <float>)` field to the `(cram)` stanza to specify per-test time limits. Tests exceeding the timeout are terminated with an error. (ocaml/dune#12041, @Alizter) ### Changed - Format long lists in s-expressions to fill the line instead of formatting them in a vertical way (ocaml/dune#10892, fixes ocaml/dune#10860, @nojb) - Switch from MD5 to BLAKE3 for digesting targets and rules. BLAKE3 is both more performant and difficult to break than MD5 (ocaml/dune#11735, @rgrinberg, @Alizter) - Print a warning when `dune build` runs over RPC (ocaml/dune#11833, @gridbugs) - Stop emitting empty module group wrapper `.js` file in `melange.emit` (ocaml/dune#11987, fixes ocaml/dune#11986, @anmonteiro)
CHANGES: ### Fixed - Stop re-running cram tests after promotion when it's not necessary (ocaml/dune#11994, @rgrinberg) - fix: `$ dune subst` should not fail when adding the version field in opam files (ocaml/dune#11801, fixes ocaml/dune#11045, @btjorge) - Kill all processes in the process group after the main process has terminated; in particular this avoids background processes in cram tests to stick around after the test finished (ocaml/dune#11841, fixes ocaml/dune#11820, @Alizter, @Leonidas-from-XIV) ### Added - `(tests)` stanzas now generate aliases with the test name. To run `(test (name a))` you can do `dune build @runtest-a`. (ocaml/dune#11558, grants part of ocaml/dune#10239, @Alizter) - Inline test libraries now produce aliases `runtest-name_of_lib` allowing users to run specific inline tests as `dune build @runtest-name_of_lib`. (ocaml/dune#11109, partially fixes ocaml/dune#10239, @Alizter) - feature: `$ dune subst` use version from `dune-project` when no version control repository has been detected (ocaml/dune#11801, @btjorge) - Allow `dune exec` to run concurrently with another instance of dune in watch mode (ocaml/dune#11840, @gridbugs) - Introduce `%{os}`, `%{os_version}`, `%{os_distribution}`, and `%{os_family}` percent forms. These have the same values as their opam counterparts. (ocaml/dune#11863, @rgrinberg) - Introduce option `(implicit_transitive_deps false-if-hidden-includes-supported)` that is equivalent to `(implicit_transitive_deps false)` when `-H` is supported by the compiler (OCaml >= 5.2) and equivalent to `(implicit_transitive_deps true)` otherwise. (ocaml/dune#11866, fixes ocaml/dune#11212, @nojb) - Add `dune describe location` for printing the path to the executable that would be run (ocaml/dune#11905, @gridbugs) - `dune runtest` can now understand absolute paths as well as run tests in specific build contexts (ocaml/dune#11936, @Alizter). - Added 'empty' alias which contains no targets. (ocaml/dune#11556 ocaml/dune#11952 ocaml/dune#11955 ocaml/dune#11956, grants ocaml/dune#4161, @Alizter and @rgrinberg) - Allow `dune promote` to properly run while a watch mode server is running (ocaml/dune#12010, @ElectreAAS) - Add `--alias` and `--alias-rec` flags as an alternative to the `@@` and `@` syntax in the command line (ocaml/dune#12043, fixes ocaml/dune#5775, @rgrinberg) - Added a `(timeout <float>)` field to the `(cram)` stanza to specify per-test time limits. Tests exceeding the timeout are terminated with an error. (ocaml/dune#12041, @Alizter) ### Changed - Format long lists in s-expressions to fill the line instead of formatting them in a vertical way (ocaml/dune#10892, fixes ocaml/dune#10860, @nojb) - Switch from MD5 to BLAKE3 for digesting targets and rules. BLAKE3 is both more performant and difficult to break than MD5 (ocaml/dune#11735, @rgrinberg, @Alizter) - Print a warning when `dune build` runs over RPC (ocaml/dune#11833, @gridbugs) - Stop emitting empty module group wrapper `.js` file in `melange.emit` (ocaml/dune#11987, fixes ocaml/dune#11986, @anmonteiro)
CHANGES: ### Fixed - Stop re-running cram tests after promotion when it's not necessary (ocaml/dune#11994, @rgrinberg) - fix: `$ dune subst` should not fail when adding the version field in opam files (ocaml/dune#11801, fixes ocaml/dune#11045, @btjorge) - Kill all processes in the process group after the main process has terminated; in particular this avoids background processes in cram tests to stick around after the test finished (ocaml/dune#11841, fixes ocaml/dune#11820, @Alizter, @Leonidas-from-XIV) ### Added - `(tests)` stanzas now generate aliases with the test name. To run `(test (name a))` you can do `dune build @runtest-a`. (ocaml/dune#11558, grants part of ocaml/dune#10239, @Alizter) - Inline test libraries now produce aliases `runtest-name_of_lib` allowing users to run specific inline tests as `dune build @runtest-name_of_lib`. (ocaml/dune#11109, partially fixes ocaml/dune#10239, @Alizter) - feature: `$ dune subst` use version from `dune-project` when no version control repository has been detected (ocaml/dune#11801, @btjorge) - Allow `dune exec` to run concurrently with another instance of dune in watch mode (ocaml/dune#11840, @gridbugs) - Introduce `%{os}`, `%{os_version}`, `%{os_distribution}`, and `%{os_family}` percent forms. These have the same values as their opam counterparts. (ocaml/dune#11863, @rgrinberg) - Introduce option `(implicit_transitive_deps false-if-hidden-includes-supported)` that is equivalent to `(implicit_transitive_deps false)` when `-H` is supported by the compiler (OCaml >= 5.2) and equivalent to `(implicit_transitive_deps true)` otherwise. (ocaml/dune#11866, fixes ocaml/dune#11212, @nojb) - Add `dune describe location` for printing the path to the executable that would be run (ocaml/dune#11905, @gridbugs) - `dune runtest` can now understand absolute paths as well as run tests in specific build contexts (ocaml/dune#11936, @Alizter). - Added 'empty' alias which contains no targets. (ocaml/dune#11556 ocaml/dune#11952 ocaml/dune#11955 ocaml/dune#11956, grants ocaml/dune#4161, @Alizter and @rgrinberg) - Allow `dune promote` to properly run while a watch mode server is running (ocaml/dune#12010, @ElectreAAS) - Add `--alias` and `--alias-rec` flags as an alternative to the `@@` and `@` syntax in the command line (ocaml/dune#12043, fixes ocaml/dune#5775, @rgrinberg) - Added a `(timeout <float>)` field to the `(cram)` stanza to specify per-test time limits. Tests exceeding the timeout are terminated with an error. (ocaml/dune#12041, @Alizter) ### Changed - Format long lists in s-expressions to fill the line instead of formatting them in a vertical way (ocaml/dune#10892, fixes ocaml/dune#10860, @nojb) - Switch from MD5 to BLAKE3 for digesting targets and rules. BLAKE3 is both more performant and difficult to break than MD5 (ocaml/dune#11735, @rgrinberg, @Alizter) - Print a warning when `dune build` runs over RPC (ocaml/dune#11833, @gridbugs) - Stop emitting empty module group wrapper `.js` file in `melange.emit` (ocaml/dune#11987, fixes ocaml/dune#11986, @anmonteiro)
CHANGES: ### Fixed - Stop re-running cram tests after promotion when it's not necessary (ocaml/dune#11994, @rgrinberg) - fix: `$ dune subst` should not fail when adding the version field in opam files (ocaml/dune#11801, fixes ocaml/dune#11045, @btjorge) - Kill all processes in the process group after the main process has terminated; in particular this avoids background processes in cram tests to stick around after the test finished (ocaml/dune#11841, fixes ocaml/dune#11820, @Alizter, @Leonidas-from-XIV) ### Added - `(tests)` stanzas now generate aliases with the test name. To run `(test (name a))` you can do `dune build @runtest-a`. (ocaml/dune#11558, grants part of ocaml/dune#10239, @Alizter) - Inline test libraries now produce aliases `runtest-name_of_lib` allowing users to run specific inline tests as `dune build @runtest-name_of_lib`. (ocaml/dune#11109, partially fixes ocaml/dune#10239, @Alizter) - feature: `$ dune subst` use version from `dune-project` when no version control repository has been detected (ocaml/dune#11801, @btjorge) - Allow `dune exec` to run concurrently with another instance of dune in watch mode (ocaml/dune#11840, @gridbugs) - Introduce `%{os}`, `%{os_version}`, `%{os_distribution}`, and `%{os_family}` percent forms. These have the same values as their opam counterparts. (ocaml/dune#11863, @rgrinberg) - Introduce option `(implicit_transitive_deps false-if-hidden-includes-supported)` that is equivalent to `(implicit_transitive_deps false)` when `-H` is supported by the compiler (OCaml >= 5.2) and equivalent to `(implicit_transitive_deps true)` otherwise. (ocaml/dune#11866, fixes ocaml/dune#11212, @nojb) - Add `dune describe location` for printing the path to the executable that would be run (ocaml/dune#11905, @gridbugs) - `dune runtest` can now understand absolute paths as well as run tests in specific build contexts (ocaml/dune#11936, @Alizter). - Added 'empty' alias which contains no targets. (ocaml/dune#11556 ocaml/dune#11952 ocaml/dune#11955 ocaml/dune#11956, grants ocaml/dune#4161, @Alizter and @rgrinberg) - Allow `dune promote` to properly run while a watch mode server is running (ocaml/dune#12010, @ElectreAAS) - Add `--alias` and `--alias-rec` flags as an alternative to the `@@` and `@` syntax in the command line (ocaml/dune#12043, fixes ocaml/dune#5775, @rgrinberg) - Added a `(timeout <float>)` field to the `(cram)` stanza to specify per-test time limits. Tests exceeding the timeout are terminated with an error. (ocaml/dune#12041, @Alizter) ### Changed - Format long lists in s-expressions to fill the line instead of formatting them in a vertical way (ocaml/dune#10892, fixes ocaml/dune#10860, @nojb) - Switch from MD5 to BLAKE3 for digesting targets and rules. BLAKE3 is both more performant and difficult to break than MD5 (ocaml/dune#11735, @rgrinberg, @Alizter) - Print a warning when `dune build` runs over RPC (ocaml/dune#11833, @gridbugs) - Stop emitting empty module group wrapper `.js` file in `melange.emit` (ocaml/dune#11987, fixes ocaml/dune#11986, @anmonteiro)
CHANGES: ### Fixed - Stop re-running cram tests after promotion when it's not necessary (ocaml/dune#11994, @rgrinberg) - fix: `$ dune subst` should not fail when adding the version field in opam files (ocaml/dune#11801, fixes ocaml/dune#11045, @btjorge) - Kill all processes in the process group after the main process has terminated; in particular this avoids background processes in cram tests to stick around after the test finished (ocaml/dune#11841, fixes ocaml/dune#11820, @Alizter, @Leonidas-from-XIV) ### Added - `(tests)` stanzas now generate aliases with the test name. To run `(test (name a))` you can do `dune build @runtest-a`. (ocaml/dune#11558, grants part of ocaml/dune#10239, @Alizter) - Inline test libraries now produce aliases `runtest-name_of_lib` allowing users to run specific inline tests as `dune build @runtest-name_of_lib`. (ocaml/dune#11109, partially fixes ocaml/dune#10239, @Alizter) - feature: `$ dune subst` use version from `dune-project` when no version control repository has been detected (ocaml/dune#11801, @btjorge) - Allow `dune exec` to run concurrently with another instance of dune in watch mode (ocaml/dune#11840, @gridbugs) - Introduce `%{os}`, `%{os_version}`, `%{os_distribution}`, and `%{os_family}` percent forms. These have the same values as their opam counterparts. (ocaml/dune#11863, @rgrinberg) - Introduce option `(implicit_transitive_deps false-if-hidden-includes-supported)` that is equivalent to `(implicit_transitive_deps false)` when `-H` is supported by the compiler (OCaml >= 5.2) and equivalent to `(implicit_transitive_deps true)` otherwise. (ocaml/dune#11866, fixes ocaml/dune#11212, @nojb) - Add `dune describe location` for printing the path to the executable that would be run (ocaml/dune#11905, @gridbugs) - `dune runtest` can now understand absolute paths as well as run tests in specific build contexts (ocaml/dune#11936, @Alizter). - Added 'empty' alias which contains no targets. (ocaml/dune#11556 ocaml/dune#11952 ocaml/dune#11955 ocaml/dune#11956, grants ocaml/dune#4161, @Alizter and @rgrinberg) - Allow `dune promote` to properly run while a watch mode server is running (ocaml/dune#12010, @ElectreAAS) - Add `--alias` and `--alias-rec` flags as an alternative to the `@@` and `@` syntax in the command line (ocaml/dune#12043, fixes ocaml/dune#5775, @rgrinberg) - Added a `(timeout <float>)` field to the `(cram)` stanza to specify per-test time limits. Tests exceeding the timeout are terminated with an error. (ocaml/dune#12041, @Alizter) ### Changed - Format long lists in s-expressions to fill the line instead of formatting them in a vertical way (ocaml/dune#10892, fixes ocaml/dune#10860, @nojb) - Switch from MD5 to BLAKE3 for digesting targets and rules. BLAKE3 is both more performant and difficult to break than MD5 (ocaml/dune#11735, @rgrinberg, @Alizter) - Print a warning when `dune build` runs over RPC (ocaml/dune#11833, @gridbugs) - Stop emitting empty module group wrapper `.js` file in `melange.emit` (ocaml/dune#11987, fixes ocaml/dune#11986, @anmonteiro)
For a given library with inlines tests, we add an alias with the name of the library for running those tests prefixed by
inline-test-.Further work on separating out the partition actions would allow us to have an alias for each partition.
For now, this will allow us to run inline tests with the runtest command. Further work on that will follow this PR.
This addresses part of #10239 the remaining part would be to add an alias to the
(test)stanza.