Nice, excellent elaborate write up @lucacasonato!

I'm not explicitly against this yet but it brings up the question of procedural vs declarative.

So far our approach has been mostly declarative, in that test definitions have been specified as objects so mixing procedural and declarative in this way might not be that great.

The Go test runner is declarative all the way through, to ignore you call the Skip method on the test context, but we have it as an option so already we've diverged to the point where there is very little purpose of a context like this.

In terms of handling our edge cases, to some degree this seems to introduce more than it solves.
The advantage of being declarative is that we "the test runner" can easily reason about things ahead of time before tests start running any test cases.

For example, we could easily figure out things like how to partition sets of tests. Like a set of tests marked as parallel safe with the same permissions can be partitioned and run in parallel based on their permission requirements.

Same applies to sanitisers altho with sanitiser failure, however it will be harder for you the user to pin-point which test is leaking resources if N tests are run in parallel; so to debug you'd want to disable test case parallelism. This isn't a problem for the exit sanitiser, only applies to ops and resources.

Potentially, we could be a lot smarter about tracking that but since this is global state so it's non trivial problem, being able to temporarily disable parallelism with a flag for debugging seems like a good compromise here.

Filtering is also difficult with a procedural model like this, because we don't know what subtests there are until in the middle of test execution.

Unlike Go we have top level execution with await, so we don't need functions like Skip to ignore tests.

We also don't need to execute code inside a function to generate a table test, we can just execute that in the top level scope, which is going to be prettier with time as we get do let blocks.

declare namespace Deno {
  export interface TestDefinition {
    fn: () => void | Promise<void>;
    name: string;
    ignore?: boolean;
    parallel?: boolean,
  }

  export interface TestCollection {
    entries: Array<TestDefinition | TestCollection>;
    name: string;
  }

  export function test(test: TestDefinition | TestCollection);
}

The basic version of Deno.test would look something like this:

Deno.test({
  name: "Array.prototype.push",
  entries: [
    {
      name: "it allows one element to be given",
      fn() {
        // ...
      },
    },
    {
      name: "it allows two elements to be given",
      fn() {
        // ...
      },
    },
  ],
]); 

Outputs:

Array.prototype.push
  it allows one element to be given ... ok
  it allows two element to be given ... ok

Also, since we swallow the console it does probably makes sense to have some sort of context available.
But I see its main use being more in line with printing diagnostics, log messages and extra things to the test harness without getting involved with the scheduling of test cases.

Haven't really thought this through yet, but maybe a console like interface.

interface TestContext {
  assert();
  log();
  debug();
  // ...
}

This is loosely based on TAP, which prints lines like these as "comments".

Hey @lucacasonato, I was just curious if the test part(s) in the output were needed? I wonder if it adds uneccessary bloat? Would be interested to hear your thoughts on this (and others too if people want to chime in, after all, this is a discussion :) )

@yacinehmito AFAIK, there are no special provisions for the beforeEach/afterEach hooks. If you are writing in BDD style, you can use (or take inspiration from) this gist where describe sections are treated as "groups", with it being test steps, and hooks being run prior/post each step.

Hi all, this discussion has been greatly enlightening, and I have some input.

To make it more understandable, I have divided my thoughts into four sections:

1. Declarative vs. Procedural API
   I share some of @caspervonb's concerns around the procedural nature of the steps API and how concepts get mixed when using test and steps.
2. Implementing hooks
   How do we implement hooks considering the "declarative vs procedural" conundrum?
   Should we even implement hooks, or should we delegate that to third party library implementers?
3. Naming
   How should we name the methods in this API considering the two points above?
   Does it make sense to have a step API?

Items 2 and 3 depend on item 1.

1. A declarative vs. a procedural API

Currently, our approach to tests requires users to declare them twice. Once to implement the tests themselves and then to explain their tests' structure, which is already implicit in the way tests are nested.

This duplication requires a lot of effort from users and requires us to write more code on our end.

By following a declarative approach, we can separate the act of declaring tests from the act of running them.

My proposed API is similar to what @yacinehmito has suggested. The only difference is that it does not require users to await on a step itself. Instead, users will declare async functions, upon which the runner will await.

Users don't need to await on a particular step because when they "declare" a test, they are not telling it to run. They're simply saying that a test exists and has an asynchronous function associated with it.

// Notice how the Deno.test callback doesn't need to return a promise.
// It doesn't need to return a promise because it _schedules_ tests (in other words, it _declares_ them)
Deno.test("database tests", (t) => {
  // Notice how in the previous example you naturally
  // used an `async` callback, not an `await` on `beforeAll`
  t.beforeAll(async () => {
      return {
        database: await setupDatabase()
      };
  });

  t.step("first step using database", async (t, { database }) => {
    await myAsyncTask();
  });
  
  t.step("second step using database", () => { /* ... */ });

  t.afterAll(async ({ database }) => {
    await database.destroy();
  });
});

Such separation:

1. Makes it trivial to implement hooks
2. Allows us to have more modular code
3. Facilitates the filtering and sanitizing of tests and their efficient scheduling
4. Opens up the possibility of third-party library implementers to hook into our declarative API to write their own runners or simply listen to our runner's events

2. Implementing hooks

2.1 How do we implement hooks?

With such an API, the beforeAll, beforeEach, afterAll, afterEach implementation becomes trivial. They schedule functions, and our runner knows when to execute those functions and which ones to filter out.

I feel like we should have these explicit hooks, as they cover all the possible points when a user may want to execute setup/teardown operations. Furthermore, these functions are already widely used in the JavaScript testing community, so everyone is familiar with them. Given its adoption by people and all other testing frameworks, I believe such an interface has been an enormous success.

2.2 Should Deno even implement hooks? If so, how?

In my view, it should.

If Deno wants to be "a productive and secure scripting environment for the modern programmer", it must come with everything users will need to test their code.

Now, everyone has a different view on what's essential. Still, I think the vast majority of people would agree that the set of crucial features includes an assert function and testing hooks for setting up and tearing down infrastructure for both individual tests and groups of tests.

I think the most important aspect up for debate is not whether Deno should implement hooks, but whether it wants to have the final word on how tests should be written.

A good balance between incentivizing standardization of good practices and having an open ecosystem would be to follow an approach similar to what Jest has done with jest-circus. Such an approach consists of implementing their own declarative API but emitting events which allow users to do whatever they want to run hooks, or tests themselves (and actually a bit more). This step takes us even further in the direction of separating the declaration of tests to their execution.

These are the advantages of such an approach:

1. We're still opinionated about how users declare tests, so there's standardization across Deno's ecosystem.
   Furthermore, because events for each hook are still emitted in a particular order, we're also opinionated on which hooks users can implement and the semantics of each.
   For example: beforeAll is always emitted before all tests within a group, even though a third party may want to do something different when it's emitted.
2. Third-party implementers can do whatever they want in terms of executing tests (and hooks).
   Suppose someone wants to schedule each group of tests to run in a particular container. In that case, they can implement a runner that listens to the test events emitted and runs each within a particular container.
   Furthermore, if third-party approaches to testing are successful, we can always bring them into the core.

3. Naming

Even though I like the step interface, I think its semantics should be different. Furthermore, I think we should make our APIs familiar with what people are already used to, especially considering the success of the most widely used APIs in the JS testing ecosystem.

That doesn't mean we necessarily need to name our APIs describe and it. I think most people actually misuse them, as these APIs are not supposed to simply mean you're "grouping" tests. These APIs exist to incentivize people to write tests that read like "given scenario X (describe) then my code (it) does Y". The problem is that most people just use describe to name a suite and it to name a test.

For example:

Expected usage (ideal):

describe("when buying alcoholic drinks" , () => {
  describe("when buyers don't look like they're 25", () => {
    it("shows a confirmation popup asking for ID", () => { /* ... */ });
  });
  
  describe("when buyers look like they're over 25", () => {
    it("immediately charges the customer", () => { /* ... */ });
  });

  it("sends telemetry data to the central management system", () => { /* ... */ });
});

Actual usage:

describe("Sales Module" , () => {
  it("Confirmation popup for under 25", () => { /* ... */ });
  it("Immediate checkout for over 25", () => { /* ... */ });
  it("Telemetry data calls", () => { /* ... */ });
});

Considering the above, I don't think we need to name our functions describe nor it. Instead, we could name them suite and test considering the actual usage patterns illustrated above.

My only strong opinion is that I do think that APIs for recursive grouping (meaning you can nest as many "describes" as you want) and individual tests must exist. Hooks should exist too (as per the explanation above). For hooks, given how commonplace and successful they are, I'd also advise we keep the same names (as they also evoke similar semantics in people's minds).

3.1 What about steps? + A concrete usage example

I think that steps have a place in tests, but I think they should be used for a different purpose. In my view, steps represent an atomic part of a test. Steps cannot have beforeEach or afterEach hooks associated with them. They exist just so that users can have a more detailed output for critical parts of a test and so that runner implementers (as described in section 2) could perform a particular operation for each step of a test.

Elastic's synthetics runner is a concrete example of how useful it is to have a step API (and another argument for having a jest-circus approach to our testing infrastructure).

Disclaimer: I do work for Elastic.

The synthetics runner (which is nothing but a test runner, really), uses the step API to indicate when it should take a screenshot of a page so that users can more easily debug their synthetics tests.

Now, if you imagine that step preserves the semantics I described above and that we have a jest-circus-like system, anyone could hook into Deno to take screenshots for each of theirs test's steps (as in the synthetics example) or record a particular state of a database, or HTML page for each step (or even send telemetry data to their servers).

For example you can write a mocha polyfill on top of the imperative API.

@lucacasonato maybe that "polyfill" would be a good addition to deno_std/testing? The Jest/Mocha BDD style is popular in JS world, and I doubt most of the users would care all that much about the implementation as long as they get familiar looking test files, so this would be a low overhead option as compared to porting/using mocha in compat mode.