Given limited resources I think the main question we should answer is "did vendor X made an effort to support feature Y". Whether that feature implementation is free of bugs is a more ambitious task and out of scope I think, at least initially. Start with something simple and then improve over time. However, if we do find/know of corner cases that a vendor fails to support I think it would be a good strategy to add that to the test suite and then make a bug report to the vendor with a pointer to the test suite. I want this test suite to be a way for users to evaluate tools and if a vendor has a lot of bugs in its implementation the user should be aware of that. Vendors should also know that the test suite exists and is being used for that purpose. Ideally it should be a part of their continuous internal testing.

@LarsAsplund I agree with your statements and the limited resources we have (for now). Especially I have been out of open source business for a while due to a project :(.

The aim of my questions was more on a structural focus on:

• how to structure / divide tests
  e.g. now we have 2008 vs. 2019 tests. Another orthogonal dimension might be the "levels" posted in this question.
• how and where to import existing tests and where to accept merge requests.

I believe for the beginning we have most existing tests in "level 2" and focusing on VHDL-2008, right? Some questions like lately raise for GHDL might touch "level 3) like std_logic vs. bit implementations, but I assume this is rare.

Level 4/5 tests are out of scope for now, right?
(But this repository might be a host for them in future.)

Ok, I understand. I like the idea of levels but I think we should set a priority order based on the user perspective.

First priority for a user would probably be if the feature is supported at all so that would be level 2.

Second priority would probably be to what extent is this feature supported. What subset of types corresponds to level 3. Can be used in code to be synthesized or just in testbenches corresponds to level 5. Level 3 is easier to test right now since it doesn't involve any synthesis tool.

I feel that levels 1 and 4 are of lower priority.

A practical implication of the level thinking could be that tests on std_logic types are separated from those working on other types. They are usually in different test cases but we could also work with naming to be able to run all tests of a specific level and have the results sorted by standard version and level.

I agree with your summary. Level 1 is more for low-level tool and parser or linter implementer. It's more like a lexer parser stress test based on valid language inputs OR proper error generations as defined in the language. (The language states that certain types of erros must be or might be detected.)

Of cause this is low priority.

As said, I agree on Levels 2 / 3 being the main focus.

I just would like to see some kind of directory structure and/or naming convention how to save test cases and how to iterate or detect what cases / combinations might be missing.

It should be somehow identifiable based on the test name or atleast the description, why test1 passes and test2 fails, but they are testing the same feature, but maybe one is used in a function the other one in a process.

We are adding lots and lots of tests, so we should think about structure and naming to avoid to many restructuring of hundreds of tests :)

Agree on naming conventions. What we have to work with is the library name, the testbench name, the test case name and the error/pass message of a check. It would also be possible to assign VUnit attributes to test cases allowing us to reference related part of the LRM. That can be used to just run tests related to a part of the LRM. We could even use that to create a HTML test report cross linking the LRM. Let me produce an example that can be discussed.

Yes, correlating tests and LRM sections would be really great. If this works, it can also be an example of VUnit being traceable.

Yes, this would be the first public test to requirement traceability I've seen. It's used within companies though since that's an important feature in safety-critical designs.