Summary

This topic was brought up by @krsriq in #82 and partially addressed in #90. This issue is to discuss possible solutions to make sure our tests are properly checking the library behavior.

Versions

Self-enclosed code snippet for reproduction

Tests are now seeded with 1 to assure test result consistency but that opens another potential issue:

In this code, $this->faker->creditCardType will always return MasterCard because generator is seeded with 1 before each test method execution so removing the last element in array will still make the test pass every time.

Possible solutions

While testing random data generation in a reproducible way is difficult, we should make sure our tests are working properly.

Retry tests

I have tried experimenting with PHPUnit --repeat 100 flag to repeat the test several times but setUp (and therefore seed(1)) is called before each repeat too.

Mark tests that require seeding

Another approach would be to introduce a custom @seed <int> annotation for test methods or classes that explicitly require seeding before test to get specific results:
https://github.com/FakerPHP/Faker/blob/main/test/Faker/Provider/ja_JP/InternetTest.php and https://github.com/FakerPHP/Faker/blob/main/test/Faker/Provider/uk_UA/PersonTest.php

This approach together with --repeat 100 flag in one of our test matrix should allow us to make sure all the tests are actually making sure that generated data is always correct.

Other notes

• How to make sure that each element is returned at least once after N repetitions?