Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FTL issues with uniform sharding and high test count #2491

Open
joehoag opened this issue Apr 16, 2024 · 1 comment
Open

FTL issues with uniform sharding and high test count #2491

joehoag opened this issue Apr 16, 2024 · 1 comment
Labels

Comments

@joehoag
Copy link

joehoag commented Apr 16, 2024

Describe the bug
We use flank to run our Android FTL tests. And we use uniform sharding (num-uniform-shards setting) instead of smart sharding because we need access to a valid shardIndex to support shard-specific users, and smart sharding sets the shardIndex to 0 for all shards.

Our test runs just recently started failing on FTL, and a quick summary of the reason that they fail is that we seem to have hit a limit (roughly 970) of the number of tests that we can include in a run. The reason for that is that flank produces a command like this for each shard:

adb shell am instrument -w -r \
  -e numShards 40 \
  -e shardIndex 2 \
  -e class com.onepeloton.callisto.instrumentationTest.flows.ActivityFeedMobileTests#test_C13590563_feed_workout_cards_class_thumbnail_JWO,class com.onepeloton.callisto.instrumentationTest.flows.CircuitsFiltersMobileTests#test_C18008311_gym_filter_length...(970 test targets)"

In other words, each shard gets passed all 970 tests, instead of just the 25 or so tests that that shard needs to run. And that super-long list causes an error in FTL:

/sdcard/run_command_7539975812237112782.sh[2]: /system/bin/app_process: Argument list too long

So, I was hoping that you could either (1) fix this problem by not passing all tests to each shard, (2) come up with a workaround, or (3) report a valid shardIndex when smart-sharding is used so that I don't have to use uniform sharding.

To Reproduce

Steps to reproduce the behavior:

  1. Create 970 tests (see the android_shards.json file below that flank produced for us).
    matrix-1rmnzi34f1nkc_android_shards.json

  2. Run on FTL with uniform sharding. I don't think that the shard count even matters, but we used 40.

Expected behavior
The test run completes.

Observed behavior
The test run never gets off the ground. All shards report

/sdcard/run_command_7539975812237112782.sh[2]: /system/bin/app_process: Argument list too long

Details (please complete the following information):

Have you tested on the latest Flank snapshot?

Would it make a difference?

Post the output of flank --version.

We are using flank v23.10.1

Additional context

Add any other context about the problem here.

We are able to work around this problem by excluding some test modules to reduce the total test count. But once the test count hits a certain number (~970), we get the "Argument list too long" failure message.

@joehoag joehoag added the Bug label Apr 16, 2024
@SelaseKay
Copy link
Collaborator

SelaseKay commented Nov 20, 2024

Hi @joehoag, can you share your flank.yml config and if possible, your app-debug.apk as well as app-debug-androidTest.apk?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants