Hey, was there any progress regarding this issue ? it seems its detrimental for the correct operation of llmperf but on the other hand no one gives it any attention.
@gracehonv

I guess maybe it might be better to move it to a different file in an overlay manner instead of modifying the token_benchmark_ray.py file..

Anyway, I have been using llmperf well since applying this commit without any errors.

I had made a commit 5 months ago to move the prompt construction outside the send loop so the benchmark doesn't get slowed down. I didn't know this affects the num_concurrent_requests mode. Let me know if I can help in some way.

@gracehonv
I rebased my code onto your commit, and it didn't slow down.

It does look like in your commit you have prepared all of the prompts before launching all the concurrent requests so there shouldn't be any slowdown. Thanks for fixing!

The y axis is the number of concurrent requests.

The pattern on the left is with the current token_benchmark_ray, the pattern on the right is with your fix.

Thanks for the fix!

@cpwan,
Thank you for your analysis and visualization related to this PR.

@avnishn @rickyyx
Would it be possible to get this PR reviewed please? Thank you.