Hi Fabian Höring,

Thnx for the question and sorry for the delay. We're currently working on getting back to you,

Thank you,
Alexander

Hi sorry for the long delay.

• Could you please elaborate on the workload you have in mind? The throughput and latency performance depends greatly on the workload. For example what’s the context and use case and what’s the main operations?
• SetCodeObject is called before ExecuteCode so the Snapshot would be created. This should minimize the latency for context creation. We verified this using benchmarks. Context has to be created for every execution to maintain per-execution isolation.
  The latency varies depending on the UDF code. If you are curious about the numbers for your code, they should be easy to using benchmark setup we have.
  Since the setup is in a dependency, that repository would need to be pulled in.
  This can be done by cloning and checking out the commit we use. Example -
  git clone https://github.com/privacysandbox/data-plane-shared-libraries.git
  cd data-plane-shared-libraries
  git checkout 89e8cf07e233779e92915fa6fbcd854f648e327c
  Or downloading the zip file for the version we use at HEAD.
  To run the benchmarks implemented, the following command can be used -
  scripts/run-benchmarks --target //src/roma/benchmark:kv_server_udf_benchmark_test --benchmark_time_unit ms

Sample output -

We would like to help you achieve an accurate measurement. If you are interested in collaborating to measure more please feel free to let us know what you think we may be able to help with.

Hello @peiwenhu,

Thanks for the information. I will come back to you with more information about the workloads.

About compîling data-plane-shared-libraries locally.
Sorry for this question but I never used Google specific tooling like bazel before.

If I do this:

cd data-plane-shared-libraries
git checkout 89e8cf07e233779e92915fa6fbcd854f648e327c

What command do I need to execute to compile this ?
How do I need to change the workspace file to actually pull my local sources ?

Hello!

What command do I need to execute to compile this ?

For running benchmarks, you can simply use the specified command -

scripts/run-benchmarks

(For running your benchmarks, you will need to modify kv_server_udf_benchmark_test and include the code you want to benchmark.)

If you run at HEAD for data-plane-shared-libraries, the following command can be used -

scripts/run-benchmarks --target //src/roma/benchmark:kv_server_udf_benchmark_test --benchmark_time_unit ms

How do I need to change the workspace file to actually pull my local sources

For running benchmarks, I don't think you have to modify your K/V server workspace. However, in general local_repository bazel rule can be used.

Let us know if any more information is needed from our side.

Thanks!

Thanks. I was able to execute the benchmarks like this:

./builders/tools/bazel-debian run //scp/cc/roma/benchmark/test:kv_server_udf_benchmark_test -- --benchmark_out=/src/workspace/dist/benchmarks/kv_server_udf_benchmark.json --benchmark_out_format=json --benchmark_time_unit=ms
./builders/tools/bazel-debian run //scp/cc/roma/benchmark/test:benchmark_suite_test -- --benchmark_out=/src/workspace/dist/benchmarks/benchmark_suite_test.json --benchmark_out_format=json --benchmark_time_unit=ms

I get the same results of ~1ms for executing an empty JS function.

I also ran the multi threaded roma workloads which give results of ~1000 request per sec.

An additional question on that. I never succeeded to get dropped requests even with a worker size of 1 and a queue size of 1. Is this expected ?

test_configuration.workers = 1;
    test_configuration.inputs_type = InputsType::kSimpleString;
    test_configuration.input_payload_in_byte = 500000;
    test_configuration.queue_size = 1;
    test_configuration.batch_size = 1;
    test_configuration.request_threads = 30;
    test_configuration.requests_per_thread = 1000;

Those tests seem to be done from the same machine which means the client can impact the server and vice versa. We probably have to run the full server and run the client with some web load injector like gatling to get representative results.

An additional question on that. I never succeeded to get dropped requests even with a worker size of 1 and a queue size of 1. Is this expected ?

In these benchmarks, for every request, we wait for the response individually. Hence, this behaviour is expected.

@peiwenhu @lx3-g @a-shruti

We have done several web load tests with Gatling (ramp-up then constant load to 100k QPS over several steps and 120 secs each).

Performance test setup (c# vs JS UDF execution)

We deployed KV server version 0.16 on our own infrastructure in a container on 1 instance with 8 cores (16 threads) & 16 GB of memory with the following components:

• Envoy proxy
• ROMA JS UDF engine with 16 workers
• In memory lookup

Javascript UDF code:

• no batching (broken in version 0.16)
• it reads the keys from the request;
• it queries the in-memory datastore to get the values of those keys;
• it replies with those values.

We then implemented the same basic logic in a c# vanilla asp.net server and executed the same web load test for comparison.

Web assembly test setup

We have deployed and benchmarked the provided c++ => WASM sample file from here (file size: 100 KB)

We also tried out the provided Microsoft templates to compile c# to WASM (dotnet 9 required, file size: 30 MB, contains c# runtime)

dotnet workload install wasi-experimental
dotnet new wasiconsole -o Ans43
dotnet build -c Release

Results

Conclusion

• Google KV JS implementation can handle 1000s of queries per second with ms second latency
• an empty c#/asp.net service can handle 10 times more queries (45k QPS vs 5k QPS) with better mean response times (3ms vs 6ms)
• c++ WASM can handle 100 QPS, there is a clear down lift in performance, seems prohibitive already
• c# WASM can handle 1 QPS, it replies but can be considered as not working
• WASM in general doesn't seem like a real alternative, so the effective mandatory UDF backend seems to be Javascript