Why do we need A/B tests?

• A/B testing is a key feature for experimentation in order to increase performance of Protected audience
• It allows us to measure the impact of technical changes
• We must be able to measure long term effects in a consistent way

To give an example of what we mean by long term effects let’s look at a complex user journey and assume that we split users per publisher website (because we have access to the hostname in PA API), on some websites we propose buying strategy A and on some publisher websites we propose buying strategy B, and that we can measure conversions like sales for each ad display.

In retargeting, we show a banner to users multiple times before they buy. For example, a user has added Nike shoes to his basket but has not converted, we will remind him of the product, through ads on several publishers. When he converts, the sale will be attributed to the publisher on which was shown the last ad and not to whatever happened before that. In other terms, it is impossible to measure the effect of a buying strategy A versus B since we will not have a single identifier across sites.

Existing mechanism with ExperimentGroupId

https://github.com/WICG/turtledove/blob/main/FLEDGE.md#21-initiating-an-on-device-auction

Optionally,perBuyerExperimentGroupIds can be specified to support coordinated experiments with buyers' trusted servers. If specified, this must also be an integer between zero and 65535 (16 bits).​

The expected workflow has been described here:
Extending FLEDGE to support coordinated experiments by abrik0131 · Pull Request #266 · WICG/turtledove

Our understanding is that this translates to:

Pros:

• buyerExperimentGroupId can be dynamically set by buyer as part of the contextual call allowing any split (see comment below, it already might not apply anymore as async calls should be used to reduce auction latency)
• 16 bits (65535 different values) is big enough to handle multiple experiments at a time
• this AB testing seems interesting to measure technical changes
• analysis can directly be done via ExperimentGroupId as it is propagated to reportWin

Cons:

• We cannot measure long term effects (as explained above) as the split must be done based on contextual signals for example by publisher domain, so basically we can only measure something that is directly attributed to this ad display, a same Chrome browser might see changes on the very same ad campaign in population A & B
• One alternative, that would allow measuring long terms effects, would be segmenting users based on geolocation, the challenge here would be to have populations of same size and same user behavior, so it will not be universally applicable but depend on the use case
• It might not be applicable to auctions where signals are resolved in an asynchronous way as in this case the contextual call and the key/value server call run in parallel

Splitting per interest group and 1st party user id

Doing a per interest group split seems appealing because for interest groups that are created on one advertiser website one could apply the same changes to the same campaigns to all 1st party users of this advertiser.

This would mainly work for single advertiser AB tests where we target users that already went to advertiser web page. It would work less well for more complex scenarios on all our traffic where we modify the behavior of multiple campaigns on multiple websites and in this case the same drawback as above, the very same user could see behavior changes in population A and B.

As we would split users during tagging phase we cannot guarantee that we really see those users again for a bidding opportunity. So we cannot guarantee an even split as for bidding we might only see n% of users of population A for bidding and a different amount for population B (some more explanation here Approach 2: Intent-to-Treat)

Pros:

• Could handle single advertiser scenarios where we consistently know the user with its 1st party id

Cons:

• For reporting we need to log the AB test population of computeBid to reportWin
  • it could be done by encoding the AB test population inside the renderUrl at the expense of k-anonymity, handling 5 AB tests in a independent way would mean 2^5 renderUrls
  • alternatively aggregated reporting like Aggregated ARA could be used
• We cannot handle a large number of AB tests in parallel, in any case less than with ExperimentGroupId
• Leakage as the same user will change for different advertiser websites which will be an issue when the behavior is changed on retargeting campaigns for multiple advertisers or when more upper funnel campaigns are used
• Additional bias because we split user based on tagging behavior and not when we get a bid opportunity

Using shared storage for AB testing

The shared-storage proposal already has a section on how to activate AB tests. The general idea is to create a unique user identifier (seed) for the Chrome browser with generateSeed, then call the window.sharedStorage.selectURL operation which takes a list of urls, hashes the user identifier to an index in this list and then returns the url for that user. The AB test population would be encoded in the url and as the number of urls is limited to 8 urls it would allow 3 bits of entropy for the user population. As different urls can be used for each call and would leak 3 bit all the time some mechanisms are in place to limit the budget per 24h per distinct number of urls (see https://github.com/WICG/shared-storage#budgeting).

As of now shared storage can only be called from a browser Javascript context and not from a Protected Audience worklet. This means the urls selection can only happen during rendering and not during bidding and therefore shared storage can only be used for pure creative AB tests and not Protected Audience bidding AB tests. So we still need a dedicated proposal to activate Protected Audience AB tests.

Proposal - Inject a low entropy global user population into computeBid

For real world scenarios a global user population would still be needed for AB tests that need to measure complex user behaviors. As injecting any form of user identifier would leak additional information we propose a low entropy user identifier and some mitigations to prevent using or combining this into a full user identifier.

Chrome could cluster all users into a low entropy UserExperimentGroupId something like 3 bits. This identifier should be randomly drawn for each ad tech and not unique to all actors to prevent that our measurement cannot be influenced by the testing of other ad techs.

As attribution is measured for each impression or click we would like this identifier to be stable for some time but it should be also shifted on a certain amount of users to prevent a large population drift over time. Long running AB tests will influence users and then user behavior will change over time introducing some bias. The usual way to solve this is restarting an AB test which cannot be done here for such a limited amount of buckets. So one idea might be to constantly rotate the population. Constantly rotating the population would be also useful to limit the effectiveness of a coordinated attack among Ad Techs to identify a user. If 1% of users get reassigned to population each day it would mean that after 14 days 14% of user might have shifted population.

If the labels are rotated every X weeks, it adds further burden to those trying to collude and update their 1st-party ID → global ID mappings

This new population id would be injected only into the generateBid function and also the trusted key/value server (to mirror current ExperimentGroupID behavior and because many of our computations are still server side, it is secure by design as it will run in a TEE without side effects).

The identifiers could only get out of the of generateBid function via existing mechanisms and that already present privacy/utility trade offs, for example:

• by adding more renderUrls at the expense of k-anonymity
• by reserving some bits of modelingSignals at the expense of handling less advertiserSignals
• by using aggregated reporting at the expense of DP noise and bucketization

If we encode the 3 bits into renderUrl this proposal seems very aligned with the proposal on shared-storage to allow 8 URLs (= 3 bits of entropy) for selectURL to activate creative AB testing (post bidding). In our case as Chrome would control the seed and the generateSeed function can not be used we would not leak more than 3 bit. So introducing any form of budget capping seems not necessary.

To prevent some cookie sync scenario where ad techs combine this new id into a full user identifier Chrome could add an explicit statement to the attestation to prevent ad techs sharing this id.

By design as we have few AB test populations we could only run a limited number of AB tests at the same time but we could reserve this for important AB tests and use the ExperimentGroupId mechanism more for technical AB tests.