Skip to content

privacysandbox/protected-auction-key-value-service

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

27 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Important

The main branch hosts live code with latest changes. It is unstable and is used for development. It is suitable for contribution and inspection of the latest code. The release-* branches are stable releases that can be used to build and deploy the system.


FLEDGE has been renamed to Protected Audience API. To learn more about the name change, see the blog post.


Privacy Sandbox Logo Protected Auction Key/Value service

State of the project

The current codebase represents the implementation of the TEE-based Key/Value service by Privacy Sandbox.

For Protected Audience, the service can be used as a BYOS KV server. Soon it can be used to communicate with Chrome and the Bidding and Auction services using V2 protocol.

For Protected App Signals, the service should be used as the ad retrieval server.

It can be integrated with Chrome and Android with the Privacy Sandbox unified origin trial and Privacy Sandbox on Android Developer Preview.

Current features

Build and deployment

  • Source code is available on Github
  • Releases are done on a regular basis
  • Binaries can be built from source code
    • C++ binary
    • [AWS & GCP] Docker container image
    • [AWS]: Nitro EIF
    • [AWS]: Reference AMI
    • Other tools
  • Server can run as a standalone local process for testing without any cloud dependency or TEE-related functionality
  • Server can be deployed to AWS Nitro enclave
  • Server can be deployed to GCP Confidential Space
  • Reference terraform available for a clean and comprehensive deployment to AWS or GCP
    • Clean: assumes the cloud environment has no preset infrastructure
    • Comprehensive: deploys all dependencies and some recommended (but not necessarily required) configuration
    • Many server behaviors can be configured by parameter flags without having to rebuild

Data loading

  • Server loads key/value data from cloud file storage
  • Server loads key/value data from cloud pub/sub services
  • Server loads data into an in-RAM representation for query serving
  • Server continuously monitors for new data and incrementally updates ("delta files") the in-RAM representation
  • Support independent data ingestion pipelining by monitoring directories in cloud file storage independently
  • Supports Flatbuffers as the data event format
  • Supports Avro and Riegeli as the data file format
  • Supports snapshot files for faster server start up
  • Users can perform compactions of delta files into snapshot files in an offline path

Read request processing

  • Support Protected Audience Key Value Server query spec: can be used as a BYOS server to serve requests from Chrome
  • Support simple key value lookups for queries
  • Users can write "user defined functions" to execute custom logic to process queries
  • User defined functions can be written in JavaScript or WASM
  • User defined functions can call "GetValues" to look up key value from the dataset

Advanced features

  • Set-as-a-value is supported
    • A key "value" pair in the dataset can be a key and a set of values
  • UDF can call "RunQuery" API to run set operations on sets (intersection, union, difference)
  • For GCP, Terraform supports deploying into an existing VPC, such as used by the Bidding and Auction services Non-prod Server logs are persisted after server shutdown
  • Data can be sharded and different servers may load and serve different shards (subset) of the dataset.
  • Sharding supports data locality, where the operator specifies "sharding key" for key value pairs so different key value pairs can have the same sharding key.

Timeline and roadmap

The following sections include the timelines for the Trusted Key Value Server for Protected Auctions. Protected Auctions refer to Protected Audiences and Protected App Signals ad targeting products.

Timelines

Beta testing General availability Enforcement
For Protected Audience

(web browser on desktop)

July 2024

The Privacy-Sandbox-provided Key Value Server implementation can

  • run as a BYOS KV server
  • support production scale traffic and common functionalities
Q4 2024

Opt-in TEE mode will be available to the Adtechs. Opt-in guidance will be published in early Q4 2024.

No sooner than Q3 2025
Beta testing General availability
For Protected Audience

(With Bidding & Auction services for Chrome or Android)

July 2024

The Privacy-Sandbox-provided Key Value Server implementation can be used with the Bidding and Auction services and

  • run as a BYOS KV server
  • support production scale traffic and common functionalities
Dec 2024

The Privacy-Sandbox-provided Key Value Server implementation can be used with the Bidding and Auction services and adtechs can opt-in TEE mode

For Protected App Signals June 2024

The Privacy-Sandbox-provided Key Value Server implementation supports Ad retrieval server functionality and protected communication for live traffic testing

Dec 2024

The implementation supports live traffic at scale

Roadmap

June 2024 Beta release

Deployment and Setup
  • For AWS, Terraform supports deploying into an existing VPC, such as the one that is used by the Bidding and Auction services
  • Internal load balancer is used for servers to send metrics to OpenTelemetry collector
    • In v0.16, the communication goes through a public load balancer
Integration with the Bidding & Auction services
  • The Bidding and Auction services can send encrypted requests to the Key Value Server for Protected App Signals
Debugging support
  • Consented Debugging is supported
  • Diagnose tool to check the cloud environment to warn for potential setup errors before the system is deployed
  • Operational playbook
  • Introduction of unsafe metrics
    • Unsafe metrics have privacy protections such as differential privacy noises
    • More metrics for comprehensive SLO monitoring
Runtime features
  • Data loading error handling
    • The system can be configured to use different error handling strategy for different dataset
Performance/Cost
  • Benchmarking tools
  • Cost explainer
  • Sets-as-values will switch to using bitsets to represent sets for faster RunQuery performance.
Support process
  • Commitment to support window for active releases

Q4 2024 Chrome-PA GA

Chrome integration
  • Update to V2 protocol to support the hybrid mode of BYOS & Opt-in TEE
  • Chrome and Key Value server can communicate in the updated V2 protocol
  • Chrome can send full publisher URL to TEE KV server under V2 protocol

H2 2024 Android-PA GA, PAS GA

User Defined Functions
  • UDF can perform Key/Value lookup asynchronously
  • Flags can be passed from the server parameters into UDF
  • One Key Value Server system can be used for multiple use cases. Multiple UDFs can be loaded. Different UDF can be selected based on the request type.
  • Canaring support for UDF: canary version UDF can be staged in machines with specific tags.
Customization support
  • First class support for customization of the system (without violating the trust model)
Debugging support
  • Diagnose tool to collect standard and necessary debug information for troubleshooting requests
Documentation
  • Complete end to end example as a template to set up the service

Breaking changes

Backward-incompatible changes are expected to be rare and will result in a major version change.

Key documents

Contribution

Contributions are welcome, and we will publish more detailed guidelines soon. In the meantime, if you are interested, open a new Issue in the GitHub repository.

Feedback

The FLEDGE K/V feature set, API and system design are under active discussion and subject to change in the future. If you try this API and have feedback, we'd love to hear it:

Announcements

General announcements

General announcements will be made on the FLEDGE mailing list on chromium.org and the Privacy Sandbox progress update page on the Android developer site, if needed, an accompanying article will be published on the Chrome Developers blog and Android Developers blog.


OpenSSF Scorecard