diff --git a/README.md b/README.md index 313773a..d70612b 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,90 @@ -# FLEDGE documentation +# Protected Auction Services documentation This GitHub repository contains proposals for the Privacy Sandbox's -FLEDGE server-side infrastructure that supports the Privacy +Protected Auction server-side infrastructure that supports the Privacy Sandbox's client-side APIs. -* [Overview of Privacy Sandbox FLEDGE services](trusted_services_overview.md) -* [Bidding and auction services API](bidding_auction_services_api.md) +Protected Auction includes [Protected Audience](https://github.com/WICG/turtledove/blob/main/FLEDGE.md) +and [Protected App Signals](https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals) +ad targeting products. + +> [!IMPORTANT] +> This repository is for documentation only. For discussions, visit the [WICG/protected-auction-services-discussion](https://github.com/WICG/protected-auction-services-discussion) repository. + +## Protected Auction services trust model + +* [Overview of trust model / Coordinators](trusted_services_overview.md) +* [Public cloud TEE requirements](public_cloud_tees.md) + +## Protected Audience auctions mixed mode +* [Protected audience auctions mixed mode](protected_audience_auctions_mixed_mode.md) + +## Bidding and Auction services + +* [Bidding and Auction services](bidding_auction_services_api.md) +* Self-serve guide + * [Bidding and Auction services onboarding and self-serve guide](bidding_auction_services_onboarding_self_serve_guide.md) +* Payload optimization + * [Bidding and Auction services payload optimization](https://github.com/privacysandbox/fledge-docs/blob/main/bidding-auction-services-payload-optimization.md) +* System design + * [Bidding and Auction services system design](bidding_auction_services_system_design.md) +* [Multi-Currency Support in Bidding and Auction services](bidding_auction_services_bid_currency.md) +* Multi seller auctions + * [Bidding and Auction services multi seller auctions](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_multi_seller_auctions.md) +* Reporting + * [Protected Audience event level reporting with Bidding and Auction services](bidding_auction_event_level_reporting.md) + * [Protected Audience event level reporting for multi seller auctions](bidding_auction_multiseller_event_level_reporting.md) +* Chaffing + * [Bidding and Auction Services: Chaffing Design](bidding_auction_services_chaffing.md) +* Protected App Signals + * [Bidding and Auction services: Protected App Signals](bidding_auction_services_protected_app_signals.md) + * [Protected App Signals egress format](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_protected_app_signals_egress.md) +* Cloud support + * AWS: [Bidding and Auction services AWS cloud support and deployment guide](bidding_auction_services_aws_guide.md) + * GCP: [Bidding and Auction services GCP cloud support and deployment guide](bidding_auction_services_gcp_guide.md) + +## Key/Value service + * [Key/Value service trust model](key_value_service_trust_model.md) +* [Key/Value service use cases](key_value_service_use_cases.md) +* [Key/Value service user-defined functions (UDFs)](key_value_service_user_defined_functions.md) +* [Key/Value sharding](key_value_service_sharding.md) +* [K/V Server Instance Sizing Guide](key_value_service_instance_sizing_guide.md) +* [Key/Value Service Cost](key_value_service_cost.md) + +## Ad Retrieval service + +* [Protected App Signals: Ad Retrieval service](https://github.com/privacysandbox/protected-auction-key-value-service/blob/main/docs/protected_app_signals/ad_retrieval_overview.md) + +## User Defined Function Execution +* [Roma Bring-Your-Own-Binary](roma_bring_your_own_binary.md) + +## Inference + +* [Bidding and Auction services: Inference Overview](inference_overview.md) + +## Server productionization + +* [Production operation](production_operation.md) +* [Debugging protected audience api services](debugging_protected_audience_api_services.md) +* [Monitoring protected audience api services](monitoring_protected_audience_api_services.md) + +## Server cost + +* [Bidding and Auction services: Cost](bidding_auction_cost.md) +* [Protected App Signals cost](protected_app_signals_cost.md) + +## Client and server integration + +### Protected Audience +* [Browser - Bidding and Auction services integration](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md) +* [Android - Bidding and Auction services integration high level document](https://developer.android.com/design-for-safety/privacy-sandbox/protected-audience-bidding-and-auction-services) + +### Protected App Signals +* [Android - Protected App Signals](https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals) -Additional explainers for FLEDGE live in other repositories. +## Related explainers -## Related client-side APIs +Additional explainers for Protected Audience API. * Chrome [FLEDGE](https://github.com/WICG/turtledove/blob/main/FLEDGE.md) proposal -* Android [FLEDGE](https://developer.android.com/design-for-safety/privacy-sandbox/fledge) proposal diff --git a/bidding-auction-services-payload-optimization.md b/bidding-auction-services-payload-optimization.md new file mode 100644 index 0000000..0a18e07 --- /dev/null +++ b/bidding-auction-services-payload-optimization.md @@ -0,0 +1,305 @@ +> FLEDGE has been renamed to Protected Audience API. To learn more about the name change, see the [blog post](https://privacysandbox.com/intl/en_us/news/protected-audience-api-our-new-name-for-fledge) + +**Author:**
+[Priyanka Chatterjee][1], Google Privacy Sandbox + +# Bidding and Auction services payload optimization + +The [Bidding and Auction (B&A) services][2] outlines a strategy for allowing +Protected Auction auctions (Protected Audience, Protected App Signals[23]) to +take place in a [trusted execution environment][3] (TEE) on a [supported cloud +platform][4]. This document describes steps for optimizing the size of payload +uploaded from clients for Protected Audience. + +As described in [Bidding and Auction (B&A) services high level architecture][21], +the seller's code in a publisher's webpage on browser or Android app sends a +unified contextual and Protected Auction HTTPS request (_unified request_) +to the seller's ad service. This request includes a contextual payload and +encrypted [`ProtectedAuctionInput`][5] data. (The contextual payload is not HPKE +encrypted; it may be TLS protected over the wire.) The seller's frontend service +orchestrates [`GetBids`][20] requests to buyers' BuyerFrontEnd services to +generate bids. The GetBids request payloads include BuyerInput and contextual +signals. + +**Note**: +* In case of Protected Audience, clients (web browser, app) prefetch + `interest groups` (referred to as `custom audiences` for Android) periodically + and stored on device. +* The [`ProtectedAuctionInput`][5] request payload is compressed, padded, and + then encrypted by the client. The payload is decrypted and decompressed by the + Bidding and Auction services. + +This document provides guidance to adtechs for **optimizing size of Protected +Audience request payload** uploaded from web browser (on desktop, Android) for +B&A services. For Android app targeting, similar payload optimization strategies +and solutions can be followed. + +**Payload optimization is a requirement for onboarding to B&A services.** + +## Top reasons for payload optimization + +The payload size optimization is important because: + +* **[Latency & network cost]** Large payloads can be expensive to transmit; + they impact overall auction **latency** and increase **network costs** for + sellers and buyers. To mitigate these effects, the total size of compressed + request payload uploaded from clients is recommended to be small (less than 60 + KB). + +* **[Seller configured per-buyer limit]** A seller may impose a `per-buyer payload size limit` + for [`ProtectedAuctionInput`][5] request payload. + +* **[Utility]** In order to achieve good **utility** from Protected Audience + auctions, enough `interest groups` must be included in `per-buyer payload size limit` + and made available for bid generation. + +## Higher payload driving factors + +The following fields in [interest groups][6] required for [generateBid()][7] can +cause [`ProtectedAuctionInput`][5] payload sizes to become large, and hence can not +be included in the umbrella request payload uploaded by clients. + + * `ads` that include `renderUrl`(s) and `ad metadata` + * _Note: There can be multiple ads per interest group_. + * `adComponents` + * _Note: There can be multiple ad components per ad in interest group_. + * `userBiddingSignals` + * _Note: There is no payload size limit on userBiddingSignals_. + +*Note:* + * The [`browserSignals`][27] generated by web browser, especially [`prevWins`][28] + in the [`browserSignals`][27] can add to payload size overhead, however + this is expected to be comparatively lower. + +## Payload optimization strategies + +Following are the high level strategies: + + * Only a small amount of information per `interest group` can be sent in the + request payload for B&A. This would ensure that enough `interest groups` fit + in a per-buyer payload limit. + + * For each `interest group` of a buyer, only the following information can be + sent in the umbrella request payload. + * `interest group` [name][24] + * [trustedBiddingSignalKeys][25] + * [browserSignals][26] + + * The buyer maintains an identifier (`adRenderId`) for each of its ads. Each ID + is a short, **8-character or less bytestring** (12-byte or less when Base64 encoded) + that identifies the ad. Buyer would generate short identifier for each `ad render url` + and `component ad render url` such that `prevWins` size included in + `browserSignals` is optimized. + + * Per buyer input that include information about `interest groups` are + **compressed** on the client. + + * For B&A auctions, `trustedBiddingSignals` is looked up from BuyerFrontEnd + service. All other information in `interest group` required for generating + bids in B&A, must be made available in `trustedBiddingSignals` fetched from + buyer's Key/Value service. + +## Payload optimization guide for sellers / SSPs + +* We recommend to sellers to set `per-buyer payload size limit` in the + range of **2 KB to 4 KB** to ensure payload for enough buyers are included in + [`ProtectedAuctionInput`][5]. + +* Sellers running B&A services, are required to incorporate [perBuyerConfig][33] + to configure `per-buyer payload size limit`. + +## Payload optimization guide for buyers / DSPs + +### Steps for payload optimization +This section discusses solutions for optimizing per-buyer input included in +compressed [`ProtectedAuctionInput`][5] request payload. + +#### Step 1 + +**The buyer creates `interest groups` based on `first party advertiser cookie`** + +Buyers running B&A services must ensure that `trustedBiddingSignalKeys` in +each `interest group` only include a `first party (1p) advertiser cookie`. This +would facilitate fetching data required by generateBid() for the interest group in +`trustedBiddingSignals` from buyer's key / value service. + +#### Step 2 + +**The buyer generates an identifier for each ad (`adRenderId`) to optimize size of ad `renderURL`.** + +* Generate an `adRenderId`, that is an identifier for an `ad renderURL`. Similarly, + also generate identifier for each `adComponent renderURL`. + +##### Size of adRenderId & adComponentRenderId + +The ad render IDs and adComponent render IDs must be [DOMString][27] that are +**8 bytes** long or less. If the identifier string is Base64 encoded, it's length +must be **12 bytes** long or less. + +#### Step 3 + +**The buyer sets priority for each `interest groups`** + +Chrome allows buyers to set a priority value (floating point number) per [interest group][6]. + +* The buyer sets a priority for each [interest group][7] when interest groups are + fetched by the client. + +* The browser uses these priorities to select the higher priority interest groups and + drop some lower-priority interest groups if per-buyer payload size limits are reached. + +Higher priority `interest groups` is expected to provide better utility. + +#### Step 4 + +**The buyer passes `adRenderId` to the browser during an interest group fetch and sets `auctionServerRequestFlags`.** + +The buyer running B&A services, must pass the following additional information +in an [`interestGroup`][6] when browser fetch interest groups from the buyer's +tagging servers. + +* The buyer also populates the `adRenderId` field in each `ads` object in an `interest group`. + +* Sets the following two flags with specific enum, that would indicate to the browser to + omit the ads, adComponents and userBiddingSignals data for this interest group from + the `ProtectedAuctionInput` request payload. This is also noted in [Protected Audience API explainer][34]. + + * The `omit-ads` enumeration causes the request to omit the `ads` and `adComponents` fields + for this interest group from the auction blob. + * The `omit-user-bidding-signals` enumeration causes the request to omit the + `userBiddingSignals` field for this interest group from the auction blob. + + ``` + 'auctionServerRequestFlags': ['omit-ads', 'omit-user-bidding-signals'] + ``` + +#### Step 5 + +**The browser sends small amount of information in each interest group, however + includes adRenderIds in `prevWins` fields of `browserSignal` objects.** + +The only information included in each interest group are the following. Note that the +browser will not include `adRenderId(s)` or `adComponentRenderId(s)` or `userBiddingSignals` +or any other information in interest groups. + * interest group `name` + * `trustedBiddingSignalKeys`: Note that this must be a first party advertiser cookie + * [browserSignals][18] + +##### Specification for browser signals + +The specification for the browser signals for `generateBid()`, is illustrated in +the following example. + +```json +{ 'topWindowHostname': 'www.example-publisher.com', + 'seller': 'https://www.example-ssp.com', + 'topLevelSeller': 'https://www.another-ssp.com', + 'joinCount': 5, + 'bidCount': 24, + 'recency': 1684134092, + 'prevWins': [[time1, adRenderId1],[time2, adRenderId2],...], + 'wasmHelper': ... /* a WebAssembly.Module object based on interest group's biddingWasmHelperURL */ + 'dataVersion': 1, /* Data-Version value from the trusted bidding signals server's response(s) if available */ +} +``` + +The browser generates [browserSignals][18] per `interest group`, that includes +includes [joinCount][28], [bidCount][29], [recency][30] and [prevWins][31]; the +remaining fields in [browserSignals](#Specification-for-browser-signals) is populated +by Bidding service. + +The `prevWins` in `browserSignals` refers to a tuple of **time-adRenderId pairs** +for a previous win for this interest group that occurred in the last 30 days. Note that +`adRenderIds` are passed in `prevWins` instead of full ad objects. + +#### Step 6 + +**The buyer's `generateBid()` is able to handle adRenderIds, reporting Ids and generate render url for bids.** + +* Bidding service passes `interest groups` to generateBid(), as received from + the browser. This implies that these `interest groups` do not include `ads` or + `adComponents` or `userBiddingSignals`. Buyer's generateBid() must be updated + to handle such `interest groups`. Note that since `ads` object is omitted, ads + attributes like `reporting Ids` are also omitted from `interest groups` object. + +* `generateBid()` ingests adRenderIds in the `prevWins` field in the + `browserSignals` object. + +* The `trustedBiddingSignals` fetched from the buyer's Key Value Service + include `adRenderID`, `ad metadata`, `reporting Ids` and `userBiddingSignals` information. + +* `generateBid()` recreates the `ad render URLs`, `adComponent render URLs` and `reporting Ids` + for the final bids after buyside auction. + + * `generateBid()` maintains a URL template with the base URL. The variable + part of the render URL is recreated using information in the `interestGroup` + and `trustedBiddingSignal` arguments. + + +#### Step 7 + +**Fetch userBiddingSignals per interest group from buyer's key / value service** + +_This section can be skipped for buyers that don't depend on user bidding signals._ + +The browser will not include `userBiddingSignals` information in `interest group`s that +are sent in `ProtectedAuctionInput` request payload. + +The `userBiddingSignals` information per `interest group` can be made available in +`trustedBiddingSignals`, if `first party advertiser cookie` is used as +`trustedBiddingSignalKeys`. + + +#### Step 8 + +**Mitigation for winning ad candidate validation** + +For Protected Audience, the render URL and and reporting Ids associated with the winning ad candidate +returned from B&A services in encrypted response, is validated in browser to ensure that it +belongs to the interest group that the browser has information about. + +Since buyer's [generateBid()][7] generate render url for the bids based on the information +in `trustedBiddingSignals`, the URL may fail validation if the browser gets out of sync +with `interest group`s in buyer's key / value service. . Therefore, it is important +to incorporate the following mitigation. + +**Mitigation:** Chrome browser launched more frequent `interest group` updates +triggered by `trustedBiddingSignals`, refer to [Intent-To-Ship][32] for more information. +With B&A, the `trustedBiddingSignals` are sent back to the BuyerFrontEnd service from +buyer's Key / Value service. B&A would indicate to the browser in encrypted AuctionResult +about how frequently or which interest groups require more frequent updates. +Buyers must incorporate this feature to keep the browser in better sync with the +`interest group` information in buyer's key / value service. This will help mitigate +the render url validation failure issue. + +## Related material + +* [Bidding and Auction services][15] +* [Chrome client design for Bidding and Auction services integration](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md) +* [Android - Bidding and Auction services integration high level document](https://developer.android.com/design-for-safety/privacy-sandbox/protected-audience-bidding-and-auction-services) + +[1]: https://github.com/chatterjee-priyanka +[2]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md +[3]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trusted-execution-environment +[4]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#supported-public-cloud-platforms +[5]: https://github.com/privacysandbox/bidding-auction-servers/blob/c98a51c7dc11de92e9c8fb719242a033e620a1b4/api/bidding_auction_servers.proto#L64 +[6]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#12-interest-group-attributes +[7]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#generatebid +[12]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#buyerinput +[15]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md +[18]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#browersignals +[20]: https://github.com/privacysandbox/bidding-auction-servers/blob/c98a51c7dc11de92e9c8fb719242a033e620a1b4/api/bidding_auction_servers.proto#L638 +[21]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#high-level-design +[22]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#11-joining-interest-groups +[23]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_protected_app_signals.md +[24]: https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L119 +[25]: https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L123 +[26]: https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L161 +[27]: https://webidl.spec.whatwg.org/#idl-DOMString +[28]: https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L178 +[29]: https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L181 +[30]: https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L198 +[31]: https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L190 +[32]: https://groups.google.com/a/chromium.org/g/blink-dev/c/eXJLbFAuSU8/m/WzCpcHaZAgAJ +[33]: https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-1-get-auction-blob-from-browser +[34]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#11-joining-interest-groups diff --git a/bidding_auction_cost.md b/bidding_auction_cost.md new file mode 100644 index 0000000..334421b --- /dev/null +++ b/bidding_auction_cost.md @@ -0,0 +1,612 @@ +# Bidding and Auction Cost + +**Authors:** +
[Akshay Pundle][31], Google Privacy Sandbox +
[Dave Garred][32], Google Privacy Sandbox +
Itay Sharfi, Google Privacy Sandbox + +# Background + +[Bidding and Auction Services][3] run in supported [Trusted Execution +Environments (TEE)][4] provided by AWS and GCP. In this document, the Privacy +Sandbox team explores the various factors that affect the cost of such a +deployment and how these can be mitigated. + +There are different tradeoffs when provisioning this system like [QPS][5], +acceptable latency, regional failover etc. Providing estimates for all +combinations is impossible, and a lot depends upon the choice the ad tech makes, +the tradeoffs that are considered etc. + +A framework has been developed to understand the costs of operating a +privacy-preserving advertising system. This framework will help ad tech +companies understand the source of costs and make informed decisions about how +to operate their businesses. + +The cost components described depend on the underlying cost structures provided +by cloud providers. The Privacy Sandbox team takes into account our current +understanding of these, but the underlying cost structures can change over time, +which would also change the cost components described in this document. + +The cost components described below scale with the throughput of the system. +Provisioned costs (e.g. number of VMs) depend on how many resources are +provisioned. Costs like network and load balancer cost scale with actual usage +and are proportional to the volume of traffic processed by the ad tech. + +The cost depends on system performance which will need to be empirically +measured. Ad techs can follow instructions in the [Cost Estimation][6] section +or participate in our [Alpha and Beta tester programs][7] to collect such data +and create detailed cost estimates. + +While it is impossible for the Privacy Sandbox team to collect representative +data since it does not operate a production system with live traffic, it can +measure resource consumption for a parameterized system with synthetic requests +and setup. Ad tech can use these to arrive at ballpark cost estimates. + +Typical cloud costs fall into the following categories. + +## Network + +Network costs can be a significant component of operating costs. Services are +usually operated inside a Virtual Private Cloud (VPC). Typically, network egress +from a VPC is charged. This means that you would pay for outbound traffic from +the services that go outside the VPC boundary, like going to the internet or to +another region but not for traffic inside the VPC. Traffic inside VPC is usually +free. Internal network components (like load balancers) that charge for data +processing will still incur cost on internal traffic. For this reason, we use a +service mesh for internal traffic (implemented in GCP. AWS implementation in +progress). This does not incur cost for traffic inside the VPC. Some network +components like NAT ([AWS][8], [GCP][9] have hourly charges, and ingress and +egress data charges based on usage. + +## Infrastructure + +Infrastructure costs include Virtual Machine (VM) costs and other services used. +VM cost depends on CPU, memory usage etc. These costs typically balance QPS, +application latency and availability needs. + +## Load balancer + +Load balancers typically charge for data processed. This includes inbound and +outbound data. If a load balancer is used on internal traffic, this processing +is still usually chargeable. + +NAT gateways are also chargeable. These are used to make outbound connections +out of the VPC and typically also charge for processed data. + +## TEE overhead + +Cloud operators may charge an overhead for TEE enclave usage. The way these +charges are levied is different and operators should use their setup to +determine how TEE costs affect the overall cost. + +GCP: [Confidential Compute charges][10] + +AWS: [Enclave pricing][11] + +# Seller costs + +
+ Fig. 1 - Seller costs +
Figure 1. Fig.1 - Seller costs
+
+ +## High level cost breakdown + +Seller costs breakdown in the following way, as depicted in Fig. 1 above. + +**C1: Request to SellerFrontEnd (SFE) network egress**: This additional cost is +incurred by the Seller ad server sending the Protected Audience auction request +to the SFE. + +**C2: LB cost**: A load balancer (LB) forwards incoming requests to the SFE +instances. This cost depends on the total processed bytes by the LB and may also +include components based on usage time. + +**C3: SFE compute and memory**: SFE processing consumes CPU and memory. + +**C4: K/V request network egress**: Requests sent to the [BYOS K/V server][12] +will result in network egress cost, accrued on the SFE side. NAT costs may also +be accrued since a connection across the VPC boundary will be needed. This cost +is 0 for TEE K/V (since it is in the same VPC). + +**C5: K/V response network egress**: This cost is accrued on the BYOS K/V side. +This system will send the response back to the SFE, which will result in network +egress cost. This cost is 0 for TEE K/V (since it is in the same VPC). + +**C6: Auction compute and memory**: Auction processing consumes CPU and memory, +bulk of which comes from running the scoreAds scripts. + +**C7: Request to Buyers network egress**: SFE sends requests to buyers to get +the bids. All these requests are subject to network egress / NAT costs. + +**C8: Response from SFE network egress**: This is the network egress cost of +sending the SFE responses back to the seller trusted server. + +**C9: Response to client**: This additional cost is incurred by the Seller ad +server sending any additional data (from B&A request processing) back to the +client. + +## Details + +### Seller untrusted server + +In the [unified request (contextual and Protected Audience)][13] flow, the +unified request is sent to the Seller ad server that runs outside the TEE. This +is an untrusted server managed by the SSP. This is the same server that +initiates an [Real-time Bidding (RTB)][14] request today, which will also +forward the appropriate remarketing requests to the SFE server, so this wouldn't +incur any additional cost to run. There may be additional processing costs +including packaging and sending the request to SFE, handling the response, +and additional traffic and processing to the Buyer RTB servers. Costs for these +should be estimated by the ad techs. + +If the seller ad server is not on the same cloud provider as the seller +frontend, the server will incur additional network costs (C1 and C8 in Fig. 1). +C1 is the cost of sending the remarketing request (along with the encrypted +payload) to the SFE. This cost is affected by the total payload which depends on +factors like the number of buyers in the request, payload per buyer (including +per buyer signals) etc. This cost can be bounded by enforcing [per-buyer +limits][15] on payload. Buyers can minimize their payload through methods +described in [Bidding and Auction Payload Optimization][16]. C8 is the cost of +additional payload sent back to the client. This payload depends on the response +size, which should be experimentally verified. + +These costs (C1 and C8) will not be there if the Seller ad server is in the same +VPC as the SFE. + +The SSP can decide which requests are valuable for remarketing, and only these +requests could be sent to the SFE. The SSP costs mentioned from here onward +depend on the number of requests the SFE services. Therefore, if the SSP's +untrusted server throttles requests, the remarketing costs will go down. + +### Seller Front End (SFE) and Auction services + +SSPs run SFE and Auction servers inside the TEE. + +The SFE servers are in charge of orchestrating the auction (including sending +requests to buyers, getting their bids, fetching real-time signals from the +seller key value server etc.). The auction servers are responsible for fetching +and running the ScoreAds functions. + +#### VM costs + +Each of these services require CPU and memory. From Fig. 1, the costs for these +are (C3 and C6 for SFE and Auction respectively). + +These costs depend on factors like the total throughput required, throughput per +[vCPU][17] of each of the servers, how the servers are provisioned, what latency +is acceptable, complexity of scoring scripts, etc. + +**Total required throughput**: Desired throughput of request being processed. +The required resources scale proportionally to this. Peak QPS may be much more +than avg. QPS. Provisioning sufficiently taking into account peak QPS will be +essential since cold boots take time. Provisioning servers for peak QPS would +incur more cost. + +**Throughput per vCPU of each of the servers**: Needs to be experimentally +determined by ad tech + +**Latency**: If degraded latency is acceptable, servers may be underprovisioned. +If this tradeoff is made, at high QPS, the servers could start to lag and +latency could be higher. + +**Other factors**: The SFE and Auction services do some additional processing +like uncompressing and decrypting (and compressing and encrypting) payloads, +fetching key-values, ad tech code scripts etc. These add processing overhead +which should be negligible, but needs to be determined experimentally by the ad +tech. + +**Complexity of** [**User-defined Functions**][18] **scripts (Auction server)**: +Running scoreAds functions requires CPU and memory. To keep costs lower, the +scripts can be made simpler / more efficient. + +For the auction servers, the above factors are all linked together in the +following way: + +The total processing required per request is the sum total of the script +execution and other overhead like encryption/decryption, compression etc. The +ScoreAds scripts are executed in parallel per ad. We get the following +relations: + + + + + +

Total processing = Per request Overhead + (#candidate ads) * (scoreAds time per ad) [vCPU-seconds]

Throughput Per vCPU = 1 / (Total processing) [requests / vCPU-second]

vCPUs needed = Total required throughput / Throughput Per vCPU [vCPUs]

+ +The cost will scale according to the number of needed vCPUs calculated above. + +Processing latency depends on the number of workers made available for parallel +executions. + + + + + +

Processing latency = Overhead + (#candidate ads) * (scoreAds time per ad) / #workers [vCPU-seconds]

+ +#### Network costs + +**SFE to BFE**: The SFE initiates calls to the Buyer Front End (BFE) servers. +This payload counts towards the SFE network egress cost and scales with the +number of buyers and the payload per buyer. Payload per buyer depends on the +number of interest groups and other associated data being sent. This cost is C7 +in Fig. 1. + +**Response from SFE to Seller ad server** + +The response from SFE to Seller ad server will incur network egress costs. These +will depend on the payload sent back, which will contain information about the +winning bid and [other reporting information][19]. This cost is C8 in Fig. 1. + +**SFE to BYOS KV servers** + +SFE will look for signals from KV servers. These request costs will also incur +network egress and NAT charges. Responses will be charged on the BYOS KV server +side. These two costs are C4 and C5 in Fig. 1. + +**Communication with Auction servers** + +Since the SFE and auction servers will be in the same private subnet, these +calls will not incur any network cost. + +### Load balancer and Network address translation (NAT) + +The SFE will have a load balancer for incoming requests. These will incur +processing costs (C2 In Fig. 1) depending on the total bytes routed. The total +bytes over a period of time depend on QPS and request payload size. The request +payload depends on a number of factors mentioned in the Seller untrusted section +above. + +A NAT is used to make outbound connections to the internet from within the VPC. +This is needed for sending requests to the BYOS K/V (if it is outside the VPC) +and to the buyer servers. NAT traffic is typically charged for bytes processed. +The cost can be estimated by estimating the traffic, and can be controlled by +limiting it. + +### Key/Value server costs + +The BYOS K/V server will accrue cost towards network egress of the responses. +This cost will depend on a few factors: + +1. Whether BYOS K/V is run in the same VPC, or not. If the K/V server is not + running in the same VPC as the SFE, it will incur network egress cost. The + cost will depend on the total data being returned. This, in turn, depends on + the QPS and amount of data returned per lookup. This cost is C5 in Fig. 1, and + was briefly mentioned in the "SFE to BYOS KV servers" section. Other BYOS K/V + server costs are left to the ad tech to determine. +1. The cost will also depend on the infrastructure of the BYOS K/V server. For + example, if the BYOS K/V runs on cloud and uses a load balancer, it will incur + processing cost for all the bytes processed. + +The compute costs for the BYOS K/V server are ad tech dependent, based on their +implementation, deployment and usage. These should be determined experimentally +by the ad tech. + +# Buyer costs + +The buyer runs Bidding Front End (BFE) service and Bidding service. The BFE +fetches signals from the buyer BYOS KV server, orchestrates calls to the Bidding +service and returns responses to the SFE. + +The Bidding servers generate the bids and are also responsible for loading the +buyer bidding scripts. + +
+ Fig. 2 - Buyer Costs +
Figure 2. Fig. 2 - Buyer Costs
+
+ +## High level cost breakdown + +Buyer costs breakdown in the following way, as depicted in Fig. 2 above. + +**C1: LB cost**: A load balancer (LB) forwards incoming requests to the BFE +instances. This cost depends on the total processed bytes by the LB and may +include a component based on usage time. + +**C2: BFE compute, memory**: Buyer Front End (BFE) processing consumes CPU and +memory. + +**C3: K/V request network egress**: Requests sent to the BYOS K/V server will +result in network egress cost. NAT costs may also be accrued since a connection +across the VPC boundary will be needed. + +**C4: K/V response network egress**: This cost is accrued on the BYOS K/V side. +This system will send the response back to the BFE, which will result in network +egress cost. + +**C5: Bidding compute and memory**: Bidding service running the generateBid +scripts consumes CPU and memory. + +**C6: Response to SFE**: The response back to the SFE will cause network egress +cost for the buyer.. + +## Load balancer + +The BFE will have a load balancer for incoming requests. These will incur +processing costs (C1 in Fig. 2) depending on the total bytes routed. This will +depend on QPS and request payload size. The payload sizes will depend on various +factors like number of IGs, Ads per IG, size of buyer signals etc. To reduce the +payload size, refer to the [Bidding and Auction Payload Optimization][16] +explainer. + +## Infrastructure + +Buyers run BFE and Bidding servers inside the TEE. + +The BFE servers are in charge of orchestration and the bidding server runs the +buyer JS scripts to generate the bids. + +#### VM costs + +Each of these services require CPU and memory. From Fig. 2, the costs for these +are C2 and C5. These costs depend on factors like the total throughput required, +throughput per vCPU of each of the servers, how the servers are provisioned, +what latency is acceptable, complexity of scoring scripts etc. + +**Total throughput required**: Desired throughput of request being processed. +The required resources scale proportionally to this. + +**Throughput per vCPU of each of the servers**: Needs to be experimentally +determined + +How the servers are provisioned: Peak QPS may be much more than avg. QPS. +Provisioning servers for peak QPS would require more cost. + +**Latency**: If degraded latency is acceptable, servers may be underprovisioned. +If this tradeoff is made, at high QPS, the servers could start to lag and +latency could be higher. + +**Other factors**: There is additional overhead to the processing done by the +BFE and Bidding services (e.g. uncompressing and decrypting payloads, fetching +key-values, ad tech code scripts, sending responses etc.). This overhead needs +to be determined experimentally by ad techs. + +**Complexity of scripts (Bidding server)**: Running generateBid functions +required CPU and memory. The more time they run for, the more resources they +consume. To keep costs lower, the scripts can be made simpler / more efficient. + +For the bidding servers, the above factors are all linked together in the +following way: + +The total processing required per request is the sum total of the script +execution and other overhead like encryption/decryption, compression etc. The +GenerateBid scripts are executed in parallel per ig. We get the following +approximate relations: + + + + + +

Total processing = Overhead + (#IGs) * (generateBid time per IG) [ vCPU-seconds]

Throughput Per vCPU = 1 / (Total processing) [requests / vCPU-second]

vCPUs needed = Total required throughput / Throughput Per vCPU [vCPUs]

+ +The cost will scale according to the number of needed vCPUs calculated above. + +Processing latency depends on the number of workers made available for parallel +executions. + + + + + +

Processing latency = Overhead + (#candidate ads) * (scoreAds time per ad) / #workers [vCPU-seconds]

+ +## Network + +**Response from BFE to SFE** + +The response from BFE to SFE will incur network egress costs. This cost is C6 in +Fig. 2. The costs will depend on the payload sent back, which will contain +information about the winning bid and other reporting information (e.g. debug +loss reports etc.). Larger win and loss reports will increase the payload back, +and hence increase this cost. + +**BFE to BYOS KV servers** + +BFE will seek signals from KV servers. BFE sends requests to the BYOS KV server +which cause network egress and NAT cost. This is C3 in Fig. 2. The response cost +(C4 in Fig. 2) will be charged on the BYOS KV server side + +**Communication with Bidding servers** + +The BFE and Bidding servers will be in the same private subnet, so these calls +will not incur any network cost. + +### Key Value server costs + +The BYOS KV server will accrue cost towards network egress of the responses. +This cost will depend on the total data being returned. This, in turn, depends +on the QPS and amount of data returned per lookup. This cost is C4 in Fig. 2, +and was briefly mentioned in the "BFE to BYOS KV servers" section. Other BYOS +K/V server costs are left to the ad tech to determine. + +# Cost estimation + +Both GCP and AWS itemize all costs needed for a B&A environment, and this can be +used to build a quite accurate estimate of running costs. In reality estimating a +range of cost for ad techs running these servers is made difficult by a number +of factors including: + +- Deployment model for each ad tech +- Estimation of compute capacity for each service. +- Estimation of payload size, including for downstream services. +- Estimation of response size. +- Identifying where pricing applies and the correct line item that should be + applied. + +## Cost test environment + +To overcome these obstacles, running B&A in a test environment over a short +period of time may be used to extrapolate monthly or annual costs (or to verify +calculated estimates). Guides are available for deploying an environment in +[GCP][20] and [AWS][21]. The test environment should replicate, as closely as +possible, the architecture that will be used in production. This includes: + +- Network configuration and load balancers. +- Correct proportion and sizing of frontend to backend instances. +- Request payloads and header information that are similar in size and + composition to production payloads. +- The Javascript module (aka "ad tech code") that is used in the backend service + should provide a similar latency and response size. + +Correct sizing and quantity of the host instances will be a significant factor +in the total cost of the environment. Load testing should be used to identify +the optimal allocation of vCPUs to frontend servers (SFE / BFE) and the backend +counterparts (Auction / Bidding). + +To facilitate identifying the billing costs attributed by the test environment, +each component should be [labeled (GCP)][22] or [tagged (AWS)][23] with unique +key value sets. These are needed to isolate the costs involved in the test +environment from those of any other resources in use within the same project +(GCP) or account (AWS). The [reference environment provided in the code +base][24] provides examples of how this can be done, the keys used there are +environment and operator. + +Some networking costs in GCP can not be labeled, particularly [Cloud NAT][25] +which is used for outbound calls from the B&A servers. Placing the test +environment in a region by itself can be used to isolate these costs. + +## Running a cost test + +Tests may be performed with any load test utility that supports gRPC (e.g. +[GHZ][26] and [wrk2][27]. Tests should provide a constant stream of calls to the +test environment at a steady rate for the period of the test. Each test run +should cover a minimum of 24 hours within the daily billing period of the cloud +service that is being used. + +- AWS: midnight UTC - midnight UTC +- GCP: midnight PST - midnight PST + +``` +RATE=60 +HOURS=48 +TOTAL_CALLS=$(( HOURS*60*60*RATE )) +ghz \ + -r $RATE \ + --total=$TOTAL_CALLS \ + --call=privacy_sandbox.bidding_auction_servers.SellerFrontEnd/SelectAd \ + --data-file ~/tmp/test-payload.txt \ + sfe-service.seller1-frontend.com:443 +``` + +_Example command line call to start a two-day test run at 60 QPS_ + +## Compiling costs for the test + +After the test is completed the results may be found in the billing explorer of +the respective cloud provider ([GCP Billing][28], [AWS Cost Explorer][29]. These +results are generally indicative of the total cost for running the environment +and may be extrapolated to find a monthly or annual cost. Note that some costs +are monthly or may not be obvious in the results found in the billing search. +These are usually minor (e.g., most DNS usage) but can be much larger depending +on how the infrastructure is configured and used (i.e., S3 or other storage +charges). + +### AWS + +In AWS, billing data is collected from AWS Cost Explorer using a daily usage +report. This breaks down cost by component and can be exported in csv-format. An +example from one run is shown below. + +
+ AWS Cost breakbown +
Figure 3. AWS Cost breakbown
+
+ +_Sample data export from AWS Cost Explorer for a two-day experiment_ + +### GCP + +A similar tool is provided by GCP in the Reports section of the Billing +interface breaking out the costs into [individual SKUs][30]. To identify the +test data, the correct date range and include filters for the desired GCP labels +(for NAT costs, the region filter may be used when the test environment is the +only NAT user in the region). + +
+ GCP Cost breakbown +
Figure 4. GCP Cost breakbown
+
+ +_Sample data export from GCP Billing Reports for a one-day experiment_ + +### Extrapolating monthly or annual cost + +These captured billing costs can be extrapolated to find the expected monthly or +annual operating costs. This cost is (roughly) linearly representative of a +single day of operation for the architecture in question computing the total +number of inbound requests. + +Since production level traffic will be higher than the test run, the "scaled up" +daily cost must be calculated for the expected QPS. In the simplest case, the +total cost for the test environment can be multiplied by a factor of +production-to-test QPS. + +$$Cost_{production} = Cost_{test} \times \frac{Rate_{production}}{Rate_{test}}$$ + +_Calculating "scaled up" daily operating cost for a production system_ + +This estimated "scaled up" daily production cost can then be multiplied to +obtain the monthly or annual production costs. + +Note that this linearity exists since most of the cost is based on data transfer +bytes or processing, both of which are roughly linear with the rate of requests. +Some additional savings will be seen in actual production costs that are not +seen here such as the NAT uptime in a GCP environment, however these costs tend +to be relatively minor (NAT uptime is $0.0014 per hour). + +Note that this assumes a production architecture similar to that under test. +More complex architectures to handle higher loads, such as a fan-out load +balancing scheme, may yield different results. + +# Future work + +Enabling accurate cost estimation is an important area that the Privacy Sandbox +team wants to support ad tech partners. To this effect, we have the current +plan: + +- Publish a load generator tool that can be used by ad techs for cost analysis + in conjunction with methods mentioned above +- Work with ad tech partners to measure and optimize cost + +[1]: http://goto.google.com/relevance-cost +[2]: http://go/cma-notice +[3]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md +[4]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trusted-execution-environment +[5]: https://en.wikipedia.org/wiki/Queries_per_second +[6]: #cost-estimation +[7]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#alpha-testing +[8]: https://aws.amazon.com/vpc/pricing/ +[9]: https://cloud.google.com/nat/pricing/ +[10]: https://cloud.google.com/compute/confidential-vm/pricing +[11]: https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave.html#nitro-enclave-pricing +[12]: https://github.com/WICG/turtledove/blob/main/Proposed_First_FLEDGE_OT_Details.md#trusted-signals +[13]: https://github.com/privacysandbox/fledge-docs/blob/833e9d819b8e3288dde5abebe3d59077aac3daba/bidding_auction_services_api.md#unified-request +[14]: https://en.wikipedia.org/wiki/Real-time_bidding +[15]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#sellers-ad-service +[16]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding-auction-services-payload-optimization.md +[17]: https://en.wikipedia.org/wiki/Central_processing_unit#Virtual_CPUs +[18]: https://github.com/privacysandbox/fledge-docs/blob/main/key_value_service_user_defined_functions.md +[19]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L131C9-L131C22 +[20]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_gcp_guide.md +[21]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_aws_guide.md +[22]: https://cloud.google.com/resource-manager/docs/creating-managing-labels +[23]: https://docs.aws.amazon.com/tag-editor/latest/userguide/tagging.html +[24]: https://github.com/privacysandbox/bidding-auction-servers/tree/main/production/deploy +[25]: https://cloud.google.com/nat/docs/overview +[26]: https://ghz.sh/ +[27]: https://github.com/giltene/wrk2/ +[28]: https://pantheon.corp.google.com/billing/ +[29]: https://console.aws.amazon.com/cost-management/home#/cost-explorer/ +[30]: https://cloud.google.com/skus +[31]: https://github.com/akshaypundle +[32]: https://github.com/dave-garred diff --git a/bidding_auction_event_level_reporting.md b/bidding_auction_event_level_reporting.md new file mode 100644 index 0000000..e6d814f --- /dev/null +++ b/bidding_auction_event_level_reporting.md @@ -0,0 +1,520 @@ +> _FLEDGE has been renamed to Protected Audience API. To learn more about the +name change, read the_ [_blog post_][1]. + +# Protected Audience event level reporting with Bidding and Auction services + +**Author:**
+[Rashmi Rao][0], Google Privacy Sandbox + +[Bidding and Auction services][2] outlines a way to allow Protected +Audience auctions to take place in a [trusted execution +environment][3] (TEE) hosted on a [supported cloud +platform][4]. + +This explainer describes the system design for event level reporting using the +Bidding and Auction services. For [high level design][5], [ad tech +specifications][6], [API][7], refer to the [Bidding and +Auction services explainer][5]. + +## Overview + +The seller and the winning buyer each have an opportunity to generate URLs for +event-level reporting based on an auction outcome. + +Reporting with Bidding and Auction services is planned to be supported in +multiple phases. + + + + + + + + + + + + + + + + + + + + + + +
Reporting support with Bidding and Auction servicesTimeline
Win reporting for Single Seller AuctionsLAUNCHED
Win reporting for Device Orchestrated Multi Seller Auctions.LAUNCHED
Win reporting for Server Orchestrated Multi Seller Auctions.Jan 2025
Private Aggregation APIJune 2025
+ +This document details the design to generate and ping win reporting URLs with +the Bidding and Auction services for single-seller auctions. The reporting URLs +are generated from JavaScript using the `reportResult()` function provided by the +seller and `reportWin()` function provided by the buyer on the Auction service. +Final pings to the URLs happen on the client. + +This design supports win reporting on Android and Chrome. The design also +supports _fenced frame reporting_, also known as _interaction reporting_ or +_beacon reporting_. + +## Background + +A Bidding and Auction service enables processing Protected Auction requests on +the server. Sellers provide the code required for scoring ads and reporting URL +generation using the `scoreAd()` and `reportResult()` functions. Buyers provide the +code for generating the bids and reporting using the `generateBid()` and +`reportWin()` functions. The flow of Bidding and Auction services is as follows: + +- The seller sends a [`SelectAd`][8] request to the SellerFrontEnd + service. +- The service orchestrates requests in parallel to the buyers' BuyerFrontEnd + services for Protected Auction bidding. +- The buyers generate bids and return the bids to the seller. +- The SellerFrontEnd service calls the Auction service with the bids and scoring + signals obtained from the seller's Key-Value service. +- The Auction service scores the bids and determines the winner. +- The Auction service returns the winning ad to the SellerFrontEnd service. +- The SellerFrontEnd service encrypts the auction result and returns a + [`SelectAdResponse`][9] to the seller's ad service. +- Seller's ad service sends the encrypted auction result back to the client. + +After scoring ads and determining a winner, the reportResult() and reportWin() +functions generate the reporting URLs. The client must ping these URLs after an +ad is rendered. + +## The reportResult specification + +The seller-provided endpoint for `scoreAd()` contains the `reportResult()` function. + +``` +reportResult(auctionConfig, sellerReportingSignals){ +… + return signalsForWinner +} +``` + +The arguments for `reportResult()` are described in the following table: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ArgumentJSON Object subfieldDescription
auctionConfigauctionSignalsContextual signals passed from seller's ad service to the SellerFrontEnd service in SelectAdRequest.AuctionConfig
signalsForWinnerContextual signals passed from seller's ad service to the SellerFrontEnd service in SelectAdRequest.AuctionConfig
sellerReportingSignalsinterestGroupOwnerDSP / buyer domain.
renderURLAd render URL of winning bid.
bidValue of bid that scored highest.
desirabilityScore of winning bid.

+ Note: Winning bid refers to the bid with the highest score.
highestScoringOtherBidThis is the value of a bid with the second highest score in the auction. This value represents eCPM (effective cost per thousand impressions), valued in US dollars.

+ Note: +
    +
  • A higher bid value can get a lower score, so it is possible this bid value is higher than "highest scoring bid".
  • +
  • Rejected bids are not included.
  • +
  • If more than one bid has the second highest score, then the 'highestScoringOtherBid' is randomly selected.
  • +
  • This value will be 0 if there was only 1 bid.
  • +
+
topWindowHostNameThis is the host name of the publisher.
+ +The `reportResult()` function returns signalsForWinner, which is used as input for +the buyer's `reportWin()` function. + +## The reportWin specification + +The buyer-provided endpoint for `generateBid()` is expected to contain the +`reportWin()` function. + +``` +reportWin(auctionSignals, perBuyerSignals, signalsForWinner, buyerReportingSignals) { + ... +} +``` + +The arguments for `reportWin() are described in the following table: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ArgumentJSON Object SubfieldDescription
auctionSignalsContextual signals passed from the seller's ad service to the +SellerFrontEnd service in SelectAdRequest.AuctionConfig.
signalsForWinnerSignals returned by seller's ReportResult() function execution.
perBuyerSignalsContextual signals passed from the seller's ad service to the SellerFrontEnd service in SelectAdRequest.AuctionConfig. This is the buyer signals for the winning buyer.
buyerReportingSignalssellerSeller's origin. This will be passed to the Auction service from the SellerFrontEnd service.
adCostAn optional field returned by generateBid(), rounded to fit into a floating point number with an 8 bit mantissa and 8 bit exponent for reporting.
interestGroupNameThe name of the interest group corresponding to the highest scoring bid.
madeHighestScoringOtherBidThis is set to true if the winning buyer was the only buyer that made bids with the second highest score.
recencyDuration of time (in minutes) from when this device joined this interest group until now. This is passed by the client.
modelingSignalsSent as an output from the generateBid() function. This is expected to be a single integer value.
joinCountThe number of times the given device has joined the same interest group in the last 30 days, while the interest group has been continuously stored.
+ +Note: + +- `buyerReportingSignals` additionally contains all the signals present in the + `sellerReportingSignals`. +- The `recency`, `modelingSignals` and `joinCount` fields are noised using the noising scheme described [here][14]. The parameters `bid`, `desirability` and `adCost` will be rounded to a floating point number with an 8 bit mantissa and 8 bit exponent. Currently the noising can be enabled by setting [ENABLE_REPORT_WIN_INPUT_NOISING][15] to true. However, this flag will be removed and noise will be applied by default in B&A from Beta 2. +- The `bid`, `score`, and `adCost` fields are stochastically rounded. +- `browserSignals` in the signature of on-device implementation has been renamed to `sellerReportingSignals` for reportResult and `buyerReportingSignals` for reportWin. The reason we changed the name of the argument is because this is actually reporting signals that is not generated by the browser in case of B&A but by the TEE based Auction server based on different parameters. The contents of browserSignals (for on-device) and seller/buyerReportingSignals are the same. +- The feature support for [buyer_and_seller_reporting_id][16] and [selectable_buyer_and_seller_reporting_id][16] in ad properties were recently introduced with on-device Protected Audience auctions. These signals are expected to be used instead of interestGroupName if present. For the server side implementation, these fields will need to be fetched from an AdTech's Key/Value server so that the payload (encrypted ProtectedAuctionInput) from the device doesn't increase. This requires a detailed design to ensure there is no significant increase in latency and costs for the Adtechs. + +## Reporting function + +In the short-term, `reportResult()` and r`eportWin()` reporting are run by calling a +`sendReportTo()` API. The `sendReportTo()` function can be called at most once +during a function's execution. Eventually, reporting will also support the +Private Aggregation API. + +The executable code in [Roma][11] contains a `sendReportTo()` +function which takes a single string argument representing a URL: + +``` +sendReportTo = function(url){ + // Stores the url in a global variable +} +``` + +## Fenced Frame reporting + +Ad techs can register URLs that correspond to events or beacons when +`reportResult()` and r`eportWin()` are run. These beacons can be used to trigger +pings using the `reportEvent` API on the client. + +The beacons can be registered by calling the `registerAdBeacon()` function, +which will be a part of the executable code in [Roma][11]: + +``` +registerAdBeacon = function(interactionUrlMap){ + // Stores the map to a global variable. +} +``` + +The input to `registerAdBeacon` is expected to be a JSON object with a mapping +of the event string to a corresponding URL. + +## Design + +The high level flow for reporting is as follows: + +- The client and buyer send the required signals for reporting to the The + SellerFrontEnd service. +- The SellerFrontEnd service sends the required signals to run `reportResult()` + and `reportWin()` to the Auction service +- Auction service executes the `reportResult()` and `reportWin()` functions in + [Roma][11] and returns the reporting URLs to The SellerFrontEnd + service. +- The SellerFrontEnd service encodes the reporting URLs using CBOR encoding for + both buyer and seller and adds it to the [`SelectAdResponse`][9]. + The `reportResult()` and `reportWin()` functions are executed serially in a + single dispatch call after the winner has been determined. +- Clients use these URLs to ping the buyer and seller reporting endpoints after + the ad is rendered. + + + +### Rationale for the design choices + +#### Why is the reporting URL generation done on the server? + +The inputs to the reporting functions include large signals such as: + +- Auction config +- Per-buyer signals + +The response payload will significantly increase (potentially by 30kb or more) +if reporting URL generation is performed on the client, leading to an increase +in the cost and latency. Server-generated reporting URLs are lightweight and +don't affect the response payload size. + +#### Why is the buyer's reportWin function executed on the seller's auction +server? + +The reporting function execution can happen only after the ads are scored and +the winner is determined. The `reportWin()` function call depends on an output of +the `reportResult()` function called `signalsForWinner()`. If `reportWin()` needs to +be executed on the buyer's Bidding service, an RPC from the SellerFrontEnd +service to BuyerFrontEnd service is needed to fetch the URLs. This adds +significant latency on the critical response path (>50ms) and additional network +costs to both buyer and seller. + +### BuyerReportingId +buyerReportingId is one of the interest group attributes which is input to reportWin via buyerReportingSignals. If set, the interestGroupName in buyerReportingSignals will not be set. The buyerReportingId is expected to be returned in the response from generateBid for every bid. This value is not obtained from the device due to [payload optimization][17]. + +GenerateBid() API changes + +``` +generateBid(interestGroup, auctionSignals, perBuyerSignals, + trustedBiddingSignals, browserSignals, directFromSellerSignals) { + ... + return {'adCost': …, + 'bid': …, + 'bidCurrency': …, + 'render': …, + 'adComponents': [...], + 'allowComponentAuction': …, + 'modelingSignals': …, + 'buyerReportingId': “...”}; +} + +``` + + +### API changes + +The reporting URLs and beacon URL map for both the seller and the buyer are sent +to the client in a [`SelectAdResponse`][9]. + +These URLs are [CBOR-encoded][12] and encrypted before sending the +response from the SellerFrontEnd service. + +``` +Message AdScore { +... +// The reporting URLs registered during the execution of reportResult() and +// reportWin(). +WinReportingUrls win_reporting_urls = 10; +... +std::string buyer_reporting_id = 19; +} + +message WinReportingUrls { + +message ReportingUrls { +// The url to be pinged for reporting win to the buyer or seller. +string reporting_url = 1; +// The map of (interactionKey, URI). +map interaction_reporting_urls = 2; +} + +// The reporting URLs registered during the execution of +// reportWin(). These URLs will be pinged from the client. +ReportingUrls buyer_reporting_urls = 1; + +// The reporting URLs registered during the execution of reportResult() of the +// seller in case of single seller auction and component seller in case of +// multi seller auctions. These URLs will be pinged from the client. +ReportingUrls component_seller_reporting_urls = 2; + +// The reporting URLs registered during the execution of reportResult() of the +// top level seller in case of multi seller auction. These URLs will be pinged +// from the client. This will not be set for single seller auctions. +ReportingUrls top_level_seller_reporting_urls = 3; +} + +``` + +#### SFE +buyer_reporting_id for the winning buyer will be sent back to the client in the [SelectAdResponse][18]. + +### Flags + +Reporting function execution is enabled by 2 flags on the server: + +- `enableReportResultUrlGeneration`: To enable executing the `reportResult()` + function +- `enableReportWinUrlGeneration`: To enable executing the `reportWin()` function. + +If `enableReportResultUrlGeneration` is set to `false`, neither of the functions +are executed. + +### Code fetch and loading + +The code executed in the auction service is loaded only once at startup time, +and periodically every few hours (as configured by the developer) in the Auction +service from endpoints where the ad tech's reporting code modules are hosted. + +The function name to be called is determined at run time before the dispatch +request. The code is periodically fetched from public endpoints provided by the +seller and buyer. The code loaded into [Roma][11] should to contain +the following functions: + +- `scoreAd()` +- `reportResult()` +- `reportWin()` for all buyers. `reportWin()` is called only for the winner. + +The `reportWin()` functions are loaded if `enableReportWinUrlGeneration` has +been set to `true`. + +Note: Try to keep `reportWin()` in a separate code module from `generateBid()`, +so that a seller can only fetch the partner buyers' `reportWin()` code modules. + +### ReportWin udf endpoint validation + +The Buyer `reportWin()` UDF URL on the Auction Service will be validated as per the following criteria: +- The URL must have the same origin as the interest group owner for Chrome (specified for an IG by the buyer on device) and buyer origin for Android (specified for a custom audience by the buyer on device). +For example, if the interest group owner is https://example.com/ab, the script must be loaded from https://example.com/foo/bar/foo/… +- If the interest group owner is https://ab.example.com, the script must be loaded from https://ab.example.com/foo/bar/foo/… +In this case, B&A would reject any script from https://example.com/foo/bar/foo/… +- No query parameters or fragments are allowed: https://url.spec.whatwg.org/#concept-url-fragment. These will be removed from the configured reportWin URL while trying to fetch the script. +- The response from the `reportWin()` endpoint must contain the response header `X-Allow-Fledge:true` (similar to Chrome). +- Redirects will be supported to HTTPS/HTTP URLs only. + +If the udf endpoint fails the validation, the buyer `reportWin()` function will not be loaded into Roma or executed on the Auction service. An error log will be printed. However the winner will still be returned to the client. + +### JavaScript wrapper +The reportResult function is called using a [wrapper JavaScript function][13] such as: + +``` +function reportResultEntryFunction(auctionConfig, sellerReportingSignals, directFromSellerSignals, enable_logging) { +… +const ps_report_result_response = { + signalsForWinner : "null", + reportResultUrl : "", + interactionReportingUrls : {}, + } +… + return { + response: ps_report_result_response, + logs: ps_logs, + errors: ps_errors, + warnings: ps_warns + } + +} + +``` +The reportWin function is called using a [wrapper JavaScript function][19] such as: + +``` +function reportWinEntryFunction(auctionSignals, perBuyerSignals, signalsForWinner, buyerReportingSignals, directFromSellerSignals, enable_logging) { +… + const ps_report_win_response = { + reportWinUrl: '', + interactionReportingUrls: {}, + sendReportToInvoked: false, + registerAdBeaconInvoked: false, + }; +… +reportWin( + auctionSignals, perBuyerSignals, signalsForWinner, + buyerReportingSignals, directFromSellerSignals); + +… + return { + response: ps_report_win_response, + buyerLogs: ps_logs, + buyerErrors: ps_errors, + buyerWarnings: ps_warns + }; +} +``` +During each function call, the ad tech can log information or error messages using `console.log` ,`console.warn` and `console.error` functions. These logs are exported in the response from the wrapper. Log capture is enabled only if user-consented debugging is enabled. + +During each function call, the ad tech can log information or error messages +using `console.log` and `console.error` functions. These logs are exported in +the response from the wrapper. Log capture is enabled only if the server side +flag `enableAdTechCodeLogging` is set to `true`. In the future, the logging will +be controlled by an additional user-consented debugging flag. + +### Client side + +Chrome and Android will extract the reporting URLs and beacon URL map from the ProtectedAudience (encrypted AuctionResult) response. Upon rendering of the ad, reporting URLs are pinged based on the client controlled flag. Clients are expected to do an additional check to verify if the domain of the reporting URLs belong to the seller's or buyer's domain. +The buyer_reporting_id returned in encrypted AuctionResult will be validated by the client with the buyerReportingId present as a part of the interest group attributes stored on-device. If the ids don’t match, the win reporting urls will be dropped. + +[0]: https://github.com/rashmijrao +[1]: https://privacysandbox.com/intl/en_us/news/protected-audience-api-our-new-name-for-fledge +[2]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md +[3]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trusted-execution-environment +[4]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#supported-public-cloud-platforms +[5]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#unified-contextual-and-fledge-auction-flow-with-bidding-and-auction-services +[6]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#specifications-for-adtechs +[7]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#service-apis +[8]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#sellerfrontend-service-and-api-endpoints +[9]: https://github.com/privacysandbox/bidding-auction-servers/blob/4a7accd09a7dabf891b5953e5cdbb35d038c83c6/api/bidding_auction_servers.proto#L441 +[10]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#521-noised-and-bucketed-signals +[11]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#roma +[12]: https://cbor.io/ +[13]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/services/auction_service/code_wrapper/seller_code_wrapper.h +[14]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#521-noised-and-bucketed-signals +[15]: https://github.com/privacysandbox/bidding-auction-servers/blob/06bf6ed1bd98916d1a9b6887518936655b4d537b/production/deploy/gcp/terraform/environment/demo/seller/seller.tf#L91 +[16]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#12-interest-group-attributes +[17]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding-auction-services-payload-optimization.md#payload-optimization-guide-for-buyers--dsps +[18]: https://github.com/privacysandbox/bidding-auction-servers/blob/722e1542c262dddc3aaf41be7b6c159a38cefd0a/api/bidding_auction_servers.proto#L370 +[19]: https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/services/auction_service/code_wrapper/buyer_reporting_udf_wrapper.h#L32 diff --git a/bidding_auction_experiment.md b/bidding_auction_experiment.md new file mode 100644 index 0000000..c0bf734 --- /dev/null +++ b/bidding_auction_experiment.md @@ -0,0 +1,147 @@ +# Bidding and Auction services experiment + +**Authors:**
+[Xing Gao](https://github.com/xinggao01), Google Privacy Sandbox
+ +This document proposes a cloud architecture to enable experiment with Bidding and Auction services. + + +To learn more about Protected Audience services and Bidding and Auction services, read the following: +* [Protected Audiences Services Overview](https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md) +* [Bidding and Auction Services System Design](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md) + +The current design only enables experiments using randomly split traffic with multiple instances (separate servers). + +Specifically, A/B testing that diverts traffic based on request data is not covered. + +## Supported Features +1. Trusted server image agnostic: all the setup is in the cloud deployment configuration, server binary doesn't need change to support experiments +2. Experiment arms can be switched on/off, and traffic can be adjusted dynamically to split the traffic +3. Runtime flags, server images, region and machine specs of control and experiment arms can be configured +4. Server metrics of control and experiment arms can be compared for evaluation + +## Architecture Overview + +This architecture uses a cloud load balancer for the front-end server to split traffic between two or more B&A server stacks. The example below shows a production and canary setup. The configuration on the seller and buyer side servers is similar. + +The configurations, including runtime flags, server images, regions, and machine specs, apply to each arm. The front-end server (i.e., SFE, BFE) only communicates with the back-end server (i.e., auction, bidding) within the same arm. + +
+ B&A Experiment Example +
Figure 1. B&A Experiment Example
+
+ +## Deploy Example in GCP +In the examples below, we demonstrate deploying a configuration for a seller. The process for configuring a buyer is similar. + +Please also see the [example config in the published repo](https://github.com/privacysandbox/bidding-auction-servers/blob/532e2938d0f82687ceda629ba50eaeae2a3a4102/production/deploy/gcp/terraform/environment/demo/seller/seller.tf#L55) +and [weight-based traffic splitting example in GCP](https://cloud.google.com/load-balancing/docs/l7-internal/traffic-management#traffic_actions_weight-based_traffic_splitting). + +### Use Case 1: Try out a new B&A version with Feature X enabled on 5% of traffic +Configure two traffic arms, with 5% directed to the experiment arm. Enable the feature flag for Feature X only in the experiment arm, leaving the control arm unaffected. + +```terraform + seller_traffic_splits = { + # control arm + "${local.environment}" = { + image_tag = "prod_4.0" + traffic_weight = 1000 + region_config = local.default_region_config + runtime_flag_override = {} + } + + # experiment arm + "feature_x" = { + image_tag = "prod_4.0" + traffic_weight = 50 + region_config = local.default_region_config + runtime_flag_override = { + FEATURE_X = "true" + } + } + } +``` +### Use Case 2: Try out a new server version on 5% of traffic +Configure two arms, split traffic between them, and use different image tags for each arm to test the new server version on 5% of traffic. + +```terraform + seller_traffic_splits = { + # control arm + "${local.environment}" = { + image_tag = "prod_4.0" + traffic_weight = 1000 + region_config = local.default_region_config + runtime_flag_override = {} + } + + # experiment arm + "new_version" = { + image_tag = "prod_4.1" + traffic_weight = 50 + region_config = local.default_region_config + runtime_flag_override = {} + } + } +``` +### Use Case 3: Run multiple experiments simultaneously +Configure multiple arms, split traffic between them, and assign different flag values to each experiment arm to run several experiments at the same time. + +```terraform + seller_traffic_splits = { + # control arm + "${local.environment}" = { + image_tag = "prod_xxx" + traffic_weight = 1000 + region_config = local.default_region_config + runtime_flag_override = {} + } + + # experiment arm 1 + "experiment_1" = { + image_tag = "prod_xxx" + traffic_weight = 50 + region_config = local.default_region_config + runtime_flag_override = { + FEATURE_Y = 1 + } + } + + # experiment arm 2 + "experiment_2" = { + image_tag = "prod_xxx" + traffic_weight = 50 + region_config = local.default_region_config + runtime_flag_override = { + FEATURE_Y = 2 + } + } + } +``` + +## Update Traffic Split Dynamically in GCP +The traffic weight split can be updated dynamically while the servers are serving requests. Here's an example using the gcloud command to ramp up traffic from a 100:1 split to 100:100: + +
+ B&A Experiment Traffic Update +
Figure 2. Update Traffic Split Dynamically in GCP
+
+ +The gcloud command used: +```bash +$ gcloud compute url-maps edit seller1-$ENV-xlb-grpc-map + +// modify the weight and save +defaultRouteAction: + weightedBackendServices: + - backendService: https://www.googleapis.com/compute/v1/projects/bas-dev-383721/global/backendServices/seller1-$ENV-xlb-backend-service + weight: 1000 + - backendService: https://www.googleapis.com/compute/v1/projects/bas-dev-383721/global/backendServices/seller1-$ENV-1-xlb-backend-service + weight: 500 +``` + + +## Deploy Example in AWS (comming soon) diff --git a/bidding_auction_multiseller_event_level_reporting.md b/bidding_auction_multiseller_event_level_reporting.md new file mode 100644 index 0000000..22ee1c7 --- /dev/null +++ b/bidding_auction_multiseller_event_level_reporting.md @@ -0,0 +1,364 @@ +> _FLEDGE has been renamed to Protected Audience API. To learn more about the name change, read the [blog post](https://privacysandbox.com/intl/en_us/news/protected-audience-api-our-new-name-for-fledge)._ + +**Author:**
+[Rashmi Rao](https://github.com/rashmijrao), Google Privacy Sandbox + +## Protected Audience Event Level Reporting for Multi Seller Bidding & Auction Services + +[Bidding and Auction services](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md) outline a way to allow Protected Audience auctions to take place in a [trusted execution environment](https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trusted-execution-environment) (TEE) hosted on a [supported cloud platform](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#supported-public-cloud-platforms). + +This explainer describes the system design for event level reporting using the Bidding and Auction services specifically in multi seller scenarios. For single-seller event level reporting refer to [this explainer](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_event_level_reporting.md#protected-audience-event-level-reporting-with-bidding-and-auction-services). + + For detailed design of multi seller auction support on Bidding and Auction Services refer to this [explainer](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_multi_seller_auctions.md#multi-seller-auction). + +For [high level design](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#unified-contextual-and-fledge-auction-flow-with-bidding-and-auction-services), [ad tech specifications](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#specifications-for-adtechs), [API](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#service-apis), refer to the [Bidding and Auction services explainer](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#unified-contextual-and-fledge-auction-flow-with-bidding-and-auction-services). + + +### Overview + +The seller and the winning buyer each have an opportunity to generate URLs for event-level reporting based on an auction outcome. + +Reporting with Bidding and Auction services is planned to be supported in multiple phases. + + + + + + + + + + + + + + + + + + + + + + + +
Reporting support with Bidding and Auction services + Timeline +
Win reporting for single seller auctions + LAUNCHED +
Win reporting for device-orchestrated multi seller auctions. + Beta 2 +
Win reporting for server-orchestrated multi seller auctions. + Beta 2 +
Private Aggregation API + Scale Testing +
+ + +This document details the design to generate win reporting urls with the Bidding and Auction Services for multi seller auctions. The urls are sent back to the client (browser, Android) and pinged from the client after the ad is rendered.The design for single seller auctions is covered in a different [explainer](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_event_level_reporting.md#protected-audience-event-level-reporting-with-bidding-and-auction-services). + +Multi seller auctions are those auctions with participation from multiple sellers and their partner buyers, where the final winning ad candidate is decided by a top level auction. There multiple types of multi seller auctions: + + + +* [Device-orchestrated component auctions](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_multi_seller_auctions.md#device-orchestrated-component-auctions) - Only for browsers +* [Server-orchestrated multi seller auctions](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_multi_seller_auctions.md#server-orchestrated-component-auctions) - For browsers and Android +* WaterFall meditation - Only for Android apps + +This document will focus on device-orchestrated and server-orchestrated multi seller auctions. + + +### Background + + +##### Device-orchestrated component auctions + +In [device-orchestrated component auctions](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_multi_seller_auctions.md#device-orchestrated-component-auctions), each component auction is orchestrated by the respective seller in the publisher web page, in the browser. Each component auction can be conducted with the Bidding and Auction services or on-device. The top-level Protected Audience auction is executed on the browser. + + +##### Server-orchestrated component auctions + +In [server-orchestrated component auctions](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_multi_seller_auctions.md#server-orchestrated-component-auctions), the top-level seller makes one request for an auction from the publisher web page to its ad service. The top-level seller’s ad service orchestrates unified requests to other sellers for Component auctions that can run on Bidding and Auction services. Then, the top-level Protected Audience auction is executed in the top-level seller's TEE-based Auction service. + + +## Design + +For a single-seller auction, reporting urls are generated for the winning buyer and seller (as explained [here](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_event_level_reporting.md#protected-audience-event-level-reporting-with-bidding-and-auction-services)). + +For multi seller auctions, event level reporting urls are generated for the top-level seller, winning component seller and winning buyer. + + \ +reportResult and reportWin execution for multi seller auctions + + +The inputs to each reportResult and reportWin function are expected to be different for each of them: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Input Signals + Provided to +

+Top Level Seller’s reportResult +

Provided to +

+Component Seller’s reportResult +

Provided to Component Buyer’s reportWin +
Auction Signals + Yes +

+(Top Level auction config) +

Yes \ +(Component level auction config) + Yes \ +(Component level auction config) +
topWindowHostname + Yes + Yes + Yes +
topLevelSeller + Yes + Yes + Yes +
componentSeller + Yes + Yes + Yes +
interestGroupOwner + Yes + Yes + Yes +
renderURL + Yes + Yes + Yes +
bid + Yes +

+(modified bid) +

Yes \ +(buyer bid) + Yes \ +(buyer bid) +
desirability + Yes +

+(Top level score) +

Yes +

+(Component level score) +

No +
topLevelSellerSignals + No + No + No +
modifiedBid + No + Yes + No +
highestScoringOtherBid + No + Yes + Yes +
madeHighestScoringOtherBid + No + No + Yes +
adCost + No + No + Yes +
interestGroupName + No + No + Yes +

+(may be included if the tuple of interest group owner, name, bidding script URL, ad creative URL, and ad creative size were jointly k-anonymous once B&A supports K-anonymity). +

seller + No + No + Yes +
joinCount + No + No + Yes +
recency + No + No + Yes +
modelingSignals + No + No + Yes +
signalsForWinner + No + No + Yes +
perBuyerSignals + No + No + Yes +
+ + +On-device Protected Audience auctions support passing output of the top level seller reportResult call as the [topLevelSellerSignals input](https://github.com/WICG/turtledove/blob/main/FLEDGE.md#51-seller-reporting-on-render) to the component seller's reportResult call. This creates a dependency in the reporting function execution for the component seller and buyer on the reportResult() execution of the top level seller for the topLevelSellerSignals. + +On-device flow for top_level_seller_signals + +This dependency may not be necessary and adtechs have not provided requirements for [supporting this](https://github.com/WICG/turtledove/issues/821). This dependency is removed for reporting URL generation in Bidding and Auction Services and the topLevelSellerSignals will not be passed to the component seller’s reportResult. + +The reporting URLs for the component auctions will be generated immediately after the component auction for both component level seller and buyer. The win reporting urls for the seller and buyer from the component auction are passed to the top level auction in the response from SFE for both [device orchestrated](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_multi_seller_auctions.md#device-orchestrated-component-auctions) and [server orchestrated](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_multi_seller_auctions.md#server-orchestrated-component-auctions) component auctions. + + +##### Device-orchestrated component auctions + +The top-level seller will receive the reporting urls for all the component sellers and winning buyers in the response from the component auctions. After the top level auction, the browser will run the top-level seller’s reportResult(). All the reporting urls will finally be pinged after the ad has rendered. + +On-device flow for top_level_seller_signals + + +###### API changes: + +For component auctions, the reporting urls will be populated in [component\_seller\_reporting\_urls](https://github.com/privacysandbox/bidding-auction-servers/blob/b27547a55f20021eb91e1e61b0d2175b4aee02ea/api/bidding_auction_servers.proto#L1173) instead of the [top\_level\_seller\_reporting\_urls](https://github.com/privacysandbox/bidding-auction-servers/blob/b27547a55f20021eb91e1e61b0d2175b4aee02ea/api/bidding_auction_servers.proto#L1178) + +##### Server-orchestrated component auctions + +The top-level seller will receive reporting urls for the component seller and winning buyer in the response from the component auctions. After the final top level auction in the top level seller's TEE based Auction service, the top level seller can run the reportResult() to generate a reporting url and return all the urls back to the client. The client will get the reporting urls for all the sellers and winning buyers and the top level seller in [AuctionResult](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L208) ciphertext in the response. + +Flow for event level reporting for server orchestrated multi seller auctions + + +###### API changes: + +For component auctions, the reporting urls will be populated in [component\_seller\_reporting\_urls](https://github.com/privacysandbox/bidding-auction-servers/blob/b27547a55f20021eb91e1e61b0d2175b4aee02ea/api/bidding_auction_servers.proto#L1173) ScoreAdsResponse instead of the [top\_level\_seller\_reporting\_urls](https://github.com/privacysandbox/bidding-auction-servers/blob/b27547a55f20021eb91e1e61b0d2175b4aee02ea/api/bidding_auction_servers.proto#L1178) + +For top-level auctions, the reporting urls will be populated in [top\_level\_seller\_reporting\_urls](https://github.com/privacysandbox/bidding-auction-servers/blob/b27547a55f20021eb91e1e61b0d2175b4aee02ea/api/bidding_auction_servers.proto#L1178) + +_Note: If the [top\_level\_seller field](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L434) in the AuctionConfig of SelectAdRequest is set, it will be considered a component auction. This field will be set by the component seller's ad server,based on an agreement with the top-level seller._ + +Once the ad has been rendered on the client, all the reporting urls will be pinged. The clients will perform a check for [enrollment](https://github.com/privacysandbox/attestation/blob/main/how-to-enroll.md)_ _and [attestation](https://github.com/privacysandbox/attestation/blob/main/README.md#core-privacy-attestations). There will be additional validation to ensure that the reporting url domain matches the domain of the AdTech on Android. diff --git a/bidding_auction_services_api.md b/bidding_auction_services_api.md index c1776a3..90a23f7 100644 --- a/bidding_auction_services_api.md +++ b/bidding_auction_services_api.md @@ -1,972 +1,2268 @@ -# Bidding and Auction Services API +**Authors:**
+[Priyanka Chatterjee][26], Google Privacy Sandbox
+Itay Sharfi, Google Privacy Sandbox + +# Bidding and Auction services [The Privacy Sandbox][4] aims to develop technologies that enable more private advertising on the web and mobile devices. Today, real-time bidding and ad auctions are executed on servers that may not provide technical guarantees of -security. Some users have concerns about how their data is handled to -generate relevant ads and in how that data is shared with ad providers. -FLEDGE ([Android][24], [Chrome][5]) provides ways to preserve privacy and limit -third-party data sharing by serving personalized ads based on previous mobile -app or web engagement. - -For all platforms, FLEDGE may require [real-time services][6]. In the initial -[FLEDGE proposal][7], bidding and auction for remarketing ad demand is -executed locally. This can demand computation requirements that may be -impractical to execute on devices with limited processing power, or may be -too slow to render ads due to network latency. - -The Bidding and Auction Services API proposal outlines a way to allow FLEDGE -computation to take place on cloud servers in a trusted execution -environment, rather than running locally on a user's device. Moving -computations to cloud servers can help optimize the FLEDGE auction, to free -up computational cycles and network bandwidth for a device. This is not a -requirement for Chrome at this point. - -This contrasts with [Google Chrome's initial FLEDGE proposal][7], which -proposes bidding and auction execution to occur locally. There are other -ideas, similar to FLEDGE, that propose server-side auction and bidding, such -as [Microsoft Edge's PARAKEET][8] proposal. - -This document focuses on the API design for FLEDGE's bidding and auction -services. Based on adtech feedback, changes may be incorporated in the -design. A document that details system design for bidding and auction services -will also be published at a later date. - -## Background - -Read the [FLEDGE services overview][6] to learn about the environment, trust -model, server attestation, request-response encryption, and other details. - -Each FLEDGE service is hosted in a virtual machine (VM) within a secure, -hardware-based trusted execution environment (TEE). Adtech platforms operate -and deploy FLEDGE services on a public cloud. Adtechs may choose the cloud -platform from one of the options that are planned. As cloud customers, -adtechs are the owners and only tenants of such VM instances. - -## Types of Auctions - -Bidding & Auction services plan to support single-seller and all types of multi-seller -auctions including [Component Auctions][25]. - -Details about multi seller auctions will be published in a separate explainer. - -## Unified Contextual & Remarketing Flow - -While this explainer talks about just the remarketing flow with Bidding & Auction Services, -we plan to update and publish an extension of this proposal for unifying the contextual and -remarketing flow. - -## Bidding and Auction system architecture - -The following is a high-level overview of the architecture of the Bidding and -Auction system for the remarketing only flow. - -![Architecture diagram.](images/bidding-auction-services-architecture.png) - -_In this diagram, one seller and one buyer are represented in the service - workflow. In reality, a single seller auction has multiple participating - buyers_. - -1. The client starts the auction process. - - The client sends a `SelectWinningAd` request to the `SellerFrontEnd` service. - This request includes the seller's auction configuration and input for each - participating buyer. -1. The `SellerFrontEnd` service orchestrates `GetBids` requests to participating - buyers’ `BuyerFrontEnd` services. -1. The `BuyerFrontEnd` services fetch real-time data from the buyer’s key-value - service required for generating bids. -1. The `BuyerFrontEnd` service sends a `GenerateBids` request to the bidding - service. The bidding service returns ad candidates with bid(s) for each IG. -1. The `BuyerFrontEnd` returns all bid(s) in IG (`AdWithBid`) to `SellerFrontEnd`. -1. Once `SellerFrontEnd` has received bids from all buyers, it requests real-time - data for render_urls (corresponding to ad with bids) from the seller’s key/value - service required to score the ads for auction. - 1. `SellerFrontEnd` sends a `ScoreAds` request to the `auction` service to - score and select a winner. The Auction service selects the the winning ad and - additional data. -1. `SellerFrontEnd` returns winning ad, server attestation metadata, other data for - for reporting purposes back to the user's device. Ad is rendered on the device. - -### Sell-side platform (SSP) system - -The following are the FLEDGE services that will be operated by an SSP, also -referred to as a Seller. Server instances are deployed so that they are -co-located in a data center within a cloud region. - -#### SellerFrontEnd service - -The `SellerFrontEnd` service orchestrates calls to other adtechs. For a -single seller auction, this service sends requests to DSPs participating in -the auction for bidding. This service also fetches real-time seller signals -and adtech proprietary code required for the auction. - -_Note: In this model and with the_ [_proposed APIs_][9], _a seller can have - auction execution on a cloud platform only when all buyers participating in - the auction also operate services for bidding on any supported cloud - platform. If required for testing purposes, the client-facing API can be - updated to support seller-run auctions in the cloud seamlessly, without - depending on a buyer's adoption of services_. - -#### Auction service - -The `Auction` service only responds to requests from `SellerFrontEnd` service and -periodically fetches private keys from Key Management Systems; other than doesn't -have outbound network access. The proposal is that code blobs required for auction is -pulled from Cloud Storage (eg. S3 / Blobstore) instance in the private VPC network as -Bidding service and cached. The code is prefetched, updated and cached periodically. - -For every ad auction request, the `Auction` service executes seller owned auction -code written in JavaScript and / or WebAssembly in a V8 sandbox instance within a TEE. -Execution of code in the sandbox within a TEE ensures that all input and output -(such as logging, file access, and disk access) is disabled, and the service has -no network or storage access. The scoring code is executed per ad within a V8 worker -process, where all scoreAd() execution for an auction request happens in parallel. -If the scoring code is in Javascript, the Javascript context is initialized before -every ScoreAd() execution; if WASM is provided with the Javascript depending on how -WASM is instantiated, the WASM can be precompiled and cached. - -The hosting environment protects the confidentiality of the Seller's proprietary -code, if the execution happens only in the cloud. - -#### Seller's key/value service - -A key/value service is a critical dependency for the auction system. The -[FLEDGE key/value service][10] receives requests from the `SellerFrontEnd` -service in this architecture. The service returns real-time seller -data required for auction that corresponds to lookup keys available in buyers' -bids (such as `ad_render_urls` or `ad_component_render_urls`). - -The seller’s key/value system may be BYOS Key/Value Service or Trusted Key/Value -Service depending on timeline. - -### Demand-side platform (DSP) system - -This section describes FLEDGE services that will be operated by a DSP, also -referred to as a buyer. Server instances are deployed such that they are -co-located in a data center within a given cloud region. - -#### BuyerFrontEnd service - -The front-end service of the system that receives requests to generate bids from -a `SellerFrontEnd` service. This service fetches real-time bidding signals, buyer -signals, and proprietary adtech code that is required for bidding. - -_Note: With the proposed APIs, a `BuyerFrontEnd` service can also receive requests -directly from the client, such as an Android app or web browser. This is -supported during the interim testing phase so that buyers (DSPs) can test / roll out -servers independently without depending on seller (SSP) adoption_. - -#### Bidding service - -The Bidding service can only respond to requests from a `BuyerFrontEnd` -service, and periodically fetches private keys from Key Management Systems; -other than doesn't have outbound network access. Code blobs required for bidding will be -pulled from Cloud Storage (eg. S3 / Blobstore) instance in the private VPC network as Bidding -service and cached. The code is prefetched, updated and cached periodically. - -For every bidding request, the service executes buyer owned bidding code written in -JavaScript and / or WebAssembly (WASM) in a V8 sandbox instance within a TEE. All input -and output (such as logging, file access, and disk access) are disabled, and the service -has no (public) network or storage access. The bidding code is executed per Custom Audience / -Interest Group within a V8 worker process, where all GenerateBid() execution for a bidding -request happens in parallel. If the bidding code is in Javascript, the Javascript context -is initialized before every GenerateBid() execution; if WASM is provided with the Javascript -depending on how WASM is instantiated, the WASM can be precompiled and cached. +security. Some users have concerns about how their data is handled to generate +relevant ads and in how that data is shared. + +Protected Auctions API provides ways to preserve privacy and limit third-party +data sharing by serving personalized ads based on previous mobile app or web +engagement. Protected Auctions include: + * Protected Audience ([Chrome][5], [Android][7]) auctions for web browser and app + * [Protected App Signals][142] for Android app + +The Bidding and Auction Services (B&A) proposal outlines a way to allow Protected +Auction computation to take place on cloud servers in a trusted execution +environment, rather than running locally on a user's device. Running workloads +in a [Trusted Execution Environment (TEE)][29] in cloud have the following benefits: + + * Scalable ad auctions. + * A scalable ad auction may include several buyers and sellers and that can + demand more compute resources and network bandwidth. + + * Lower latency of ad auctions. + * Server to server communication on the cloud is faster than multiple + device to server calls. + * Adtech code can execute faster on servers with higher computing power. + + * Higher utility of ad auctions. + * Servers have better processing power, therefore adtechs can run compute + intensive workloads on a server for better utility. + * Lower latency of ad auctions also positively impact utility. + + * Security protection + * [trusted execution environment][29] can protect confidentiality of adtech + code and signals. + + * System health of the user's device. + * Ensure better system health of user's device by freeing up computational + cycles and network bandwidth. + +This document focuses on [timeline and roadmap][127], [high level design][130], +[API][131] for Bidding and Auction services. + + +## Useful information + +### Client integration +Clients imply web browsers and Android platforms. In this context: + * Web browsers imply browsers on desktop and Android devices. + * Android implies Android apps. + +_Chrome and Android announced to integrate with Bidding and Auction services. +See [blog][27]._ + +#### Browser +Refer to [browser API for Bidding and Auction services][54]. + +In case of browser, B&A request and response payload are [Concise Binary Object Representation (CBOR)][122] +encoded. Following are the web schemas used by browsers. +* [auction_request.json][123] is the web schema corresponding to [ProtectedAudienceInput][9] message (B&A request). + * [interest_group.json][124] is the web schema for [BuyerInput.InterestGroup][82] message. +* [auction_response.json][125] is the web schema corresponding to [AuctionResult][84] message (B&A response). + +##### Near drop-in replacement + +Bidding and Auction services integrate into [Protected Audience API for browsers][28] and +can scale as a near drop-in replacement for adtechs who already adopted Protected Audience +API for on-device execution and are interested in exploring a server side solution. + + * [Interest Groups (Custom Audience) creation and management][158] can stay the same. + * User defined functions (UDF) developed by adtechs: + * The function signatures of UDFs for bid generation, scoring and reporting can + stay exactly the same. + * Buyer's code for [generateBid][69]() would mostly work. However, certain + updates will be required for [payload optimization][51], see [here][159] for + more details. + * Key-value services: + * Seller key-value service do not require additional updates. + * Buyer key-value service require some additional updates. The trusted bidding signals + may need to include more information to support [payload optimization][51], see + [here][159] for more details. + * Seller integration: + * Seller's code on client would require additional changes to call an API (provided + by client) to fetch encrypted B&A request payload (`B&A request ciphertext`) and + include that in ad request to seller's ad service. + * Seller's ad service would require additional changes to call B&A. + * Seller's ad service would require additional changes to handle encrypted B&A response + (`B&A response ciphertext`) and send that back to the client. + * Seller's ad service would require additional changes to call an API (provided by + client) to decrypt `B&A response ciphertext`. + +#### Android app + * Protected Audience: Refer to [Android's integration][7] document. + * Protected App Signals: Refer to [Protected App Signals][152] document. + +In case of Android app, the B&A request and response payload are based on protobufs. + +#### Privacy Preserving Ads +[Ad Selection API][8] is a proposal by Microsoft Edge browser and similar to Protected +Audience. The proposal supports [server-side bidding and auction][141] in [trusted execution environment][29]. + + +### Related documents +Following explainers are recommended to adtechs onboarding to B&A. You may see +also see the [full list][147] of documents related to Protected Auctions and service +trust model. -This environment protects the confidentiality of a buyer's proprietary code, if -the execution happens only in the cloud. + * [Protected Audience auctions mixed mode][143]: + Mixing of on-device and B&A auctions together is referred as mixed mode. + This explainer is a highly recommended read for sellers ([SSPs][148]). -#### Buyer’s key/value Service + * [Bidding and Auction services self-serve guide][144]: + The guide for adtech onboarding, coordinator enrollment, user-defined-function specifications, + cloud deployment, testing and scalability guidance. + **Adtechs who wants to onboard to B&A, must refer to the self-serve guide.** -A buyer's key/value service is a critical dependency for the bidding system. -The [FLEDGE key/value service][10] receives requests from the `BuyerFrontEnd` -service in this architecture. The service returns real-time buyer data required -for bidding, corresponding to lookup keys. + * [Payload optimization][51]: + Buyers ([DSPs][149]) onboarding to B&A must refer to this explainer for payload + optimization strategies. **Payload optimization is a requirement for buyers using B&A**. + This is also recommended to the sellers ([SSPs][148]) as an optimization for + configuring [per buyer payload][150] limits. -The buyer’s key/value system may be BYOS Key/Value Service or Trusted Key/Value -Service depending on timeline. + * Adtech code execution + * [Roma Bring Your Own Binary (BYOB)][151]: Recommended for buyers ([DSPs][149]) + who may want to use BYOB for generateBid() execution. + _Note: We will publish the guidance for incorporating BYOB for generateBid()._ -### Dependencies + * [Javascript and WASM based execution in Roma][153]: Adtechs may refer to the system + design of code fetch and code execution engine. -Through techniques such as prefetching and caching, the following dependencies -are in the non-critical path of ad serving. + * [Multi seller auctions design][55]: + Sellers ([SSPs][148]) may refer to this explainer to understand multi seller + auctions design with B&A. -- A key management system required for FLEDGE service attestation. Learn more - in the [FLEDGE Services public document][6]. -- A [K-anonymity service][11] ensures the ad is served to K unique users. This - service is a dependency for the `BuyerFrontEnd` service in this architecture. - The details are not included in this document, but will be covered in a - subsequent document. + * Reporting + * [Event level reporting design][135]: Adtechs may refer to this explainer to + understand event level reporting url generation with B&A. -## API + * Monitoring and TEE debugging + * [Monitoring design][145]: Describes the monitoring infra design and + [list of metrics][139] available for adtechs. -This API proposal for FLEDGE services is based on the gRPC framework. -[gRPC][12] is an open source, high performance RPC framework built -on top of HTTP2 that is used to build scalable and fast APIs. gRPC uses -[protocol buffers][13] as the [interface description -language][14] and underlying message interchange format. + * [Adtech consented debugging design][146]: Decribes design and provides guidance + to adtechs for using consented debugging. -FLEDGE services expose RPC API endpoints. In this document, the proposed API -definitions use [proto3][15]. + * Cost guidance + * [B&A cost guidance][154]: Provides an overview of cost breakdown of the + system and cost estimation guidance. -All communications between FLEDGE services are RPC and are encrypted. All -client-to-server communication is also encrypted. Refer to the [FLEDGE services -explainer][6] for more information. + * [Cost guidance for Protected App Signals][155] -A client, such as an Android app or web browser, can call FLEDGE services using -RPC or HTTPS. A proxy service hosted in the same VM instance as the FLEDGE -service converts HTTPS requests to RPC. Details of this service are out of scope -of this document. +### Github discussion -Requests to FLEDGE services and corresponding responses are encrypted. Every -request includes an encrypted payload (`request_ciphertext`) and a raw key -version (`key_id`) which corresponds to the public key that is used to encrypt -the request. The service that decrypts the request will have to use private -keys(corresponding to the same key version) to decrypt the request. +Please file an issue in [Github/WICG/protected-auction-services-discussion][156] +for feedback and discussion. -### Public APIs +### Code repository -A client such as an Android app or web browser can access public APIs. Clients -can send RPC or HTTPS to FLEDGE service. +Bidding and Auction services code is available in [Github](https://github.com/privacysandbox/bidding-auction-servers) +and a new code version is released every few weeks. + * Refer to [releases page][157] for the available code versions, prod and non-prod image hashes + of the services corresponding to each released version. + * The prod images are approved by the Coordinators and must be used in production environment. + * The non-prod images do not require Coordinator approval and must be used in + B&A services that enables TEST_MODE and disables attestation. -#### Protocol buffer <-> JSON / YAML +Adtechs will have to deploy the binaries and configurations to a supported +public cloud platform. -Given gRPC APIs are defined as protocol buffers, following are some options to -convert protocol buffers to JSON or YAML. +### Supported public cloud platforms -[Proto3][15] has a built-in JSON serializer and parser. These libraries have -multi-language support and can be used to convert protocol buffer message -objects to JSON and vice-versa. +Bidding and Auction services will be available within the [Trusted Execution Environment][29](TEE) +on AWS and GCP 2023 onwards. -- C++: - - [json_util.h][16] - - `MessageToJsonString()`: Converts Protobuf message to JSON. - - `JsonStringToMessage()`: Converts JSON to Protobuf message. -- Java: - - [`JsonFormat.Printer`][17]: Converts Protobuf message to JSON format. - - [`JsonFormat.Parser`][17]: Converts JSON to Protobuf message. +More cloud platforms may be supported eventually. See the [Public Cloud TEE requirements explainer][138] +for more details. -_Learn more about_ [_Proto3 to JSON mapping_][18]. +_Note: SSP and DSP can operate Bidding and Auction services on different cloud +platforms that are supported. For multi seller auctions, SSPs can operate on +different cloud platforms._ -YAML can also be used with HTTPS. The open source tool [gnostic][19] can -convert protocol buffers to and from YAML Open API descriptions. +Bidding and Auction services will not support a “Bring Your Own Server” model, +similar to what is made available to Protected Audience’s Key/Value server. +**Bidding and Auction services can only be deployed within approved TEE cloud +environments.** This is valid for Alpha, Beta, Scale testing programs and beyond. -Also, you may refer to the [protoc-gen-openapi plugin][20] to generate Open -API output corresponding to a proto definition. +#### AWS support +Bidding and Auction services runs in [Nitro Enclaves][30] on AWS. Refer +[here][52] for more details. -#### SelectWinningAd +#### GCP support +Bidding and Auction services runs in [Confidential Space][31] +([Confidential Computing][32]) on GCP. Refer [here][53] for more details. -The Seller FrontEnd service (`SellerFrontEnd`) exposes an API endpoint -(`SelectWinningAd`). A client such as Android device or web browser sends a -`SelectWinningAd` RPC or HTTPS request to `SellerFrontEnd`. After processing -the request, the `SelectWinningAdResponse` includes the winning ad that would -be rendered on the user's device. +#### Azure support +Subject to additional diligence and coordination with Microsoft, we expect to reach the Alpha milestone for B&A on +[Azure][167] in September 2025, and the Beta milestone 6 months after ad techs test the Alpha version and provide feedback. -_Note: - * _Seller Key Value service endpoint (`scoring_signals_url`) __is not passed from the device - to server side__; this endpoint is ingested from SellerFrontEnd service configuration at - service startup_ to prewarm network client / connection. - * _Buyer Key Value service endpoint (`bidding_signals_url`) __is not passed from the device__; this - endpoint is ingested from BuyerFrontEnd service configuration at service startup to prewarm - network client / connection_. +### Types of ad auctions -_Note: The following API is designed for the desired end state, where sellers - and all buyers operate auction and bidding services (respectively) in the - cloud_. +Bidding and Auction services supports single-seller and [Multi seller auctions][55] +for web and app traffic. -Following is the `SelectWinningAd` API definition. -``` -syntax = "proto3"; +## Timeline and roadmap -// Seller's FrontEnd service. -service SellerFrontEnd { - // Selects a winning ad for the Publisher ad slot that would be - // rendered on the user's device. - rpc SelectWinningAd(SelectWinningAdRequest) - returns (SelectWinningAdResponse) { - option (google.api.http) = { - post: "/v1/selectwinningad" - body: "*" - }; - } -} +The following sections include the timelines for ad techs interested in testing +Bidding and Auction services for Protected Auctions. Protected Auctions refer to +[Protected Audiences](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md) +and [Protected App Signals](https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals#timeline) +ad targeting products. -// SelectWinningAdRequest is sent from user's device to the -// SellerFrontEnd Service. -message SelectWinningAdRequest { - // Unencrypted request. - message SelectWinningAdRawRequest { - enum ClientType { - UNKNOWN = 0; - - // An Android device with Google Mobile Services (GMS). - // Note: This covers apps on Android and browsers on Android. - ANDROID = 1; - - // An Android device without Google Mobile Services (GMS). - ANDROID_NON_GMS = 2; - - // Any browser. - BROWSER = 3; - } - - // AuctionConfig required by the seller for an ad auction. - // The auction config includes contextual signals and other data required - // for auction. This config is passed from client to SellerFrontEnd service - // in the umbrella request (SelectWinningAdRequest). - message AuctionConfig { - // Custom seller inputs for advertising on Android. - message CustomSellerInputsForAndroid { - // To be updated later if any custom fields are required to support - // Android. - } - - // Custom seller inputs for advertising on the web. - message CustomSellerInputsForBrowser { - // Optional. Component auction configuration can contain additional - // auction configurations for each seller's "component auction". - // Represents a serialized string that can be deserialized to a JSON object. - string component_auctions = 1; - - // The Id is specified by the seller to support coordinated experiments - // with the seller's Key/Value services. - int32 experiment_group_id = 2; - } - - // Optional. Custom seller inputs. - oneof CustomSellerInputs { - CustomSellerInputsForAndroid custom_seller_inputs_android = 1; - CustomSellerInputsForBrowser custom_seller_inputs_browser = 2; - } - - /*..........................Contextual Signals.........................*/ - // Contextual Signals refer to seller_signals and auction_signals - // derived contextually. - - // Seller specific signals that include information about the context - // (e.g. Category blocks Publisher has chosen and so on). This can - // not be fetched real-time from Key-Value Server. - // Represents a serialized string that is deserialized to a JSON object - // before passing to Adtech script. - string seller_signals = 3; - - // Information about auction (ad format, size). - // This information is required for both bidding and auction. - // Represents a serialized string that is deserialized to a JSON object - // before passing to Adtech script. - string auction_signals = 4; - } - - // Optional. Required by Android to identify an ad selection request. - int64 ad_selection_request_id = 1; - - // BuyerInput per buyer. - // The key in the map corresponds to buyer Id that can identify a buyer - // participating in the auction. Buyer Id can be eTLD+1; i.e. domain address - // (ETLD+1) of the global load balancer of Buyer Frontend Service. - // The value corresponds to BuyerInput ingested by the buyer for bidding. - map input_per_buyer = 2; - - // Includes configuration data required in Remarketing ad auction. - // Some data in AuctionConfig is passed to BuyerFrontEnd. - AuctionConfig auction_config = 3; - - // Publisher website or app that is part of Buyer KV lookup url. - string publisher_hostname = 4; - - // Type of end user's device / client, that would help in validating the - // integrity of an attested client. - // Note: Not all types of clients will be attested. - ClientType client_type = 5; - - // Field representing Android client attestation data will be added later. - } +### Testing phases - // Encrypted SelectWinningAdRawRequest. - bytes request_ciphertext = 1; +The supported testing phases for Bidding and Auction services are Alpha, Beta, and Scale. +These phases are rolling windows and specific timelines may vary for every ad tech. - // Version of the public key used for request encryption. The service - // needs use private keys corresponding to same key_id to decrypt - // 'request_ciphertext'. - string key_id = 2; -} +#### Alpha testing -message SelectWinningAdResponse { - // Unencrypted response. - message SelectWinningAdRawResponse { - // The ad that will be rendered on the end user's device. - // Note: This could be an int32 instead given an ad can be uniquely identified - // by the Buyer. In that case, the device would keep the mapping of the ad - // identifier to ad_render_url. - string ad_render_url = 1; - - // Score of the winning ad. - float score = 2; - - // Custom Audience (a.k.a Interest Group) name. - string custom_audience_name = 3; - - // Bid for the winning ad candidate, generated by a buyer participating in - // the auction. - float bid_price = 4; - - // The version of the binary running on the server and Guest OS of Secure - // Virtual Machine. The client may validate that with the version - // available in open source repo. - string server_binary_version = 5; - } +During this phase, ad techs onboard, set up, and run services on **non-production user opt-in traffic**. + +The goal of this phase is to complete end-to-end functional testing from client to server. This would require a seller integrating with clients (web browsers, Android) and one or more partner buyers; and a buyer integrating with at least one partner seller. + +For alpha testing, check the following: +* [Coordinator integration](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_onboarding_self_serve_guide.md#step-3-enroll-with-coordinators) is not required. +* Debug build and production build would be available for testing. Ad techs can run Bidding and Auction services [locally (debug mode)](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_onboarding_self_serve_guide.md#local-testing) or in TEE using [test mode](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_onboarding_self_serve_guide.md#test-mode). +* As an interim milestone, ad techs can start [testing](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_onboarding_self_serve_guide.md#step-6-test) their Bidding and Auction setup, even before fully integrating with clients or partner ad techs. + * Buyers or DSPs can independently test their setup, before fully integrating with partner sellers. + * Sellers can independently test their setup with a real partner buyer or fake buyer, before fully integrating with the seller's ad server or clients. +* Clients do not necessarily need to enable a traffic experiment. Testing can be done with test or developer-owned devices. +* Ad tech testing or experimentation can be enabled on user opt-in traffic (that is, on a small set of users). + + +#### Beta testing + +During this phase, clients (web, Android) would enable Bidding and Auction APIs on +**limited stable (end user) devices**. Ad techs can run services on +**limited stable production traffic**. + +For beta testing, check the following: +* [Enrollment with coordinators](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_onboarding_self_serve_guide.md#step-3-enroll-with-coordinators) is required. +* Ad techs must run Bidding and Auction services in TEE in production mode. + * Sellers must [integrate with clients](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_onboarding_self_serve_guide.md#step-5-integration-with-clients) and at least one partner buyer for conducting protected auctions. + * The buyer must integrate with at least one seller. +* Clients would enable and ramp experiments to enable Bidding and Auction APIs on a small percentage of real end user devices. + * _Note: Clients would determine the ramp percentage for an experiment._ +* Ad techs can enable and ramp experiments on a limited stable production or live traffic for **initial performance analysis**. + * _Note: Seller or SSP would determine the ramp percentage for an experiment and align with partner buyers._ + + +#### Scale testing + +During this phase, ad techs can run services on **full stable, production scale testing**. + +For scale testing, check the following: +* [Enrollment with Coordinators](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_onboarding_self_serve_guide.md#step-3-enroll-with-coordinators) is required. +* Adtechs must run Bidding and Auction services in TEE, in production mode. + * Sellers must [integrate with clients](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_onboarding_self_serve_guide.md#step-5-integration-with-clients) and at least one partner buyer for conducting protected auctions. + * The buyer must integrate with at least one seller. +* Clients would ramp experiments to enable Bidding and Auction APIs on a larger percentage of real end user devices. +* Ad techs can enable and ramp experiments on full stable, production or live traffic. At this point, ad techs can do **advanced performance, utility, and [cost](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_cost.md) analysis**. + + +#### **Timelines** + +The following timelines show the availability of Bidding and Auction services. + + + + + + + + + + + + + + + + + + + + + + + + + + + +
API + Alpha testing + Beta testing + Scale testing +
Protected Audience +

+(web browser on desktop and Android) +

July 2023 onwards +

+Available +

    + +
  • Beta 1: Nov 2023 onwards +
  • Beta 2: April 2024 onwards +Production experiment ramp:
      + +
    • Chrome browser enabled B&A APIs for 1% stable user traffic.
    +
+ +
Jan 2025 +

+Production experiment ramp:

    + +
  • Chrome browser will enable B&A APIs for 100% stable user traffic.
+ +
Protected Audience (Android) + Available + April 2024 onwards + Jan 2025 +
Protected App Signals (Android) + Available + September 2024 onwards + Jan 2025 +
+ + + +#### **Roadmap** + +_Note: The common server side features are common to all types of Protected Auctions on web and Android apps._ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Timeline + Common server side features + Protected Audience for web browser on desktop and Android + Protected Audience and Protected App Signals (PAS) on Android Apps +
July 2023 +
    + +
  • Privacy and security protections:
      +
    • Bidding and Auction services running in a trusted execution environment. +
    • Encryption of request or response payload between client and server and TEE based servers. +
    • Padding of request or response payload between client and server.
    - // Encrypted SelectWinningAdRawResponse. - bytes response_ciphertext = 1; -} -``` +
  • Server trust model: + +
  • Adtech UDF execution support + +
  • Public cloud support:
      +
    • AWS support for Bidding and Auction services. +
    • GCP support for Bidding and Auction services. +
    • Cloud regional support.
    + +
  • Payload optimization + +
  • Reporting:
      +
    • Event level reporting: Generation of event level reporting URLs and registered beacons for Fenced Frame reporting in Bidding and Auction services.
    + +
  • Local server debugging:
      +
    • A debug binary build of servers will be available that can provide access to TEE server logs of different verbosity levels. For Google Cloud Platform, these logs will be exported to Cloud Logging.
    + +
  • Testing tools: -##### BuyerInput -`BuyerInput` data corresponding to each buyer participating in the auction -is passed in the umbrella request (`SelectWinningAdRequest`) from the client -to the `SellerFrontEnd` service. +
+ -// A BuyerInput includes data that a buyer (DSP) requires to generate bids. -message BuyerInput { - // CustomAudience (a.k.a InterestGroup) includes name, the set of ads - // corresponding to this audience, Buyer Key Value lookup keys, user bidding - // signals and other optional fields. - message CustomAudience { - // Name or tag of Custom Audience (a.k.a Interest Group). - string name = 1; +
    - // Keys to lookup from Buyer Key/Value service. - // NOTE: CA / IG name would be another lookup key besides the keys in this - // field when the Buyer KV lookup happens from BuyerFrontEnd. It is - // recommended to a Buyer that CA / IG name is not added to - // `bidding_signals_keys` so that less redundant data is shipped server - // side. Client (Android or Browser) should be check and remove IG / CA name - // from `bidding_signals_keys` so redundant / duplicate data is not sent - // over the wire. - repeated string bidding_signals_keys = 2; - } +
  • Data format for Android APIs: +
+ +
November 2023 +
    + +
  • Public cloud support:
      +
    • Cross cloud communication between SSP and DSPs.
    - // The Custom Audiences (a.k.a Interest Groups) corresponding to the buyer. - repeated CustomAudience custom_audiences = 1; +
  • Server trust model:
      +
    • Coordinator integration on GCP
    + +
  • Productionisation of servers; refer to our Github for up-to-date information. + +
+
January 2024 + + +
    + +
  • Protected App Signals
      +
    • Client and Bidding and Auction integration to support Protected App Signals. +
    • Combined PAS and PA support. +
    • Reporting: Event-level win reporting. +
    • Cloud support: GCP support.
    + +
March 2024 + +
April 2024 + +
    + +
  • Reporting:
      +
    • BuyerReportingId
    + +
  • Reporting and debugging:
      +
    • forDebugOnly (debug reporting) for single seller auctions.
    + +
    + +
  • Protected Audience:
      +
    • Custom audience (interest group) origin support
    + +
  • Protected App Signals:
      +
    • B&A and Ad Retrieval service integration:
        +
      • TEE to TEE communication. +
      • Support for ad retrieval through contextual path.
      - // First party user_id owned by the Buyer. - // This can be an additional bidding signal key for Buyer Key Value lookup. - // E.g.- Buyer may use this to lookup user_bidding_signal from Key Value server - // for each IG/CA. - string buyer_1p_user_id = 2; - - // Buyer may provide additional contextual information that could help in - // generating bids. This is derived from contextual response. - // Represents a serialized string that is deserialized to a JSON object before - // passing to Adtech script. - string buyer_signals = 3; +
    • Adtech UDF execution support:
        +
      • Javascript and Javascript with inlined WASM support.
      +
    + +
July 2024 + + +
    + +
  • Security protections:
      +
    • OHTTP Encapsulation Media Type
    - // Signals about the user's device. - // Required for bidding. - oneof DeviceSignals { - // A JSON string constructed by Android containing contextual - // information that SDK or app knows about and that adtech's bidding - // code can ingest. - string android_signals = 4; - - // A JSON string constructed by the browser, containing information that - // the browser knows about and that adtech's bidding code can ingest. - string browser_signals = 5; - } +
  • Multi cloud support on Android
      +
    • Android devices can fetch public / encryption keys for more than one cloud platform to encrypt the ProtectedAuctionInput for B&A.
    + +
  • Multiseller auction:
      +
    • Waterfall mediation
    - // Custom buyer inputs for advertising on Android. - message CustomBuyerInputsForAndroid { - // To be updated later if any custom fields are required to support Android. - } +
  • Debugging: + +
  • Protected App Signals:
      +
    • Cloud support: AWS support. +
    • B&A and Inference integration
    + +
August 2024 + + + + + +
September 2024 + + + + +
October 2024 + + + + +
November 2024 +
    + +
  • Adtech UDF execution support
      +
    • Multiple versions of adtech code blobs
    + +
  • Data version header +
+ +
+ + +
Jan 2025 +
    + +
  • K-Anonymity Integration
      +
    • Including K-Anon status in reporting +
    • K-Anon support for Private Aggregate reporting
    + +
  • Priority vector:
      +
    • This can help filter interest groups and reduce unnecessary executions in Bidding service to optimize latency and cost.
    + +
  • TEE key / value service integration for Protected Audience
+ + - // Custom buyer inputs for advertising on browsers. - message CustomBuyerInputsForBrowser { - // The Id is specified by the buyer to support coordinated experiments with - // the buyer's Key/Value services. - int32 experiment_group_id = 1; - } +
+ + + +
March 2025 + + + + + + +
April 2025 + + + + +
June 2025 +
    + +
  • Adtech UDF execution support:
      +
    • Publisher / subscriber message queue - B&A integration: This will support emergency rollouts of adtech code stored in cloud storage.
    + +
+ + + +
September 2025 + +
    +
  • Azure Alpha support for Bidding and Auction Services
  • +
+
+ + +
October 2025 and beyond +
    + +
  • Adtech UDF execution support:
      +
    • Code blob signing and verification
    + +
  • Parallelization of contextual and Bidding and Auction auctions +
  • + +
+ +**Further details to be added in future updates**. + +
    + +
  • Optimizations related to multi slot ads +
  • Support for ad size +
  • Reporting and debugging:
      +
    • forDebugOnly (debug reporting) for server orchestrated component auctions
    +
+ +
    + +
  • Waterfall Mediation optimization with server side truncation
  • +
  • Protected App Signals:
      + +
    • Protected App Signals and Protected Audience isolation support in Bidding service on supported cloud platforms.
    +
+ +
+ + +## Onboarding and alpha testing guide + +Following is the guide for onboarding to Bidding and Auction services and +participating in Alpha testing. + +### Guidance to sellers / SSPs: + * Refer to [Spec for SSP][88] section. + * Develop [ScoreAd][67]() for Protected Audience auction. + * Develop [ReportResult][75]() for event level reporting. + * Setup [Seller's Key/Value service][68]. + * Add support such that seller's code on publisher web page calls + [browser API][54] to fetch encrypted [ProtectedAudienceInput][66]. Then + includes encrypted ProtectedAudienceInput in the request to seller's ad + server. + * Add support in [Seller's ad server][20] to send [SelectAd][35] request to + Bidding and Auction services for Protected Audience auctions. + * Add support for [forwarding client metadata][90] in the request to Bidding + and Auction services (SellerFrontEnd). + * Review [Logging][93] section. + * Bidding and Auction services code and configurations is open sourced to + [Github repo][59]. + * Refer to the [README][104] for build / packaging information. + * Refer to the [README for deployment on AWS][106] or [README for deployment on GCP][105]. + * Refer to [example config on AWS][101] or [example config on GCP][103] for the + Terraform config required for cloud deployment. The config requires update of some parameter + values (that vary per adtech) before deployment to cloud. + * Deploy [SellerFrontEnd][21] and [Auction][23] server instances to your + preferred [cloud platform that is supported][98]. + * Set up experiments for ad auctions and target user opt-in traffic. Include + one or more partner buyers in the same experiment. + * [Chrome browser][54] supports a flag `FledgeBiddingAndAuctionKeyURL`. Users' browsers that + enable the flag can be targeted. + * The flag would point to the public key service endpoint in [key management systems][10]. This + is required to fetch public keys to encrypt [ProtectedAudienceInput][66]. + * [Enroll with coordinators][85] and / or run servers in `TEST_MODE`. + + During onboarding, it is recommended to start with Option 1 and then switchover to Option 2. + However, getting started with Option 2 is also feasible. + + * Option 1: Run Bidding and Auction services in `TEST_MODE`. + * `TEST_MODE` supports [cryptographic protection](#client--server-and-server--server-communication) + with hardcoded public-private key pairs, while disabling TEE server attestation. + During initial phases of onboarding, this would allow adtechs test Bidding and Auction server workloads + even before integration with Coordinators. + * Set `TEST_MODE` flag to `false` in seller's Bidding and Auction server configurations ([AWS][101], [GCP][103]). + * The following options are available for testing the auction flow with Bidding and Auction services. + * Option A: Using [secure invoke][136] tool. + * The tool uses hardcoded public keys to encrypt payloads and then sends requests to TEE based Bidding and Auction services. + The corresponding private keys of the same version are hardcoded / configured in Bidding and Auction services such that + the encrypted payloads can be correctly decrypted. + * The tool can generate [SelectAdRequest][35] payload for communication with TEE based SellerFrontEnd if a plaintext + request payload is supplied. The payload will include [ProtectedAudienceInput][66] ciphertext that is encrypted with + hardcoded public keys. + * The tool also has the capability to decrypt the response received from Bidding and Auction services and print out the + out human readable plaintext response. + * Refer to the [README][137] for more information about the tool. + + * Option B: End to end flow from Chrome browser. + * With this option, the `FledgeBiddingAndAuctionKeyURL` flag should be set to the following endpoint. The + endpoint would serve a public key such that corresponding private keys of the same version are known to + the Bidding and Auction services. + + ```FledgeBiddingAndAuctionKeyURL/http%3A%2F%2Flocalhost%3A8000%2Fkey``` + + The above endpoint can be configured to serve a public key with similar to the following. + + ```$ mkdir -p /tmp/bna && cd /tmp/bna && echo '{ "keys": [{ "id": "40", "key": "87ey8XZPXAd+/+ytKv2GFUWW5j9zdepSJ2G4gebDwyM="}]}' > key && python3 -m http.server 8000``` + + * Option 2: [Enroll with coordinators][85]. + * During Alpha, Google Privacy Sandbox Engineers will act as Coordinators and operate the + [key management systems][10]. Reach out to your Privacy Sandbox partner to get enrolled with + Alpha Coordinators. + * Integration of Bidding and Auction server workloads with Alpha Coordinators would enable TEE server attestation + and allow fetching live encryption / decryption keys from [public or private key service endpoints][10] in Bidding + and Auction services. + * The flag supported by Chrome browser `FledgeBiddingAndAuctionKeyURL` should point to the + public key service endpoint of [key management systems][10] run by Alpha Coordinators. This would allow the + browser to fetch live public keys for encryption of [ProtectedAudienceInput][66] payload. + +### Guidance to buyers / DSPs: + * Refer to [Spec for DSP][89] section. + * Develop [GenerateBid][69]() for bidding. + * Develop [ReportWin][76]() for event level reporting. + * Setup [Buyer's Key/Value service][70]. + * If your Key/Value server supports filtering of interest groups, refer to + this [section][91] and [metadata forwarding][90]. + * [Optimise payload][51]. + * Review [Logging][93] section. + * Bidding and Auction services code and configurations is open sourced to + [Github repo][59]. + * Refer to the [README][104] for build / packaging information. + * Refer to the [README for deployment on AWS][106] or [README for deployment on GCP][105]. + * Refer to [example config on AWS][100] or [example config on GCP][102] for the Terraform config + required for cloud deployment. The config requires update of some parameter values (that vary + per adtech) before deployment to cloud. + * Deploy [BuyerFrontEnd][22] and [Bidding][42] server instances to your + preferred [cloud platform that is supported][98]. + * [Enroll with coordinators][85] or run servers in `TEST_MODE`. + + During onboarding, it is recommended to start with Option 1 and then switchover to Option 2. + However, getting started with Option 2 is also feasible. + + * Option 1: Run Bidding and Auction services in `TEST_MODE`. + * `TEST_MODE` supports [cryptographic protection](#client--server-and-server--server-communication) + with hardcoded public-private key pairs, while disabling TEE server attestation. + During initial phases of onboarding, this would allow adtechs test Bidding and Auction server workloads + even before integration with Coordinators. + * Set `TEST_MODE` flag to `false` in buyer's Bidding and Auction server configurations ([AWS][100], [GCP][102]). + * **The following tool can facilitate testing buyer's Bidding and Auction services ([BuyerFrontEnd][22], [Bidding][42]) + and bid generation flow independently.** + * [Secure invoke][136] tool. + * The tool uses hardcoded public keys to encrypt payloads and then sends requests to TEE based Bidding and Auction services. + The corresponding private keys of the same version are hardcoded / configured in Bidding and Auction services such that + the encrypted payloads can be correctly decrypted. + * The tool can generate [GetBidsRequest][36] payload for communication with TEE based BuyerFrontEnd if a plaintext + request payload is supplied. + * The tool also has the capability to decrypt the response received from TEE based BuyerFrontEnd and print out the + out human readable plaintext response. + * Refer to the [README][137] for more information about the tool. + + * Option 2: [Enroll with coordinators][85]. + * During Alpha, Google Privacy Sandbox Engineers will act as Coordinators and operate the + [key management systems][10]. Reach out to your Privacy Sandbox partner to get enrolled with + Alpha Coordinators. + * Integration of Bidding and Auction server workloads with Alpha Coordinators would enable TEE server attestation + and allow fetching live encryption / decryption keys from [public or private key service endpoints][10] in Bidding + and Auction services. + * Reach out to partner SSPs to include in experiments for Protected Audience auctions. + * _Note: Buyers can also independently start integrating and testing the bidding flow before they are included in a + seller supported ad auction experiments._ + +### Enroll with coordinators + +Adtechs would have to enroll with two Coordinators running [key management systems][10] that +provision keys to Bidding and Auction services after server attestion. + +Adtechs should only enroll with the Coordinators for the specific cloud platform where +they plan to run Bidding and Auction services. + +#### Enrollment with AWS coordinators + +An adtech should provide their **AWS Account Id** to both the Coordinators. + +The Coordinators would create IAM roles. After adtechs provide the AWS account Id, they would +attach that information to the IAM roles and include in an allowlist. Then the Coordinators would +let adtechs know about the IAM roles and that should be included in the B&A server Terraform +configs that fetch cryptographic keys from [key management systems][10]. + +Following config parameters in [buyer][100] or [seller][101] server configs would include the IAM +roles information provided by the Coordinators. + * PRIMARY_COORDINATOR_ACCOUNT_IDENTITY + * SECONDARY_COORDINATOR_ACCOUNT_IDENTITY + +#### Enrollment with GCP coordinators + +An adtech should provide [**IAM service account email**][107] to both the Coordinators. + +The Coordinators would create IAM roles. After adtechs provide their service account email, the Coordinators +would attach that information to the IAM roles and include in an allowlist. Then the Coordinators would let +adtechs know about the IAM roles and that should be included in the B&A server Terraform configs that +fetch cryptographic keys from [key management systems][10]. + +Following config parameters in [buyer][102] or [seller][103] server configs would include the IAM roles +information provided by the Coordinators. + * PRIMARY_COORDINATOR_ACCOUNT_IDENTITY + * SECONDARY_COORDINATOR_ACCOUNT_IDENTITY + +## Specifications for adtechs + +### Near drop-in replacement + +Bidding and Auction services integrate into [Protected Audience API for browsers][28] and +can scale as a near drop-in replacement for adtechs who already adopted Protected Audience +API and are interested in exploring a server side solution. + + * Interest Groups (Custom Audience) creation and management can stay the same. + * For [payload optimization][51], some additional fields will be supported + in Interest Group. + + * Key-value services can stay nearly the same. + * Seller's key-value service can stay the same. + * Buyer's key-value service instances can stay the same; however to support + [payload optimization][51], `trusted_bidding_signals` may need to include + additional data. + + * Code developed by adtechs following the guidance in [Protected Audience API for browsers][28] + will mostly work with Bidding and Auction services. **The function signatures + can stay exactly the same.** + * Seller's code for [ScoreAd][67]() can stay the same. + * Seller's code for [ReportResult][75]() can stay the same. + * Buyer's code for [GenerateBid][69]() would mostly work. However, certain + updates will be required for [payload optimization][51]. + * Buyer's code for [ReportWin][76]() can stay the same. + +### Spec for SSP + +#### scoreAd() + +The [Auction service][23] exposes an API endpoint ScoreAds. The [SellerFrontEnd service][21] sends a +ScoreAdsRequest to the Auction service for running an auction. ScoreAdsRequest includes bids from each buyer +and other required signals. The code for auction, i.e. `ScoreAd()` is prefetched from Cloud Storage, cached +and precompiled in Auction service. After all ads are scored, the Auction service picks the highest scored ad +candidate and returns the score and other related data for the winning ad in ScoreAdsResponse. + +_Note: If an SSP develops `scoreAd()` following web platform's [Protected Audience API explainer][37], +that should also work as-is for execution in the Auction service._ + +Adtech's scoreAd function signature is as follows. + +``` +scoreAd(adMetadata, bid, auctionConfig, trustedScoringSignals, bid_metadata) { + ... + return {desirability: desirabilityScoreForThisAd, + allowComponentAuction: true_or_false}; } ``` -#### GetBids +##### Arguments -The `BuyerFrontEnd` service exposes an API endpoint `GetBids`. The -`SellerFrontEnd` service sends encrypted `GetBidsRequest` to the `BuyerFrontEnd` -service that include `BuyerInput` and other data. After processing the request, -`BuyerFrontEnd` returns `GetBidsResponse`, which includes bid(s) for each Interest Group. -Refer to [`AdWithBid`][22] for more information. +* `adMetadata`: Arbitrary metadata provided by the buyer. +* `bid`: A numerical bid value. +* `auctionConfig`: This would include `sellerSignals` (auctionConfig.sellerSignals) and + `auctionSignals` (auctionConfig.auctionSignals). +* `trustedScoringSignals`: trustedScoringSignals fetched from seller's Key/Value service + * Note: Only the signals required for scoring the ad / bid is passed to `scoreAd()`. +* `bid_metadata`: This refers to an object created in the Auction service based on the render_url + of the bid and other information known to the Auction service. -The communication between the `BuyerFrontEnd` service and the `SellerFrontEnd` -service is TEE to TEE communication and is end-to-end encrypted. +``` +{ 'topWindowHostname': 'www.example-publisher.com', + 'interestGroupOwner': 'https://www.example-dsp.com', + 'renderUrl': 'https://cdn.com/render_url_of_bid', + 'adComponents': ['https://cdn.com/ad_component_of_bid', + 'https://cdn.com/next_ad_component_of_bid', + ...], + 'dataVersion': 1, /* Data-Version value from the trusted scoring signals server's response */ +} +``` -_Note: Temporarily, as adtechs test these systems, clients can call - `BuyerFrontEnd` services directly using the API below_. +#### Seller BYOS Key/Value service + +_Note: BYOS Key/Value is only supported for Chrome. Protected Auctions using data from Android devices +are required to use the [Protected Auction Key/Value service][166]._ + +The [SellerFrontEnd service][21] looks up `trustedScoringSignals` from seller's Key/Value service. The +base url (domain) for Key/Value service is configured in [SellerFrontEnd service][21] so that the +connection can be prewarmed. All `render_urls` corresponding to all bids from buyers participating +in an auction are encoded, then batched and looked up in a single request. The lookup url and +response are in the same format as described in [Chrome Protected Audience explainer][40]. + +The lookup url is in the following format: ``` -syntax = "proto3"; +?renderUrls=..&adComponentRenderUrls=,.. -// Buyer’s FrontEnd service. -service BuyerFrontEnd { - // Returns bids for each Interest Group / Custom Audience. - rpc GetBids(GetBidsRequest) returns (GetBidsResponse) { - option (google.api.http) = { - post: "/v1/getbids" - body: "*" - }; - } -} +Where , ... are substituted +``` -// GetBidsRequest is sent by the `SellerFrontEnd` Service to the `BuyerFrontEnd` -// service. -message GetBidsRequest{ - // Unencrypted request. - message GetBidsRawRequest { - // Whether this is a fake request from SellerFrontEnd service - // and should be dropped. - // Note: `SellerFrontEnd` service will send chaffs to a few other buyers - // not participating in the auction. This is required for privacy reasons - // to prevent seller from figuring the buyers by observing the network - // traffic to `BuyerFrontEnd` Services, outside of TEE. - bool is_chaff = 1; - - // Buyer Input for the Buyer that includes keys for Buyer Key Value lookup - // and other signals for bidding. - BuyerInput buyer_input = 2; - - // Information about auction (ad format, size) derived contextually. - // Represents a serialized string that is deserialized to a JSON object - // before passing to Adtech script. - // Copied from Auction Config in SellerFrontEnd service. - string auction_signals = 3; - - // Publisher website or app that is part of Buyer KV lookup url. - string publisher_hostname = 4; - } - - // Encrypted GetBidsRawRequest. - bytes request_ciphertext = 1; +The response is in the following format: - // Version of the public key used for request encryption. The service - // needs use private keys corresponding to same key_id to decrypt - // 'request_ciphertext'. - string key_id = 2; +``` +{ 'renderUrls': { + 'https://cdn.com/render_url_of_some_bid': arbitrary_json, + 'https://cdn.com/render_url_of_some_other_bid': arbitrary_json, + ...}, + 'adComponentRenderUrls': { + 'https://cdn.com/ad_component_of_a_bid': arbitrary_json, + 'https://cdn.com/another_ad_component_of_a_bid': arbitrary_json, + ...} } +``` -// Response to GetBidsRequest. -message GetBidsResponse { - // Unencrypted response. - message GetBidsRawResponse { - // Includes ad_render_url and corresponding bid value pairs for each IG. - // BuyerFrontEnd will return K-Anonymized ads to SellerFrontEnd for scoring. - // Represents a JSON object. - repeated AdWithBid bids = 1; - } - - // Encrypted GetBidsRawResponse. - bytes response_ciphertext = 1; +The `trustedScoringSignals` passed to `scoreAd()` is as follows: + +``` +{ + 'renderUrl': {'https://cdn.com/render_url_of_bidder': arbitrary_value_from_signals}, + 'adComponentRenderUrls': { + 'https://cdn.com/ad_component_of_a_bid': arbitrary_value_from_signals, + 'https://cdn.com/another_ad_component_of_a_bid': arbitrary_value_from_signals, + ...} } ``` -##### AdWithBid +#### reportResult() + +Event level win reporting would work with Bidding and Auction services and function signatures +can be the same as described in web platform's [Protected Audience API explainer][43]. + +The reporting urls for the seller and registered ad beacons (for Fenced Frame reporting) would be +generated in Auction service and returned to the client in encrypted [AuctionResult][84]. The client +will ping the seller's reporting endpoint using the reporting url. -The bid for an ad candidate, includes `ad`, `bid`, `render`, -`allow_component_auction` and `custom_audience_name`. This is returned -in [`GetBidsResponse`][23]. +_Note: Refer to the [event level reporting explainer][135] for the detailed design._ ``` -syntax = "proto3"; +reportResult(auctionConfig, reporting_metadata) { + ... + registerAdBeacon({"click", clickUrl,"view", viewUrl}); + sendReportTo(reportResultUrl); + return signalsForWinner; +} -// Bid for an ad candidate. -message AdWithBid { - // Metadata of the ad, this will be passed to Seller's scoring function. - // Represents a serialized string that is deserialized to a JSON object - // before passing to Adtech script. - string ad = 1; - - // Bid price corresponding to an ad. - float bid = 2; +``` + +##### Arguments + +* `auctionConfig`: This would include `sellerSignals` (auctionConfig.sellerSignals) and + `auctionSignals` (auctionConfig.auctionSignals). +* `reporting_metadata`: This refers to an object created in the Auction service based + on the information known to the Auction service. + +#### Seller service configurations + +Server configurations are based on [Terraform][16] and is open sourced to [Github repo][59] +for cloud deployment. + +The configurations include environment variables and parameters that may vary per seller. +These can be set by the seller in the configuration before deployment. The configurations also +include urls that can be ingested when the service starts up for prewarming the connections. + +Refer to the [README for deployment on AWS][106] or [README for deployment on GCP][105]. Refer to +[example config on AWS][101] or [example config on GCP][103] for the Terraform config required +for deployment to the cloud. The config requires update of some parameter values (that vary +per adtech) before deployment to cloud. + +Following are some examples of data configured in service configurations. + +##### SellerFrontEnd service configurations + +* _Seller Key/Value service endpoint (scoring_signals_url)_: This endpoint is configured in SellerFrontEnd + service configuration and ingested at service startup to prewarm connections to seller's Key/Value service. + +* _BuyerFrontEnd endpoint_: The domain address of BuyerFrontEnd services operated by Buyers that this Seller + has partnered with. This is ingested at service startup to prewarm connection. + +* _Auction service endpoint_: The domain address of Auction service. This is ingested at service startup to + prewarm connection. + +* _Seller origin_: The origin of the seller. + * _Note: The seller origin information is also passed by the seller's ad Service in SelectAd request and + SellerFrontEnd validates that with the origin information configured._ + +* _Map of {InterestGroupOwner, BuyerFrontEnd endpoint}_: Map of InterestGroupOwner (buyer origin) to + BuyerFrontEnd domain address. - // Ad render url that identifies an ad creative. - string render = 3; +* _Global timeout for Buyer_: This information can be used to set a timeout on each buyer; however, will be + overridden by the `buyer_timeout_ms` passed by the seller's ad service to SellerFrontEnd in SelectAd request. + +* _Private Key Hosting service_ and _Public Key Hosting service_ endpoints in [key management systems][10]. + +##### Auction service configurations + +* _Cloud Storage endpoint_: The endpoint of Cloud Storage from where seller's code is hot reloaded by the + Auction service. - // Whether component auction is allowed. - bool allow_component_auction = 4; +* Private Key Hosting service and Public Key Hosting service endpoints in [key management systems][10]. - // Name of the Custom Audience / Interest Group this ad belongs to. - string custom_audience_name = 5; -} -``` +### Spec for DSP -### Internal APIs +#### generateBid() -Internal APIs refer to the interface for communication between FLEDGE services -within a SSP system or DSP system. +The [Bidding service][42] exposes an API endpoint GenerateBids. The [BuyerFrontEnd service][22] sends +GenerateBidsRequest to the Bidding service, that includes required input for bidding. The code +for bidding, i.e. `generateBid()` is prefetched from Cloud Storage, cached and precompiled in Bidding service. +After processing the request, the Bidding service returns the GenerateBidsResponse which includes +bids that correspond to each ad, i.e. [AdWithBid][49]. -#### GenerateBids +The function can be implemented in Javascript (or WASM driven by Javascript) or compiled into a [standalone binary][162]. The specifcation for both is described in detail below. -The `Bidding` service exposes an API endpoint `GenerateBids`. The -`BuyerFrontEnd` service sends `GenerateBidsRequest` to the `Bidding` service, -that includes required input for bidding. The code for bidding is prefetched from -Cloud Storage and cached in Bidding service. After processing the request, the `Bidding` -service returns the `GenerateBidsResponse` which includes bids that correspond to each ad -(`AdWithBid`). +##### generateBid() Javascript/WASM spec -The communication between the `BuyerFrontEnd` service and `Bidding` service -occurs between each service’s TEE and request-response is end-to-end -encrypted. The communication also happens over private VPC network. +_Note: If a DSP develops `generateBid()` following web platform's [Protected Audience API explainer][41], +that should also execute in Bidding service. However, certain updates will be required for [payload optimization][51]._ -**Adtech's GenerateBid function should be as follows. Note "deviceSignals" refers to -"browserSignals" or "androidSignals".** ``` -generateBid(interestGroup, auctionSignals, perBuyerSignals, - trustedBiddingSignals, deviceSignals) { +generateBid(interestGroup, auctionSignals, perBuyerSignals, trustedBiddingSignals, deviceSignals) { ... return {'ad': adObject, 'bid': bidValue, 'render': renderUrl, + 'adComponents': ["adComponentRenderUrlOne", "adComponentRenderUrlTwo"], 'allowComponentAuction': false}; } - ``` +``` + +###### Arguments + +* `interestGroup`: The InterestGroup (Custom Audience) object. Refer InterestGroup data structure to + understand what is sent in this object from the client. + * The InterestGroup is serialized and passed to generateBid() exactly as-sent, _except_ for the following divergences: + ** `DeviceSignals` are serialized and passed separately, see below. + ** `component_ads` are serialized in a field named `adComponentRenderIds` + ** `bidding_signals_keys` are serialized in a field named `trustedBiddingSignalsKeys` to align with the On-Device interestGroup spec. + * _Note: To reduce payload over the network and further optimize latency, our goal is to minimize + the information sent in this object. We will work with Adtechs for the long term to reduce the + amount of information sent in this object and try to find a solution to fetch those on the server + side._ + +* `auctionSignals`: Contextual signal that is passed from seller's ad service to SellerFrontEnd in SelectAd + request. + +* `perBuyerSignals`: Contextual signal generated by the buyer during Real Time Bidding that is passed + from seller's ad service to SellerFrontEnd in SelectAd request. + +* `trustedBiddingSignals`: Real time signals fetched by BuyerFrontEnd service from Buyer's Key/Value + service. + * _Note: Only the `trustedBiddingSignals` required for generating bid(s) for the `interestGroup` are + passed to `generateBid()`_. + +* `deviceSignals`: This refers to `browserSignals` or `androidSignals`, built by the client (browser, + Android). This includes Frequency Cap (statistics related to previous win of ads) for the user's device. + +##### generateBid() Binary spec +The signature for the GenerateBid binary is specified as proto objects. That means, the input to the GenerateBid binary will be a proto object and the output will be a proto object. The function signature looks like this - ``` -syntax = "proto3"; +GenerateProtectedAudienceBidResponse generateBid(GenerateProtectedAudienceBidRequest); +``` + +The definition for these high level protos along with nested types is specified in the [API code][160]. This is different from the Javascript signature - the parameters and return values are encapsulated in high level proto objects. These differences are discussed as follows. -// Bidding service operated by buyer. -service Bidding { - // Generate bids for ads in Custom Audiences (a.k.a InterestGroups) and - // filters ads. - rpc GenerateBids(GenerateBidsRequest) returns (GenerateBidsResponse) { - option (google.api.http) = { - post: "/v1/generatebids" - body: "*" - }; +###### Arguments +* `GenerateProtectedAudienceBidRequest`: This is the request object that encapsulates all the arguments for the generateBid() UDF (similar to the parameters in the JS spec for generateBid like interestGroup, deviceSignals, etc.). +``` +message GenerateProtectedAudienceBidRequest { + ProtectedAudienceInterestGroup interest_group = 1 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = {description: + 'This will be prepared by the Bidding service based on the data received' + ' in the BuyerInput from the device.' +}]; + + string auction_signals = 2 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = {description: + 'Auction signals are sent by the seller in the Auction Config. This can' + ' be encoded any way by the seller and will be passed as-is to the' + ' generateBid() UDF.' +}]; + + string per_buyer_signals = 3 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = {description: + 'Per buyer signals are sent by the seller in the Auction Config. This can' + ' be encoded any way by the seller and will be passed as-is to the' + ' generateBid() UDF.' +}]; + + string trusted_bidding_signals = 4 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = {description: + 'This will be passed as the JSON response received from the buyer\'s' + ' key/value server.' +}]; + + oneof ProtectedAudienceDeviceSignals { + ProtectedAudienceAndroidSignals android_signals = 5 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = {description: + 'This will be prepared by the Bidding server based on information' + ' passed by the Android app.' +}]; + + ProtectedAudienceBrowserSignals browser_signals = 6 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = {description: + 'This will be prepared by the Bidding server based on information' + ' passed by the browser on desktop or Android.' +}]; + + ServerMetadata server_metadata = 7 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = {description: + 'This will be prepared by the Bidding server and will contain config' + ' information about the current execution environment.' +}]; } } +``` + * `ServerMetadata`: The server passes additional config information for the current execution in the ServerMetadata message. This will inform the binary if logging or debug reporting functionality is available for the current execution. +``` +message ServerMetadata { + option (privacysandbox.apis.roma.app_api.v1.roma_mesg_annotation) = {description: + 'Config information about the current execution environment for a' + ' GenerateBidRequest.' +}; + + bool debug_reporting_enabled = 1 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = {description: + 'A boolean value which indicates if event level debug reporting is' + ' enabled or disabled for the request. Adtechs should only return debug' + ' URLs if this is set to true, otherwise the URLs will be ignored and' + ' creating these will be wasted compute.' +}]; + + bool logging_enabled = 2 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = {description: + 'A boolean value which indicates if logging is enabled or disabled for' + ' the request. If this is false, the logs returned from the RPC in the' + ' response will be ignored. Otherwise, these will be outputted to the' + ' standard logs or included in the response.' +}]; +} +``` -// Generate bids for all Custom Audiences (a.k.a InterestGroups) corresponding -// to the Buyer. -message GenerateBidsRequest { - // Unencrypted request. - message GenerateBidsRawRequest { - // Custom Audience (a.k.a Interest Group) for bidding. - message CustomAudienceForBidding { - // Unique string that identifies the Custom Audience (a.k.a Interest - // Group) for a buyer. - // The object "name" is part of InterestGroup JSON object that is an argument to - // GenerateBid. - string name = 1; - - // The object "ads" is part of top level InterestGroup JSON object that is an - // argument to GenerateBid. This object contains multiple ad objects. Each ad - // contains "renderUrl" and "metadata" as objects. - // This is a JSON string corresponding to JSON array. - string ads = 2; - - // User bidding signal that may be ingested during bidding and/or filtering. - // This is part of InterestGroup JSON object that is an argument to GenerateBid; - // corresponding key in JSON is `userBiddingSignals`. - // Represents a serialized string that is deserialized to a JSON object. - string user_bidding_signals = 3; - - /*********************** Optional Fields **************************/ - // Optional. This field may be populated for browser but not required - // for Android at this point. - // - // This field contains the various ad components (or "products") that - // can be used to construct Ads Composed of Multiple Pieces. Each entry - // is an object that includes both a rendering URL and arbitrary - // metadata that can be used at bidding time. - // NOTE: This should be fetched from Buyer Key Value server using - // `bidding_signals_keys`. - repeated string ad_components = 4; - } - - // Custom Audience (a.k.a Interest Group) is an input to bidding code. - repeated CustomAudienceForBidding custom_audience_for_bidding = 1; - - /********************* Common inputs for bidding ***********************/ - - // Information about auction (ad format, size) derived contextually. - // Copied from Auction Config in SellerFrontEnd service. - // Represents a serialized string that is deserialized to a JSON object - // before passing to Adtech script. - string auction_signals = 2; - - // Optional. Buyer may provide additional contextual information that - // could help in generating bids. Not fetched real-time. - // Represents a serialized string that is deserialized to a JSON object before - // passing to Adtech script. - // - // Note: This is passed in BuyerInput. - string buyer_signals = 3; - - // Real Time signals fetched from buyer’s Key/Value service. - string bidding_signals = 4; - - // Signals about client device. - // Copied from Auction Config in SellerFrontEnd service. - oneof DeviceSignals { - // A JSON string constructed by Android containing information - // that SDK or app knows about and that adtech's bidding code - // can ingest. - // The serialized string can be deserialized to a JSON object. - string android_signals = 5; - - // A JSON string constructed by the browser, containing information that - // the browser knows about and that adtech's bidding code can ingest. - // The serialized string can be deserialized to a JSON object. - string browser_signals = 6; - } - } - - // Encrypted GenerateBidsRawRequest. - bytes request_ciphertext = 1; +* `GenerateProtectedAudienceBidResponse`: This is the response field expected from the generateBid UDF. It contains the bid(s) for ad candidate(s) corresponding to a single Custom Audience (a.k.a Interest Group) (similar to the return values from the JS spec for generateBid). - // Version of the public key used for request encryption. The service - // needs use private keys corresponding to same key_id to decrypt - // 'request_ciphertext'. - string key_id = 2; +``` +message GenerateProtectedAudienceBidResponse { + repeated ProtectedAudienceBid bids = 1 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = {description: + 'The generateBid() UDF can return a list of bids instead of a single bid.' + ' This is added for supporting the K-anonymity feature. The maximum' + ' number of bids allowed to be returned is specified by the seller. When' + ' K-anonymity is disabled or not implemented, only the first candidate' + ' bid will be considered.' +}]; + + LogMessages log_messages = 2 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = {description: + 'Adtechs can add logs to the response if logging was enabled in the' + ' request. Logs will be printed out to the console in case of non-prod' + ' builds and added to the server response in case of debug consented' + ' requests.' +}]; +} +``` + * `DebugReportUrls`: URLs to support debug reporting, when auction is won and auction is lost. There is no [forDebuggingOnly][163] method/API and the debug URLs for a bid have to be directly included by the binary in the proto response. These can be added in the DebugReportUrls field in the [ProtectedAudienceBid][165] proto. The [format][164] for the URLs stays the same as the browser definition and they will be pinged in the exact same way. +``` +message DebugReportUrls { + string auction_debug_win_url = 1 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = + {description:'URL to be triggered if the Interest Group wins the auction. If undefined' + ' or malformed, it will be ignored.' + }]; + + string auction_debug_loss_url = 2 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = { + description:'URL to be triggered if the Interest Group loses the auction. If' + ' undefined or malformed, it will be ignored.' + }]; } +``` -// Encrypted response to GenerateBidsRequest with bid prices corresponding -// to all eligible Ad creatives. -message GenerateBidsResponse { - // Unencrypted response. - message GenerateBidsRawResponse { - // Bids corresponding to ads. Each AdWithBid object contains bid for ad per - // IG (CA). Note GenerateBid() per IG returns bid for one ad per IG (though for - // component auction this would be slightly different). - repeated AdWithBid bids = 1; - } - - // Encrypted GenerateBidsRawResponse. - bytes response_ciphertext = 1; + * `LogMessages`: The standard logs from the binary [are not exported for now][161] (This will be added later on in 2025). For now, any logs from the binary will be discarded. As a workaround, the GenerateProtectedAudienceBidResponse proto includes the log_messages field for logs and error messages. +``` +message LogMessages { + option (privacysandbox.apis.roma.app_api.v1.roma_mesg_annotation) = + {description: 'Logs, errors, and warnings populated by the generateBid() UDF.'}; + + repeated string logs = 1 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = + {description: 'Optional list of logs.'}]; + + repeated string errors = 2 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = + {description: 'Optional list of errors.'}]; + + repeated string warnings = 3 [(privacysandbox.apis.roma.app_api.v1.roma_field_annotation) = + {description: 'Optional list of warnings.'}]; } + ``` +The logs, errors and warnings in this proto will be printed to the cloud logs in non_prod builds, and included in the server response in case of consented debug requests. + -#### ScoreAds +#### reportWin() -The `Auction` service exposes an API endpoint `ScoreAds`. The `SellerFrontEnd` -service sends a `ScoreAdsRequest` to the `Auction` service for running auction; -ScoreAdsRequest includes bids from each buyer and other required signals. The code for -auction is prefetched from Cloud Storage and cached in Auction service. After -all ads are scored, the `Auction` service picks the highest scored ad candidate -and returns the score and other related data for the winning ad in `ScoreAdsResponse`. +Event level win reporting would work with Bidding and Auction services and function signatures +can be the same as described in web platform's [Protected Audience API explainer][43]. -The communication between the `SellerFrontEnd` service and `Auction` service -occurs within each service’s TEE and request-response is end-to-end -encrypted. The communication also happens over private VPC network. +**ReportWin() will be executed in Auction service**, reporting url for the buyer and registered ad +beacons (for Fenced Frame reporting) would be generated in Auction service and returned to the client +in encrypted [AuctionResult][84]. The client will ping the buyer's reporting endpoint using the +reporting url. + +_Note: Refer to the [event level reporting explainer][135] for the detailed design._ -**Adtech's ScoreAd function should be as follows.** -*NOTE: -*1. AuctionConfig would include sellerSignals and auctionSignals, i.e. - auctionConfig.sellerSignals and auctionConfig.auctionSignals can be derived in - ScoreAd script. -*2. The object deviceSignals may include either browserSignals or androidSignals. ``` -scoreAd(adMetadata, bid, auctionConfig, trustedScoringSignals, deviceSignals) { +reportWin(auctionSignals, perBuyerSignals, signalsForWinner, reporting_metadata) { ... - return {desirability: desirabilityScoreForThisAd, - allowComponentAuction: false}; + registerAdBeacon({"click", clickUrl,"view", viewUrl}); + sendReportTo(reportWinUrl); + return; } + ``` +##### Arguments + +* `auctionSignals`: Contextual signal generated by the seller. +* `perBuyerSignals`: Contextual signal generated by the buyer. +* `signalsForWinner`: Object returned by seller's ReportResult(). +* `reporting_metadata`: This refers to an object created in the Auction service + based on the information known to the Auction service. + +#### Buyer BYOS Key/Value service + +_Note: BYOS Key/Value is only supported for Chrome. Protected Auctions using data from Android +devices are required to use the [Protected Auction Key/Value service][166]._ + +The [BuyerFrontEnd service][22] looks up biddingSignals from Buyer's BYOS Key/Value service. The base url +(domain) for Key/Value service is configured in BuyerFrontEnd service so that the connection can be +prewarmed. All lookup keys are batched together in a single lookup request. The lookup url and response +are in the same format as described in [Chrome Protected Audience explainer][40]. + +The lookup url is the following format: + +``` +/getvalues?hostname=&experimentGroupId=&keys=,.. + + +Where , , , key_1>,... are substituted + +Note: If keys are the same as InterestGroups names, then those are not looked up more than once. ``` -syntax = "proto3"; -// Auction service operated by the seller. -service Auction { - // Scores all top ad candidates returned by each buyer participating - // in the auction. - rpc ScoreAds(ScoreAdsRequest) returns (ScoreAdsResponse) { - option (google.api.http) = { - post: "/v1/scoreads" - body: "*" - }; +The response is in the following format: + +``` +{ 'keys': { + 'key1': arbitrary_json, + 'key2': arbitrary_json, + ...}, + 'perInterestGroupData': { + 'name1': { + }, + ... } } +``` -// Scores top ad candidates of each buyer. -message ScoreAdsRequest { - // Unencrypted request. - message ScoreAdsRawRequest { - /**************** Inputs to JavaScript auction code module ****************/ - // Ad with bid. - // This includes an ad object comprising ad render url and ad metadata, - // bid corresponding to the ad, the name of Custom Audience (a.k.a Interest - // Group) the ad belongs to. - // The ad (metadata) and bid will be converted to a JSON objects before passing - // as inputs to the scoring function. The Custom Audience name is not - // required as an input for scoring but is required to be returned in response back - // to client. - // Note: Every ad is scored in a different process in an isolated Sandbox - // within the TEE. - // Note: The ad_render_url and CA/IG name in AdWithBid object is not passed to - // ScoreAd() but kept in Auction grpc service for tracking which ad is being - // scored and which IG/CA it belongs to. - repeated AdWithBid ad_bids = 1; - - /*....................... Contextual Signals .........................*/ - // Contextual Signals refer to seller_signals and auction_signals - // derived contextually. - - // Seller specific signals that include information about the context - // (e.g. Category blocks Publisher has chosen and so on). This can - // not be fetched real-time from Key-Value Server. - // This is passed to ScoreAd() in AuctionConfig JSON object, the key in JSON - // being "sellerSignals". - // Note: This is passed by client in AuctionConfig in SelectWinningAdRequest - // to SellerFrontEnd service. This data is copied from AuctionConfig. - // The serialized string can be deserialized to a JSON object. - string seller_signals = 2; - - // Information about auction (ad format, size). This information - // is available both to the seller and all buyers participating in - // auction. - // This is passed to ScoreAd() in AuctionConfig JSON object, the key in JSON - // being "auctionSignals". - // Note: This is passed by client in AuctionConfig in SelectWinningAdRequest - // to SellerFrontEnd service. This data is copied from AuctionConfig. - // The serialized string can be deserialized to a JSON object. - string auction_signals = 3; - - /*....................... Real time signals .........................*/ - // Real-time signals fetched from seller Key Value Service. - // Represents a JSON string as fetched from Seller Key Value service. - // Note: The keys used to look up scoring signals are ad_render_urls and - // ad_component_render_urls that are part of the bids returned by buyers - // participating in the auction. - string scoring_signals = 4; - - // Publisher website or app required to construct device_signals for ScoreAd() - // per ad / bid in the Auction service. - string publisher_hostname = 5; - - // InterestGroupOwner (Buyer) that Seller has partnered with and can identify. - // This is required to construct device_signals for ScoreAd() per ad / bid. - string interest_group_owner = 6; - - /************************ Custom auction parameters ***********************/ - // Custom parameters for seller code execution. - - // Custom auction params for advertising on Android. - message CustomAuctionParamsForAndroid { - // To be updated later if any custom fields are required to support - // Android. - } - - // Custom auction params for advertising on web. - message CustomAuctionParamsForBrowser { - // Optional. Component auction configuration can contain additional - // auction configurations for each seller's "component auction". - // The serialized string can be deserialized to a JSON object. - string component_auctions = 1; - } - - // Optional. Custom parameters for auction. - oneof CustomAuctionParams { - CustomAuctionParamsForAndroid custom_auction_params_android = 7; - - CustomAuctionParamsForBrowser custom_auction_params_browser = 8; - } - } +_Note: The `trustedBiddingSignals` passed to `generateBid()` for an Interest Group (Custom Audience) is +the value corresponding to each lookup key in the Interest Group but not the entire response. +Following is an example, if key1 is the lookup key in an interest group, then the following is passed +to `generateBid()` in `trustedBiddingSignals`._ - // Encrypted ScoreAdsRawRequest. - bytes request_ciphertext = 1; +``` + 'key1': arbitrary_json +``` - // Version of the public key used for request encryption. The service - // needs use private keys corresponding to same key_id to decrypt - // 'request_ciphertext'. - bytes key_id = 2; -} +##### Filtering in buyer's Key/Value service + +Filtering interest groups in buyer's Key/Value service can help reduce number +of interest groups for bidding; and therefore optimize latency and reduce cost of +Bidding service. + +To support filtering of interest groups in buyer's BYOS Key/Value service, metadata +received from the client will be forwarded in the HTTP request headers of the +`trustedBiddingSignals` lookup request. + +Refer to [metadata forwarding][90] for more details. + +#### Buyer service configurations + +Server configurations are based on [Terraform][16] and is open sourced to [Github repo][59] for +cloud deployment. + +The configurations will include environment variables and parameters that may vary per buyer. +These can be set by the buyer in the configuration before deployment. The configurations also +include urls that can be ingested when the service starts up for prewarming the connections. + +Refer to the [README for deployment on AWS][106] or [README for deployment on GCP][105]. Refer to +[example config on AWS][100] or [example config on GCP][102] for the Terraform config required +for deployment to the cloud. The config requires update of some parameter values (that vary +per adtech) before deployment to cloud. + +Following are some examples of data configured in service configurations. + +##### BuyerFrontEnd service configurations + +* _Buyer's Key/Value service endpoint (bidding_signals_url)_: This endpoint is configured in BuyerFrontEnd + service configuration and ingested at service startup to prewarm connections to buyer's Key/Value service. + +* _Bidding service endpoint_: The domain address of Bidding service. This is ingested at service startup to + prewarm connection. + +* _Private Key Hosting service_ and _Public Key Hosting service_ endpoints in [key management systems][10]. + +##### Bidding service configurations + +* _Cloud Storage endpoint_: The endpoint of Cloud Storage from where buyer's code is hot reloaded by the + Bidding service. + +* _Private Key Hosting service_ and _Public Key Hosting service_ endpoints in [key management systems][10]. + +### Metadata forwarding + +#### Metadata added by client + +Browser will forward the following metadata in the request headers of [unified request][83]. + +* `Accept-Language` +* `User-Agent` +* `IP / geo information` + +#### Metadata forwarded by seller's ad service + +Seller's ad service will forward the metadata in the following non-standard HTTP +headers in the request to SellerFrontEnd service. + +* `X-Accept-Language` +* `X-User-Agent` +* `X-BnA-Client-IP` + +_Note: If the seller's ad service uses a gRPC client to send request to SellerFrontEnd, +the standard `User-Agent` header may be altered by gRPC. Due to that reason, non-standard +HTTP headers are required so that it cannot be overridden or altered by the gRPC client._ + +Seller's ad service can do one of the following to forward the metadata: + * If the seller's ad service sends SelectAd as HTTPS to SFE, the header can + be [forwarded][46]. + * If the seller's ad service sends SelectAd as gRPC, metadata needs to be + [created and added][47]. -// Encrypted response that includes winning ad candidate. -message ScoreAdsResponse { - // Identifies the winning ad belonging to a Custom Audience / Interest Group. - message AdScore { - // Score of the ad determined during the auction. Any value that is zero or - // negative indicates that the ad cannot win the auction. The winner of the - // auction would be the ad that was given the highest score. - // The output from ScoreAd() script is desirability that implies score for - // an ad. - float desirability = 1; - - // Ad creative render url. - string render = 2; - - // Name of Custom Audience / Interest Group the ad belongs to. - string custom_audience_name = 3; - - // Bid corresponding to the winning ad. - float buyer_bid = 4; - - /***************** Only relevant to Component Auctions *******************/ - // Additional fields for Component Auctions. - - // Optional. Arbitrary metadata to pass to top level seller. - // This is also optional for Component Auctions. - string ad_metadata = 5; - - // Optional for Android, required for Web in case of component auctions. - // If the bid being scored is from a component auction and this value is not - // true, the bid is ignored. If not present, this value is considered false. - // This field must be present and true both when the component seller scores - // a bid, and when that bid is being scored by the top-level auction. - bool allow_component_auction = 6; - - // Optional for Android, required for Web in case of component auctions. - // Modified bid value to provide to the top-level seller script. If - // present, this will be passed to the top-level seller's scoring function - // instead of the original bid, if the ad wins the component auction and - // top-level auction respectively. - // This is optional, relevant for Component Auctions. - float bid = 7; - } - // The response includes the top scored ad along with other related data. - // Unencrypted response. - message ScoreAdsRawResponse { - // Score of the winning ad in the auction. - AdScore ad_score = 1; - } - - // Encrypted ScoreAdsRawResponse. - bytes response_ciphertext = 1; -} -``` +#### Metadata forwarded by SellerFrontEnd service + +SellerFrontEnd will [add metadata to gRPC][47] request sent to BuyerFrontEnd service. + +* `X-Accept-Language` +* `X-User-Agent` +* `X-BnA-Client-IP` + +SellerFrontEnd will [forward][46] the metadata in the request headers to seller's +Key/Value service. Seller's Key/Value service may ingest this information to +generate scoring signals. Seller's Key/Value service may use `X-BnA-Client-IP` +header to monitor requests from TEE based SellerFrontEnd service. + +* `Accept-Language` +* `User-Agent` +* `X-BnA-Client-IP` + +#### Metadata forwarded by BuyerFrontEnd service + +BuyerFrontEnd will [forward][46] the metadata in the request headers to buyer's +Key/Value service. + +This may help with filtering of interest groups (custom audiences) in buyer's +Key/Value service. Buyer's Key/Value service may use `X-BnA-Client-IP` header to +monitor requests from TEE based BuyerFrontEnd service. + +* `Accept-Language` +* `User-Agent` +* `X-BnA-Client-IP` + + +## High level design + +![Architecture diagram.](images/unified-contextual-remarketing-bidding-auction-services.png) + +_Note:_ +* _All arrows in the diagram are bidirectional, implying request and response._ +* _Based on feedback from Adtechs, we may incorporate further optimization in + the flow while ensuring it is privacy safe._ + +Seller's code (in the Publisher web page or app) sends one [unified request][83] +for contextual and Protected Audience auctions to seller's ad service. + +Then the seller’s ad service makes two *sequential requests*. + * [Existing flow] The seller may send real-time bidding (RTB) requests to + a select partner buyers for contextual bids, then conducts the contextual + auction. + * [Server side Protected Audience flow] The seller sends a [SelectAd][35] + request to SellerFrontEnd service to start the Protected Audience auction if + seller determines there is incremental value in conducting the auction. The + request payload includes encrypted [ProtectedAudienceInput][9], AuctionConfig + and other required information. The encrypted [ProtectedAudienceInput][9] can + only be decrypted by an attested service running in [trusted execution environment][29], + in this case the [SellerFrontEnd service][21]. + +_Unified Contextual and Protected Audience Auctions Flow_ is important to +optimize e2e auction latency. + +### Unified request + +A seller’s single request for the contextual and server-side Protected Audience +auction, from their code on a publisher site or app. The request includes +contextual payload and encrypted [ProtectedAudienceInput][9]. The seller's code +on the publisher site or app calls the client API to get the encrypted +[ProtectedAudienceInput][9] before sending the unified request. + +### Components + +#### Browser API for Bidding and Auction services + +Refer to [browser API and integration][54] design. + +#### Sell-side platform (SSP) system + +The following are the Protected Audience services that will be operated by an SSP, +also referred to as a Seller. + +##### Seller's ad service + +With the *Unified Contextual and Protected Audience Auction* flow, the seller's +ad service will receive one request from the client. The request would include +contextual request payload and encrypted [ProtectedAudienceInput][9] from the +client. + +The encrypted [ProtectedAudienceInput][9] includes Interest Group (Custom +Audience) information on the user's device. The size of [ProtectedAudienceInput][9] +is required to be small; and there may be a per-buyer size limit set by the +client or the seller. Refer to [payload optimization][51] explainer for the +guidance around optimizing [ProtectedAudienceInput][9] payload size. + +Refer to more details [here][77]. + +##### SellerFrontEnd service + +The front-end service of the system that runs in the [trusted execution environment][29] +on a supported cloud platform. The service receives requests from [seller's ad service][20] +to initiate Protected Audience auction flow. Then the service orchestrates +requests (in parallel) to Buyers / DSPs participating in the auction for bidding. + +This service also fetches real-time scoring signals required for the auction and +calls [Auction service][23] for Protected Audience auction. + +Refer to more details [here][78]. + +##### Auction service + +The Auction service runs in the [trusted execution environment][29] on a supported +cloud platform. This service responds to requests from the [SellerFrontEnd service][21] +and doesn't have access to arbitrary untrusted endpoints. + +The Auction service prefetches [code blobs](#adtech-code) owned by seller from +Cloud Storage (or an endpoint provided by the seller). The code is prefetched +at service startup, periodically thereafter and cached. More than one code +version can be supported to facilitate experimentation by adtechs. + +SSP's code for scoring ads can be written in JavaScript and / or WebAssembly (WASM). +The code runs in a custom V8 sandbox within the TEE that has tighter security +restrictions; that can not log information or has no disk or network access in +production mode. For a ScoreAds request from SellerFrontEnd, SSP's scoring +code is executed per ad within a separate V8 worker thread but all execution can +happen in parallel. Between two executions in the service, there is no state saved. + +_Note: The hosting environment protects the confidentiality of the Seller's code, +if the execution happens only in the cloud._ + +Refer to more details [here][79]. + +##### Seller's Key/Value service + +A seller's Key/Value service is a critical dependency for the auction system. +The Key/Value service receives requests from the [SellerFrontEnd service][21]. +The service returns real-time seller data required for auction that corresponds +to lookup keys available in buyers' bids (such as `ad_render_urls` +or `ad_component_render_urls`). + +_Note: For a Protected Auction using data from Chrome, the sellers’s Key/Value +system may be BYOS Key/Value Service or trusted Key/Value service depending on timeline. +For a Protected Auction using data from Android, the seller must use the trusted +Key/Value service._ + +#### Demand-side platform (DSP) system + +This section describes Protected Audience services that will be operated by a +DSP, also called a buyer. + +##### BuyerFrontEnd service + +The front-end service of the system that runs in the [trusted execution environment][29] +on a supported cloud platform. This service receives requests to generate bids +from a [SellerFrontEnd service][21]. This service fetches real-time bidding +signals that are required for bidding and calls Bidding service. + +Refer to more details [here][80]. + +##### Bidding service + +The Bidding service runs in the [trusted execution environment][29] on a supported +cloud platform. This service responds to requests from [BuyerFrontEnd service][22] +and doesn't have access to arbitrary untrusted endpoints. + +The Bidding service prefetches [code blobs](#adtech-code) owned by the buyer from +Cloud Storage (or an endpoint provided by the seller). The code is prefetched at +service startup, periodically thereafter and cached. More than one code version +can be supported to facilitate experimentation by adtechs. + +Buyer's code for generating bids can be written in JavaScript and / or WebAssembly (WASM). +The code runs in a custom V8 sandbox within the TEE that has tighter security restrictions; +that can not log information or has no disk or network access in production. For a +GenerateBids request from BuyerFrontEnd, buyer's code is executed per Interest Group +(Custom Audience) within a separate V8 worker thread but all execution can happen in +parallel. Between two executions in the service, there is no state saved. + +_Note: This environment protects the confidentiality of a buyer's code, if the +execution happens only in the cloud._ + +Refer to more details [here][81]. + +##### Buyer’s Key/Value service + +A buyer's Key/Value service is a critical dependency for the bidding system. The +Key/Value service receives requests from the [BuyerFrontEnd service][22]. The +service returns real-time buyer data required for bidding, corresponding to +lookup keys. + +_Note: For a Protected Auction using data from Chrome, the buyer’s Key/Value +system may be BYOS Key/Value Service or trusted Key/Value service depending on timeline. +For a Protected Auction using data from Android, the buyer must use the trusted +Key/Value service._ + +### Flow + +* Clients (browser, Android) builds encrypted [ProtectedAudienceInput][9]. + * Client prefetch a set of public keys from the [key management systems][10] + in the non request path every 7 days. The public keys are used for encrypting + [ProtectedAudienceInput][9]. + + * This encrypted data includes Interest Group (Custom Audience) information + for different buyers, i.e. [BuyerInput][82]. On the client, the BuyerInput(s) + are compressed, then ProtectedAudienceInput is encrypted and padded. The + padding may be done such that the size of padded ProtectedAudienceInput + ciphertext falls in one of the 7 different size buckets. + * From privacy perspective, 7 size buckets is chosen because this would + leak less than 3 bits of information in the worst case. + +* Seller's code in publisher page on the browser sends to seller's ad service. + * Web: Seller's code in publisher webpage on the browser sends HTTP request to (untrusted) + seller's ad service. + * __[Existing request, but excluding 3P Cookie]__ Contextual payload. + * Seller's code in publisher webpage asks browser for encrypted [ProtectedAudienceInput][9] + to be included in request. + * [Client adds metadata][94] to HTTP request headers. + + * Android: Seller's code in publisher SDK in Android sends HTTP request to (untrusted) + seller's ad service. + * __[Existing request]__ Contextual payload. + * Seller's code in publisher SDK asks Android for encrypted [ProtectedAudienceInput][9] + to be included in request. + +* Seller's ad service makes two requests. + * __[Existing flow]__ May send Real Time Bidding (RTB) requests to partner buyers + for contextual bids and then conduct a contextual auction to select a + contextual ad winner. + + * Sends SelectAd request to SellerFrontEnd service if there is incremental value + in conducting the Protected Audience auction. The request payload includes + encrypted [ProtectedAudienceInput][9], AuctionConfig and other + required information. + * Encrypted [ProtectedAudienceInput][9] should be `Base64` encoded string of bytes + if SelectAd request is sent as HTTPS request. + * AuctionConfig includes contextual signals like seller_signals, auction_signals; + per buyer signals / configuration and other data. Contextual ad winner + may be part of seller_signals. + * __If the SelectAd request is sent after contextual auction concludes, the + `seller_signals` may include information about the contextual auction winner that + can filter Protected Audience bids during Protected Audience auction.__ + * [Forwards client metadata][95] in non-standard HTTP headers in the request to + SellerFrontEnd service. + + __Note: It is upto the seller / SSP to decide whether SelectAd request should be sent + after contextual / RTB auction concludes or in parallel with contextual auction. The seller's + ad server may incorporate traffic shaping and determine incremental value / demand for + calling SellerFrontEnd for an ad request. The seller's ad server may call SellerFrontEnd while + the contextual auction is running and this can optimize overall auction latency even further; + however in this case contextual ad winner can not take part in Protected Audience auction to + filter Protected Audience bids. If contextual signals (`buyer_signals`, `seller_signals` and + `auction_signals`) are required for bidding, auction and reporting, that should be sent in + SelectAdRequest.__ + +* Protected Audience auction kicks off in Bidding and Auction Services. + * The SellerFrontEnd service decrypts encrypted [ProtectedAudienceInput][9] using + decryption keys prefetched from [key management systems][10]. + + * The SellerFrontEnd service orchestrates GetBids requests to participating buyers’ + BuyerFrontEnd services in parallel. + * [SellerFrontEnd adds metadata to gRPC request][96] sent to BuyerFrontEnd service. + * Buyers in `buyer_list` (in AuctionConfig) as passed by the seller and that + have non empty [BuyerInput][82] in [ProtectedAudienceInput][9] receive + GetBids request. + + * Within each buyer system: + * The BuyerFrontEnd service decrypts GetBidsRequest using decryption keys + prefetched from [key management systems][10]. + * The BuyerFrontEnd service fetches real-time data (`trustedBiddingSignals`) from + the buyer’s Key/Value service required for generating bids. + * [Forwards the metadata][97] in the request header to buyer's Key/Value service. + Buyer's Key / Value service may ingest this information for optional + [filtering of interest groups][91] and / or monitoring requests from Bidding and + Auction services. + * The BuyerFrontEnd service sends a GenerateBids request to the Bidding service. + * The Bidding service returns ad candidates with bid(s) for each `InterestGroup`. + * The Bidding service deserializes and splits `trustedBiddingSignals` such that + `generateBid()` execution for an `InterestGroup` can only ingest + `trustedBiddingSignals` for the interest group / bidding signal key. + * The BuyerFrontEnd returns all bid(s) ([AdWithBid][49]) to SellerFrontEnd. + + * Once SellerFrontEnd has received bids from all buyers, it requests real-time data + (`trustedScoringSignals`) for all `ad_render_urls` and `ad_component_render_urls` + (corresponding to ad with bids) from the seller’s Key/Value service. These + signals are required to score the ads during Protected Audience auction. + + * SellerFrontEnd sends a ScoreAdsRequest to the Auction service to score ads + and selects a winner. + * Auction service deserializes and splits scoring signal such that `scoreAd()` + execution for an ad can only ingest `trustedScoringSignals` for the ad. + + * The Auction service selects the winning ad, generates reporting urls and + returns ScoreAdsResponse to SellerFrontEnd service. + + * SellerFrontEnd returns winning ad, other metadata and reporting urls as an + encrypted [AuctionResult][84] back to seller's ad service. + + * There are a few cases when a fake, padded, encrypted [AuctionResult][84] will + be returned to the client. For such cases, contextual ad winner should be + rendered on the client. + + * The `is_chaff` field in AuctionResult will be set and that would indicate + to the client that this is a fake Protected Audience auction result. Following + are the possible scenarios when this would be set. + 1. Protected Audience auction returns no winner; this may be a possible + scenario when contextual ad winner (part of `seller_signals`) + participate in Protected Audience auction to filter all bids. Therefore, + the contextual ad wins and should be rendered on the client. + 2. When none of the participating buyers in the auction return any bid. + + * The `error` field in AuctionResult is for the client to ingest and handle. + Following are the possible scenarios when this would be set. + 1. The required fields in [ProtectedAudienceInput][9] doesn't pass validation. + 2. There is failure in de-compression of [BuyerInput][82]. + 3. The private keys (decryption keys) are compromised and revoked. In this + scenario, [ProtectedAudienceInput][9] ciphertext can be decrypted with a + compromised key but the request won't be processed further. The error field + would be set in AuctionResult to indicate to the client this corresponding + public key with the same version shouldn't be used again for encryption. + The AuctionResult will be encrypted and returned in the response. + + * SellerFrontEnd will propagate downstream service errors back to seller's ad service + and will also return gRPC / HTTP errors when required fields in SelectAdRequest.AuctionConfig + are missing. + +* Seller's ad service returns the encrypted [AuctionResult][84] back to the client. + * Contextual ad and / or encrypted ProtectedAudienceInput will be sent back + to the client. In case the contextual ad wins, a padded (chaff) but encrypted + [AuctionResult][84] will be sent in response. + +* Seller's code in publisher web page or app receives the response from seller's + ad service and passes the encrypted [AuctionResult][84] (`auction_result_ciphertext`) + to the client. + +* Only the client (browser, Android) would be able to decrypt [AuctionResult][84] + ciphertext. + +* Ad is rendered on the device. + +### Client <> server and server <> server communication + +#### Client <> seller ad service communication + +Client would send a HTTPS request ([unified request][83]) to seller's ad service. + +The [ProtectedAudienceInput][9] included in the unified request will be encrypted on the client +using a protocol called [Oblivious HTTP][50] that is based on bidirectional [Hybrid Public Key Encryption][48](HPKE). +The Protected Audience response, i.e. [AuctionResult][84] will also be encrypted in SellerFrontEnd using +[Oblivious HTTP][50]. + +The seller's ad service will not be able to decrypt or have access to [ProtectedAudienceInput][9] or +[AuctionResult][84] in plaintext. + +##### Data format +* For the web platform, request ([ProtectedAudienceInput][9]) and response ([AuctionResult][84]) +payload will be [Concise Binary Object Representation (CBOR)][122] encoded. Refer to +[web platform schemas][126]. The request will be CBOR encoded on the browser and decoded in +TEE based SellerFrontEnd service. The response will be CBOR encoded in SellerFrontEnd service +and decoded on the browser. + +* For android, request ([ProtectedAudienceInput][9]) and response ([AuctionResult][84]) +payload will be binary protobuf. + +##### Compression, encryption, padding +* For both web and Android, the BuyerInput(s) in [ProtectedAudienceInput][9] will be compressed. + Then ProtectedAudienceInput will be encrypted and padded. An exponential padding scheme will be + used. + +#### Seller ad service <> SellerFrontEnd communication + +[Seller's ad service][20] can send gRPC or HTTPS request to SellerFrontEnd service. There would be an [Envoy Proxy][45] +service instance hosted with [SellerFrontEnd][21] for HTTPS to gRPC translation. + * If seller's ad service sends HTTPS request to SellerFrontEnd, [ProtectedAudienceInput][9] ciphertext should + be `Base64` encoded; similarly the response to seller's ad service would be `Base64` encoded. This encoding is not + required if seller's ad service and SellerFrontEnd communication is gRPC. + +The communication between seller's ad service and SellerFrontEnd service would be protected by TLS / SSL that +provide communications security by encrypting data sent over the untrusted network to an authenticated peer. + +#### Communication between Bidding and Auction Services + +All communication between services running in [trusted execution environment][29] is protected by TLS / SSL +and the request and response payloads are encrypted using bidirectional [Hybrid Public Key Encryption][48](HPKE). + +**The TLS / SSL session terminates at the load balancer or the first hop in-front of a service, therefore the data over +the wire from the load balancer to service needs to be protected; hence the request-response is end-to-end encrypted using +bidirectional HPKE.** + +_For client metadata forwarded in the requests, refer [here][90]._ + +### Payload compression + +Most request/response payload sent over the wire should be compressed. +* The [BuyerInput(s)][82] in [ProtectedAudienceInput][9] will be compressed on the client using `gzip`. + The payload needs to be compressed first and then encrypted. In SellerFrontEnd, [ProtectedAudienceInput][9] + will be decrypted first and then decompressed. + +* Seller's ad service can compress [SelectAd][35] request payload when calling SellerFrontEnd service to save + network bandwidth cost and reduce latency; `gzip` is accepted by SellerFrontEnd service. The + [AuctionResult][84] will be compressed using `gzip`, then encrypted and returned in SelectAdResponse. + +* Request to BuyerFrontEnd service from SellerFrontEnd will be compressed first using `gzip` and then encrypted. + In BuyerFrontEnd service, the request will have to be decrypted first and then decompressed. + * _Note: This would be similar for communication between BuyerFrontEnd <> Bidding services and + SellerFrontEnd <> Auction services._ + +* For Key/Value Server HTTP lookup, the [Accept-Encoding HTTP request header][38] would be set to specify the + correct compression algorithm, i.e. `gzip`. The Key/Value server may set the [Content-Encoding Representation Header][39] + to specify the content encoding (i.e. `gzip`) used to compress the response payload. + * _Note:_ + * It is recommended to compress Key/Value server response to optimize Key/Value lookup latency and reduce + network bandwidth cost for adtechs. + * The request payload to Key/Value service need not be compressed given the size is expected to be small. + * The request-response payload between SellerFrontEnd <> seller's Key/Value services and + BuyerFrontEnd <> buyer's Key/Value services do not require additional encryption using [HPKE][48]. + However, the communication between these services is over TLS that provide communications security + by encrypting data sent over the untrusted network to an authenticated peer. + +### Payload optimization + +The size of compressed [ProtectedAudienceInput][9] should be small. Refer to +[payload optimization][51] explainer for more details. + +### Service code and framework + +Bidding and Auction services are developed in C++. The service configurations required for cloud deployment +are based on [Terraform][16]. + +The service framework is based on gRPC. [gRPC][12] is an open source, high performance RPC framework built +on top of HTTP2 that is used to build scalable and fast APIs. gRPC uses [protocol buffers][13] as the +[interface description language][14] and underlying message interchange format. + +Bidding and Auction services code and configurations are open sourced to [Github repo][59]. + +### Adtech code + +Adtech code for `generateBid()`, `scoreAd()`, `reportResult()`, `reportWin()` can follow +the same signature as described in the [Protected Audience API for the browser][28]. + +_Note:_ + * Code can be Javascript only or WASM only or WASM instantiated with Javascript. + * If the code is in Javascript, then Javascript context is initialized before every execution. + * No limit on code blob size. + * More than one version of code can be supported to facilitate adtech experimentation. + * Adtech can upload their code to Cloud Storage supported by the Cloud Platform. + * Code is prefetched by Bidding / Auction services running in [trusted execution environment][29] + from the Cloud Storage bucket owned by adtech. + +Refer to more details [here][86]. + +### Cloud deployment + +Bidding and Auction services are deployed by adtechs to a [public cloud platform][98] so that they are +co-located within a cloud region. + +Servers can be replicated in multiple cloud regions and the availability Service-Level-Objective (SLO) +will be decided by adtechs. + +#### SSP system + +There will be a Global Load balancer for managing / routing public traffic to [SellerFrontEnd service][21]. +Traffic between SellerFrontEnd and Auction service would be over private VPC network. To save cost, +SellerFrontEnd and Auction server instances will be configured in a [service mesh][87]. + +#### DSP system + +There will be a Global Load balancer for managing / routing public traffic to [BuyerFrontEnd services][22]. +Traffic between BuyerFrontEnd and Bidding service would be over private VPC network. To save cost, +BuyerFrontEnd and Bidding server instances will be configured in a [service mesh][87]. + +### Logging + +#### Debug / non-prod build + +Bidding and Auction server logs will be available with debug (non-prod) build / mode. The debug binaries +can be built with higher [level of verbose logging](https://github.com/google/glog#verbose-logging). +For GCP, these logs will be exported to [Cloud Logging][65]. + +The [context logger](#logcontext) in Bidding and Auction servers supports logging `generation_id` +passed by the client in encrypted [ProtectedAudienceInput][9] and optional +(per) `buyer_debug_id` and `seller_debug_id` passed in [`SelectAdRequest.AuctionConfig`][35] for +an ad request. The `adtech_debug_id` (`buyer_debug_id` or `seller_debug_id`) can be an internal +log / query id used in an adtech's non TEE based systems and if available can help the adtech trace +the ad request log in Bidding and Auction servers and map with the logs in their non TEE based systems. + +Logs from adtech's code can be made available in debug mode. + +#### Production build + +Bidding and Auction servers will support safe logging in production mode and the logs will be +exported to Cloud Logging / Cloud Watch. Refer [Debugging Protected Audience API services][60] +for more details. + +Logs from adtech's code can be made available if [adtech / user consented debugging][92] is enabled. + +### Dependencies + +Through techniques such as prefetching and caching, the following dependencies are in the non-critical +path of ad serving. + +#### Key management systems + +The [key management systems][10] are required for Protected Audience service attestation and cryptographic +key generation. Learn more in the [Overview of Protected Audience Services Explainer][6]. The +key management systems will be deployed to all supported public clouds. Services in the key management +systems will be replicated in multiple cloud regions. + +All services running in TEE prefetch encryption and decryption keys from key management systems at service +startup and periodically in the non critical path. All communication between a service in TEE and another +service in TEE is end-to-end encrypted using Hybrid Public Key Encryption and TLS. Refer [here][11] for more +details. + +## Service APIs + +Refer to Bidding and Auction services APIs [in open source repo][121]. + +### Client <> server data + +Following section describes the data that flows from client (e.g. browser, Android) to Bidding and Auction +Services through [Seller Ad service][20] and the data received by client from Bidding and Auction Services. + +#### ProtectedAudienceInput + +ProtectedAudienceInput is built and encrypted by client (browser, Android). Then sent to [Seller Ad service][20] in +the [unified request][83]. This includes per [BuyerInput](#buyer-input) and other required data. + +Refer to [ProtectedAudienceInput message][108]. + +#### BuyerInput + +BuyerInput is part of [ProtectedAudienceInput][9]. This includes data for each buyer / DSP. + +Refer to [BuyerInput message][109]. + +#### BrowerSignals +Information about an Interest Group known to the browser. These are required to +generate bid. + +Refer to [BrowserSignals message][110]. + +#### AndroidSignals +Information passed by Android for Protected Audience auctions. This will be +updated later. + +Refer to [AndroidSignals message][111]. + +#### AuctionResult + +Protected Audience auction result returned from SellerFrontEnd service to the client through the Seller +Ad service. The data is encrypted by SellerFrontEnd service and decrypted by the client. The +Seller Ad service will not be able to decrypt the data. + +In case the contextual ad wins, an AuctionResult will still be returned that includes fake data +and has is_chaff field set to true. Clients should ignore AuctionResult after decryption if +is_chaff is set to true. + +Refer to [AuctionResult message][112]. + +### Public APIs + +#### SellerFrontEnd service and API endpoints + +The SellerFrontEnd service exposes an API endpoint (SelectAd). The Seller Ad service would send +a SelectAd RPC or HTTPS request to SellerFrontEnd service. After processing the request, +SellerFrontEnd would return a SelectAdResponse that includes an encrypted AuctionResult. + +The AuctionResult will be encrypted in SellerFrontEnd using [Oblivious HTTP][50] that is based on +bidirectional [HPKE][48]. + +Refer to the [API][113]. + +#### LogContext + +Context for logging requests in Bidding and Auction servers. This includes `generation_id` +passed by the client in encrypted [ProtectedAudienceInput][9] and optional +(per) `buyer_debug_id` and `seller_debug_id` passed in [`SelectAdRequest.AuctionConfig`][35]. +The `adtech_debug_id` (`buyer_debug_id` or `seller_debug_id`) can be an internal log / query id +used in an adtech's non TEE based systems and if available can help the adtech trace the ad request +log in Bidding and Auction servers and map with the logs in their non TEE based systems. + +Refer to the [LogContext message][114]. + +#### BuyerFrontEnd service and API endpoints + +The BuyerFrontEnd service exposes an API endpoint GetBids. The SellerFrontEnd service sends +encrypted GetBidsRequest to the BuyerFrontEnd service that includes BuyerInput and other data. +After processing the request, BuyerFrontEnd returns GetBidsResponse, which includes bid(s) for +each Interest Group. Refer to [AdWithBid][49] for more information. + +The communication between the BuyerFrontEnd service and the SellerFrontEnd service is TEE to TEE +communication and is end-to-end encrypted using [HPKE][48] and TLS/SSL. The communication will happen +over public network and that can also be cross cloud networks. + +Refer to the [API][115]. + +##### AdWithBid + +The AdWithBid for an ad candidate, includes `ad` (i.e. ad metadata), `bid`, `render` (i.e. ad render url), +`allow_component_auction` and `interest_group_name`. This is returned in GetBidsResponse by +BuyerFrontEnd to SellerFrontEnd. + +Refer to the [AdWithBid message][116]. + +### Internal API + +Internal APIs refer to the interface for communication between Protected Audience services within a SSP +system or DSP system. + +#### Bidding service and API endpoints + +The Bidding service exposes an API endpoint GenerateBids. The BuyerFrontEnd service sends +GenerateBidsRequest to the Bidding service, that includes required input for bidding. The code for +bidding is prefetched from Cloud Storage and cached in Bidding service. After processing the request, +i.e. generating bids, the Bidding service returns the GenerateBidsResponse to BuyerFrontEnd service. + +The communication between the BuyerFrontEnd service and Bidding service occurs between each service’s TEE +and request-response is end-to-end encrypted using [HPKE][48] and TLS/SSL. The communication also happens +over a private VPC network. + +Refer to the [API][117]. + +#### Auction service and API endpoints + +The Auction service exposes an API endpoint ScoreAds. The SellerFrontEnd service sends a +ScoreAdsRequest to the Auction service for running auction; ScoreAdsRequest includes bids from +each buyer and other required signals. The code for auction is prefetched from Cloud Storage and +cached in Auction service. After all ads are scored, the Auction service picks the highest scored +ad candidate and returns the score and other related data for the winning ad in ScoreAdsResponse. + +The communication between the SellerFrontEnd service and Auction service occurs within each service’s +TEE and request-response is end-to-end encrypted using [HPKE][48] and TLS/SSL. The communication also +happens over a private VPC network. + +Refer to the [API][118]. + +#### WinReporting Urls + +Refer to [WinReportingUrls message][119]. + +#### DebugReporting Urls + +Refer to [DebugReportingUrls message][120]. [4]: https://privacysandbox.com -[5]: https://developer.chrome.com/docs/privacy-sandbox/fledge/ +[5]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md [6]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md -[7]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md -[8]: https://github.com/microsoft/PARAKEET -[9]: #selectwinningad -[10]: https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md -[11]: https://github.com/WICG/turtledove/blob/main/FLEDGE_k_anonymity_server.md +[7]: https://developer.android.com/design-for-safety/privacy-sandbox/protected-audience-bidding-and-auction-integration +[8]: https://github.com/WICG/privacy-preserving-ads/tree/main?tab=readme-ov-file#ad-selection-api-proposal +[9]: #protectedaudienceinput +[10]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#key-management-systems +[11]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#fledge-services [12]: https://grpc.io [13]: https://developers.google.com/protocol-buffers [14]: https://en.wikipedia.org/wiki/Interface_description_language [15]: https://developers.google.com/protocol-buffers/docs/proto3 -[16]: https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.util.json_util -[17]: https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/util/JsonFormat +[16]: https://www.terraform.io/ +[17]: https://github.com/privacysandbox [18]: https://developers.google.com/protocol-buffers/docs/proto3#json -[19]: https://github.com/google/gnostic -[20]: https://github.com/google/gnostic/tree/main/cmd/protoc-gen-openapi -[21]: #scoreads -[22]: #adwithbid -[23]: #getbid +[19]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#client-to-service-communication +[20]: #seller-ad-service +[21]: #sellerfrontend-service +[22]: #buyerfrontend-service +[23]: #auction-service [24]: https://developer.android.com/design-for-safety/privacy-sandbox/fledge [25]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#21-initiating-an-on-device-auction +[26]: https://github.com/chatterjee-priyanka +[27]: https://developer.chrome.com/blog/bidding-and-auction-services-availability +[28]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md +[29]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trusted-execution-environment +[30]: https://aws.amazon.com/ec2/nitro/nitro-enclaves/ +[31]: https://cloud.google.com/blog/products/identity-security/announcing-confidential-space +[32]: https://cloud.google.com/confidential-computing +[33]: https://github.com/privacysandbox +[34]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md +[35]: #sellerfrontend-service-and-api-endpoints +[36]: #buyerfrontend-service-and-api-endpoints +[37]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#23-scoring-bids +[38]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding +[39]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding +[40]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#31-fetching-real-time-data-from-a-trusted-server +[41]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#32-on-device-bidding +[42]: #bidding-service +[43]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#5-event-level-reporting-for-now +[44]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#35-filtering-and-prioritizing-interest-groups +[45]: https://www.envoyproxy.io/ +[46]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded +[47]: https://github.com/grpc/grpc-go/blob/master/Documentation/grpc-metadata.md#constructing-metadata +[48]: https://datatracker.ietf.org/doc/rfc9180/ +[49]: #adwithbid +[50]: https://datatracker.ietf.org/wg/ohttp/about/ +[51]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding-auction-services-payload-optimization.md +[52]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_aws_guide.md +[53]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_gcp_guide.md +[54]: https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md +[55]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_multi_seller_auctions.md +[56]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md +[57]: https://github.com/privacysandbox/fledge-docs#bidding-and-auction-services +[58]: https://github.com/privacysandbox/fledge-docs#server-productionization +[59]: https://github.com/privacysandbox/bidding-auction-servers +[60]: https://github.com/privacysandbox/fledge-docs/blob/main/debugging_protected_audience_api_services.md +[61]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#adtech-code-execution-engine +[62]: https://github.com/privacysandbox/fledge-docs/blob/main/monitoring_protected_audience_api_services.md +[63]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#fake-requests--chaffs-to-dsp +[64]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#code-blob-fetch-and-code-version +[65]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_gcp_guide.md#cloud-logging +[66]: #protectedaudienceinput +[67]: #scoread +[68]: #seller-byos-key-value-service +[69]: #generatebid +[70]: #buyer-byos-keyvalue-service +[71]: #sellerfrontend-service-configurations +[72]: #auction-service-configurations +[73]: #buyerfrontend-service-configurations +[74]: #bidding-service-configurations +[75]: #reportresult +[76]: #reportwin +[77]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#sellers-ad-service +[78]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#sellerfrontend-service +[79]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#auction-service +[80]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#buyerfrontend-service +[81]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#bidding-service +[82]: #buyerinput +[83]: #unified-request +[84]: #auctionresult +[85]: #enroll-with-coordinators +[86]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#adtech-code-execution-engine +[87]: https://en.wikipedia.org/wiki/Service_mesh +[88]: #spec-for-ssp +[89]: #spec-for-dsp +[90]: #metadata-forwarding +[91]: #filtering-in-buyers-keyvalue-service +[92]: https://github.com/privacysandbox/fledge-docs/blob/main/debugging_protected_audience_api_services.md#adtech-consented-debugging +[93]: #logging +[94]: #metadata-added-by-client +[95]: #metadata-forwarded-by-sellers-ad-service +[96]: #metadata-forwarded-by-sellerfrontend-service +[97]: #metadata-forwarded-by-buyerfrontend-service +[98]: #supported-public-cloud-platforms +[100]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/aws/terraform/environment/demo/buyer/buyer.tf +[101]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/aws/terraform/environment/demo/seller/seller.tf +[102]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/gcp/terraform/environment/demo/buyer/buyer.tf +[103]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/gcp/terraform/environment/demo/seller/seller.tf +[104]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/packaging/README.md +[105]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/gcp/terraform/environment/demo/README.md +[106]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/aws/terraform/environment/demo/README.md +[107]: https://cloud.google.com/iam/docs/service-account-overview +[108]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L27 +[109]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L51 +[110]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L118 +[111]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L136 +[112]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L143 +[113]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L210 +[114]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L369 +[115]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L380 +[116]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L459 +[117]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L503 +[118]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L638 +[119]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L867 +[120]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L892 +[121]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto +[122]: https://cbor.io/ +[123]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/services/seller_frontend_service/schemas/auction_request.json +[124]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/services/seller_frontend_service/schemas/interest_group.json +[125]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/services/seller_frontend_service/schemas/auction_response.json +[126]: #web-platform-schemas +[127]: #timeline-and-roadmap +[128]: #onboarding-and-alpha-testing-guide +[129]: #specifications-for-adtechs +[130]: #high-level-design +[131]: #service-apis +[132]: #data-format +[133]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_multi_seller_auctions.md#device-orchestrated-component-auctions +[134]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_multi_seller_auctions.md#server-orchestrated-component-auction +[135]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_event_level_reporting.md +[136]: https://github.com/privacysandbox/bidding-auction-servers/tree/main/tools/secure_invoke +[137]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/tools/secure_invoke/README.md +[138]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/public_cloud_tees.md +[139]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/monitoring_protected_audience_api_services.md#list-of-metrics +[140]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/monitoring_protected_audience_api_services.md +[141]: https://github.com/WICG/privacy-preserving-ads/blob/main/Auction%20&%20Infrastructure%20Design.md#infrastructure-design-elements +[142]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_protected_app_signals.md +[143]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/protected_audience_auctions_mixed_mode.md +[144]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_onboarding_self_serve_guide.md +[145]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/monitoring_protected_audience_api_services.md +[146]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/debugging_protected_audience_api_services.md +[147]: https://github.com/privacysandbox/protected-auction-services-docs/tree/main?tab=readme-ov-file#protected-auction-services-documentation +[148]: https://en.wikipedia.org/wiki/Supply-side_platform +[149]: https://en.wikipedia.org/wiki/Demand-side_platform +[150]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding-auction-services-payload-optimization.md#payload-optimization-guide-for-sellers--ssps +[151]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/roma_bring_your_own_binary.md +[152]: https://developers.google.com/privacy-sandbox/private-advertising/protected-audience/android/protected-app-signals +[153]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_system_design.md#adtech-code-execution-engine +[154]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_cost.md +[155]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/protected_app_signals_cost.md +[156]: https://github.com/WICG/protected-auction-services-discussion +[157]: https://github.com/privacysandbox/bidding-auction-servers/releases +[158]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#1-browsers-record-interest-groups +[159]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding-auction-services-payload-optimization.md#payload-optimization-guide-for-buyers--dsps +[160]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/udf/generate_bid.proto +[161]: https://github.com/privacysandbox/data-plane-shared-libraries/blob/main/docs/roma/byob/sdk/docs/udf/Communication%20Interface.md#standard-output-stdout +[162]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/roma_bring_your_own_binary.md +[163]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#71-fordebuggingonly-fdo-apis +[164]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#711-post-auction-signals +[165]: https://github.com/privacysandbox/bidding-auction-servers/blob/722e1542c262dddc3aaf41be7b6c159a38cefd0a/api/udf/generate_bid.proto#L261 +[166]: https://github.com/privacysandbox/protected-auction-key-value-service +[167]: https://azure.microsoft.com/ \ No newline at end of file diff --git a/bidding_auction_services_aws_guide.md b/bidding_auction_services_aws_guide.md new file mode 100644 index 0000000..dd76f85 --- /dev/null +++ b/bidding_auction_services_aws_guide.md @@ -0,0 +1,371 @@ +> FLEDGE has been renamed to Protected Audience API. To learn more about the name change, see the [blog post](https://privacysandbox.com/intl/en_us/news/protected-audience-api-our-new-name-for-fledge) + +**Authors:**
+[Daniel Kocoj][49], Google Privacy Sandbox + +# Bidding and Auction services AWS cloud support and deployment guide +_This document proposes a cloud architecture for FLEDGE Bidding and Auction services on Amazon Web Services (AWS) and also documents the packaging and deployment process for the Bidding and Auction services on AWS._ + +## Cloud Architecture for FLEDGE Bidding and Auction services +_This document proposes a cloud architecture for FLEDGE Bidding and Auction services on Amazon Web Services (AWS) with Nitro Enclave Trusted Execution Environments (TEE). The goal of this document is to enable server operators to gain familiarity with the methods of and requirements for running the Bidding and Auction services in AWS. This document will be updated as more features are added and the system evolves._ + +To learn more about FLEDGE services and the Bidding and Auction services, read the following documents: +* [FLEDGE Services Overview][1] +* [Bidding and Auction Services High Level Design and API][2] + +### Availability timeline +The FLEDGE Bidding and Auction services will be open-sourced in Q2 2023. In addition to the code for each service, we will open source Terraform configurations and scripts that will allow ad techs to easily deploy the services proposed in this document. + + +#### Comparison to Google Cloud Platform (GCP) +While this document focuses on AWS, [a similar document was published for GCP in Q2 2023][53]. We intend to provide full GCP and AWS support for these services. Most implementation details will be, at a high level, similar. A few notable exceptions (ultimately with no impact on functionality) are: + +1. GCP is based on a single [virtual machine instance][5], so a parent-child instance relationship will not be required to host the running trusted code in GCP. +1. The [Envoy component][4] runs inside the Seller Frontend TEE in GCP. +1. Each EC2 instance in AWS runs the following outside TEE: a health-checking script, and an instance of Envoy in a container. These support the [service mesh][3] for AWS. No such infrastructure is needed for mesh on GCP. + +### Sell-side platform + +The seller will operate a SellerFrontEnd service and an Auction service. These services will be responsible for orchestrating the auction and will communicate with a seller-operated Key/Value service and multiple seller-configured BuyerFrontEnd services. See [here][6] for a more detailed overview. + +The [trusted execution environment][7]-based SellerFrontEnd service will receive its requests from a [Seller Ad service][8]. These requests can be either HTTP or [gRPC][9]. See the [Envoy component][10] to learn how HTTP requests are translated to gRPC. The cloud system will also send network requests to all configured buyer front ends and key/value services. The following diagram provides an overall view of the system. + +![Seller AWS Cloud Architecture Diagram](images/seller_aws_diagram.svg) + +### Demand-side Platform + +A buy-side ad tech will be responsible for operating a BuyerFrontEnd service, a Bidding service, and a Key/Value service. A request begins with a gRPC message from a SellerFrontEnd service. The major subsequent steps then include fetching data from the buyer-operated Key/Value service and generating bids for the ads present in the request. [See a more detailed overview][11]. The following diagram provides an overall view of the system. + +![Buyer AWS Cloud Architecture Diagram](images/buyer_aws_diagram.svg) + +### Component overview + +#### Cloud region +The Bidding and Auction services are regional services, where a ‘cloud region’ refers to a particular geographic area as [defined by AWS][12]. We’ll open source Terraform configurations to support deployment in multiple regions in different geographies. Ad techs can deploy services in any region supported by AWS. The Terraform configurations include parameters that the ad tech can update before deploying to a different cloud region. + +This document uses the AWS us-west-1 region as an example. + +#### IAM Policies +IAM stands for [Identity and Access Management][13]. It’s used to securely control who has access to your AWS resources. You can use IAM to create and manage users, groups, and permissions. You can also use IAM to audit access to your resources. + +The default Terraform configuration places each [EC2 Host][14] into an IAM role. This IAM role can be used to determine permissions for the EC2 Host via an IAM policy document. It’s also used to set up admin access, such as SSH. This way, an ad tech can ssh into an EC2 host and inspect the environment. + +#### VPC +A [virtual private cloud][15] (VPC) is an isolated resource within a public cloud. Sellers and buyers should start with 1 VPC per region. A VPC is critical for security purposes, and the proposed IAM authentication and authorization model will heavily rely on the VPC configuration. + +#### Subnet +Each buyer and seller should have a public and private subnet. By hosting most components in the private subnet, the service has extra protection from internet traffic. However, the buyer and seller’s load balancers must face the public internet through an [internet gateway][16]. + +You can create outbound connections from the private subnet using the [NAT gateway][17], configured to route through the public subnet’s internet gateway. + +#### Network policies +[Security groups][18] are used to control which types of traffic flow to which ports in the VPC. Key security group rules allow for egress/ingress traffic to flow from the Load Balancers to the EC2 Instances, and EC2 instances to send network requests to external services. + +#### Load balancer +Each front-end service (SFE and BFE) has a load balancer in front. Each load balancer only applies to a particular subnet, so it is regional. The front-end service load balancers are in the public subnet while the Bidding and Auction load balancers, if and when enabled, are in the private subnets. To have a truly global system, it is the ad tech's responsibility to provide a global-scale load balancer that can route requests to its corresponding regional front end load balancers. + +The SellerFrontEnd load balancer accepts both HTTP and gRPC traffic over TLS, while the BFE load balancer (and Auction and Bidding load balancers, if enabled) take gRPC traffic only. Both of the front end services’ load balancers accept internet traffic (the load balancers for the Bidding and Auction services, if enabled, are internal-only). Each load balancer has health checks configured for its [target group][19]. A *target group* consists of an autoscaled number of EC2 hosts running a service inside a Nitro Enclave. + +#### Service Mesh + +> [!IMPORTANT] +> On September 24, 2024, Amazon [announced][54] that new customers of Amazon Web Services (AWS) will be unable to onboard and use the AWS App Mesh service. This change was effective the same day as the announcement. Existing accounts currently using AWS App Mesh will have to stop using it by September 2026. +> +> The Privacy Sandbox Bidding & Auction (B&A), Trusted Key/Value (TEE K/V) and Retrieval Services utilize AWS App Mesh for cost-efficient inter-service communication. To ensure smooth onboarding of our prospective customers considering AWS, the Privacy Sandbox team has immediately updated our services to stop the use of AWS App Mesh and replaced them with Load Balancers. There is ***no immediate functional impact to Privacy Sandbox users currently using or considering deployment of our services to AWS***. There is also no impact to users currently using or considering deployment of Privacy Sandbox services to Google Cloud Platform (GCP). +> +> Users are advised that the change from AWS App Mesh to Load Balancers may result in a change to AWS pricing. The Privacy Sandbox team is continually evaluating cost-effective solutions for the ecosystem. Users who have experience with solutions or suggestions for us to consider are welcome to do so via [GitHub][55]. + + +The SellerFrontEnd and BuyerFrontEnd both communicate with the Auction and Bidding services, respectively, using AWS AppMesh, a service mesh. This bypasses any need for an internal load balancer and saves on dedicated load balancer costs, though a flag exists in the terraform configurations to toggle use of service mesh (the default) vs internal load balancers. Specifically, the SellerFrontEnd and BuyerFrontEnd send out requests as normal, and the AWS AppMesh Envoy container intercepts these, querying the AWS AppMesh Envoy Management Service to find routes to available backend services. Each front end relies on its Envoy sidecar to distribute its requests to the appropriate backend service, using a round-robin algorithm amongst healthy instances. Each AppMesh is particular to the region in which it is deployed. Because the requests are internal to the VPC and subnet, they are sent as plaintext gRPC. +Each service runs a health-checking script on the parent EC2 instance (outside the enclave) to support the health checking features of AWS Cloud Map and AWS Auto-Scaling Group. AWS Cloud Map is a constituent component of the service mesh which provides service discovery (tells the mesh which instances resolve to which IPs). + +#### EC2 Host +The [EC2][20] Host is also known as an ‘instance.’ The instance runs a [Nitro Enclave][21] that contains a running service. The load balancer forwards traffic to a port on the instance and the traffic is then subsequently forwarded by a proxy, over VSOCK, to the listening port inside the Nitro Enclave. + +##### Envoy +Inside the seller’s front-end service EC2 Host, we provide an instance of the open source [Envoy proxy][22]. This is solely used to convert HTTP traffic to gRPC that the SellerFrontEnd service can consume. Envoy will then forward the transformed request directly to the VSOCK proxy port. If the seller ad service wishes to send gRPC traffic itself, the load balancer is configured to bypass the Envoy listening port and send traffic directly to the VSOCK proxy port. The envoy configuration is not included in the TEE and can thus be modified directly by the ad tech. + +##### VSOCK Proxy +Each EC2 host uses the open source [VSOCK proxy][23] to communicate with the services running inside the Nitro Enclave, automatically forwarding network traffic through [*proxify*][24] to ports bound by enclave-encapsulated services. The network security policy applied to the EC2 host must thus allow traffic on ports that are listened to by the enclave’s services. An important limitation of the proxy is that it currently only supports TCP traffic. VSOCK proxy communication is bidirectional and supports TLS. + +##### DNS +The [Route 5][25] Resolver handles domain name resolution at the [private subnet address][26]. + +##### Instance Type Recommendations +By default, ad techs are free to use any [instance type][27] that supports Nitro Enclaves and meets the ad tech’s performance requirements. This includes the c5a, c6i, c6a instance types (all sizes). A recommended starter instance for functionality is c6i.24xlarge. Please take note that for AWS Nitro Enclaves, there are rules governing how much vCPU and memory is dedicated to each instance – please see the limitations described for [*/etc/nitro_enclaves/allocator.yaml*][28]. + +#### Parameter Store +[AWS Parameter Store][29] is a fully managed service that makes it easy for you to store and retrieve configuration data in a central location. The default Terraforms store the servers’ runtime flags in Parameter Store and fetch the flags on server startup. An ad tech can simply modify the parameter store and restart the server for the new flag to take effect. + +#### Amazon S3 Usage +[Amazon S3][30] (a cloud object storage service) buckets are used to store the ad tech’s proprietary code modules that are required for the Bidding and Auction services. The Bidding and Auction services will communicate with S3 via an S3 VPC endpoint to fetch the code modules. Each S3 bucket must be configured to allow READ access to the EC2 host IAM role. Due to relaxed code module fetch latency requirements, the ad tech can host its code in an S3 bucket in a single region. + +#### Terraform-based Service Configuration +We will provide [Terraform][31] configurations for the Bidding and Auction services. Terraform is used to describe the cloud resource composition (via [Infrastructure as Code][32]) that is required for a fully-functional bidding and auction system and the provided Terraform configurations can be modified by the ad tech with no limitations. In fact, the Bidding and Auction services will be [configured via Terraform][33] so it is expected that the Ad Tech will interact with Terraform throughout [the deployment process][48]. + +## Guide: Package, Deploy, and Run a Service +_This section documents the packaging and deployment process for the Bidding and Auction services. The goal is for technical users to gain an understanding of how to deploy a functioning cloud environment with the service(s) of their choice._ + +### Overview +In order to create a functioning service in AWS, there are two major steps: +1. Packaging: this step involves the creation of the [Amazon Machine Image][34] (AMI) containing the service’s code in a Nitro Enclave. +2. Deployment: this step involves running Terraform to bring up the individual cloud components, including the AMI from step 1. + +### Step 0: Prerequisites +Use a Linux-based operating system to follow these instructions. Other systems have not been tested. +1. Install [`git`][35]. +2. Download the source code from the [Github repository][51]. +3. Run: + ``` + git submodule update --init + ``` + This command and all suggested commands in this document should be run from the project root directory. +4. Install [Docker][37], which is required to: + 1. Build the code. _NOTE: the code relies on the [Bazel][36] build system which is included in the default Docker images used by the build scripts._ + 2. Build the production images with attestable hashes. + 3. Run tests and binaries locally. + + To verify that docker is installed and runs, try building the code using one of the tools installed by Docker:
`builders/tools/bazel-debian info workspace`
should return an output of the Bazel workspace location. +5. You may need to provide python3 (a packaging dependency) at `/opt/bin/python3` +6. Install the [aws-cli][38] and set up [AWS CLI environment variables][39] with your credentials. If you don’t already have credentials, you must [create an AWS account][40] and then create an IAM User for yourself via the AWS GUI by completing the steps at `IAM > Users > Add users`. +

+Note that the aws-cli is not used directly, and is instead used by our toolchain via a docker image. Thus, you do not necessarily have to install aws-cli locally, but because the packaging and deployment steps will be using the aws-cli, we need to create an Access Key pair. To create credentials, in the AWS GUI complete the steps at:
+`IAM > Users > (your user) > Security credentials > Access keys > Create access key` +
+Then locally create the file ~/.aws/credentials with contents: + ``` + [default] + aws_access_key_id = + aws_secret_access_key = + ``` + Additionally, the AWS access key and secret environment variables are required in your shell environment for the server deployment process to work with our tooling. + ``` + export AWS_ACCESS_KEY_ID= + export AWS_SECRET_ACCESS_KEY= + export AWS_DEFAULT_REGION=us-west-1 + ``` + +#### Local testing +After installing the prerequisites, you should be able to test the server. To bring the server up locally: +1. Run `builders/tools/bazel-debian build` to build each server. +2. Start the server with the artifact returned by Bazel. +3. Test the server following the steps [here][41]. +4. (Optional) Run the built binary with the `--helpfull` flag to inspect the required flags. +
+ +**Startup Scripts** + +The scripts in `tools/debug` contain example startup commands and can be run directly. + +**Bidding Server** +``` +builders/tools/bazel-debian build services/bidding_service:server && ./bazel-bin/services/bidding_service/server +``` + +**Buyer Frontend Server** +``` +builders/tools/bazel-debian build services/buyer_frontend_service:server && ./bazel-bin/services/buyer_frontend_service/server +``` + +**Auction Server** +``` +builders/tools/bazel-debian build services/auction_service:server && ./bazel-bin/services/auction_service/server +``` + +**Seller Frontend Server** +``` +builders/tools/bazel-debian build services/seller_frontend_service:server && ./bazel-bin/services/seller_frontend_service/server +``` + +### Step 1: Packaging +#### Step 1.1: Configuring a Test Build (Optional) + +The file `config.bzl` presents a flag for non_prod (non-attestable) builds, `non_prod_build`. You may modify the value of the `GLOG_v` key to increase your log level for more verbose logs. + +To build a seller front end service, you may want to modify the *envoy.yaml* configuration file to expose whichever ports you need via the *socket_address* fields. The *gRPC_cluster* port must match the port passed via the *[SERVICE]_PORT* flag. + +#### Step 1.2: Building the Amazon Machine Image (AMI) +*Buy-side ad techs will only need to deploy the Buyer Front-end Server and the Bidding Server in production, while sell-side ad techs will only need to deploy the Seller Front-end Server and the Auction Server. However, when testing, ad techs may want to deploy all of the servers to better understand the message flows and structures.* +To deploy to AWS for testing, we suggest building an AMI for each service. The script to do so can be found at: +``` +production/packaging/build_and_test_all_in_docker +``` +The script takes flags to specify which service and which region to build, for example: + +``` +production/packaging/build_and_test_all_in_docker \ +--service-path auction_service --with-ami us-west-1 --platform aws --instance aws \ +--build-flavor --no-tests --no-precommit +``` + +> **Note:** +> - Switch `prod` to `non_prod` for a debugging build that turns on all vlog. +> - After the AMI is built, the PCR0 will be saved at `dist/aws/${SERVICE_PATH}.json`. `${SERVICE_PATH}` is what is specified in the `--service-path` in the build command. + +The PCR0 of `--build-flavor prod` and `--build-flavor non_prod` should match the PCR0 in the +[Release notes](https://github.com/privacysandbox/bidding-auction-servers/releases).The PCR0 hash will be validated by the Coordinators (Key Management Systems) to provision keys as part of server attestation. This is only relevant for `prod` images, not images built with `non_prod`. + +If the *--with-ami* flag is specified, the script will try to build an AMI in AWS. This will fail if you do not have your AWS Credentials configured, so take care to set that up ahead of time. + +The script outputs the AMI ID of each service you build. You will need these IDs when deploying the cloud setup, so record them (or check your AMI page in AWS). You will have to update the AMI ID in the service’s *service_vars.tf* file.

+ +The following diagram documents the major steps involved in building an AMI. ad techs are free to edit this script as it is not included in the Trusted Execution Environment. + +![Packaging Script Flow Diagram](images/bidding_auction_services_packaging.svg) + +##### AMI Structure +A key part of the AMI is the [Nitro Enclave][42]. This is a wrapper (built with *nitro-cli build-enclave*) for a Docker image and contains the trusted server binary and startup scripts. Ad techs can modify code and settings outside of the enclave. The enclave cannot be modified, because it requires attestation. Thus, the supplied Terraform cloud architecture may be changed to meet the ad tech’s needs. + +The AMI represents the base image of an EC2 parent instance. The parent hosts a VSOCK proxy (and in the case of the Seller Front End, also an instance of Envoy to enable http to gRPC transformation). This VSOCK proxy is used to communicate with the enclave with any port the enclave binds. All enclave ingress and egress traffic flows through the VSOCK proxy. Note that the parent runs *deploy/aws/terraform/services/autoscaling/instance_init_script.tftpl* on startup (as part of its autoscaling config) which prepares the instance and starts up the Nitro Enclave. + +### Step 2: Deployment +#### Step 2.1: Terraform Setup +Install Terraform, following the instructions [here][43]. +##### Terraform Layout +The Terraform is across three main folders in production/deploy/aws/terraform: +``` +├── environment +│ └── demo +├── modules +│ ├── buyer +│ └── seller +└── services + ├── autoscaling + ├── backend_services + ├── iam_group_policies + ├── iam_groups + ├── iam_role_policies + ├── iam_roles + ├── load_balancing + ├── networking + ├── security_group_rules + ├── security_groups + └── ssh +``` +##### Terraform Services Directory +This directory contains all of the individual components of a full stack: networking, load balancing, etc.. +##### Terraform Modules Directory +This directory contains the seller and buyer modules, which compose the objects found in services/ and apply defaults. Please read the variable descriptions and defaults. +##### Terraform Environment Directory +This directory contains example setups of sellers and buyers; subdirectories of *environment* (such as *setup_1*) are where you should run *terraform apply*. As an ad tech, this is where you will write (or reuse) *.tf* files. Review *setup_1/us-west-1.tf* as an example. This file contains all of the ad tech-specific details such as runtime flags, region, and domain addresses. The Terraform variable descriptions in the buyer and seller *service_vars.tf* tiles contain the complete details of each variable. + +#### Step 2.2: Configure Terraform Variables + +For recommended configurations, please see [here][53]. + +Terraform variables are split into two major categories: +1. Those for the seller (definitions, defaults found in *production/deploy/aws/terraform/modules/seller/service_vars.tf*). +2. Those for the buyer (definitions, defaults found in *production/deploy/aws/terraform/modules/buyer/service_vars.tf*). + +The seller module will bring up a SellerFrontEnd and Auction service, while the buyer module will bring up a BuyerFrontEnd and Bidding service. You can have multiple buyers for every seller, if you follow the configuration example in *setup_2/us-west-1.tf*. + +Suppose you want to bring up a test environment in which you support multiple regions. In this case, you should make your environment directory look like the following, with each *.tf* file containing region-specific parameters: +``` +├── environment + └── multi_region_setup + ├── us-central-1.tf + ├── us-east-1.tf + └── us-west-1.tf +``` + +#### Step 2.3: Apply Terraform +After modifying the provided implementations to your desired parameters (including updating all defaults, such as AMI IDs, in *modules/*/service_vars.tf*), you should run the following in your desired *environment/setup_n* directory to bring the entire system up. + +``` +terraform init && terraform apply +``` + +You can then tunnel through the Terraform-created SSH instances to enter EC2 hosts in the private subnet using this [guide][44]. The Terraform configurations automatically create a [bastion host][45] with the correct network security policies to support SSH into the individual EC2 host instances, bypassing the load balancer and allowing server operators to inspect the environment and output of the Nitro Enclaves. + +#### Step 2.4: Upload Code Modules +Ad techs must use an S3 bucket to host proprietary code modules. The bucket name will be required by the Terraform configuration so that a bucket and S3 VPC Endpoint can be created. The Bidding and Auction services will automatically fetch updates from the bucket, but it is the ad tech’s responsibility to upload their code modules to the bucket. Note that to upload to the bucket, the ad tech must modify the bucket permissions to allow their own proprietary endpoints WRITE access. This is most easily done through IAM permissions. See the AWS S3 [permission guide for details][46]. The Terraform configuration will allow the VPC’s instances READ access to the bucket by default. + +Instead of using a bucket, during alpha and early beta testing server operators may specify an arbitrary code module endpoint to fetch (via an HTTPS call) in the Terraform configuration. Only a single code module is supported. Later in beta testing, ad techs will be able to host multiple different code modules in a single bucket and specify the module to use at the individual request level. + +#### Step 2.5: Test the Service + +##### Option 1: secure_invoke +Please see the secure_invoke [README][52]. This tool is bundled with the Bidding and Auction services. + +##### Option 2: grpcurl +Use [grpcurl][47] to send a gRPC request to the load balancer address you configured in the Terraform. Requests must be addressed to port 443 so that the load balancer can terminate the TLS connection. When testing locally running services, plaintext must be used because there is no TLS connection termination support. + +**Note:** if providing a `sample_request.json`, keep in mind that the `SelectAdRequest` will still require a protected_audience_ciphertext (see `secure_invoke` in Option 1 for instructions on how to generate a ciphertext payload). Additionally, grpcurl will not be able to decrypt the `AuctionResult` ciphertext. + +*Local service: list grpc endpoints* +``` +grpcurl -plaintext localhost: list +``` + +*Local service: send query* +``` +grpcurl -plaintext -d '@' localhost: privacy_sandbox.bidding_auction_servers./ < sample_request.json +``` + +*AWS service: list grpc endpoints* +``` +grpcurl dns:///:443 list +``` + +*AWS service: send query* +``` +grpcurl -d '@' dns:///:443 privacy_sandbox.bidding_auction_servers./ < sample_request.json +``` + + +[1]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md +[2]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md +[3]: https://aws.amazon.com/app-mesh/ +[4]: #envoy +[5]: https://cloud.google.com/compute/docs/instances +[6]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#sell-side-platform-ssp-system +[7]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trusted-execution-environment +[8]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#seller-ad-service +[9]: https://grpc.io/ +[10]: #envoy +[11]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#demand-side-platform-dsp-system +[12]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html +[13]: https://aws.amazon.com/iam/ +[14]: #ec2-host +[15]: https://aws.amazon.com/vpc/ +[16]: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html +[17]: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html +[18]: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-security-groups.html +[19]: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html +[20]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html +[21]: https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave.html +[22]: https://www.envoyproxy.io/docs.html +[23]: https://github.com/privacysandbox/control-plane-shared-libraries/tree/main/cc/aws/proxy +[24]: https://github.com/privacysandbox/control-plane-shared-libraries/tree/main/cc/aws/proxy +[25]: https://aws.amazon.com/route53/ +[26]: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#AmazonDNS +[27]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html +[28]: https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave-cli-install.html#install-cli +[29]: https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html +[30]: https://aws.amazon.com/s3/ +[31]: https://developer.hashicorp.com/terraform/intro +[32]: https://www.terraform.io/use-cases/infrastructure-as-code +[33]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#service-configuration +[34]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html +[35]: https://git-scm.com/book/en/v2/Getting-Started-Installing-Git +[36]: https://bazel.build/ +[37]: https://docs.docker.com/get-docker/ +[38]: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html +[39]: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html +[40]: https://portal.aws.amazon.com/billing/signup/iam +[41]: #step-25-test-the-service +[42]: https://aws.amazon.com/ec2/nitro/nitro-enclaves/ +[43]: https://github.com/privacysandbox/fledge-key-value-service/blob/main/docs/deploying_on_aws.md#set-up-terraform +[44]: https://github.com/privacysandbox/fledge-key-value-service/blob/main/docs/deploying_on_aws.md#ssh-into-ec2 +[45]: https://aws.amazon.com/solutions/implementations/linux-bastion/ +[46]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html +[47]: https://github.com/fullstorydev/grpcurl +[48]: #guide-package-deploy-and-run-a-service +[49]: https://github.com/dankocoj-google +[50]: #load-balancer +[51]: https://github.com/privacysandbox/bidding-auction-servers +[52]: https://github.com/privacysandbox/bidding-auction-servers/tree/main/tools/secure_invoke +[53]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_gcp_guide.md +[53]: https://github.com/privacysandbox/bidding-auction-servers/tree/main/production/deploy/aws/terraform/environment/demo +[54]: https://aws.amazon.com/blogs/containers/migrating-from-aws-app-mesh-to-amazon-ecs-service-connect/ +[55]: https://github.com/WICG/protected-auction-services-discussion/issues/ diff --git a/bidding_auction_services_bid_currency.md b/bidding_auction_services_bid_currency.md new file mode 100644 index 0000000..e2369f5 --- /dev/null +++ b/bidding_auction_services_bid_currency.md @@ -0,0 +1,292 @@ +**Author:**
+[Alek Kundla][0], Google Privacy Sandbox + +# Bidding and Auction Services Bid Currency Guide + +This document provides guidance to adtechs for multi currency support of their bids in [Bidding and Auction services (B&A)](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md). + +You may refer to B&A services timeline and roadmap [here](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#timeline-and-roadmap). + +You may refer to the high level architecture and design [here](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#high-level-design). + +You may also refer to [privacy considerations](https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#privacy-considerations), [security goals](https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#security-goals) and the [trust model](https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trust-model). + +If you have any questions, please submit your feedback using the Privacy Sandbox feedback [form](https://docs.google.com/forms/d/e/1FAIpQLSePSeywmcwuxLFsttajiv7NOhND1WoYtKgNJYxw_AGR8LR1Dg/viewform) or file a github issue [here](https://github.com/WICG/protected-auction-services-discussion). + + +## Overview + +Multi-currency support means allowing buyers to bid in their own preferred currency, and yet submit their bids to participate in auctions conducted by sellers in the seller’s preferred currency, which may be different from the buyer’s. Broadly, this is accomplished by specifying the potential currencies upfront, annotating the currency of bids throughout the system, and enabling sellers and buyers to provide real-time exchange rates. + +The following sequence diagrams illustrate the effects of currency on the system. The example scenario used supposes an InterestGroup with information in JPY, a DSP which bids in EUR, and an SSP which evaluates bids in USD. + +Bid Currency support is common to both Protected Audience and Protected App Signals (PAS) AdWithBids. + + +### Advertiser and Publisher Setup + +![Diagram for Bid Currency Advertiser and Publisher Setup](images/bid_cur_ad_pub_setup.png "Diagram for Bid Currency Advertiser and Publisher Setup") + + +Advertiser and DSP Setup: + + + +* Advertisers operate their business in some currency, say “JPY”. +* Advertisers set information used for bidding (maybe a bid and a budget, for example) on each Interest Group/Custom Audience (IG/CA) in its own currency, say “JPY”. Advertisers must communicate with its buyer to do this. +* Buyer operates in some other currency, let’s say “EUR”. + * Buyer ensures that the IG/CA has a currency tag in its key-value server lookup keys, so that it can lookup real-time conversion rates from the currency of the IG/CA’s bidding information (JPY here) to its own bidding currency (EUR here). +* Buyer pushes the IGs/CAs to a device. + +Separately, Publisher and Seller Setup: + + + +* Publisher operates its business in some currency, say “USD” +* Publishers communicate this to the Seller’s Ad Server (SAS) through some channel. SAS chooses then to run the auction in “USD”. + + +### Buyer Flow + +![Diagram for Bid Currency Buyer Flow](images/bid_cur_buyer_flow.png "Diagram for Bid Currency Buyer Flow") + + + + +* Seller's code in client (browser, Android) makes ad requests to Seller Ad Server. +* Seller Ad server adds to the AuctionConfig its own currency, each buyer’s expected currency, and real-time conversion rates, in this case from EUR to USD. +* The Seller’s Seller Front-End server (SFE) sends a request for bids to each buyer’s Buyer Front-End server (BFE). +* BFE performs the Key-Value lookup for each IG/CA. IG/CAs which have a currency tag in their keys can use this to fetch a conversion rate. + * The Key-Value server response, in addition to a conversion rate (JPY to EUR here), can include any information relevant to bidding for the IG/CA, in JPY, since the buyer’s script will have access to a conversion rate. +* The IG/CAs and the bidding signals both are passed into the Buyer’s `generateBid()` UDF (user defined function). This UDF needs to be able to handle the information in JPY and a conversion rate. It must return a bid in the buyer’s specified currency (EUR here). +* This process takes place for each buyer. The results are sent back to the SFE, which then executes currency checking. + * Bids which have a currency which does not match the buyer\_currency are rejected. + + +### Seller Flow + +![Diagram for Bid Currency Seller Flow](images/bid_cur_seller_flow.png "Diagram for Bid Currency Seller Flow") + + + + +* Seller Front-End (SFE) looks up each AdWithBid’s scoring signals from Seller Key-Value server, this is totally unchanged. +* SFE makes the ScoreAdsRequest to the Auction server, containing AdWithBids (in EUR), and the seller\_signals, which contain a real-time conversion rate (set by the SAS) (from EUR to USD in this case). +* The Seller’s `scoreAd()` needs to accept AdWithBids in the buyer’s specified bidding currency (EUR here) and the conversion rate from buyer currency to seller currency (EUR to USD here) in seller\_signals. + * `scoreAd()` should also set an `incomingBidInSellerCurrency` on each bid, if applicable, to aid reporting; in this case this would be multiplying each `AdWithBid`’s `buyer_bid` by the EUR->USD conversion rate. + * In case of a component auction, `scoreAd()` must return bids in the seller\_currency (USD here). +* The Seller’s `reportResult()` script now gets the original bid and original currency alongside all of the information it currently gets + * The Buyer’s `reportWin()` gets this same new information +* In case of component auction, the Auction server checks each `AdScore`’s bid currency against the seller\_currency. +* The result is returned to the Seller Ad Server. + + +## Interest Groups / Custom Audiences + +DSPs may wish to bid in one currency (say, EUR) but store values to be used for bidding in another currency, say JPY. DSPs may put `'JPY'` in the `trustedBiddingSignalsKeys` and retrieve a conversion rate from the buyer KV server. + + +## AuctionConfig + +Sellers may set the following new fields in the `AuctionConfig`: + + + +* Their own currency in `AuctionConfig.seller_currency` +* Each participating DSP’s currency in `AuctionConfig.per_buyer_config.buyer_currency` + +Sellers may set a conversion rate from each buyer’s currency to their own (seller) currency in `sellerSignals` (which already exists in the `AuctionConfig`). These fields are optional and have no defaults. These fields must contain three uppercase letters; SFE will reject requests with invalid currency fields. [Currency checking behavior is specified below](#currency-checking). + + +## generateBid() + +`generateBid()` has always included the `bidCurrency` field in the response. + +This is to be used to specify the currency of the bid. + + +## Metrics + +B&A will use the existing bid rejection metric, kAuctionBidRejectedCount, when an `AdWithBid` or `AdScore` is rejected for currency mismatch. + +[kAuctionBidRejectedCount is partitioned by kSellerRejectReasons](https://github.com/WICG/turtledove/blob/main/FLEDGE_extended_PA_reporting.md#reporting-api-informal-specification). B&A will add two new rejection reasons to the list: + + + + + + + + + + + + + + + + + + +
Enum value + Rejection Reason + Use +
8 + BidFromGenBidFailedCurrencyCheck + Tracks AdWithBids rejected in the SFE for mismatch with the buyer_currency. +
9 + BidFromScoreAdFailedCurrencyCheck + Tracks AdScores with modified bids rejected in the auction service for mismatch with the seller_currency. +
+ + +These rejection reasons will also be included as a part of Private Aggregate Reporting with B&A. They correspond to enum values 9 and 10 in [the Reporting API informal specification of the FLEDGE API Explainer](https://github.com/WICG/turtledove/blob/main/FLEDGE_extended_PA_reporting.md#reporting-api-informal-specification), which are unlikely to change since this would represent a change in the API. + +_B&A reserves the right to change the integer values of this enum in B&A in the future to fully align with this API spec._ + + +## Currency Checking + +Currency checking in B&A matches that of the Chrome implementation. [To quote that explainer](https://github.com/WICG/turtledove/blob/main/FLEDGE.md#36-currency-checking): + + + +* If participants in the auction need to deal with multiple currencies, they can optionally take advantage of automated currency checking. All of it operates on currency tags, which are required to contain 3 upper-case ASCII letters. +* If the `generateBid()` method returns a `bidCurrency`, and the `perBuyerConfig` for that buyer specifies a `buyerCurrency`, their consistency will be checked, and if there is a mismatch, the bid will be dropped. Both the `buyerCurrency` for that buyer and returned `bidCurrency` must be present for checking to take place; if one or both are missing the currency check does not take place and the bid is passed on as-is. The returned `bidCurrency` will be passed to `scoreAd()`'s `browserSignals.bidCurrency`, with unspecified currency rendered as `'???'`. +* Currency checking after `scoreAd()` happens only [inside component auctions; please refer to this section](#component-auctions). +* `sellerCurrency` is used in currency checking for component auctions and also has an extensive effect on how reporting behaves. [Please see the reporting section for more details](#reporting). +* `sellerCurrency`'s one other effect is with respect to checking the value of `incomingBidInSellerCurrency`, which is not checked unless the `AdWithBid.bidCurrency` matches the `sellerCurrency` and its value is set. In this case, the `incomingBidInSellerCurrency` must match the original `AdWithBid.bid`, else the bid will be rejected. +* For each AdWithBid dropped, B&A will increment the `kAuctionBidRejectedCount` metric and record the rejection reason with it as `BidFromGenBidFailedCurrencyCheck`. If all `AdWithBid`s are dropped, the request will finish with grpc status code `INVALID_ARGUMENT` and the message `"All bids rejected for failure to match buyer currency."` + + +## In the Auction Server + + +### scoreAd() + +bidMetadata (the fifth argument to scoreAd(), formerly called deviceSignals in B&A & browserSignals on Chrome) now passes the bid’s currency in the field "bidCurrency". + +For all auctions, `scoreAd()` now includes a new field in the response, `incomingBidInSellerCurrency`. For component auctions, the field `bidCurrency` is also to be used. + + + +* `incomingBidInSellerCurrency`: (optional) Provides a conversion of a bid in a multi-currency auction to the seller's own currency. See below. +* `bid` (already extant; the modified bid for component auctions) +* `bidCurrency`: Currency of the modified bid above (therefore for component auctions only too). + + +#### `incomingBidInSellerCurrency` + + + +* If `sellerCurrency` is set, `scoreAd()` for an auction is responsible for converting bids not already in `sellerCurrency` to `sellerCurrency`, via the `incomingBidInSellerCurrency` field of its return value. A bid already explicitly in the seller's currency cannot be changed by `incomingBidInSellerCurrency` (passing an identical value is a no-op; **passing a different one rejects the bid**). +* If neither the original bid is explicitly in `sellerCurrency` nor an `incomingBidInSellerCurrency` is specified, a value of 0 is used as the converted value. + + +### Component Auctions + + + +* If no modified bid is specified on the `AdScore` from `scoreAd()`, the original `AdWithBid`’s `bid` and `bid_currency` are set on the `AdScore`. +* If the `seller_currency` is specified, then the currency of the `AdScore` (if specified, and whether from `scoreAd()` or not) is checked against it. AdScores will be rejected for mismatch. + + +#### Metrics + +For each `AdScore` rejected for failing to match the `seller_currency`, [kAuctionBidRejectedCount](https://source.corp.google.com/h/team/android-privacy-sandbox-remarketing/fledge/servers/bidding-auction-server/+/main:services/common/metric/server_definition.h;l=197;drc=fd1abfdac744124967378137d336ad65f034251a) is incremented and the rejection reason set to `BidFromScoreAdFailedCurrencyCheck`. + + +## Reporting + + +##### Win Reporting + +The following fields have have been added to the `buyer/sellerReportingMetadata` parameter of `reportResult()` and `reportWin()`: + + + +* `bidCurrency`, denoting the currency of the `bid` field already present. +* `highestScoringOtherBidCurrency`, denoting the currency of the `highestScoringOtherBid` already present. + +These will be populated according to the this [table](https://github.com/WICG/turtledove/blob/main/FLEDGE.md#53-currencies-in-reporting) in the FLEDGE explainer: + +To summarize this table: + + + +* For both `reportResult/Win()`, and regardless of whether `sellerCurrency` is set, the `bid` reported is the original value from `buyer_bid`, and the `bidCurrency` is the currency from `AdWithBidMetadata.bid_currency`. +* For both `reportResult/Win()`: + * When the `sellerCurrency` is not set, `highestScoringOtherBid` is the original value from `buyer_bid` and `highestScoringOtherBidCurrency` will be set to`'???'` + * When `sellerCurrency` is set, `highestScoringOtherBid` is the converted value from `incomingBidInSellerCurrency` and `highestScoringOtherBidCurrency` is `sellerCurrency` + + +##### Debug Reporting + +`forDebuggingOnly` functionality is expanded to include the following three fields: + + + +* `"${winningBidCurrency}"` is the currency tag of the winning bid. +* `"${highestScoringOtherBidCurrency}"` is the currency tag of `highestScoringOtherBid`. +* `"${topLevelWinningBidCurrency}"` is the currency tag of `highestScoringOtherBid`. + +These, as well as `winningBid`, `highestScoringOtherBid`, and `topLevelWinningBid` will be filled out according to this table (https://github.com/WICG/turtledove/blob/main/FLEDGE.md#53-currencies-in-reporting). To summarize: + + + +* When `sellerCurrency` is unset: + * The bids are all original values from `buyer_bid`. + * The currencies are all `'???'`. +* When `sellerCurrency` is set: + * The bids are all converted values from `incomingBidInSellerCurrency`. + * The currencies are all `sellerCurrency`. + * For component auctions, `topLevelWinningBidCurrency` is the `sellerCurrency` of the top-level auction. + + +## AuctionResult + +AuctionResult is unchanged for single-seller auctions. For multi-seller auctions, the bid for the `scoreAd()` result will always be passed back, in the `bid` field. When specified, its currency will go in the new field `bid_currency`. + + +## Server-Orchestrated Multi-Seller Auctions + + +### API Updates + +B&A has added a new field to the `AuctionConfig` called `per_component_seller_config`, which is analogous to `perBuyerConfig`. It maps each component seller to a configuration object. The config has one field, `expected_component_seller_currency`, a string containing a currency code, representing the currency in which the top-level-seller expects the component seller to deliver `AuctionResult`s. + +The name of the seller in the map must match what the component seller sends back in its `AuctionResult.auction_params.component_seller`. + +The top-level-seller is expected to agree upon expected currencies from each of its component sellers beforehand, and this communication is out-of-band from B&A’s perspective. This matches the way that sellers already communicate expected currencies with buyers. + + +#### Validation + +The expected component seller currency field is optional and is validated by the same rule which governs `seller_currency` and `buyer_currency`. + + +### Currency Checking + +As `AuctionResult`s carry bid currencies, each `AuctionResult`’s bid currency is checked against the top-level seller’s specified `expected_currency` for its component seller. [This check operates based on the same rules as the AdWithBid currency checking described above](#currency-checking). + +However, when an AuctionResult is rejected for currency mismatch, no metric is logged; a client-visible error is instead recorded. + + +### Checking `incomingBidInSellerCurrency` + +If the incoming `AuctionResult` (from the component seller) has a bid currency, and this bid currency matches the top-level seller’s seller currency, then if the top level seller’s scoreAd() outputs an `incomingBidInSellerCurrency`, it must equal the `AuctionResult.bid`. Otherwise the bid will be rejected. + +[This check operates based on the same rules as the AdScore incomingBidInSellerCurrency checking described above](#incomingbidinsellercurrency). + + +### Github issues + +Adtechs can file issues and feature requests on [Github](https://github.com/WICG/protected-auction-services-discussion). + + +## Related publications + +Refer to related publications on [Github](https://github.com/privacysandbox/protected-auction-services-docs). + +[0]: https://github.com/akundla-google + diff --git a/bidding_auction_services_chaffing.md b/bidding_auction_services_chaffing.md new file mode 100644 index 0000000..e50c74b --- /dev/null +++ b/bidding_auction_services_chaffing.md @@ -0,0 +1,311 @@ +# Bidding and Auction Services: Chaffing Design +**Author:**
+[Neil Vohra](https://github.com/neilv-g), Google Privacy Sandbox + +## Overview + +This document explains [Bidding and Auction Services](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md)’ (B&A) design for chaffing, a security solution to prevent malicious actors from inferring which buyers a user is associated with by observing network traffic. + +This is achieved by sending out chaff requests, which are ‘fake’ requests sent to buyers that aren’t participating in an auction for a user. + +This document covers: + + + +1. Background on B&A flow +2. The motivations for chaffing +3. Design for chaffing +4. [Estimated effects on cost](?tab=t.0#bookmark=id.hl33mwpzg98z) for adtechs + +Chaffing builds upon the [privacy considerations](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/trusted_services_overview.md#privacy-considerations) and [security goals](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/trusted_services_overview.md#security-goals) that have been previously described in other explainers. Note that all intra-server communication is encrypted, and messages in transit can only be decrypted by B&A servers running in Trusted Execution Environments (TEEs). + +_Note: Chaffing will be enabled by default on production builds and cannot be disabled. For non-production builds, the servers will expose a runtime flag that adtechs can toggle while testing to disable chaffing._ + + +## Background + +The following describes the flow of Protected Auctions with B&A: + + + +1. The client (browser, Android app) sends an encrypted B&A payload ([ProtectedAuctionInput](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L65)), to the seller's ad service. \ + \ +This includes [buyer_input](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L72) for buyers that have interest groups stored on the client. Note that the seller can also specify a set of buyers to be included in encrypted B&A payload (refer [payload optimization)](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding-auction-services-payload-optimization.md#payload-optimization-guide-for-sellers--ssps). +2. The Seller Ad Service sends [SelectAdRequest](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L544) to B&A (Seller Frontend service), The SelectAdRequest includes encrypted B&A payload and AuctionConfig. \ + \ +In the [AuctionConfig](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L722) field, the seller's ad service can provide a [list of buyers](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L569) that may participate in the auction. +3. The Seller Frontend Service (SFE) processes the request and determines the subset of buyers (each of whom are running Buyer Frontend Services (BFE)) to invoke based on the intersection of the following: + 1. The [buyer_input](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L72) in the ProtectedAuctionInput generated by the client + 2. The [buyer_list](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L569) suggested by the Seller Ad Service + 3. Partner buyer domains configured in the SFE’s runtime config. + +For an overview of the B&A architecture, see the B&A high level design diagram [here](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#high-level-design). + +Without chaffing, a malicious actor (be it an SSP acting in bad faith or a network snooper) could monitor an SFE’s network traffic and easily correlate which buyers are invoked for a given SelectAd request. The data in the `buyer_input` is considered highly sensitive information because it represents the user’s interests, thus it must be protected. + +To mitigate this threat, the SFE will send out additional chaff requests. If a network snooper were to observe the outgoing network traffic of the SFE, they would now see an additional `n` outgoing network requests that are indiscernible from the ‘real’ requests. However, buyers will be able to differentiate chaff and ‘real’ requests using a field in the plaintext request. Using this, buyers will know to not spend server resources processing a chaff request, and malicious actors are unable to learn the buyers specified in the client’s ciphertext. + + +## Design + +There are a few different problems to solve for a proper chaffing implementation on B&A: + + + +1. [Seller side] Determine a set of buyers to send chaff requests to +2. [Seller side] Generate chaff requests +3. [Buyer side] Process chaff requests in a manner similar to real requests +4. [Buyer side] Generate chaff responses + +Chaffing will be implemented in two phases: + + + +* Phase 1 will focus on basic security measures by getting an initial chaffing implementation in place on B&A, and that is launched in 2024. +* Phase 2 will revisit some of the solutions from Phase 1 and implement solutions with proper anti-abuse mitigations necessary to make chaffing a more secure solution. Phase 2 is planned to launch in early 2025. + + +## Phase 1 + + +### Ensuring determinism + +Imagine a chaffing solution where the SFE randomly chooses a set of buyers to send chaff requests to. If a client’s ciphertext is run through the SFE 1000 times, a malicious actor can see which buyers are invoked each of the 1000 times, effectively sidestepping chaffing. To prevent this, determinism is introduced in the system by seeding server side random number generators (RNGs). + +The system will seed RNGs with the [generation_id](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L49), a UUID that is generated and stored by clients within the protected auction ciphertext. Unlike the auction config, the protected auction ciphertext is encrypted by the client and cannot be read by the Seller Ad Service. The ciphertext is an OHTTP encapsulated payload that is encrypted using a public key vended by the coordinators, and only attested Protected Auction Servers running within a TEE are given the private keys. Therefore, it is guaranteed that the Seller Ad Service can never view or decipher any of the contents of the ciphertext. The plaintext value for this field will also not be returned to adtechs client side when running an ad auction. + +The [generation_id](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L49) is propagated from the SFE to the BFE today ([1](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L49), [2](https://github.com/privacysandbox/data-plane-shared-libraries/blob/main/src/logger/logger.proto#L52)), so it can be used throughout the system where deterministic behavior is needed. + + +### [SFE] Calculate how many chaff requests to send + +Where + + + +* participating_buyers is the set of buyers who will receive non-chaff requests (i.e. the buyers in the intersection of a) the client ciphertext, b) `buyer_list` in the `auction_config`, and c) the SFE’s runtime config) +* `buyers_in_sfe_runtime_config` is the [set of partner buyers that the seller's SFE is configured to communicate with](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/production/deploy/gcp/terraform/environment/demo/seller/seller.tf#L66) +* `buyers_in_auction_config` is the [list of buyers passed into the SelectAd request](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L569) + +the SFE will use the following bounds for the number of chaff requests to send: + + + +* If `participating_buyers` == 0, the bounds will be: + * [2, `|(buyers_in_sfe_runtime_config ∩ buyers_in_auction_config) - participating_buyers|`]. +* If `participating_buyers` >= 1, the bounds will be: + * [1, `|(buyers_in_sfe_runtime_config ∩ buyers_in_auction_config) - participating_buyers|`]. + +The SFE will choose a random number between these bounds and will always send at least 1 chaff request. The minimum number of requests (real and chaff) the SFE will send is always 2; it must be a non-static lower bound for the number of chaff requests to send. If the lower bound was static, the system would leak information for the case where the SFE sends 0 real requests and the ‘static lower bound’ number of requests. The malicious actor would then know the buyers in the ciphertext have no overlap with `|buyers_in_sfe_runtime_config| ∩ |buyers_in_auction_config|`. + +Please read below for the cost implications of this solution. + + +### [SFE] Identify chaff request candidates + +As mentioned above, chaff request candidates are buyers in the intersection of: + + + +1. Buyers the SFE is configured to send GetBids requests to. +2. Buyers in the `buyer_list` in the `auction_config`. +3. Buyers not in the protected auction ciphertext sent by the client (because the SFE will send real requests to these buyers if possible). + + + + + +**Chaffing in B&A** + +Given those chaff request candidates, the SFE selects candidates by: + + + +1. Taking the hash of the [generation ID](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L49) and use it to seed a random number generator (RNG) +2. Getting the list of buyers chaff requests can be sent to. +3. Shuffling the list, and send chaff requests to the first `n` buyers in the list where `n` is the number of chaff requests it will send. + + +### [SFE] Generate chaff requests + +The SFE will set the [is_chaff](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L262) field in the GetBids request so the BFE can differentiate the request from real traffic. All other fields will be empty, so the chaff request will be padded to make them seem similarly sized to a ‘real’ request. + + +#### Request format + +The format of the [request_ciphertext](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L1040-L1041) field in the GenerateBidsRequest proto will change to accommodate for chaffing. Today, the payloads only consist of the [GenerateBidsRawRequest](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L945) proto bytestrings. Going forward, the format will be as follows: + + + +* 1 byte for request metadata: + * 4 bits for the version number (e.g. 16 possible values) \ +Version number represents the request format, in case the request format needs to be changed in the future. + * 4 bits for the compression algorithm, if any, used (e.g. 16 different values) +* 4 bytes for the payload size +* X bytes for the `GetBidsRawRequest` payload +* Y bytes of padding + + + + + + + + + + + + + + + + + + + + + + + +
+BFE Request Format +
1 byte + 4 bytes + X bytes + Y bytes +
Version +

+number +

Compression +

+algorithm +

Payload +

+size +

GetBidsRequest +

+Payload +

Padding +
4 bits + 4 bits + +
+ + +For Phase 1, the version number and compression algorithm bits will all be zero. The version number will be zero until the request format is changed. Compression before encryption is enabled with B&A. + + +#### Request size + +For Phase 1, lower and upper bounds for chaff request size are hardcoded and will be tightened in Phase 2. The SFE will choose a random number between these bounds. + + +### [BFE] Process chaff requests + +To mimic processing a real request, the BFE will ‘hold’ onto chaff requests before sending back a response so that the response time is noised. Similar to the request, the response size will also be padded to mirror a real response. + + +#### Response format + +The BFE response format will mirror the request format defined above: + + + + + + + + + + + + + + + + + + + + + + + + +
BFE Response Format +
1 byte + 4 bytes + X bytes + Y bytes +
Version +

+number +

Compression +

+algorithm +

Payload +

+size +

GetBidsRequest +

+Payload +

Padding +
4 bits + 4 bits + +
+ + + +## Phase 2 + +Phase 2 will implement dynamic solutions for some of the problems from Phase 1. These new solutions will more closely simulate real-time traffic, making it harder for malicious actors to learn anything about clients. + +**We are seeking ecosystem feedback on following proposals:** + + +### [SFE] Generate chaff requests + +For Phase 2, tighter bounds will be established for chaff `GenerateBids` request sizes from per-buyer historical request data. The SFE will maintain a per-buyer set of the last `n` real outgoing `GenerateBids` request sizes. This will be a moving window of the requests maintained in memory. When the SFE wants to send a particular buyer a chaff request, it can use some statistic (average, random number between the minimum and maximum) over the window to determine the request size. + +The SFE already knows the set of buyers it can invoke on startup (via its runtime config), so there is a bounded set of buyers the SFE would maintain these windows for. On server startup, the SFE will alternate between hardcoded default request sizes to add some randomness while the windows are being populated for each buyer. + + +### [BFE] Process chaff requests + +Having the BFE hold onto a chaff request and generating a chaff response are problems that can be solved using the same solution. Each BFE host can keep its own moving window of the respective metrics (request duration and response size) for the last `n` requests (real and chaff) it has served. When serving a chaff request, it can use some statistic over the window and use those values to serve a chaff request. Given that these windows will be empty on startup, the BFE will default to alternating pairs of hardcoded min/max values for these metrics until the windows have been fully populated. + +This solution is on a host-by-host basis since the windows are localized to the BFE hosts. And because the BFE hosts are behind a global load balancer, a malicious seller has no way to guarantee hitting the same BFE host twice. The solution also very closely follows real-time traffic, also making chaff requests indiscernible from real requests. There is also the option to maintain these windows on a per-seller basis, similar to the previous solution. + + +# Cost estimation + +Although the SFE will determine chaff request candidates by considering all buyers 1) in the `AuctionConfig` and 2) SFE’s runtime config and 3) not in the ciphertext, it is expected that sellers will not overpopulate the buyer list in the `AuctionConfig`. I.e. it is expected that the set of buyers in the `AuctionConfig` will be much less than the set of buyers in the SFE server for the average request. + +This solution is designed to keep B&A operating costs low without compromising on protecting user data. With chaffing, B&A operating costs can be estimated to increase by low single-digit percentages for both sellers and buyers (more so sellers). We are actively seeking ecosystem feedback on the proposals above. + + + +--- + + + +# Preventing replay attacks + +Separately from the Phase 1 and Phase 2 solutions above, B&A servers also need to implement a solution to prevent replay attacks using a client’s ciphertext such that a malicious actor can’t repeatedly run a request through the system to learn more about its contents. However, Chrome and Android clients both have valid use cases where they may need to reuse a previously generated ciphertext blob. This is because ciphertexts are expensive to generate client side, and there are also valid use cases (ex. intermittent network issue resulting in a retry) where an SFE may see a given ciphertext blob more than once. + + +## Only process recently created ciphertexts + +In the `ProtectedAuctionInput`, a new `request_timestamp_ms` field has been added that represents when the ciphertext was created or requested to be created. On Chrome, the client API will be updated to take a timestamp argument (i.e. the time will be set by adtechs), and Chrome will include this timestamp as a part of the encrypted ciphertext sent to the SFE. On Android, this timestamp will come from the device. + +The SFE will examine the timestamp and reject any requests not within ± `n` seconds of the server side time. `n` will be large enough such that it accounts for the client ciphertext reuse use case. + +Note: this solution is not meant to mitigate all potential side channel attacks; it is possible for a malicious actor to replay a request through the system within the `n` second window or even reset the clock time on a server to match the ciphertext creation time. This mitigation is meant to be a security hardening measure. That being said, this attack vector of repeatedly resetting the system time for every request is not scalable, especially when running on public cloud platforms. + + +## Fixed Size Responses + +One of the potential mitigations against the attack of replayed requests is to use fixed size responses for all server to server communication. Even if the request payload is replayed, the response payload will be padded such that it is always the same size to ensure no leak of bits. Fixed size responses is a mitigation still being explored which we are seeking ecosystem feedback on. + +Note: For padding, fixed bucket sizes would be used instead of exponential bucket sizes to reduce impact on payload sizes and network costs for SSPs and DSPs, so the system would use another bucketing scheme. For server to server communication in a private network within the same availability zone, there is no network cost incurred. diff --git a/bidding_auction_services_gcp_guide.md b/bidding_auction_services_gcp_guide.md new file mode 100644 index 0000000..3585ccf --- /dev/null +++ b/bidding_auction_services_gcp_guide.md @@ -0,0 +1,372 @@ +> FLEDGE has been renamed to Protected Audience API. To learn more about the name change, see the [blog post](https://privacysandbox.com/intl/en_us/news/protected-audience-api-our-new-name-for-fledge) + +**Authors:**
+[Daniel Kocoj][58], Google Privacy Sandbox + +# Bidding and Auction services GCP cloud support and deployment guide + +## Cloud Architecture for Protected Audience Bidding and Auction services + +This document proposes a cloud architecture for Protected Audience Bidding and Auction services on Google Cloud Platform (GCP). The goal of this document is to enable server operators to gain familiarity with the methods of and requirements for running Bidding and Auction services in GCP. This document will be updated as more features are added and the system evolves. + +To learn more about Protected Audience services and Bidding and Auction services, read the following: + + - [Protected Audiences Services Overview][1] + - [Bidding and Auction Services High Level Design and API][2] + - [Bidding and Auction Services System Design][3] + +To compare the GCP implementation to the AWS implementation, refer to the [Bidding and Auction services AWS cloud support and deployment guide][4]. + +The Protected Audience Bidding and Auction services implementations on GCP and AWS are functionally similar. One key difference is that while AWS uses an internal load balancer to route traffic to the Bidding and Auction services, GCP uses a service mesh. Another key difference is that the GCP packaging pipeline creates a Docker image, while the AWS packaging pipeline creates an Amazon Machine Image. + +### Availability timeline + +The Protected Audience Bidding and Auction services will be open-sourced in June 2023. In addition to the code for each service, we will open source Terraform configurations and scripts that allow ad techs to easily deploy the services proposed in this document. + +### Sell-side platform + +The seller operates a SellerFrontEnd service and an Auction service. These services are responsible for orchestrating the auction and communicate with a seller-operated Key/Value service and partner-operated BuyerFrontEnd services. [Learn more about how these services work][5]. + +A [trusted execution environment (TEE)][6] provides a level of assurance for data integrity, data confidentiality, and code integrity using hardware-backed techniques for increased security guarantees for code execution and data protection. The TEE-based SellerFrontEnd service receives its requests from a [Seller Ad service][7]. These requests can be either HTTP or [gRPC][8]. See the [Envoy component][9] to learn how HTTP requests are translated to gRPC. Then network requests are sent to all configured buyer front ends and Key/Value services. The following diagram provides an overall view of the system. + +
+ Seller GCP Architecture +
Figure 1. Seller GCP Architecture
+
+ + +### Demand-side platform + +A buy-side ad tech is responsible for operating a BuyerFrontEnd service, a Bidding service, and a Key/Value service. A request begins with a gRPC message from a SellerFrontEnd service. The major subsequent steps include fetching data from the buyer-operated Key/Value service and generating bids for the ads that are present in the request. [Learn more about the demand-side platform system][10]. The following diagram provides an overall view of the system. + +
+ Buyer GCP Architecture +
Figure 2. Buyer GCP Architecture
+
+ + +### Component overview + +#### Cloud region + +The Bidding and Auction services are regional services, where a ‘cloud region' refers to a particular geographical location as [defined by GCP][11]. We'll open source Terraform configurations to support deployment in **multiple regions in different geographies**. Ad techs can deploy services in any region supported by GCP. The Terraform configurations include parameters that an ad tech can update before deploying to a different cloud region. Note that even though the services are regional, the Terraform configurations show how to make use of global load balancers and service meshes to achieve multi-region coverage. + +#### IAM Policies + +[Identity and Access Management][12] (IAM) is used to securely control who has access to your GCP resources. You can use IAM to create and manage users, groups, and permissions. You can also use IAM to audit access to your resources. + +The default Terraform configuration requires a [service account][13] for each GCE instance. The service account is bound to an IAM role, which the GCE instance inherits. + +#### VPC + +A [virtual private cloud][14] (VPC) is an isolated resource within a public cloud. Sellers and buyers should start with 1 global VPC. A VPC is critical for security purposes and provides configurable firewall rules (which can be found in the Terraform modules). + +#### Subnet + +Each buyer and seller should have 1 private subnet per region. By hosting most components in the private subnet, the service has extra protection from internet traffic. However, the buyer and seller's load balancers must face the public internet. Like VPCs, subnets rely on firewall rules for network security. + +#### Network policies + +[Firewall rules][15] are used to control which types of traffic flow to which ports in the VPC. Key security group rules allow for egress and ingress traffic to flow from the load balancers to GCE Instances, and GCE instances to send network requests to external services. + +#### Managed instance group + +Each region uses a [managed instance group][16] to host its GCE instances. The managed instance group uses an instance template to create or destroy instances based on health checks and autoscaling requirements. Autoscaling is based on CPU utilization and is configurable in the terraform variables, including specifying a minimum and maximum number of instances per instance group. Health checks are configured such that managed instance groups will detect mal-performing VM instances and [auto-heal][17]. + +#### GCP Backend Service + +A [GCP Backend Service][18] is a global resource consisting of all regional managed instance groups for a particular service. Backend services work in tandem with load balancers and are responsible for routing a request to the correct instance. + +#### Load balancer + +The buyer and seller frontend services both use a [Global External Load Balancer][19]. + +The SellerFrontEnd load balancer accepts both HTTP and gRPC traffic over TLS, while the BuyerFrontEnd load balancer accepts gRPC traffic only. Both services' load balancers accept internet traffic. Load balancers terminate TLS, then start a new TLS session with the GCE instance provided by the load balancer's GCP Backend Service. Note that root certificates for the Load Balancer and Backend Service are not verified. + +#### Service mesh + +The SellerFrontEnd and BuyerFrontEnd both communicate with the Auction and Bidding services, respectively, via the [Traffic Director gRPC proxyless service mesh][20]. This bypasses any need for an internal load balancer and saves on dedicated load balancer costs. Specifically, the SellerFrontEnd and BuyerFrontEnd use gRPC's built-in [xDS capabilities][21] to query the Traffic Director control plane service and find routes to available [backend services][22]. Each front end relies on the Traffic Director to distribute its requests to the appropriate backend service, based on utilization and regional availability. While the service mesh itself is a global resource, it automatically routes requests based on region. Because the requests are internal to the VPC and subnet, they are sent as plaintext gRPC. + +#### GCE Host + +The [GCE][23] Host is also known as an ‘instance.' The instance runs in a [Confidential Computing Space][24], which is created via a Docker image containing the service binary and startup scripts. The instance's open ports are identical to those exposed by the Docker image and are the only ports on which incoming connections are supported; connections initiated from inside the Confidential Space workload support both ingress and egress. Individual GCE instances belong to a [Managed Instance Group][25]. + +##### Confidential Space image version +In order to support incoming connections, the minimum Confidential Space image version is [`230600`][60] under the `confidential-space-images` project and in the `confidential-space` and `confidential-space-debug` families. + +##### Envoy + +Inside the seller's front-end service GCE Host Confidential Space, we provide an instance of the open source [Envoy proxy][26]. This is solely used to convert HTTP traffic to gRPC that the SellerFrontEnd service can consume. Envoy terminates TLS and then forwards the plaintext request to the seller's port. The envoy configuration is included in the TEE and can only be modified in a limited way by the operator (such as by providing TLS keys and certificates). + +##### DNS + +[Cloud Domains][27] and [Cloud DNS][28] are supported for use with global external load balancers. The load balancers can automatically use [Google-managed TLS certificates][29] and don't require operators to provide regional domains. The Terraform configurations require a GCP Domain, [DNS Zone][30], and certificate resource ID. + +##### Instance type recommendations + +By default, ad techs are free to use any [instance type][31] that supports Confidential Compute ([N2D or C2D][32] and that meets the ad tech's performance requirements. A recommended starter instance for functionality is _n2d-highcpu-128_. Take note of the [regional availability limitations][33] for Confidential Computing. + +#### Cloud Logging +The GCE host uses the `tee-container-log-redirect` [metadata variable][59] to redirect all stdout and stderr output to [Cloud Logging][61] in both production and debugging environments. This allows service operators to use [Logs Explorer][62] to view logs across all of their services. The production Bidding and Auction service binaries will be built with `VLOG=0`, so only limited information will be logged compared to debugging binaries (which can be built with any `VLOG` level). + +#### Secret Manager + +[GCP Secret Manager][34] is a fully managed service that makes it easy for you to store and retrieve configuration data (including secrets, such as TLS information, and runtime flags, such as ports) in a central location. The default Terraform configurations store the servers' runtime flags in Secret Manager and fetch the flags on server startup. An ad tech can modify the secret and restart the server for the new flag value to take effect. + +#### Cloud Storage + +[Google Cloud Storage][35] (GCS, a cloud object storage service) [buckets][36] are used to store the ad tech's proprietary code modules that are required for Bidding and Auction services. The Bidding and Auction services communicate with GCS via a [Private Service Connect][37] endpoint to fetch the code modules. Each bucket must be configured to allow READ access to the GCE host IAM role. Due to relaxed code module fetch latency requirements, the ad tech can host its code in a GCS bucket in a single region if desired. + +### Terraform-based Service Configuration + +We will provide [Terraform][38] configurations for the Bidding and Auction services. Terraform is used to describe the cloud resource composition (via [Infrastructure as Code][39] that is required for a fully-functional bidding and auction system and the provided Terraform configurations can be modified by the ad tech with no limitations. In fact, the Bidding and Auction services are [configured via Terraform][40], so it is expected that the ad tech interacts with Terraform throughout the [deployment process][41]. + +## Guide: Package, deploy, and run a service + +_This section documents the packaging and deployment process for the Bidding and Auction services. The goal is for technical users to gain an understanding of how to deploy a functioning cloud environment with the service(s) of their choice. After the Bidding and Auction services code is open sourced, much more detail will be added to this section so that it can serve as a complete how-to guide_. + +### Overview + +In order to create a functioning service in GCP, there are two major steps: + +1. Packaging: The creation of the Docker image containing the service's code. +1. Deployment: Running Terraform to bring up the individual cloud components, including the image from step 1. + +### Step 0: Prerequisites + +#### Local Setup + +Use a Linux-based operating system to follow these instructions. Other systems have not been tested. + +1. [Install][42] [`git`][42]. +2. Download the source code from the [Github repository][63]. +3. Run + +`git submodule update --init`. + +This command and all suggested commands in this document should be run from the project root directory. + +4. Install [Docker][44], which is required to: + 1. Build the code. _NOTE: the code relies on the [Bazel][43] build system which is included in the default Docker images used by the build scripts._ + 2. Build the production images with attestable hashes. + 3. Run tests and binaries locally. + +To verify that Docker is installed and runs, try building the code using one of the tools installed by Docker. Running builders/tools/bazel-debian info workspace should return an output of the Bazel workspace location. Make sure that bazel info workspace and bazel-debian info workspace have different outputs, or your local builds will conflict with the packaging toolchain. + +5. Create a symbolic link that points to your python3 installation from /opt/bin/python3. +6. Install [gcloud][45] and [initialize your environment][46]. + +#### Local testing + +After installing the prerequisites, you should be able to test the server. To run the server locally: + +1. Run `builders/tools/bazel-debian build` to build each server. +2. Start the server with the artifact returned by Bazel. +3. Test the server following [these steps][47]. +4. Optional: Run the built binary with the `--helpfull` flag to inspect the required flags. + +**Startup Scripts** + +The scripts in `tools/debug/start_*` contain example startup commands and can be run directly. See the [README][50] for more detail. + +#### GCP project setup + +1. Create a billable [project][48]. This may belong to an organization but can also be standalone. All work will proceed within a single project. +2. Register a domain name via [Cloud Domains][27]. Both seller and buyer services require a dedicated domain. +3. In your project, create an [Artifact Registry][51] repository, then [authenticate your account][52]. You will build and upload the docker images used for testing. +4. [Install Terraform](#step-21-terraform-setup) and follow the remainder of the [project setup using Terraform][49]. + +### Step 1: Packaging + +#### Step 1.1: Configure a test build (Optional) + +The file `config.bzl` presents a flag for non_prod (non-attestable) builds, `non_prod_build`. You may modify the value of the `GLOG_v` key to increase your log level for more verbose logs. + +To build a seller front end service, you may want to modify the `envoy.yaml` configuration file to expose whichever ports you need via the `socket_address` fields. The `gRPC_cluster` port must match the port passed via the _`_PORT`_ flag. + +#### Step 1.2: Building the GCP Confidential Space Docker Image + +_Buy-side ad techs only need to deploy the Buyer Front-End service and the Bidding service in production, while sell-side ad techs only need to deploy the Seller Front-end Server and the Auction Server. However, when testing, ad techs may want to deploy all of the servers to better understand the message flows and structures_. + +To deploy to GCP for testing, we suggest building a docker image for each service. A script to do so can be found at: + +`production/packaging/build_and_test_all_in_docker` + +This script takes flags to specify which service and which region to build. For example: + +``` +production/packaging/build_and_test_all_in_docker --service-path _service --instance gcp --platform gcp --gcp-image-tag --gcp-image-repo -docker.pkg.dev// --build-flavor --no-tests --no-precommit +``` +> **Note**: +> - Switch `prod` to `non_prod` for a debugging build that turns on all vlog. +> - `` must match `environment` in the terraform deployment (see Step 2). + + +The script uploads the service (configured via the `service-path` flag) docker image (tagged with the `gcp-image-tag` flag) to the Artifact Registry repository provided by the `gcp-image-repo` flag. The GCE managed instance group template Terraform resources then take as input an image path, which you can provide via a string of the following format: `/:`. + +### Step 2: Deployment + +#### Step 2.1: Terraform setup + +Install Terraform, following [these instructions][54]. + +##### Terraform layout + +The Terraform lies across three main folders within the `production/deploy/gcp/terraform` directory: + +``` +. +├── environment +│ └── demo +├── modules +│ ├── buyer +│ └── seller +└── services + ├── autoscaling + ├── load_balancing + ├── networking + └── security +``` + +##### Terraform Services directory + +This directory contains all of the individual components of a full stack: networking, load balancing, etc.. + +##### Terraform Modules directory + +This directory contains the seller and buyer modules, which compose the objects found in `services/` and apply defaults. Take note of the variable descriptions and defaults. + +##### Terraform Environment directory + +This directory contains example setups of sellers and buyers. Subdirectories of the `environment` directory (such as `setup_1`) are where you should run `terraform apply`. As an ad tech, this is where you write (or reuse) _`.tf`_ files. Review `setup_1/multi-region.tf` as an example. This file contains all of the ad tech-specific details such as runtime flags, region, and domain addresses. The Terraform variable descriptions in the buyer and seller `service_vars.tf` files contain the complete details of each variable. + +#### Step 2.2: Configure Terraform variables + +For recommended configurations, please see [here][65]. + +Terraform variables are split into two major categories: + +1. Those for the seller (definitions, defaults found in `production/deploy/gcp/terraform/modules/seller/service_vars.tf`). +1. Those for the buyer (definitions, defaults found in `production/deploy/gcp/terraform/modules/buyer/service_vars.tf`). + +The seller module brings up a SellerFrontEnd and Auction service, while the buyer module brings up a BuyerFrontEnd and Bidding service. You can have multiple buyers for every seller. + +#### Step 2.3: Apply Terraform + +After modifying the provided implementations to your desired parameters (including updating all defaults in `modules/*/service_vars.tf`), you should run the following in your desired environment or setup directory to bring the entire system up. + +``` +terraform apply +``` + +You can see the output of each GCE instance via the [serial logging][55] console. + +#### Step 2.4: Upload Code Modules + +Ad techs must use a GCS bucket to host proprietary code modules. The bucket name is required by the Terraform configuration so that a bucket and Private Service Connect Endpoint can be created. Bidding and Auction services automatically fetch updates from the bucket, but it is the ad tech's responsibility to upload their code modules to the bucket. Note that to upload to the bucket, the ad tech must modify the bucket permissions to allow their own proprietary endpoints WRITE access. This is most easily done through IAM permissions. See the GCP GCS [permission guide for details][56]. The Terraform configuration allows the VPC's instances READ access to the bucket by default. + +Instead of using a bucket, during alpha and early beta testing server operators may specify an arbitrary code module endpoint to fetch (via an HTTPS call) in the Terraform configuration. Only a single code module is supported. Later in beta testing, ad techs will be able to host multiple different code modules in a single bucket and specify the module to use at the individual request level. + +#### Step 2.5: Test the Service + +##### Option 1: secure_invoke +Please see the secure_invoke [README][64]. This tool is bundled with the Bidding and Auction services. + +##### Option 2: grpcurl +Use [grpcurl][57] to send a gRPC request to the load balancer address you configured in the Terraform. Requests must be addressed to port 443 so that the load balancer can terminate the TLS connection. When testing locally-running services, disable the `TLS_INGRESS` flags to bypass TLS requirements. + +**Note:** if providing a `sample_request.json`, keep in mind that the `SelectAdRequest` will still require a protected_audience_ciphertext (see `secure_invoke` in Option 1 for instructions on how to generate a ciphertext payload). Additionally, grpcurl will not be able to decrypt the `AuctionResult` ciphertext. + + + + + + + + + + + + + + + + + + + + + + + +

Task

Command

Local service: List grpc endpoints

grpcurl -plaintext localhost: list

Local service: Send query

grpcurl -plaintext -d '@' localhost: privacy_sandbox.bidding_auction_servers./ < sample_request.json

GCP service: List grpc endpoints

grpcurl dns:///:443 list

GCP service: Send query

grpcurl -d '@' dns:///:443 privacy_sandbox.bidding_auction_servers./ < sample_request.json

+ +[1]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md +[2]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md +[3]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md +[4]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_aws_guide.md +[5]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#sell-side-platform-ssp-system +[6]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trusted-execution-environment +[7]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#seller-ad-service +[8]: https://grpc.io/ +[9]: #envoy +[10]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#demand-side-platform-dsp-system +[11]: https://cloud.google.com/compute/docs/regions-zones +[12]: https://cloud.google.com/iam/docs/ +[13]: https://cloud.google.com/compute/docs/access/service-accounts +[14]: https://cloud.google.com/vpc/docs +[15]: https://cloud.google.com/firewall/docs/firewalls +[16]: https://cloud.google.com/compute/docs/instance-groups +[17]: https://cloud.google.com/compute/docs/tutorials/high-availability-autohealing +[18]: https://cloud.google.com/load-balancing/docs/backend-service +[19]: https://cloud.google.com/load-balancing/docs/https +[20]: https://cloud.google.com/traffic-director/docs/proxyless-configure-advanced-traffic-management +[21]: https://grpc.github.io/grpc/core/md_doc_grpc_xds_features.html +[22]: #gcp-backend-service +[23]: https://cloud.google.com/compute/docs +[24]: https://cloud.google.com/compute/confidential-vm/docs/about-cvm +[25]: #managed-instance-group +[26]: https://www.envoyproxy.io/docs.html +[27]: https://cloud.google.com/domains/docs/overview +[28]: https://cloud.google.com/dns +[29]: https://cloud.google.com/load-balancing/docs/ssl-certificates/google-managed-certs +[30]: https://cloud.google.com/dns/docs/zones +[31]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html +[32]: https://cloud.google.com/compute/docs/general-purpose-machines#n2d_machines +[33]: https://cloud.google.com/compute/confidential-vm/docs/locations +[34]: https://cloud.google.com/secret-manager +[35]: https://cloud.google.com/storage +[36]: https://cloud.google.com/storage/docs/creating-buckets +[37]: https://cloud.google.com/vpc/docs/about-accessing-google-apis-endpoints +[38]: https://developer.hashicorp.com/terraform/intro +[39]: https://www.terraform.io/use-cases/infrastructure-as-code +[40]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#service-configuration +[41]: #guide-package-deploy-and-run-a-service +[42]: https://git-scm.com/book/en/v2/Getting-Started-Installing-Git +[43]: https://bazel.build/ +[44]: https://docs.docker.com/get-docker/ +[45]: https://cloud.google.com/sdk/gcloud +[46]: https://cloud.google.com/sdk/docs/install-sdk#initializing_the +[47]: #step-25-test-the-service +[48]: https://cloud.google.com/resource-manager/docs/creating-managing-projects +[49]: https://github.com/privacysandbox/bidding-auction-servers/tree/main/production/deploy/gcp/terraform/environment/demo/project_setup_utils#overview +[50]: https://github.com/privacysandbox/bidding-auction-servers/tree/main/tools/debug#running-servers-locally/ +[51]: https://cloud.google.com/artifact-registry/docs +[52]: https://cloud.google.com/artifact-registry/docs/docker/pushing-and-pulling#cred-helper +[53]: https://cloud.google.com/endpoints/docs/openapi/enable-api +[54]: https://github.com/privacysandbox/fledge-key-value-service/blob/main/docs/deploying_on_aws.md#set-up-terraform +[55]: https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-using-serial-console +[56]: https://cloud.google.com/storage/docs/access-control/iam-permissions +[57]: https://github.com/fullstorydev/grpcurl +[58]: https://github.com/dankocoj-google +[59]: https://cloud.google.com/compute/confidential-vm/docs/reference/cs-options#cs-metadata +[60]: https://cloud.google.com/compute/confidential-vm/docs/work-with-confidential-space-images#open_inbound_ports +[61]: https://cloud.google.com/logging +[62]: https://cloud.google.com/logging/docs/view/logs-explorer-interface +[63]: https://github.com/privacysandbox/bidding-auction-servers +[64]: https://github.com/privacysandbox/bidding-auction-servers/tree/main/tools/secure_invoke +[65]: https://github.com/privacysandbox/bidding-auction-servers/tree/main/production/deploy/gcp/terraform/environment/demo diff --git a/bidding_auction_services_multi_seller_auctions.md b/bidding_auction_services_multi_seller_auctions.md new file mode 100644 index 0000000..b2b11f1 --- /dev/null +++ b/bidding_auction_services_multi_seller_auctions.md @@ -0,0 +1,801 @@ +> FLEDGE has been renamed to Protected Audience API. To learn more about the name change, see the [blog post](https://privacysandbox.com/intl/en_us/news/protected-audience-api-our-new-name-for-fledge) + +**Authors:**
+[Jaspreet Arora][2], Google Privacy Sandbox
+[Priyanka Chatterjee][1], Google Privacy Sandbox + +# Bidding and Auction Services Multi Seller Auctions +This document explains the proposed multi-seller auction flows using [Bidding and Auction services][3]. +We propose following types of multi-seller auctions: +* Device-orchestrated component auction +* Server-orchestrated component auction + +This explainer provides building blocks for ad techs to participate in multi-seller auctions with their +ad management solution. While this explainer does not provide step-by-step instructions for each specific +ad tech’s integration, we aim to provide options to meet your auction use cases. + +_Note: SSP and DSP can operate Bidding and Auction services on different cloud platforms that are supported. +For multi seller auctions, SSPs can operate on different cloud platforms._ + +Refer to this [section][5] for related explainers. You can share feedback on the design by filing a [GitHub issue][4]. + +Additional questions remain open for comment. As we build this technology, we’ll provide more detailed +setup instructions. + +_Note: We will publish reporting designs for multi seller auctions in a different explainer. If you +developed reporting scripts using [Chrome on-device Protected Audience specification][27], that would work +with [Bidding and Auction services][3]._ + +## Key terms and concepts +### Seller +A party running an ad auction, likely to be an ad tech supply-side platform (SSP). + +### Buyer +A party bidding for ad space in an ad auction, likely to be a demand-side platform (DSP), or the advertiser +itself. Ad space buyers own and manage Protected Audience interest groups. + +### Seller’s ad service +A server owned by a seller to orchestrate real-time bidding (RTB) requests and Protected Audience auction +requests to Bidding and Auction services. + +### Unified request +A seller’s single request for the contextual and server-side Protected Audience auction, from their code +on a publisher site. This was described in the Bidding and Auction services high-level design and API +explainer. +* The seller’s ad service makes two sequential requests. + * [Existing flow] The seller sends real-time bidding (RTB) requests to buyers for contextual bids, + then conducts the contextual auction. + * [Server side Protected Audience flow] The seller sends a SelectAd request to SellerFrontEnd service + to start the Protected Audience auction. The request payload includes encrypted Protected Audience data, + contextual signals, buyer_list, seller_origin, per_buyer_timeout and other necessary information. +* The SellerFrontEnd conducts the Protected Audience auction. It returns an encrypted response that + includes either the winning bid, or an encrypted padded response if the contextual bid filters all + Protected Audience bids. For privacy reasons, the seller's ad service will be unable to determine if + the payload has a winning Protected Audience ad or not, so it should also pass the contextual winner + to the client. + +### Contextual auction +In this explainer, we use "Contextual auction" to mean everything that does not involve cross-site data. +This includes the contents of the web page or URL where the ad will appear, which is the traditional +sense of "contextual." But for our purposes, first-party data that the publisher provides about their +user, or Seller-Defined Audience data, are also bundled into what we are calling "contextual". + +### Component auction +An auction conducted by a single seller among a collection of buyers, as part of a multi-seller auction. +The auction winners then participate in a top-level auction. In a unified flow, the top-level auction +includes the winners for the contextual auction and/or the winner for the Protected Audience auction. +Each Component auction follows the single seller unified flow specified in Bidding and Auction Services +Design. + +### Top-level auction +The auction which decides the final winner, conducted after individual seller auctions (component +auctions). + +Note: The top-level auction runner cannot use Protected Audience data to bring its own demand. +Therefore buyers can not participate in the top-level auction. The top-level auction scores winning bids +from component auctions. + +### Multi-seller auction +Auctions with participation from multiple sellers and their partner buyers, where the final winning ad +candidate is decided by a top level auction. + +### Top-level seller +The sell-side ad tech or the publisher conducting the top-level auction. This ad tech specifies the +scoreAd() script and auction config for scoring the winning candidates from Component auctions. + +### Component-level seller +The sell-side ad techs that integrate with a top-level seller and conduct their own Component auction. +The winners from those auctions are scored in the top-level auction. + +## Timeline +We expect to make Multi seller auction support with Bidding and Auction services available for testing +by September 2023. + +## Services overview + +![Architecture diagram](images/unified-contextual-remarketing-bidding-auction-services.png) + +_This type of multi seller auction will be supported for web advertising with Bidding and Auction services._ + +The Bidding and Auction services allow Protected Audience computation to take place on cloud servers +with a [trusted execution environment][6] (TEE), rather than locally on a user's device. Each Bidding +and Auction service is hosted in a virtual machine (VM) within a secure, hardware-based TEE. Ad tech +platforms operate and deploy Protected Audience services on a public cloud. + +### Sell-side services +The sell-side services include the SellerFrontEnd service and the Auction service. The SellerFrontEnd +service orchestrates requests in parallel to buyers by communicating with multiple BuyerFrontEnd +Services. This service also fetches real-time scoring signals from the seller's Key/Value service +required for the auction and calls the [Auction service][7] for Protected Audience auction. The Auction +service will execute the seller-owned code to score the bids from their seller partners. Review a +detailed [overview of the sell-side services][8]. + +### Buy-side services +The buy-side services include the BuyerFrontEnd service and the Bidding service. The BuyerFrontEnd +service communicates with the Buyer-operated Key/Value services to fetch bidding signals and call +the Bidding service to generate bids. The Bidding service will execute the buyer-owned code to +generate the bids for the interest groups. Review a detailed [overview of the buy-side services][9]. + +The seller’s ad service starts the Protected Audience auction on the Auction service by sending a +request to the SellerFrontEnd Service with the encrypted Protected Audience data (from the browser), +and contextual signals in plaintext. Read about the Protected Audience auction [flow using the +Bidding and Auction Services][10]. + +## Design +The [Bidding and Auction services API][11] will be extended to support multi-seller auction flows, +with an updated API. There are two different ways in which a multi-seller auction could make use +of these services, based on where the top-level auction takes place. + +[Device-orchestrated Component auction][13]: Each component auction is orchestrated by the respective +seller in the publisher web page, in the browser. Each component auction can be conducted with +the Bidding and Auction services or on-device. The top-level Protected Audience auction is executed +on the browser. + +_Note: If all component sellers run auctions on-device, the component auction flow should be the same +as described in [Chrome Protected Audience on-device explainer][15]._ + +[Server-orchestrated Component auction][14]: The top-level seller makes one request for an auction, +from the publisher web page, to its ad service, The top-level seller’s ad service orchestrates unified +requests to other sellers for Component auctions that can run on Bidding and Auction services. Then, +the top-level Protected Audience auction is executed in the top-level seller's TEE-based Auction service. + +### Device-orchestrated Component auctions + +![device orchestrated comp auction](images/device-orchestrated-comp-auction1.png) + +#### High-level overview +Each component seller sends a unified request for its specific Component auction. Each sellers' code +calls the browser API to obtain the encrypted Protected Audience request payload. Then, they forward +this payload to their own ad servers as a [unified request][16]. Requests can be sent from the client +in parallel. + +Each component seller decides where its auction takes place. This means sellers can run a Protected +Audience component auction on-device or with the Bidding and Auction services. + + * If a seller decides to run a Protected Audience component auction on-device, their partner buyers + that participate in the auction must generate bids on-device. If a seller decides to run a Protected + Audience component auction in the Protected Audience Auction service, their participating partner + buyers must generate bids server side in the Protected Audience Bidding service. + + * If a seller decides to run a Protected Audience component auction on-device, the seller code in the + publisher web page on browser sends contextual request to seller's ad service, the contextual response + include contextual signals required to build Protected Audience auction config, then the Protected + Audience component auction executes on-device. + +The contextual auction winner and encrypted Protected Audience response from each Component auction are +made available to the browser. The top-level seller conducts the final contextual auction with the +contextual auction winners from the contextual half of the component auctions, and populates the auction +config (scoring signals and auction signals). This can happen on-device or on the top-level seller’s ad +service. Then the top-level seller's ScoreAd() function scores the Protected Audience bids received from +the Protected Audience half of the component auctions. This takes place on the device. + +#### Flow +The flow describes Protected Audience component auction in Bidding and Auction services or on the browser, +followed by top level auction on the browser. + +* Following is the flow if Protected Audience component auction happens server side: + + * Each component-level seller’s code on a publisher’s web page on browser calls respective seller's ad + service for a Component auction with one [unified request][16], as per the [Contextual and Protected Audience + auction flow][17]. If the seller decides to run server side Protected Audience auction in Bidding and Auction services, + then a [unified request][16] is sent; if the seller decides to run Protected Audience auction on the browser, + the contextual request is sent to seller ad service. The requests fan out in parallel from the publisher web page + for each Seller. + * The unified request includes contextual payload and encrypted Protected Audience data. + * The seller's ad service uses the contextual payload to run a contextual auction. + * The seller's ad service forwards the encrypted Protected Audience data and contextual signals to the TEE-based + SellerFrontEnd service to conduct the Protected Audience auction. + * This flow is similar to the Protected Audience auction described in [Bidding and Auction services][10] + with one special handling for which bids are skipped for scoring if corresponding _allow_component_auction_ + field is set to false in the [AdWithBid][19] response from Buyer. + * For the bids that are scored, the _allowComponentAuction_ flag and _modified bid_ field would be returned + by the scoreAd() script. For the winning Protected Audience bid, these will be returned to the client + in the encrypted Protected Audience response. + + * Each component seller's ad service sends its contextual ad winner ad and/or encrypted Protected Audience + response from component auctions back to the browser. + + * If the seller's component Protected Audience auction does not produce any winner, an encrypted, padded + (fake) Protected Audience response is returned from the TEE-based SellerFrontEnd service to the seller's + ad service. + + * The seller code in the publisher web page passes the encrypted Protected Audience response to the browser. + _Note_: Only the browser API can decrypt the response. The client-side design details are documented in the + [Browser API design for Bidding and Auction services integration][20]. + +* Following is the flow if Protected Audience component auction happens on-device: + + * Each component-level seller’s code on a publisher’s web page on browser calls respective seller's ad service + for a contextual auction. This response includes contextual ad winner and contextual signals (that are part + of the auction config) required for component-level Protected Audience auction. + + * Component-level Protected Audience auction happens on-device if the buyers and sellers determine if there is + incremental value for Protected Audience. The highest scored bid and other data from this auction are made + available for the top-level ScoreAd() on-device. See Chrome on-device explainer for details. + See the explainer for [Protected Audience API on browser][21] for more details. + +* The code in a publisher’s page can pick the contextual winner on the browser or sends contextual ad winners + to the top-level seller's ad service to pick the contextual ad winner. + + * Contextual signals (seller signals and auction signals) are part of the top-level Protected Audience auction + config and are required for the top-level Protected Audience auction. + +* Top-level Protected Audience auction scores Protected Audience bids that are made available from component + auctions. The top-level Protected Audience auction happens in the browser. The client-side design details are + documented in the [Browser API design for Bidding and Auction services integration][20]. + + * The browser may fetch scoring signals for top level auction as done today in [top-level Protected Audience + auctions in the browser][21]. However, the top-level seller can decide whether their script requires scoring + signals for the top-level auction. + +#### Sequence diagram + +![device orchestrated comp auction seq](images/device-orchestrated-comp-auction-seq2.png) + +#### Data flow + +![device orchestrated comp auction data flow](images/device-orchestrated-comp-auction-data-flow1.png) + +#### API changes +_Note: The Bidding and Auction services will conduct a component auction or a top-level auction, based on the +input to the [SelectAd RPC][22]_. + +##### AuctionConfig +The following new fields will be added to the plaintext AuctionConfig object sent to the SellerFrontEnd service by the Seller Ad server: + +* _string top_level_seller(optional)_: It’s similar to the seller field passed in the single seller auctions and will signal the + Bidding and Auction services that this is a Component auction. This is passed in the deviceSignals field to the generateBid() and reportResult() code by ad-techs. The component auction output includes other information required by the client for the top-level auction (for example, allowComponentAuction). + +``` +syntax = "proto3"; + +// Plaintext. Passed by the untrusted Seller service for configuring options for // the auction. +message AuctionConfig { + // Existing fields .... + + // Optional, but must be specifed if this is meant to be consumed for + // Component auctions. If this value is not present, the auction will be + // treated as single seller. + string top_level_seller = 9; +} +``` + +##### AuctionResult +The following new fields will be added to the encrypted AuctionResult object, sent in the SelectAdResponse +from the SellerFrontEnd service: + +* _float bid (optional)_: This will be used to provide a modified bid value for the top-level seller + scoring script. The original bid value will be returned in the “bid” field in AuctionResult. This will be + populated from the “bid” field of the component seller’s scoreAd() code as described in the Chrome Protected + Audience explainer. If this is not present, the original bid value will be passed to the top-level scoring + script. + +* _string ad_metadata (optional)_: This will be used to pass the ad's metadata to the top-level seller's scoring function. + This will be populated from the “ad” field of the component seller’s scoreAd() script as described in the Chrome + Protected Audience explainer. If this is not present, an empty string will be passed to the top-level seller + scoring script. + +* _string top_level_seller_: This will contain the value passed by the seller’s ad server in the auction config object. This will be used by the device to validate that this is only used in a top-level auction run by the specified top-level seller. + + +The updated definition for AuctionResult will be as follows: +``` + syntax = "proto3"; + + message AuctionResult { + // Existing fields .... + // Bid price corresponding to an ad. + float bid = 6; + + // Optional name/domain for top-level seller in case this + // is a component auction. + string top_level_seller = 15; + + // Optional. Metadata of the ad, this will be passed to top-level + // seller's scoring function. Represents a serialized string that is + // deserialized to a JSON object before passing to script. If this is + // not present, an empty string will be passed to the top-level + // seller scoring script. + string ad_metadata = 16; + + // Reporting metadata required for top level auction will be updated + // later. + } +``` + +_Note: Additional metadata will be sent back to the client for reporting (URL generation and report URL ping) after +the top-level auction on the browser. The API changes will be published at a later date in a different explainer._ + + +#### Ad Tech Specifications +##### Component Seller Ad Server +The seller's ad server orchestrates the flow to conduct the component auction as follows - +1. Sends a SelectAdRequest to SellerFrontEnd service with a B&A payload ciphertext and an auctionConfig as described in the [API changes][28]. + +_Note: For a component auction, the auctionConfig must contain a `topLevelSeller` field with the domain of the top level seller which will call runAdAuction on the browser._ + +2. Includes the encrypted AuctionResult responses (returned from the SelectAd request in step 1) sent back in the HTTP response body back to the JavaScript Fetch initiated by the browser with "adAuctionHeaders: true". +3. Attaches base64url encoded SHA-256 hash of each encrypted response included in an HTTP response header (Ad-Auction-Result header) back to the [browser][29]. + +_Note: This is required by the browser to ensure that the encrypted AuctionResult response is in fact coming from the correct seller origin. Refer [here][29] for more details._ + +Following is an example of Ad-Auction-Result header: +``` +Ad-Auction-Result: ungWv48Bz-pBQUDeXa4iI7ADYaOWF3qctBD_YfIAFa0=, +9UTB-u-WshX66Xqz5DNCpEK9z-x5oCS5SXvgyeoRB1k= +``` + +#### Integration with Publisher bidding frameworks + +Following are design examples for Publisher bidding frameworks such as [Prebid.js][31] integration with browser APIs for [device-orchestrated component auctions](#device-orchestrated-component-auctions). + +> **Note:** We have received feedback from the ecosystem on how a multi-seller Protected Audience auction with Bidding and Auction Services (B&A) can be integrated with publisher bidding frameworks. To enable implementation, and foster discussion of a new feature, we are presenting one possible way that B&A can work with bidding frameworks. +> +> In this section, we provide an approach for [Prebid.js][31], which is a popular publisher bidding library used by the ecosystem, but a similar approach can be applied to other bidding libraries. Note that we describe a possible way Prebid.js can be updated to work with a multi-seller auction using Bidding and Auction services, but the exact implementation is up to the Prebid.js team. +> +> We are seeking feedback on this integration, and your comments and suggestions on the approach are welcome. To provide feedback, please visit the Protected Auction Services Discussion repository on GitHub. + +##### Background + +[Prebid.js][31] provides integration with on-device component auctions through modules like the [PAAPI module](https://docs.prebid.org/dev-docs/modules/paapi.html), and [topLevelPAAPI](https://docs.prebid.org/dev-docs/modules/topLevelPaapi.html) module. An example flow to demonstrate its working is presented below. + + +##### [On-device component auctions](https://github.com/WICG/turtledove/blob/main/FLEDGE.md#24-scoring-bids-in-component-auctions) with Prebid.js + +> *Note: Prebid integration with browser to support on-device component auctions is completed and available.* + +![alt_text](images/prebid_pa_on_device.png "Prebid fledge flow with on-device API") + +1. Prebid.js calls seller adapter [buildRequests()](https://docs.prebid.org/dev-docs/bidder-adaptor.html#building-the-request) for contextual payload +2. [buildRequests()](https://docs.prebid.org/dev-docs/bidder-adaptor.html#building-the-request) returns ServerRequestObjects +3. Prebid module sends requests to seller ad servers +4. Seller ad server returns contextual bids +5. Prebid.js invokes [interpretResponse()][30] in the seller adapter with serverResponse. +6. [interpretResponse()](https://docs.prebid.org/dev-docs/bidder-adaptor.html#building-the-request:~:text=Browsing%2DTopics%27%2C%20%27%3F1%27%29%3B-,Interpreting%20the%20Response,-The%20interpretResponse%20function) returns [contextual bids and component auctionConfig(s)](https://docs.prebid.org/dev-docs/modules/paapi.html#:~:text=return%20%7Bbids%2C%20fledgeAuctionConfigs%7D%3B) to Prebid +7. Prebid.js module provides component auctionConfig(s) to top-level seller JS +8. Top-level seller JS invokes [runAdAuction()](https://github.com/WICG/turtledove/blob/main/FLEDGE.md#24-scoring-bids-in-component-auctions:~:text=optionally%2C%20a%20boolean%20*/...%2C%0A%7D%3B-,const%20result%20%3D%20await%20navigator.runAdAuction(myAuctionConfig)%3B,-//%20If%20%60result%60%20is) to kick off top-level scoreAd() +9. Top-level seller JS receives winner from top-level auction + +##### Device-orchestrated component auctions with Prebid.js + +The current Prebid modules for Protected Audience auction support can be extended to support the B&A device-orchestrated component auctions API. + +The following section presents a hypothetical integration flow for Prebid.js with the B&A device orchestrated component auctions. + +This concept envisions a possible set of changes to [Prebid.js][31] and seller modules to illustrate how Prebid.js seller modules can use B&A services for Protected Audience component auctions. These changes are an informal example and are not meant to be prescriptive. + +> *Note: The Browser APIs are available for Prebid to integrate with browser to support [device-orchestrated component auctions](#device-orchestrated-component-auctions) for B&A .* + +###### Flow +![alt_text](images/prebid_pa_component_1.png "Prebid Device orchestrated auctions") + +1. Prebid.js calls seller adapter [buildRequests()](https://docs.prebid.org/dev-docs/bidder-adaptor.html#building-the-request) for contextual payload +2. [NEW] Seller Prebid adapter [buildRequests()](https://docs.prebid.org/dev-docs/bidder-adaptor.html#building-the-request) invokes [navigator.getInterestGroupAdAuctionData()](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-1-get-auction-blob-from-browser) to fetch B&A ciphertext (encrypted request payload for B&A auctions) +3. [NEW] Browser returns a **B&A ciphertext** to [buildRequests()](https://docs.prebid.org/dev-docs/bidder-adaptor.html#building-the-request) +4. [NEW] [buildRequests()](https://docs.prebid.org/dev-docs/bidder-adaptor.html#building-the-request) includes the **B&A ciphertext** in the data field of [ServerRequestObjects](https://docs.prebid.org/dev-docs/bidder-adaptor.html#the-output-of-buildrequests-serverrequest-objects) +5. Prebid.js module sends requests to Seller Ad Servers with the **B&A ciphertext** +6. [NEW] Seller ad server calls their B&A for component PA auction +7. [NEW] Seller's B&A returns an encrypted **AuctionResult** +8. [NEW] Seller ad server returns B&A **AuctionResult** and contextual bids to browser +9. Prebid.js calls [interpretResponse](https://docs.prebid.org/dev-docs/bidder-adaptor.html#interpreting-the-response)() in the seller adapter with [serverResponse](https://docs.prebid.org/dev-docs/bidder-adaptor.html#interpreting-the-response:~:text=//%20const%20serverBody%20%3D-,serverResponse,-.body%3B%0A%20%20%20%20//%20const). +10. [NEW] interpretResponse() returns [contextual bids and component auctionConfig(s)](https://docs.prebid.org/dev-docs/modules/paapi.html#:~:text=function-,interpretResponse,-%28resp%2C) with the B&A **AuctionResult** to Prebid.js + +*[1-10 for each seller adapter]* + +11. Prebid.js module provides component auctionConfig(s) to top-level seller JS +12. Top-level seller JScalls runAdAuction() to kick off top level scoreAd() with all component auctionConfig(s) +13. Top-level seller JS receives winner from top level auction + +###### Changes in Prebid.js +[Prebid.js][31] would need to incorporate the following changes in the [paapiForGpt](https://docs.prebid.org/dev-docs/modules/paapi.html#bid-adapter-integration) module - While calling the seller ad server, include the [adAuctionHeader in the request](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-3-get-response-blobs-to-browser) to prepare the browser to look for Ad-Auction-Result HTTP response headers. + +###### Changes in Seller Prebid Module +1. If the fledgeForGpt module is enabled (PAAPI eligibility passed in [ortb2Imp.ext.ae](https://prebid-docs.atre.net/dev-docs/modules/fledgeForGpt.html#adunit-configuration)), invoke [navigator.getInterestGroupAdAuctionData()](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-1-get-auction-blob-from-browser) to fetch the B&A payload ciphertext with the top level seller. +2. Parse the response from [navigator.getInterestGroupAdAuctionData()](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-1-get-auction-blob-from-browser). It contains requestID and request (the B&A payload ciphertext). +3. Save the requestId field for later use in [interpretResponse](https://docs.prebid.org/dev-docs/bidder-adaptor.html#interpreting-the-response)(). +4. Include the B&A payload ciphertext in the data field of [ServerRequestObjects](https://docs.prebid.org/dev-docs/bidder-adaptor.html#the-output-of-buildrequests-serverrequest-objects) from [buildRequests()](https://docs.prebid.org/dev-docs/bidder-adaptor.html#building-the-request). +5. Construct and return component auctionConfig objects (per ad slot) in [interpretResponse](https://docs.prebid.org/dev-docs/bidder-adaptor.html#interpreting-the-response)() using the AuctionResult ciphertexts from the server response and requestId field saved in 3. + +###### Changes in Seller Ad Server +> *Note: These changes are not Prebid specific, but required for any seller's ad servers to support device-orchestrated component auctions with B&A services.* + +If a B&A ciphertext is received in the call from Prebid, seller's ad server sends a [SelectAdRequest](https://github.com/privacysandbox/bidding-auction-servers/blob/6830804052778404bdf6758fbd165dd644e91539/api/bidding_auction_servers.proto#L521) to SellerFrontEnd service [as described](#component-seller-ad-server) in the previous section. + +##### Device-orchestrated component auctions with Prebid.js with Browser API optimizations + +> *Note: The following section presents a hypothetical integration flow for Prebid.js with the B&A device orchestrated component auctions with an additional proposed latency optimization - an update in the Browser API \ +getInterestGroupAdAuctionData to accept more than one seller origin and return a map of seller origin to B&A payload ciphertext.* + +###### Flow +![alt_text](images/prebid_pa_component_2.png "Optimized Prebid Multiseller auction flow") + +1. [**NEW**] [Prebid.js](Prebid.js) calls [navigator.getInterestGroupAdAuctionData](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-1-get-auction-blob-from-browser)() with **all partner seller origins** +2. [**NEW**] **Browser** returns **B&A payload ciphertexts (one per seller origin)** to Prebid.js +3. [**NEW**] Prebid.js calls seller adapter [buildRequests()](https://docs.prebid.org/dev-docs/bidder-adaptor.html#building-the-request) with **B&A ciphertext** for this seller +4. [NEW] [buildRequests()](https://docs.prebid.org/dev-docs/bidder-adaptor.html#building-the-request) includes **B&A ciphertext** in the data field of [ServerRequestObjects](https://docs.prebid.org/dev-docs/bidder-adaptor.html#the-output-of-buildrequests-serverrequest-objects) +5. Prebid.js module sends requests to Seller Ad Servers with the **B&A ciphertext** +6. [NEW] Seller ad server calls their B&A for component PA auction +7. [NEW] Seller's B&A returns encrypted **AuctionResult** +8. [NEW] Seller ad server returns B&A **AuctionResult** and contextual bids to browser +9. Prebid.js calls [interpretResponse](https://docs.prebid.org/dev-docs/bidder-adaptor.html#interpreting-the-response)() in the seller adapter with [serverResponse](https://docs.prebid.org/dev-docs/bidder-adaptor.html#interpreting-the-response:~:text=//%20const%20serverBody%20%3D-,serverResponse,-.body%3B%0A%20%20%20%20//%20const). +10. [NEW] interpretResponse() returns [contextual bids and component auctionConfig(s)](https://docs.prebid.org/dev-docs/modules/paapi.html#:~:text=function-,interpretResponse,-%28resp%2C) with the B&A **AuctionResult** to Prebid.js + +*[3-10 for each seller adapter]* + +11. Prebid.js module provides component auctionConfig(s) to top level seller Js +12. Top level seller Js calls runAdAuction() to kick off top level scoreAd() with all component auctionConfig(s) +13. Top level seller Js receives winner from top level auction + +###### Changes in Chrome +Currently, the [navigator.getInterestGroupAdAuctionData()](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-1-get-auction-blob-from-browser) method accepts a single seller origin and returns a B&A ciphertext for this seller. Chrome will expose API changes which will enable fetching the B&A payload ciphertexts for multiple sellers at once. These API updates will be available in 2025. + +###### Changes in Prebid.js +[Prebid.js][31] would need to accommodate the following changes in the [paapiForGpt](https://docs.prebid.org/dev-docs/modules/paapi.html#bid-adapter-integration) module - + +1. If the fledgeForGpt module is enabled, invoke [navigator.getInterestGroupAdAuctionData()](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-1-get-auction-blob-from-browser) to fetch the B&A payload ciphertexts with **all partner seller origins** and the top level seller. +2. Pass the corresponding seller’s B&A payload ciphertext when invoking the [buildRequests()](https://docs.prebid.org/dev-docs/bidder-adaptor.html#building-the-request) function in the seller module. +3. While calling the seller ad server, include the [adAuctionHeader in the request](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-3-get-response-blobs-to-browser) to prepare the browser to look for Ad-Auction-Result HTTP response headers. + +###### Changes in Seller Prebid Module +1. Parse the response from [navigator.getInterestGroupAdAuctionData()](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-1-get-auction-blob-from-browser). It contains requestID and request (the B&A payload ciphertext). +1. Save the requestId field for later use in [interpretResponse](https://docs.prebid.org/dev-docs/bidder-adaptor.html#interpreting-the-response)(). +2. Include the B&A payload ciphertext in the data field of [ServerRequestObjects](https://docs.prebid.org/dev-docs/bidder-adaptor.html#the-output-of-buildrequests-serverrequest-objects) from [buildRequests()](https://docs.prebid.org/dev-docs/bidder-adaptor.html#building-the-request). +3. Construct and return component auctionConfig objects (per ad slot) in [interpretResponse](https://docs.prebid.org/dev-docs/bidder-adaptor.html#interpreting-the-response)() using the AuctionResult ciphertexts from the server response and requestId field saved in 3. + + +###### Changes in Seller Ad Server +> *Note: These changes are not Prebid specific, but required for any seller's ad servers to support device-orchestrated component auctions with B&A services.* + +If a B&A ciphertext is received in the call from Prebid, seller's ad server sends a [SelectAdRequest](https://github.com/privacysandbox/bidding-auction-servers/blob/6830804052778404bdf6758fbd165dd644e91539/api/bidding_auction_servers.proto#L521) to SellerFrontEnd service [as described](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_multi_seller_auctions.md#component-seller-ad-server) in the previous section. + +#### Open Questions +The following is an open question for the top level Seller: + +In Device-orchestrated Component Auction, the top level auction would score Protected Audience (winning) bids from +Component Auctions. In this case, are real-time scoring signals required to be fetched from the Seller's Key/Value +service? + +### Server-orchestrated component auction + +![server orchestrated comp auction](images/server-orchestrated-comp-auction1.png) + +_This type of multi seller auction will be supported for app and web advertising with Bidding and Auction services._ + +#### High-level overview +The top-level seller code in the publisher web page in the browser would send a [unified request][16] to the seller's +ad service that includes separate contextual request payload for each seller participating in the auction and one +encrypted Protected Audience request payload. The top-level seller’s code in the publisher web page asks the browser +for this encrypted Protected Audience data to include in the unified request. + +The top-level seller's ad service orchestrates the multi-seller auction, by sending contextual and server side Protected +Audience auction requests in parallel to other component sellers' ad services. This orchestration is new work to be done +by the top-level seller; it will not be handled by the open-sourced Bidding and Auction services code that runs inside a +TEE. All sellers, including the top-level seller, execute respective [unified flows][10] for a contextual auction followed +by Protected Audience auction if there is demand for Protected Audience as determined by buyers and the seller. + +The top-level seller's ad service receives the contextual auction winners and encrypted Protected Audience winners from +each Component auction. The top-level seller conducts the final contextual auction on their server, and prepares the +contextual signals for the top-level Protected Audience auction. The top-level seller conducts the top-level Protected +Audience auction in its TEE-based Auction service. + +After the top-level Protected Audience auction, the SellerFrontEnd service will send an encrypted response payload to +the top-level seller's ad service in [SelectAd][22] response. The top-level seller's ad service will be unable to determine +if the encrypted Protected Audience response payload has a winning Protected Audience ad or not. The seller's ad service +would pass both the top-level contextual ad winner and the encrypted Protected Audience response back to the client. + +#### Server-orchestrated flow +* The top-level seller code on a publisher web page sends one request to the top-level seller's ad service for contextual + and Protected Audience auction. The request payload includes: + + * Separate contextual request payloads for each participating seller. + + * One Protected Audience request payload for all component-level sellers. This payload is encrypted and padded by the + browser, so that it’s the same size across all users and auctions. + +* The top-level seller server orchestrates unified requests in parallel for each component-level seller partner. The + unified request payload includes the contextual request payload and the encrypted Protected Audience payload **meant + only for the partner seller**. + + * The top-level seller's ad service calls SellerFrontEnd service to obtain encrypted payload for other sellers; + this request payload also includes a list of partner sellers for the multi-seller auction. + + * The SellerFrontEnd service running in TEE decrypts the encrypted Protected Audience data and creates different + payloads for each seller by adding different random noise to the input and re-encrypting the Protected Audience + data. This ensures different encrypted data, in other words ciphertext is sent to each partner seller. + + * The SellerFrontEnd service returns different encrypted Protected Audience data mapped to the list of sellers back + to the seller's ad service. + +* All sellers (including the top-level seller) execute respective [unified flow][10] in parallel. Each seller conducts a + contextual auction with their respective ad service. If buyers and sellers determine if there is incremental demand + for Protected Audience, the seller conducts a Protected Audience auction in TEE-based Auction service. Within the + [Protected Audience auction flow][10]: + + * Bids are skipped for scoring based on the allow_component_auction field in the [AdWithBid][19] response from Buyer. + + * For scored bids, the allowComponentAuction flag and modified bid would be returned by the ScoreAd() script. For the + winning Protected Audience bid, these will be returned to the client in the encrypted Protected Audience response. + +* Each component-level seller partner returns a contextual bid and / or an encrypted Protected Audience response to the + top-level seller's ad service. + + * _Note: The top-level seller's ad service can not decrypt encrypted Protected Audience responses received from + Component auctions; it can only be decrypted within an attested TEE-based service._ + +* The top-level seller conducts a top-level contextual auction in its seller's ad service. The top-level seller also + generates contextual signals (part of the AuctionConfig) that are required for the top-level Protected Audience auction. + +* The top-level seller conducts the top-level Protected Audience auction incorporating Protected Audience bids in the + TEE-based Auction server. + + * The top-level seller sends all encrypted Protected Audience responses received from Component auctions to TEE-based + SellerFrontEnd service. The TEE-based SellerFrontEnd service decrypts them. If there’s a Protected Audience ad winner + from component auctions, the SellerFrontEnd service sends a request to the Auction service for top-level Protected + Audience auction. If there's no Protected Audience ad winner from component auctions, the SellerFrontEnd service + would return an encrypted padded (fake) response to the top level seller's ad service. + + * The SellerFrontEnd service returns a fixed-size encrypted padded Protected Audience response back to the seller's ad + service. This is true whether or not the Protected Audience auction produced a winning ad. + + * The seller’s ad service won’t be able to determine if the encrypted payload has a winning Protected Audience bid or + not, so it should also pass the contextual winner to the client. + +* The top-level seller returns the final contextual bid and / or Protected Audience winner back to the browser. + +* Seller code in the publisher web page passes an encrypted Protected Audience response to the browser. + + * The seller code in the publisher web page can not decrypt the encrypted Protected Audience response, only the + browser API will be able to decrypt that. + +#### Sequence diagram + +![server orchestrated comp auction seq](images/server-orchestrated-comp-auction-seq2.png) + +#### Data flow + +![server orchestrated comp auction data flow](images/server-orchestrated-comp-auction-data-flow2.png) + +#### API changes + +##### AuctionResult +In a device-orchestrated component auction flow, the auction result contains a top level seller field which the device matches to make sure the component auction is only used by the appropriate top-level seller. In this case, the AuctionResult will also include an AuctionParams object which contains the following fields: + +* _string component_seller_: This field will be used make sure the top level seller is not able to use the result ciphertext of a component seller for another component seller. +* _string ciphertext_generation_id_: This field will be used to make sure the component auction result is only used for the intended Protected Audience Auction initiated on the device and cannot be replayed. + +``` +syntax = "proto3"; + +message AuctionResult { + // Existing fields including auction_config.... + + message AuctionParams { + // The generation id in the request ciphertext. This is to ensure that + // the result from a component auction can only be used in the same + // top level auction as other component auctions conducted from the original + // ciphertext. This is checked in the top level seller SFE and the request + // is rejected in case of mismatch of result generation IDs. + string ciphertext_generation_id = 1; + + // Component Seller domain. + // This field is used to tie the auction result to the specific seller who + // conducted the auction and is used in the top level auction as input + // to the scoreAd function. + string component_seller = 2; + } + + // This is only populated for component auctions. + AuctionParams auction_params = 18; +} + +``` + +##### SelectAdResponse +The SelectAdResponse for Server component Auctions will contain the following additional fields: + +* _string key_id_: Version of the public key used for encryption by component seller SFE. The top-level seller SFE needs to use private keys corresponding to same key_id to decrypt 'component_auctions_results'. + +``` +syntax = "proto3"; + +message SelectAdResponse { + // Existing fields + + // Encrypted AuctionResult from FLEDGE auction. May contain a real candidate + // or chaff, depending on ScoreAd() outcomes. + bytes auction_result_ciphertext = 1; + + // ------------- New fields -------------- + + // Version of the public key used for encryption by component seller SFE. The + // top-level seller SFE needs to use private keys corresponding to same key_id + // to decrypt 'component_auctions_results'. + string key_id = 3; + + } +``` + +##### AuctionConfig +The auction config for component auctions will include the cloud platform information for the top level seller to make sure the results are encrypted with the correct public key which can be decrypted on the top level seller’s SFE. + +``` +syntax = "proto3"; + +message AuctionConfig { + // Existing fields including auction_config.... + + // Optional. specifies the cloud provider for the top-level seller. If this + // field is populated along with top-level seller field, this will be + // treated as a server orchestrated component auction, and the AuctionResult + // object in SelectAdResponse will be encrypted with a public key instead + // of the shared encryption context. + EncryptionCloudPlatform top_level_cloud_platform = 11; +} +``` + +##### SelectAdRequest + +The following new fields will be added to the [SelectAdRequest][22] object to conduct a top-level auction in +the Bidding and Auction service with "Server-Orchestrated Component auction": + + * _repeated ComponentAuctionResult component_auction_results_: For conducting a top-level auction in top-level Seller's Auction service, the + top-level seller can pass in the encrypted responses from component-level Protected Audience auctions. Along with + this, the seller would also have to populate the AuctionConfig for passing the contextual signals for the top-level + auction. The seller can leave the existing [protected_audience_ciphertext][22] field empty; it will be ignored otherwise. + + The updated definition for SelectAdRequest will be as follows: + + ``` + syntax = "proto3"; + + message SelectAdRequest { + // Existing fields .... + + message ComponentAuctionResult { + // Encrypted AuctionResult from a server component auction. + bytes auction_result_ciphertext = 1; + + // Version of the public key used for encryption. The service + // needs to use private keys corresponding to same key_id to decrypt + // 'auction_result'. + string key_id = 2; + } + // List of encrypted SelectAdResponse from component auctions. + // This may contain Protected Audience auction bids from the component level auctions, + // that will be scored by the top level seller's ScoreAd(). + // If this field is populated, this is considered a top-level auction and the other + // protected_audience_ciphertext field (if populated) will be ignored. + repeated ComponentAuctionResult component_auction_results = 5; + } + ``` + +##### GetComponentAuctionCiphers + +A new API endpoint will be added to the SellerFrontEnd service: _GetComponentAuctionCiphertexts_. + +For server-orchestrated Component auctions, this endpoint can be used to get encrypted Protected Audience payloads +for partner component sellers. The API uses the encrypted Protected Audience data sent from the device to create +unique / distinct ciphertexts for the partner sellers. + +``` +syntax = "proto3"; + +// SellerFrontEnd service operated by Seller. +service SellerFrontEnd { + // Returns encrypted Protected Audience request payload for component level sellers for + // component auctions. +rpc GetComponentAuctionCiphertexts(GetComponentAuctionCiphertextsRequest) returns (GetComponentAuctionCiphertextsResponse) { + option (google.api.http) = { + post: "/v1/getComponentAuctionCiphertexts" + body: "*" + }; + +// Request to fetch encrypted Protected Audience data for each seller. +message GetComponentAuctionCiphertextsRequest { + // Encrypted ProtectedAudienceInput from the device. + bytes protected_audience_ciphertext = 1; + + // List of partner sellers that will participate in the server orchestrated + // component auctions. + repeated string component_sellers = 2; + } + +// Returns encrypted Protected Audience data for each seller. The ciphertext +// for each seller is generated such that they are unique. +message GetComponentAuctionCiphertextsResponse { + // Map of sellers passed in request to their encrypted ProtectedAudienceInput. + // Key ID is baked into the payload as done on device. + map seller_component_ciphertexts = 1; +} +``` + +_Note: For event level win reporting, urls will be generated on the server side and sent back to the client for pinging +from the device. The API changes around reporting URL generation will be published at a later date in a different explainer._ + +#### Ad tech Specification + +Following are the specifications for ad tech owned code to support multi seller component auctions. Ad tech's code for +[scoreAd()][24] and [GenerateBid()][25] would work with the following changes. + +_Note: If scoring happens in TEE-based Auction service, bid_metadata is built by Auction service._ + +#### Component-level seller + +##### ScoreAd() + +The seller [scoreAd()][24] code needs to return the allowComponentAuction flag and a bid. + +``` +scoreAd(adMetadata, bid, auctionConfig, trustedScoringSignals, bidMetadata) { + ... + return {... + //Modified bid value to provide to the top-level seller script. + 'bid': modifiedBidValue, + // If this is not true, the bid is rejected in top-level auction. + 'allowComponentAuction': true + }; +} +``` + +#### Top-level seller + +##### ScoreAd() + +The [bid_metadata][26] / browerSignal passed to [ScoreAd()][24] for top-level Protected Audience auction would also include +componentSeller which is the seller for the Component auction. + +#### Buyer + +[GenerateBid()][25] should set the allowComponentAuction flag to true. If this is not present or set to ‘false’, the bid will be +skipped in the Protected Audience component auction. + +``` +generateBid(interestGroup, auctionSignals, perBuyerSignals, trustedBiddingSignals, browserSignals) { + ... + return {... + 'allowComponentAuction': true, + }; +} +``` + +## Related material +* [Bidding and Auction services][3] +* [Bidding and Auction services payload optimization](https://github.com/privacysandbox/fledge-docs/blob/main/bidding-auction-services-payload-optimization.md) +* [Bidding and Auction services system design explainer][18] +* [Bidding and Auction services AWS cloud support and deployment guide](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_aws_guide.md) +* [Bidding and Auction services GCP cloud support and deployment guide](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_gcp_guide.md) +* [Protected Audience services](https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md) +* [Chrome client design for Bidding and Auction services integration](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md) +* [Android - Bidding and Auction services integration high level document](https://developer.android.com/design-for-safety/privacy-sandbox/protected-audience-bidding-and-auction-services) + +[1]: https://github.com/chatterjee-priyanka +[2]: https://github.com/jasarora-google +[3]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md +[4]: https://github.com/privacysandbox/fledge-docs/issues +[5]: #related-material +[6]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trusted-execution-environment +[7]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#auction-service +[8]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#sell-side-platform-ssp-system +[9]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#demand-side-platform-dsp-system +[10]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#flow +[11]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#service-apis +[12]: #api-changes +[13]: #device-orchestrated-component-auctions +[14]: #server-orchestrated-component-auctions +[15]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#24-scoring-bids-in-component-auctions +[16]: #unified-request +[17]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#unified-contextual-and-fledge-auction-flow-with-bidding-and-auction-services +[18]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md +[19]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#adwithbid +[20]: https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md +[21]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#24-scoring-bids-in-component-auctions +[22]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#sellerfrontend-service-and-api-endpoints +[23]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#protectedaudienceinput +[24]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#scoread +[25]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#generatebid +[26]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#arguments +[27]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md#5-event-level-reporting-for-now +[28]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_multi_seller_auctions.md#api-changes +[29]: https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-3-get-response-blobs-to-browser +[30]: https://docs.prebid.org/dev-docs/bidder-adaptor.html#building-the-request:~:text=Browsing%2DTopics%27%2C%20%27%3F1%27%29%3B-,Interpreting%20the%20Response,-The%20interpretResponse%20function +[31]: https://prebid.org/ \ No newline at end of file diff --git a/bidding_auction_services_onboarding_self_serve_guide.md b/bidding_auction_services_onboarding_self_serve_guide.md new file mode 100644 index 0000000..3033c9b --- /dev/null +++ b/bidding_auction_services_onboarding_self_serve_guide.md @@ -0,0 +1,1129 @@ +> FLEDGE has been renamed to Protected Audience API. To learn more about the name change, see the [blog post](https://privacysandbox.com/intl/en_us/news/protected-audience-api-our-new-name-for-fledge) + +**Authors:**
+[Alek Kundla][4], Google Privacy Sandbox
+[Jaspreet Arora][3], Google Privacy Sandbox
+[Daniel Kocoj][2], Google Privacy Sandbox
+[Priyanka Chatterjee][1], Google Privacy Sandbox + +# Bidding and Auction services onboarding and self-serve guide + +This document provides guidance to adtechs to onboard to [Bidding and Auction +services (B&A)][5]. + +You may refer to B&A services timeline and roadmap [here][6]. +You may refer to the [high level architecture and design [here][7]. +You may also refer to [privacy considerations][8], [security goals][9] and the +[trust model][10]. + +If you have any questions, please submit your feedback using the Privacy Sandbox +feedback [form][11] or file a github issue [here][12]. + +Following are the steps that adtechs need to follow to onboard, integrate, deploy, +test, scale and run B&A services in non-production and production environments. + +## Step 1: Enroll with Privacy Sandbox and onboard to B&A +* Refer to the [guide][13] to enroll with Privacy Sandbox and enroll using the form + documented [here][124]. This is a prerequisite for onboarding to B&A. + +* Onboard to B&A (Protected Auction service) and enroll with [Coordinators][37] by filling out this + [form][125]. + + _Note:_ + * _Adtechs need to choose one of the currently [supported cloud platforms][27] to run B&A services._ + * _Set up a cloud account on a preferred [cloud platform][27] that is supported._ + * _Provide the specific information related to their cloud account in the intake form._ + * _Adtechs can onboard more than one cloud account (e.g. AWS account IDs or GCP service accounts) for the + same enrollment site._ + +## Step 2: Setup for ad auctions + +### Seller + + * Refer to [spec for SSP][15]. + * Develop [scoreAd][16]() for Protected Audience auction. + * Develop [reportResult][17]() for event level reporting. + * Setup [seller's key/value service][18]. + * Add support such that the seller's code on the publisher web page calls [browser API][19] + and/or [Android API][20] to fetch encrypted [ProtectedAudienceInput][21]. Then includes + encrypted [ProtectedAudienceInput][21] in the request to the seller's ad server. + * Add support in the [seller's ad server][22] to send [SelectAd][23] requests to + Bidding and Auction services for Protected Audience auctions. + * Refer to [SelectAdRequest][68] and [SelectAdResponse][69] in [B&A service API][24]. + * Add support for [forwarding client metadata][25] in the request to Bidding + and Auction services (SellerFrontEnd). + * Review [logging][26]. + +### Buyer + +Buyers could configure B&A to support Protected Audience (PA) both for Chrome and Android traffic +and/or [Protected App Signals][120] (PAS) for Android only. +Most of the configuration is shared between the two use cases. We will indicate what parts are specific +to one or the other. + + * Refer to [spec for DSP][28] for the PA configuration and integrate with [the PAS specific changes][117] for PAS. + * Develop [generateBid][29]() for PA bidding or [generateBid][118]() for PAS. + * Develop [reportWin][30]() for event level reporting (for both PA and PAS). + * For PA: + * Setup [buyer's Key/Value service][31]. + * If your Key/Value server supports filtering of interest groups, refer to this + section and [metadata forwarding][25]. + * For PAS: + * If you are planning to use use contextual ads retrieval: + * Setup the [buyer's trusted Key/Value service][121] + * If you are planning TEE ads retrieval: + * Setup the [buyer's trusted Ad Retrieval service][122] + * Develop [prepareDataForAdRetrieval][123]() + * Develop the [handleAdsFetchRequest][124][] UDF to retrieve the ads stored in the K/V server + * [Optimize payload][32]. + * Review [logging][26]. + +### Cloud platforms + + * Adtechs need to choose one of the currently [supported cloud platforms][27] to run + B&A services. Refer to the corresponding cloud support explainer for details: + * [AWS support][33] + * Adtechs must set up an [AWS account][34], create IAM users and security + credentials. + * [GCP support][35] + * Create a GCP Project and generate a cloud service account. Refer to the + [GCP project setup][36] section for more details. + +## Step 3: Coordinator integration + +**_Step 1 covers enrollment with Coordinators as part of onboarding to B&A. This step + provides further guidance around coordinator integration._** + +The Coordinators run [key management systems (key services)][43] that provision keys to +Bidding and Auction services running in a [trusted execution environment (TEE)][44] after +service attestation. Integration of Bidding and Auction server workloads with +Coordinators would enable TEE server attestation and allow fetching live +encryption / decryption keys from [public or private key service endpoints][43] in +Bidding and Auction services. + +_Note:_ + * _Refer to [B&A release notes][126] for the prod images allowed by Coordinators + associated with a specific release._ + * _Adtechs can only use production images attested by the key management systems_ + _(Coordinators) in production. This is required during ([Beta testing][38], + [Scale testing][39] and [beyond][40]) with live traffifc._ + * _Adtechs must use [Test mode](#test-mode) with debug (non-prod) images. This would + disable attestation of TEEs._ + * _Without successfully onboarding to the Coordinator, adtechs will not be able to run_ + _attestable services in TEE and therefore will not be able to process production data_ + _using B&A services._ + * _Key management systems are not in the critical path of B&A services. The_ + _cryptographic keys are fetched from the key services in the non critical_ + _path at service startup and periodically every few hours, and cached in-memory_ + _server side._ + * _Sellers can call buyers on a different cloud platform. The SellerFrontEnd_ + _service fetches public keys from all supported cloud platforms, this is_ + _important to encrypt request payloads for BuyerFrontEnd service._ + +**After onboarding to B&A, Coordinators will provide url endpoints of key services to +the adtechs. Adtechs must incorporate those in B&A server configurations. Refer below +for more information.** + +### Test mode + +Test mode supports [cryptographic protection][45] with hardcoded public-private key +pairs, while disabling TEE server attestation. Adtechs can use this mode with debug / non-prod +B&A service images for debugging and internal testing. + +During initial phases of onboarding, this would allow adtechs to test Bidding and Auction +server workloads even before integration with Coordinators. + + * Set TEST_MODE flag to `true` in seller's Bidding and Auction server configurations + ([AWS][46], [GCP][47]) when using debug / non-prod service images. + * _Note: TEST_MODE should be set to `false` in production with prod service images_. + +### Amazon Web Services (AWS) + +An adtech should provide their AWS Account Id to both the Coordinators. + +The Coordinators would create IAM roles. After adtechs provide the AWS account Id, +they would attach that information to the IAM roles and include in an allowlist. +Then the Coordinators would let adtechs know about the IAM roles and that should be included +in the B&A server Terraform configs that fetch cryptographic keys from key management systems. + +Adtechs must set IAM roles information provided by the Coordinators in the following parameters +in buyer or seller server configs for AWS deployment: + * PRIMARY_COORDINATOR_ACCOUNT_IDENTITY + * SECONDARY_COORDINATOR_ACCOUNT_IDENTITY + +### Google Cloud Platform (GCP) + +An adtech should provide IAM service account email to both the Coordinators. + +The Coordinators would create IAM roles. After adtechs provide their service account email, +the Coordinators would attach that information to the IAM roles and include in an allowlist. +Then the Coordinators would let adtechs know about the IAM roles and that should be included +in the B&A server Terraform configs that fetch cryptographic keys from key management systems. + +Adtechs must set IAM roles information provided by the Coordinators in the following parameters +in buyer or seller server configs for GCP deployment: + * PRIMARY_COORDINATOR_ACCOUNT_IDENTITY + * SECONDARY_COORDINATOR_ACCOUNT_IDENTITY + +## Step 4: Build, deploy services + +Follow the steps only for your preferred cloud platform to build and deploy B&A +services. + +### B&A code repository + +Bidding and Auction services code and configurations are open sourced to [Github repo][52]. + +The hashes of Coordinator approved images of B&A services will be published to +the [B&A release page][53]. + +### Prerequisites + +The following prerequisites are required before building and packaging B&A services. + * AWS: Refer to the prerequisite steps [here][54]. + * GCP: Refer to the prerequisite steps [here][55]. + +### Build service images + +To run B&A services locally, refer [here][56]. + +Adtechs must build images of B&A services to generate hashes specific to a +supported cloud platform and then verify that those match the ones published on +the [B&A release page][53]. + * AWS: Refer to the detailed instructions [here][57]. + * GCP: Refer to the detailed instructions [here][58]. + +### Deploy services + +_Note: The configurations set the default parameter values. The values that are_ +_not set must be filled in by adtechs before deployment to the cloud._ + +#### AWS + + * Refer to understand Terraform configuration setup and layout here. + * Refer to the [README][106] for deployment. + * Follow the steps [here][107] to create configurations for different environments + for deployment. + * Seller: To deploy [SellerFrontEnd (SFE)][108] and [Auction][109] server + instances, copy paste [seller/seller.tf][46] and update custom values. + * Buyer: To deploy [BuyerFrontEnd (BFE)][110] and [Bidding][111] server + instances, copy paste [buyer/buyer.tf][87] and update custom values. + * _Note: Adtechs are not required to update the default configurations before_ + _B&A services deployment._ + +#### GCP + + * Refer to understand Terraform configuration setup and layout [here][112]. + * Refer to the [README][113] for deployment. + * Follow the steps [here][114] to create configurations for different environments + for deployment. + * Seller: Deploy [SellerFrontEnd (SFE)][108] and [Auction][109] server instances, + copy paste [seller/seller.tf][47] and update custom values. + * Buyer: Deploy [BuyerFrontEnd (BFE)][110] and [Bidding][111] server instances, + copy paste [buyer/buyer.tf][89] and update custom values. + * _Note: Adtechs are not required to update the [default configurations][115]_ + _before B&A services deployment._ + +## Step 5: Integration with clients + +### Integration with web browser + +Refer to [browser and B&A integration][59] for more details. + +#### Chrome web browser + +Chrome web browser provides a flag `FledgeBiddingAndAuctionKeyURL` that should +be set to the public key endpoint that can serve a public key such that +corresponding private keys of the same version are known to the Bidding and +Auction services. + +For end-to-end functional testing from the browser with B&A services running in +[Test mode][42], the endpoint can be set to the following: + +``` +FledgeBiddingAndAuctionKeyURL/http%3A%2F%2Flocalhost%3A8000%2Fkey +``` + +The above endpoint can be configured to serve a public key similar to the +following: + +``` +$ mkdir -p /tmp/bna && cd /tmp/bna && echo '{ "keys": [{ "id": "40", "key": "87ey8XZPXAd+/+ytKv2GFUWW5j9zdepSJ2G4gebDwyM="}]}' > key && python3 -m http.server 8000 +``` + +When B&A services run in [production][60] and serve live traffic from Chrome browser, +the flag `FledgeBiddingAndAuctionKeyURL` needs to be set to the public key service +endpoint in [key management systems][43] in production, run by Coordinators. Note +the following: + * The public key service endpoint would vary per cloud platform. + * Clients (browser, Android) will configure the public key service endpoints + corresponding to those cloud platforms where the sellers are running B&A + services. This is because clients encrypt [ProtectedAudienceInput][21] data using + public keys prefetched from the endpoint that would be decrypted in + SellerFrontEnd service running in a trusted execution environment, run by the + seller. + * _Note: Chrome will enable multi cloud support by [Beta2][61]._ + +### Integration with Android + +Refer to [Android and B&A integration][62] for more details. This section will be +updated at a later date. + +## Step 6: Test + +### Payload generation tool + +The [secure invoke][63] tool is a payload generation tool that uses hardcoded public +keys to encrypt payloads and then sends requests to TEE-based B&A services. The +corresponding private keys of the same version are hardcoded / configured in B&A +services such that the encrypted payloads can be correctly decrypted. Refer to +the [README][64] for more information about the tool. + * The tool works with B&A services running in [test mode][42]. + * The tool can also work with B&A services when [Coordinators][65] are enabled. + The tool has the ability to use custom live public keys for encryption that + can be fetched from the public key service endpoint of [key management systems][43] + running on a supported cloud platform. Refer [here][67] for more details. + +#### Seller + + * The tool can generate [SelectAdRequest][68] payload for communication with TEE + based SellerFrontEnd if a plaintext request payload is supplied. The payload + will include [ProtectedAudienceInput][21] ciphertext that is encrypted with + hardcoded public keys. + * The tool also has the capability to decrypt the response received from Bidding + and Auction services and print out the out human readable plaintext response. + +#### Buyer + + * The tool can generate [GetBidsRequest][70] payload for communication with TEE based + BuyerFrontEnd if a plaintext request payload is supplied. + * The tool also has the capability to decrypt the response received from TEE + based BuyerFrontEnd and print out the out human readable plaintext response. + +### Local testing + +Adtechs can test the services locally. This step is optional. + +Refer to the [README][71] to understand how B&A services are built and run locally. + * Refer here to [test buyer services][72] (BuyerFrontEnd, Bidding) locally. + * Refer here to [test seller services][73] (SellerFrontEnd, Auction) locally. + +### Functional testing + +It is recommended to do function testing during the initial [onboarding phase][74]. + +For functional testing, B&A services must be enabled in [Test mode][42] so that +Coordinators can disabled. + +There are couple of options for functional testing: + * Option 1: Using the [secure invoke][63] tool. + In this case, B&A services can be tested without real client (web browser, + Android app) integration. B&A services running in [Test mode][42] can receive + encrypted requests generated by the tool and the response from B&A services + can be decrypted by the tool. + + * Option 2: [End to end testing][75] from Chrome browser. + + ### Load testing + + To determine scalability requirements to support desired throughput, adtechs + may run a few load tests. + +It is recommended to do load testing before running services in production, +however this step is optional. + +_NOTE: Buyers can run load tests independently for buyside auction (bid generation)._ +_However, sellers may need to coordinate with at least one partner buyer to conduct_ +_load tests for final auction of bids returned by buyer(s). Alternatively they can_ +_deploy a fake BFE server for these load tests. We recommend setting up fake buyer_ +_services so that they have the same expected range of latencies as with real buyer_ +_services._ + +Steps: + * Refer to the B&A [load testing guide][76], that includes guidance to conduct load + tests along with recommended [load testing tools][77]. + * Gather a few encrypted sample requests using the [secure_invoke][63] tool. + Refer to secure_invoke tool [guide][64] for more information. + * Deploy B&A services on one of the [supported cloud platforms][27]. + * Start with about the largest instance type of each service with which you + plan to run. + * If you don't have a preference initially, start with a VM instance of 32 + virtual CPUs (cores) with high memory. The Auction and Bidding servers + especially require instances with >1 GB memory per vCPU. + * Set Auction, BuyerFrontEnd, Bidding services to the same VM instance type initially. + * You can use the standard memory instances for SellerFrontEnd. + * Bring up one instance for each service. + * Make sure there is an adequate resource quota in the region you’re testing. + +## Step 7: Traffic experiments + +### Non-production traffic experiments + +Non-production traffic experiments can be enabled during the initial adtech +onboarding phase, also known as the Alpha testing phase. + +**Alpha testing phase is a rolling window.** When an adtech onboards, they are +recommended to do the initial setup, integration, service deployment and functional +testing during this phase. + +_Note: The timelines for testing for web and Android apps may be different._ +_However, every adtech that onboards to B&A is recommended to do Alpha testing_ +_before enabling production traffic experiments._ + +### Production traffic experiments + +#### Experiments for web + +Adtechs can participate in B&A's [Beta 1 and Beta 2][38] testing with +**limited stable production traffic** for Protected Audience auctions on the web. + +Chrome browser has enabled [Origin Trial][78] tokens for testing with B&A. Refer +[here][79] to register. + +#### Experiments for Android apps + +This section will be updated at a later date. + +## Step 8: Multi-region support + +Multi-regional replication is recommended for scaling, fault tolerance and +achieving [higher service availability][85] in production. + +### AWS + +It is recommended to be familiar with [AWS regions and availability zones][86]. +Region is a physical location where data centers are clustered. Availability +zone is one or more discrete data centers interconnected with high-bandwidth, +low-latency networking in an AWS Region. + +Sellers can configure multiple regions in [seller][46] deployment configuration. +Buyers can configure multiple regions in [buyer][87] deployment configuration. +Refer to the following for an example: + +``` +locals { + region = ["us-central1", "us-west1"] + environment = "prod" +} +``` + +### GCP + +It is recommended to be familiar with [GCP regions and zones][88]. A region is a +specific geographical location where adtechs can host their resources. Regions +have three or more zones. + +Sellers can configure multiple regions in [seller][47] deployment configuration. +Buyers can configure multiple regions in [buyer][89] deployment configuration. Refer +to the following for an example: + +``` +# Example config provided cloud region us-central1 and "asia-south-1". +# Similarly, more regions can be added. +"us-central1" = { + collector = { + ... + } + backend = { + ... + } + frontend = { + ... + }, +"asia-south1" = { + collector = { + ... + } + backend = { + ... + } + frontend = { + ... + } +} +``` + +## Step 9: Customize load balancing + +Following section provides an overview of cloud infrastructure setup for B&A +and provides scalability and performance recommendations. + +### AWS + +* Adtechs may refer to the [load balancing configuration][90]. + * _Note: Adtechs are not required to update this configuration._ +* The frontend services (SellerFrontEnd, BuyerFrontEnd) are load balanced by + [application load Balancer][91]. +* The backend services (Auction, Bidding) are load balanced by internal load + balancer. + * _As an optimization, frontend and backend services for both seller and buyer_ + _will be configured in [AWS App Mesh][92], this will eliminate the need of internal_ + _load balancers. This optimization will be available by [Scale Testing][39] phase._ +* The default load balancing algorithm used is `least_outstanding_requests`. + +### GCP + +* Adtechs may refer to the [load balancing configuration][93]. + * _Note: Adtechs are not required to update this configuration._ +* The frontend services (SellerFrontEnd, BuyerFrontEnd) are load balanced by + [global external application load balancer][94]. +* The backend services (Auction, SellerFrontEnd) are co-located in a + [service mesh][95] that is facilitated by the [Traffic Director][96]. +* The default locality load balancing algorithm is `ROUND_ROBIN`; this is set + in [load balancing configuration][93]. This controls load balancing to machines + within a specific zone. +* Traffic distribution is based on **load balancing algorithm (policy) and load balancing mode.** + +In a few specific scenarios, the type of request protocol is also a factor for traffic distribution. Following are the protocol types supported by B&A services. + * SellerFrontEnd service can receive external traffic that may be HTTP or gRPC. + Since the service depends on the gRPC framework, HTTP traffic is translated to gRPC by Envoy Proxy. + * The following server to server communication would always be gRPC. + * SellerFrontEnd and Auction services + * SellerFrontEnd and BuyerFrontEnd services + * BuyerFrontEnd and Bidding services + +#### Load balancing modes + +##### Utilization mode + +The recommended load balancing strategy is [utilization mode][97] which aims to +make each “server group” serve a fixed percentage of its capacity (defaults to 80%). +In this context, a “server group” refers to host machines in the same zone within +the same region. Capacity is calculated for each “server group,” but not for +individual machines (the locality load balancing policy distributes traffic to +individual host machines). + +**Caveat** +*In our observations, utilization mode works best for traffic sent via gRPC. +However, for traffic sent via HTTP 2 or HTTP 1.1, utilization mode tends to favor +a specific zone for all load levels and that may in turn result in machines not +in the favorited zone sitting idle and their capacity being wasted.* + +*[As per GCP's documentation, Google Cloud may prefer a specific zone if average utilization of all VMs is less than 10% in the region.][98]* + +**Recommendation** + * _If SFE receives external traffic via gRPC from the seller's ad server, the_ + _recommendation is to configure utilization mode for the global load balancer_ + _for SFE._ + * _If SFE receives external traffic via HTTP, the recommendation is to manually_ + _[specify a single zone][100] for SFE instances in each region. This will reduce_ + _redundancy protection, but will result in even incoming traffic distribution_ + _across hosts. In this case, it is not required to specify a single zone for_ + _Auction service._ + * _Do not attempt to set `max_rate_per_instance` for utilization mode in the_ + _[seller's configuration][47]. This is a no-op._ + +The following is an example for the seller to specify zones in a given region +for the SellerFrontEnd service in the seller's configuration. The `frontend` +zone setting is highlighted. + +``` +# Example config provided for us-central1 region. +"us-central1" = { + ... + frontend = { + machine_type = "n2d-standard-64" + min_replicas = 1 + max_replicas = 2 + zones = ['us-central1-a'] + max_rate_per_instance = null # Null signifies no max. + } +} +``` + +##### Rate Mode + +[Rate mode][101] allows the specification of a `max_rate_per_instance` in requests +per second (RPS), that would control the traffic that each server instance receives. + +This is an alternate mode for load balancing that requires carefully determining +the rate. However, it doesn't lead to zone favoritism for HTTP traffic, that may +be observed with utilization mode. + +**Recommendation** +If the SellerFrontEnd service receives HTTP traffic and the recommendation for +[utilization mode][99] doesn't suffice, the following are the steps to configure rate +mode for the global load balancing distributing traffic to SellerFrontEnd service. + +Adtechs can set the `max_rate_per_instance` per service in the seller's +[configuration][47] for frontend (SellerFrontEnd) service only. In this case, it +would be important to determine the target RPS per SellerFrontEnd instance. + +``` +# Example config provided for us-central1 region. +"us-central1" = { + ... + frontend = { + machine_type = "n2d-standard-64" + min_replicas = 1 + max_replicas = 2 + zones = ['us-central1-a'] + # Note: 1000 is random, not suggested rate. The rate needs to be + # determined. + max_rate_per_instance = 1000 + } +} +``` + +###### Determine optimal maximum rate per virtual machine instance +Refer to the following steps to determine the maximum rate per virtual machine (VM) +instance. + +Adtechs can refer to the [scalability and performance tuning][102] section to +choose an instance size. + * Start with one VM instance each with a few different sizes (types) that you + intend to run with. + * Send a very low amount of traffic, say 30 QPS, through the system. + * Use the metrics dashboards to record P95 and P99 latencies through the load + balancer and through the frontend service. These would be the baseline + latency numbers. + * _Note: Latencies tend to go down when you add more instances, even when_ + _single instances are not overwhelmed or close to the threshold. So P95s_ + _and P99s may in practice be better than these numbers._ + * Ramp up traffic + * If you are using a synchronous load-generation tool like [wrk2][103]: + * Set the desired rate to something high. + * Use the metrics dashboards to observe incoming and outgoing RPS. This + will give you the maximum RPS that can be supported with this setup at + the cost of higher latency. + * If you are using an asynchronous load generation tool like [ghz][104]: + * The goal is to send as much (but not more than) what the servers can + handle. + * Use the metrics dashboards to track responses with status OK (200) + and the P95 latency. If you have a high proportion of non-200 responses + or a P95 latency that is too high, dial back the QPS until requests are + all succeeding. + * Once you have an idea of what the servers can handle successfully at any + latency, dial back the QPS until you see latency numbers you find acceptable. + +_Note: It is not recommended to set this mode for all services in [main][93] load_ +_balancing configuration. This is because the communication between other B&A services_ +_would be gRPC, hence default utilization mode is expected to work better. Additionally,_ +_not all types of services (frontend, backend) would require the same amount of capacity._ + +### Load Shedding + +It is recommended for sellers to devise load shedding mechanisms in the seller's +ad server based on contextual signals, size of encrypted [ProtectedAudienceInput][21] +and other parameters. This can help determine whether to throttle traffic to B&A. + +Note: + * Within B&A, SellerFrontEnd service calls buyers only if there is corresponding + buyer input (or Interest Groups) in [ProtectedAudienceInput][21] and the buyer + is also present in `buyers_list` in `auction_config` in [SelectAdRequest][68] as + suggested by the seller. + * Other advanced throttling and replay attack mitigations are planned but not + incorporated in B&A, refer here for more information. + +## Step 10: Scalability and performance tuning + +For scaling your services, and tuning latency and performance, it is important to +determine the following for every B&A service: + * Optimal type of virtual machine (VM) instance + * Minimum and maximum number of replicas per cloud region + * Multi-regional replication + +### Configuration update + +After determining the values stated above, sellers need to update a few parameters +in [seller][47] deployment configuration. Buyers also need to update the same +parameters in [buyer][89] deployment configuration. + +The parameters are as follows and need to be set for `backend` and `frontend` +services, per cloud region. + * `machine_type` (type of VM instance) + * Refer to the scaling strategies section to determine the instance type for + different services. + * `min_replicas` + * `max_replicas` + +The following parameters are optional, per cloud region: + * Setting `zones` explicitly may be useful in a particular scenario, refer to + [utilization mode][99] for more information. + * Setting `max_rate_per_instance` may be useful when using rate mode for load balancers. + Refer to [rate mode][105] for more information. + +``` +# Example config provided cloud region us-central1. Multi regional +# replication can be supported similarly. +"us-central1" = { + ... + backend = { + machine_type = "n2d-standard-64" + min_replicas = 1 + max_replicas = 5 + zones = null # Null signifies no zone preference. + max_rate_per_instance = null # Null signifies no max. + } + frontend = { + machine_type = "n2d-standard-16" + min_replicas = 1 + max_replicas = 2 + zones = ['us-central1-a'] + max_rate_per_instance = null # Null signifies no max. + }, +"asia-south1" = { + collector = { + ... + } + backend = { + ... + } + frontend = { + ... + } +} +``` + +### Scaling Strategies + +The following scaling and latency strategy will help determine the size and +number of instances required to serve scaled traffic with an acceptable latency. +This would involve the following: + * Vertical scaling + * Horizontal scaling + +#### Vertical scaling + +Selecting a VM instance type (virtual CPUs, memory) to support a reasonable RPS. +We recommend scaling up till the point of diminishing returns in performance +(when the gain in RPS for an acceptable P95 latency is too little to justify the +increase in instance size/cost). + +The instance types can be estimated and fine-tuned with load testing. The formulae +documented below can help ad-techs choose the types of instances you can use to +run the load tests. + +**Recommendation** +*A good rule of thumb is to have smaller frontend instances serving larger backend instances. +Usually the backend services, that is Bidding Service or Auction service, will require more +compute power for the same RPS compared to the frontend services (SellerFrontEnd or BuyerFrontEnd services). +The Auction and Bidding services also require instances with >1 GB memory per vCPU.* + +##### Determine baseline latency + +We need to find the baseline latency of a B&A request for your payload and scripts. +This is the best case latency that is observed when the services are not loaded. +This can be used to determine when the server’s performance starts degrading. + * Send a few individual requests (<=1RPS) to SFE/BFE. + * Note down the overall P95 latency from the Seller/Buyer dashboards on the cloud + service or the load test client. + * If `generateBid()` or `scoreAd()` latency seems too high, this would mean + that your Bidding or Auction service instance needs more compute power. The + recommendation is to use a larger instance type for Bidding or Auction services. + * If buyer or seller key/value service lookup latency seems too high compared + to expected or reported latency metrics, this would mean that the BFE or SFE + services need more network bandwidth for downloading larger responses. + +##### Determine backend instance size + +The goal is to determine the instance type of backend services (Auction, Bidding) +for 'x' RPS. + +To determine the size of instances for your setup, you can start with a target RPS +of 50-200. + * Choose frontend instances of the same type as backend instances. This helps + ensure that frontend instances are not the bottleneck during this test. + * Slowly ramp up load on this setup till the P95 latency is in an acceptable + range. Once the latency suffers too much, ramp down the load till it becomes + acceptable again. This is this setup’s maximum capacity. + * This maximum capacity is usually driven by the size of the backend instances. + We can then jump to the next section for determining the smallest frontend + instances that can support this load. + * Alternatively, if we want to continue scaling till we can achieve the maximum + capacity equal to the RPS target, we can start to scale up or scale down the + server instances. + * If the maximum capacity was less than the required RPS - monitor P95 latency + metrics to determine which server is the bottleneck - probably the Bidding or + Auction server. Try increasing the instance’s size and test again. + * Repeat this process until you either: + * Reach an instance so large that you do not want to run such an instance + regardless of its performance. + * Reach diminishing returns in performance (the gain in RPS for an acceptable + P95 latency is too less for the increase in size/cost). At this point, + we can refer to the horizontal scaling section to increase the number + of instances instead of size. For eg. 2 Bidding service instances for + 1 Auction service instance. + * If the maximum capacity is more than the required RPS - try decreasing the + instance size and test again. Repeat this process until you reach a P95 + latency performance that is not acceptable. + * Note the following: + * `generateBid() `/ `scoreAd()` latency seems too high - This means that your + Bidding/Auction service instance needs more compute power. Try to increase + the size of the instance. + * `generateBid()` / `scoreAd()` latency latency is fine but Bidding / Auction + service latency is high - The requests are waiting in the ROMA queue. Use + a small ROMA queue for best latency performance (<100). + * Throughput seems low - Increase the ROMA queue size if the tolerance for + latency is higher. + +##### Determine frontend instance size for ‘x’ RPS + +The goal is to determine the instance type of frontend services (SellerFrontEnd, BuyerFrontEnd) for 'x' RPS. + +After your backend instance size is fixed for a target RPS, we can determine the +smallest frontend instances that can support this RPS, since smaller B&A frontend +service instances can usually support larger B&A backend service instances. + * Reduce the size of the instance for your frontend. Test again. Repeat this + process until you reach a P95 latency performance that is not acceptable. + * If the latency for any of the following is high, this would imply corresponding + frontend (BFE / SFE) instances need more compute power. Try to increase the + size of the instance. + * Real time signals from key/value service + * Bidding initiated call + * Auction initiated call + +#### Estimate an instance size + +##### Seller + +###### SellerFrontEnd service + +The SellerFrontEnd (SFE) services’ memory requirements scale with the average +number of ads per request, and the average size of `trustedScoringSignals` per ad. + + + + + + + + + + + + + + +
Number of ads per ad requestTarget RPS per instanceSize of `trustedScoringSignals` per adTotal memory required for a SellerFrontEnd instance
ARS~ A * R * S
+ +###### Auction Service + +The Auction service usually requires higher compute power than the SFE to support +the same RPS, since every `scoreAd()` execution happens in isolation per ad. + + + + + + + + + + + + + +
Number of ads per ad requestTarget RPS per instancep95 latency of `scoreAd()` and reporting executionsTotal vCPUs required in an Auction instance
ARL (ms)~ A * R * L / 1000 + + (Total scoring and reporting executions per second) * + (latency for each scoring execution) + + Note: Reporting urls are generated in Auction service after all scoring executions. + The latency overhead is very small. Refer [here][116] for more details. +
+ +###### Determine SFE / Auction instances ratio + +A **single** SFE instance can support a higher RPS with multiple Auction server +instances without any degradation in performance. To figure out how many Auction +service instances can be served by a single SFE service instance, you can run the +following test - + * Deploy your services with instance sizes as determined in the last step. Set + the min Auction service instances to 1 and max number of instances to 2 or 3. + Set the min and max SFE service instances to 1. Use a low utilization number + for the autoscaler (eg. 60%). + * Set up a mock Buyer BFE service (or coordinate with a buyer) to support a high + load (3-5X RPS) with a P95 latency range as expected for live traffic. + * Start loading the service with a small RPS X (eg. 50 or 100 RPS). + * Slowly start to ramp up the load from X to 2X or 3X RPS. + * The autoscaler should automatically bring up more Auction service instances + as required. The P95 latency might degrade while the autoscaler brings up more + instances, but wait for the server performance to stabilize to an acceptable + level of latency. + * Keep increasing the load (and increasing the Bidding service instances) as + long as the overall latency is within the acceptable range. + * Note the number of Auction instances and the RPS that can be handled by a + single SFE service instance. + * This should give you the max RPS that a single SFE and Auction services + combination can serve. + * For example, a single small SFE instance might be able to serve 3X RPS + with 3 Auction service instances. + +##### Buyer + +###### BuyerFrontEnd service + +The BuyerFrontEnd (BFE) services’ memory requirements scale with the average +number of interest groups per request, and the average size of `trustedBiddingSignals` +per interest group. + + + + + + + + + + + + + + +
Number of Interest Groups per buyer per ad requestTarget RPS per instanceSize of `trustedBiddingSignals` per Interest GroupTotal memory required for a BuyerFrontEnd instance (M)
IRS~ I * R * S
+ +###### Bidding service + +The Bidding service usually requires higher compute power than the BFE to support +the same RPS, since every `generateBid()` execution happens in isolation per +interest group. + + + + + + + + + + + + + +
Number of Interest Groups per buyer per ad requestTarget RPS per instancep95 latency of a `generateBid()` executionTotal vCPUs required in a Bidding instance
IRL (ms)~ I * R * L / 1000 + + (Total `generateBid()` executions per second * latency for each + `generateBid()` execution) +
+ +###### Determine BFE / Bidding instances ratio + +A **single** BFE service instance can support a higher RPS with multiple Bidding +service instances without any degradation in performance. To figure out how many +Bidding service instances can be served by a single BFE service instance, you +can run the following test - + * Deploy your services with instance sizes as determined in the last step. + Set the min Bidding service instances to 1 and max number of instances to 2 + or 3. Set the min and max BFE service instances to 1. Use a low utilization + number for the autoscaler (eg. 60%). + * Start loading the service with a small RPS X (eg. 50 or 100 RPS). + * Slowly start to ramp up the load from X to 2X or 3X RPS. + * The autoscaler should automatically bring up more Bidding service instances + as required. The P95 latency might degrade while the autoscaler brings up + more instances, but wait for the server performance to stabilize to an + acceptable level of latency. + * Keep increasing the load (and increasing the Bidding service instances) as + long as the overall latency is within the acceptable range. + * Note the number of Bidding instances and the RPS that can be handled by a + single BFE service instance. + * This should give you the max RPS that a single BFE and Bidding services + combination can serve. + * For example, a single small BFE instance might be able to serve 3X RPS with + 3 Bidding service instances. + +#### Horizontal Scaling + +After determining instance type and per instance RPS, adtechs can add more such +instances to support a higher RPS per cloud region and globally across cloud +regions. The SLO (Service Level Objective) for availability of B&A services run +by adtechs would be decided by them. Depending on the service availability SLO, +servers need to be replicated within a cloud region and in more than one region. + +Once the size of instances and the ratio of frontend to backend services has been +determined using the previous steps, we can add more server instances as required +for supporting a higher RPS by configuring the autoscaler - + +**Minimum instances** - you should bring up at least the number of B&A service +instance combinations required to serve your average QPS. + +**Maximum instances** - This should be set to handle the peak QPS for your setup. + +**Autoscaler threshold** - This is the metric used to determine when the autoscaler +should start adding more instances to handle the load on your setup. Ideally, the +autoscaler should bring up new instances to seamlessly handle any extra load +without performance degradation. Some cloud providers might also overflow some +traffic to other regions while the autoscaler brings up new instances. The threshold +is usually based on CPU utilization and can be determined using the following steps - + + * Set up your services with instance sizes as determined in the last step and + allow the autoscaler to start more instances with a high threshold. We want + the server performance to suffer before the autoscaler kicks in. You will + need an asynchronous load generation tool to overwhelm the servers (like + [ghz][104]) to send more requests than the servers can handle. + * Start loading your servers up to the RPS determined in the last step (X). + * At this point, note the utilization of each service instance. + * Slowly start to ramp up the load from X to 2X RPS. + * Notice where the performance of the instances starts degrading or requests + start getting dropped, and note down the utilization percent of each service + at that point. + * Figure out how long(t) the autoscaler takes to start up a new instance and + recover. Note the utilization of the overwhelmed instances time before the + instances were overwhelmed and/or requests started dropping. + * Use this utilization threshold for the autoscaler to start bringing up new + instances. + +## Step 11: Determine service availability + +Adtechs should determine service availability (uptime) of B&A services they +operate. Higher availability can be achieved with multi regional replication. + +Given B&A are real time services for ad selection, 3 nines uptime or above is +recommended. + +Adtechs would deploy, run and maintain their B&A instances in production. The +availability service level objective (SLO) of the instances need to be monitored +by adtechs. + +## Step 12: Enable debugging and monitor services + +Refer to the [debugging explainer][80] to understand how user consented debugging can +be used in production. + +Refer to the [monitoring explainer][81] to understand how cloud monitoring is +integrated and service [metrics][82] that are exported for monitoring by adtechs. + +## Known issues + +Prewarmed connections have a keep alive timeout, and the host machine brings +down the connections. This means if the services are sitting idle and not serving +traffic for around 20 mins, the first request will result in a failed response. + * B&A plans to handle warming up the connection in the critical path for such + cases. However that is not supported now. + * _Note: Setting up connections in the request path would lead to higher_ + _latency._ + +## Github issues + +Adtechs can file issues and feature requests on [Github][83]. + +## Related publications + +Refer to related publications on [Github][84]. + +[1]: https://github.com/chatterjee-priyanka +[2]: https://github.com/dankocoj-google +[3]: https://github.com/jasarora-google +[4]: https://github.com/akundla-google +[5]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md +[6]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#timeline-and-roadmap +[7]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#high-level-design +[8]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#privacy-considerations +[9]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#security-goals +[10]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trust-model +[11]: https://docs.google.com/forms/d/e/1FAIpQLSePSeywmcwuxLFsttajiv7NOhND1WoYtKgNJYxw_AGR8LR1Dg/viewform +[12]: https://github.com/privacysandbox/fledge-docs/issues +[13]: https://developers.google.com/privacy-sandbox/private-advertising/enrollment +[14]: #enrollwithcoordinators +[15]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#spec-for-ssp +[16]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#scoread +[17]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#reportresult +[18]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#seller-byos-keyvalue-service +[19]: https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md +[20]: https://developer.android.com/design-for-safety/privacy-sandbox/protected-audience-bidding-and-auction-integration +[21]: https://github.com/privacysandbox/bidding-auction-servers/blob/b27547a55f20021eb91e1e61b0d2175b4aee02ea/api/bidding_auction_servers.proto#L58 +[22]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#sellers-ad-service +[23]: https://github.com/privacysandbox/bidding-auction-servers/blob/b27547a55f20021eb91e1e61b0d2175b4aee02ea/api/bidding_auction_servers.proto#L288 +[24]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto +[25]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#metadata-forwarding +[26]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#logging +[27]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#supported-public-cloud-platforms +[28]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#spec-for-dsp +[29]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#generatebid +[30]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#reportwin +[31]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#buyer-byos-keyvalue-service +[32]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding-auction-services-payload-optimization.md +[33]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_aws_guide.md +[34]: https://docs.aws.amazon.com/signin/latest/userguide/introduction-to-iam-user-sign-in-tutorial.html +[35]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_gcp_guide.md +[36]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_gcp_guide.md#gcp-project-setup +[37]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#deployment-by-coordinators +[38]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#beta-testing +[39]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#scale-testing +[40]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#fast-follow +[41]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#alpha-testing +[42]: #testmode +[43]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#key-management-systems +[44]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trusted-execution-environment +[45]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#client--server-and-server--server-communication +[46]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/aws/terraform/environment/demo/seller/seller.tf +[47]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/gcp/terraform/environment/demo/seller/seller.tf +[48]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#enrollment-with-aws-coordinators +[49]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#enrollment-with-gcp-coordinators +[50]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#enroll-with-coordinators +[51]: https://docs.google.com/forms/d/e/1FAIpQLSduotEEI9h_Y8uEvSGdFoL-SqHAD--NVNaX1X1UTBeCeEM-Og/viewform +[52]: https://github.com/privacysandbox/bidding-auction-servers +[53]: https://github.com/privacysandbox/bidding-auction-servers/releases +[54]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_aws_guide.md#step-0-prerequisites +[55]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_gcp_guide.md#step-0-prerequisites +[56]: https://github.com/privacysandbox/bidding-auction-servers/tree/main/tools/debug#running-servers-locally +[57]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_aws_guide.md#step-1-packaging +[58]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_gcp_guide.md#step-1-packaging +[59]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#browser---bidding-and-auction-services-integration +[60]: #productiontrafficexperiments +[61]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#beta-2-february-2024 +[62]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#android---bidding-and-auction-services-integration +[63]: https://github.com/privacysandbox/bidding-auction-servers/tree/main/tools/secure_invoke +[64]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/tools/secure_invoke/README.md +[65]: #step3:enrollwithcoordinators +[66]: #cloudplatforms +[67]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/tools/secure_invoke/README.md#using-custom-keys +[68]: https://github.com/privacysandbox/bidding-auction-servers/blob/b27547a55f20021eb91e1e61b0d2175b4aee02ea/api/bidding_auction_servers.proto#L300 +[69]: https://github.com/privacysandbox/bidding-auction-servers/blob/b27547a55f20021eb91e1e61b0d2175b4aee02ea/api/bidding_auction_servers.proto#L453 +[70]: https://github.com/privacysandbox/bidding-auction-servers/blob/b27547a55f20021eb91e1e61b0d2175b4aee02ea/api/bidding_auction_servers.proto#L491 +[71]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/tools/debug/README.md +[72]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/tools/debug/README.md#test-buyer-stack +[73]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/tools/debug/README.md#test-seller-stack +[74]: #non---productiontrafficexperiments +[75]: #integrationwithwebbrowser +[76]: https://github.com/privacysandbox/bidding-auction-servers/tree/main/tools/load_testing +[77]: https://github.com/privacysandbox/bidding-auction-servers/tree/main/tools/load_testing#recommended-load-testing-tool +[78]: https://chromestatus.com/feature/4649601971257344 +[79]: https://developer.chrome.com/origintrials/#/view_trial/2845149064591310849 +[80]: https://github.com/privacysandbox/fledge-docs/blob/main/debugging_protected_audience_api_services.md +[81]: https://github.com/privacysandbox/fledge-docs/blob/main/monitoring_protected_audience_api_services.md +[82]: https://github.com/privacysandbox/fledge-docs/blob/main/monitoring_protected_audience_api_services.md#proposed-metrics +[83]: https://github.com/WICG/protected-auction-services-discussion +[84]: https://github.com/privacysandbox/protected-auction-services-docs +[85]: #step11:determineserviceavailability +[86]: https://aws.amazon.com/about-aws/global-infrastructure/regions_az/ +[87]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/aws/terraform/environment/demo/buyer/buyer.tf +[88]: https://cloud.google.com/compute/docs/regions-zones +[89]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/gcp/terraform/environment/demo/buyer/buyer.tf +[90]: https://github.com/privacysandbox/bidding-auction-servers/tree/c346ce5a79ad853c622f64ffd5082e3d1a4457d6/production/deploy/aws/terraform/services/load_balancing +[91]: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html +[92]: https://aws.amazon.com/app-mesh/ +[93]: https://github.com/privacysandbox/bidding-auction-servers/blob/b27547a55f20021eb91e1e61b0d2175b4aee02ea/production/deploy/gcp/terraform/services/load_balancing/main.tf +[94]: https://cloud.google.com/load-balancing/docs/https +[95]: https://en.wikipedia.org/wiki/Service_mesh +[96]: https://cloud.google.com/traffic-director +[97]: https://cloud.google.com/load-balancing/docs/backend-service#utilization_balancing_mode +[98]: https://cloud.google.com/load-balancing/docs/backend-service#:~:text=If%20the%20average,the%20load%20balancer +[99]: #utilizationmode +[100]: https://github.com/privacysandbox/bidding-auction-servers/blob/9a17ea6e12bdd0720c1a6ab3e4b4932ebd66621d/production/deploy/gcp/terraform/environment/demo/seller/seller.tf#L152 +[101]: https://cloud.google.com/load-balancing/docs/backend-service#rate_balancing_mode +[102]:#step10:scalabilityandperformancetuning +[103]: https://github.com/privacysandbox/bidding-auction-servers/tree/main/tools/load_testing#wrk2 +[104]: https://ghz.sh/ +[105]: #ratemode +[106]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/aws/terraform/environment/demo/README.md +[107]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/aws/terraform/environment/demo/README.md#using-the-demo-configuration +[108]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#sellerfrontend-service +[109]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#auction-service +[110]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#buyerfrontend-service +[111]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#bidding-service +[112]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_gcp_guide.md#step-2-deployment +[113]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/gcp/terraform/environment/demo/README.md +[114]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/gcp/terraform/environment/demo/README.md#using-the-demo-configuration +[115]: https://github.com/privacysandbox/bidding-auction-servers/tree/b27547a55f20021eb91e1e61b0d2175b4aee02ea/production/deploy/gcp/terraform/services +[116]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_event_level_reporting.md#rationale-for-the-design-choices +[117]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_protected_app_signals.md#buyer-ba-services +[118]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_protected_app_signals.md#preparedataforadretrieval-udf +[119]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_protected_app_signals.md#generatebid-udf +[120]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_protected_app_signals.md +[121]: https://github.com/privacysandbox/protected-auction-key-value-service/blob/main/docs/tee_kv_server_overview.md +[122]: https://github.com/privacysandbox/protected-auction-key-value-service/blob/main/docs/ad_retrieval_overview.md +[123]: https://github.com/privacysandbox/protected-auction-key-value-service/blob/main/docs/protected_app_signals/ad_retrieval_overview.md#udf-api +[124]: https://developers.google.com/privacy-sandbox/private-advertising/enrollment#how_to_enroll +[125]: https://docs.google.com/forms/d/e/1FAIpQLSduotEEI9h_Y8uEvSGdFoL-SqHAD--NVNaX1X1UTBeCeEM-Og/viewform +[126]: https://github.com/privacysandbox/bidding-auction-servers/releases diff --git a/bidding_auction_services_protected_app_signals.md b/bidding_auction_services_protected_app_signals.md new file mode 100644 index 0000000..2c2aab0 --- /dev/null +++ b/bidding_auction_services_protected_app_signals.md @@ -0,0 +1,793 @@ +# Bidding and Auction Services: Protected App Signals + +**Authors:**
+[Salman Malik][1], Google Privacy Sandbox
+[Priyanka Chatterjee][2], Google Privacy Sandbox
+Trenton Starkey, Google Privacy Sandbox
+ +## Overview + +This document explains [Bidding and Auction Services](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_system_design.md#overview) (B&A) design for ad targeting with [Protected App Signals](https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals). + +The [Bidding and Auction Services](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md) (B&A) allows **Protected Auctions** to take place in [trusted execution environment](https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trusted-execution-environment) (TEE) based servers on a [supported cloud platform](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#supported-public-cloud-platforms). Protected Auction refers to ad selection and scoring of bids in TEE based services for ad targeting. Protected Auction includes support for Protected Audience and Protected App Signals data flows. + +Protected App Signals data is curated on Android devices, refer [here](https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals#curate-protected) for more information. To conduct a Protected Auction utilizing Protected App Signals, seller SDKs collect encrypted Protected Auction data using the [getAdSelectionData](https://developer.android.com/design-for-safety/privacy-sandbox/protected-audience-bidding-and-auction-services#run-auction) API. Encrypted `ProtectedAuctionInput` [[1](https://github.com/privacysandbox/bidding-auction-servers/blob/06bf6ed1bd98916d1a9b6887518936655b4d537b/api/bidding_auction_servers.proto#L448-L449), [2](https://github.com/privacysandbox/bidding-auction-servers/blob/06bf6ed1bd98916d1a9b6887518936655b4d537b/api/bidding_auction_servers.proto#L53-L58)] comprises Protected App Signals. The whole payload, including the Protected App Signals data, is encrypted on the device and can only be decrypted within the TEE. It is important to note that the encrypted `ProtectedAuctionInput` (ciphertext) may also encompass `ProtectedAudience` data. + +Seller's SDK will send the payload and other first party ad request data to their own ad server, which then sends a [SelectAdRequest](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L300-L304) to B&A and includes the encrypted payload and the auction configuration. Upon receiving the [SelectAdRequest](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L300-L304), B&A framework will facilitate ad selection for the participating buyers. During ad selection, buyer custom logic (UDFs / user defined functions) will do [ad retrieval][3] and compute bids using Protected App Signals and other published provided data. The ad and bid from the buyer will then be sent to the seller for final scoring and for the seller to select a winner. Note that Protected App Signals and Protected Audience data flows will run in parallel but independent to select ads and compute bids. Then the seller would get bids for ads using Protected App Signals and Protected Audience flows that will be scored together to decide on a winning ad. The winning ad will be encrypted and then decrypted on the device with the results being returned to the caller for rending. + + +### B&A Self-Serve Guide + +Refer to B&A's [self-serve guide](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_onboarding_self_serve_guide.md) for more information about onboarding to B&A services, specifications for adtechs, server infrastructure, scaling and performance tuning recommendations. + + +## Design + +B&A supports two mutually exclusive modes for Protected App Signals Ad +Selection: + +1. Ad selection from ads that are retrieved from a TEE based Ad Retrieval + Service. +1. Ad selection from ads passed via the contextual path (RTB response) to SFE. + +### Auction With TEE Retrieved Ads + +Following is an overview of Protected App Signals architecture with B&A with +Top-K Ads retrieved via a TEE based Ad Retrieval Service. + +
+ Protected App Signals Architecture Diagram +
Figure 1.Protected App Signals Architecture (with retrieval service)
+


+ + +Following is the flow: + +1. [Android client] + * Using Android's privacy preserving ad-relevance APIs (PPAPI), Android client collects Protected App Signals relevant to the user. Refer [here](https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals#curate-protected) for more information. + * Seller's Android SDK calls PPAPI to get `ProtectedAuctionInput` ciphertext data including the [encoded](https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals#on-device-encoding) representation of the Protected App Signals. + * _Note: The payload is compressed, encrypted and padded, refer [here](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#client--seller-ad-service-communication) for more information about client <> server communication._ + * Seller's Android SDK sends an ad request to the seller's ad service that includes ProtectedAudienceInput ciphertext. + * _Note: Seller's SDK may also send device metadata along with the request, like device's geo information, language and user-agent._ + +2. [Seller's ad service] + * Seller's ad service may run an auction similar to today's contextual auction. At this point, the contextual signals are also made available. + * Seller's ad service sends [SelectAdRequest](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L304) to [SellerFrontend (SFE) service](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#sellerfrontend-service) that includes `ProtectedAuctionInput` ciphertext, [AuctionConfig](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L305) and [type of client](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L191). Seller's ad service can forward the device metadata to SFE if available, refer [here](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#metadata-forwarding) for more information about metadata forwarding. + +3. [SellerFrontEnd (SFE) service] + * Upon receiving a [SelectAdRequest](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L304) from the [seller's ad service](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#sellers-ad-service), [SFE](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#sellerfrontend-service) decrypts and decompresses ProtectedAuctionInput ciphertext that includes Protected App Signals data. SelectAdRequest payload also includes seller's [AuctionConfig](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L305). + * SFE parses the buyer list from buyer_list[[3](https://github.com/privacysandbox/bidding-auction-servers/blob/06bf6ed1bd98916d1a9b6887518936655b4d537b/api/bidding_auction_servers.proto#L326-L329)] provided by the seller in [AuctionConfig](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L305) and checks for intersection with the buyer domains configured on the server as part of the BUYER_SERVER_HOSTS config [[4](https://github.com/privacysandbox/bidding-auction-servers/blob/06bf6ed1bd98916d1a9b6887518936655b4d537b/production/deploy/gcp/terraform/environment/demo/seller/seller.tf#L62)]. + * SFE sends GetBidsRequest[[5](https://github.com/privacysandbox/bidding-auction-servers/blob/06bf6ed1bd98916d1a9b6887518936655b4d537b/api/bidding_auction_servers.proto#L493-L495C9)] to the Buyer Frontend (BFE) service of each buyer to collect ads / bids from the shortlisted buyers. + * GetBidsRequest includes buyer inputs for Protected App Signals and Protected Audience, buyer signals (contextual signals) and other required data. + * If [device metadata](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#metadata-forwarding) is available, SFE would also [add the metadata in GetBids gRPC request](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#metadata-forwarded-by-sellerfrontend-service) to BFE. + +4. [BuyerFrontEnd (BFE) service] + * Each buyer’s BFE checks for the presence of ProtectedAppSignalsBuyerInput[[6](https://github.com/privacysandbox/bidding-auction-servers/blob/06bf6ed1bd98916d1a9b6887518936655b4d537b/api/bidding_auction_servers.proto#L541-L542)] in GetBidsRequest. If present, BFE will send a GenerateProtectedAppSignalsBidsRequest[[7](https://github.com/privacysandbox/bidding-auction-servers/blob/06bf6ed1bd98916d1a9b6887518936655b4d537b/api/bidding_auction_servers.proto#L771-L772)] to the bidding service. GenerateProtectedAppSignalsBidsRequest includes buyer inputs, buyer (contextual) signals and other required data. + * _Note: If buyer input for ProtectedAudience is present in GetBidsRequest, then bid generation [flow](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#flow) for Protected Audience is executed in parallel._ + +5. [Bidding service] +Bidding Service performs the following in sequence to retrieve a Protected App Signals ad and corresponding bid: + + * Prepares the data required for retrieving ads from buyer’s Ad Retrieval service. This is achieved by calling into buyer provided prepareDataForAdRetrieval [UDF](#preparedataforadretrieval-udf). + * Calls buyer's [Ad Retrieval service][3] to fetch top k-ads and associated metadata (optionally including trusted bidding signals). +The ad retrieval request payload includes data from the previous step as well as buyer signals required for bidding and [device metadata](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#metadata-forwarding) if available. + * _Note: Device metadata is passed in as part of the request payload to ad retrieval service._ + * Feeds the ads and metadata to the buyer provided generateBid [UDF](#generatebid-udf) to get a single Protected App Signals ad with bid. + * Returns this ad and corresponding bid to BFE. + +7. [BFE] + * BFE handles GenerateProtectedAppSignalsBidsResponse and returns the ad and bid to the seller’s SFE service. + If Protected Audience ads are also participating in the auction then BFE waits for bids for both Protected Audience and Protected App Signals + ads before returning all the ads and bids to SFE. + +8. [SFE] + * SFE waits for all the buyer bids, collects scoring signals from seller’s KV service for all the render URLs in the bids received from BFEs + and sends these ads and bids along with the scoring signals to the auction service. + +10. [Auction service] + * Auction [service](#auction-service) scores all the ads / bids and chooses a winner. + * _Note:_ + * _Both Protected Audience and Protected App Signals bids are scored in this single auction._ + * _Each bid is scored in isolation._ + * Seller and buyer's reporting urls and registered beacons are also generated in the auction service once a winning ad / bid is selected. Refer [here](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_event_level_reporting.md) for more information. + * Auction service returns information about winning ad / bid and reporting urls back to SFE. + +9. [SFE] + * SFE sends back the encrypted [AuctionResult](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L208) in SelectAdResponse to the seller's ad service. +This response can contain a winner or can contain a chaff (e.g. if no buyers ended up participating in the auction or if no buyers generated a bid for candidate ads). + +10. [Seller's ad service] + * Seller's ad service includes the [AuctionResult](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L208) ciphertext in ad response to the client. +This payload is decrypted by the seller on the device. + +12. [Android client] + * The encrypted response is decrypted. If there is no chaff ad is rendered on the app. + * Adtech's reporting endpoints are pinged from the client. + +### Auction With Contextual Ads + +Following is an overview of Protected App Signals architecture with B&A with +contextual ads participating in the auction. + +
+ Protected App Signals Architecture with KV service +
Figure 2.Protected App Signals Architecture Diagram (with KV service)
+


+ +Following is the flow: + + +1. [Android client] + * Using Android's privacy preserving ad-relevance APIs (PPAPI), Android client collects Protected App Signals payload Refer [to the signal curation explainer](https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals#curate-protected) for more information. + * Seller's Android SDK calls PPAPI to get `ProtectedAuctionInput` ciphertext data including the [encoded](https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals#on-device-encoding) representation of the Protected App Signals. + * Note: The payload is compressed, encrypted and padded, refer [to the B&A explainer](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#client--seller-ad-service-communication) for more information about client <> server communication. + * Seller's Android SDK sends an ad request to the seller's ad service that includes `ProtectedAudienceInput` ciphertext. + * _Note: Seller's SDK may also send 1p data, device metadata and contextual signals._ +2. [Seller's ad service] + * Seller's ad service may run an auction similar to today's contextual auction. At this point, the contextual signals are also made available. Seller Ad Service also returns [contextual ad render ids](https://github.com/privacysandbox/bidding-auction-servers/blob/64cf212087572e4f0c1eac56083e49489d85e06a/api/bidding_auction_servers.proto#L472), as part of [PerBuyerConfig](https://github.com/privacysandbox/bidding-auction-servers/blob/64cf212087572e4f0c1eac56083e49489d85e06a/api/bidding_auction_servers.proto#L418), that should participate in the Protected App Signals auction. + * Seller's ad service sends [SelectAdRequest](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L304) to [SellerFrontend (SFE) service ](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#sellerfrontend-service)that includes `ProtectedAuctionInput` ciphertext, [AuctionConfig](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L305) and[ type of client](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L191). Seller's ad service can forward the device metadata to SFE if available, refer [here](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#metadata-forwarding) for more information about metadata forwarding. +3. [SellerFrontEnd (SFE) service] + * Upon receiving a [SelectAdRequest](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L304) from the[ seller's ad service](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#sellers-ad-service), [SFE](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#sellerfrontend-service) decrypts and decompresses ProtectedAuctionInput ciphertext that includes Protected App Signals data. SelectAdRequest payload also includes seller's [AuctionConfig](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L305). + * Then SFE parses the buyer list from buyer_list[[3](https://github.com/privacysandbox/bidding-auction-servers/blob/06bf6ed1bd98916d1a9b6887518936655b4d537b/api/bidding_auction_servers.proto#L326-L329)] provided by the seller in [AuctionConfig](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L305) and checks for intersection with the buyer domains configured on the server as part of the BUYER_SERVER_HOSTS config [[4](https://github.com/privacysandbox/bidding-auction-servers/blob/06bf6ed1bd98916d1a9b6887518936655b4d537b/production/deploy/gcp/terraform/environment/demo/seller/seller.tf#L62)]. + * SFE sends GetBidsRequest[[5](https://github.com/privacysandbox/bidding-auction-servers/blob/06bf6ed1bd98916d1a9b6887518936655b4d537b/api/bidding_auction_servers.proto#L493-L495C9)] to the Buyer Frontend (BFE) service of each buyer to collect ads / bids from the shortlisted buyers. + * GetBidsRequest includes buyer inputs for Protected App Signals and Protected Audience, buyer signals (contextual signals) and other required data. If [device metadata](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#metadata-forwarding) is available, SFE would also[ add the metadata in GetBids gRPC request](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#metadata-forwarded-by-sellerfrontend-service) to BFE. +4. [BuyerFrontEnd (BFE) service] + * Each buyer’s BFE checks for the presence of ProtectedAppSignalsBuyerInput[[6](https://github.com/privacysandbox/bidding-auction-servers/blob/06bf6ed1bd98916d1a9b6887518936655b4d537b/api/bidding_auction_servers.proto#L541-L542)] in GetBidsRequest. If present, BFE will send a GenerateProtectedAppSignalsBidsRequest[[7](https://github.com/privacysandbox/bidding-auction-servers/blob/06bf6ed1bd98916d1a9b6887518936655b4d537b/api/bidding_auction_servers.proto#L771-L772)] to the bidding service. GenerateProtectedAppSignalsBidsRequest includes buyer inputs (which [includes](https://github.com/privacysandbox/bidding-auction-servers/blob/64cf212087572e4f0c1eac56083e49489d85e06a/api/bidding_auction_servers.proto#L612) contextual ad render ids), buyer (contextual) signals and other required data. + * Note: If buyer input for ProtectedAudience is present in GetBidsRequest, then bid generation [flow](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#flow) for Protected Audience is executed in parallel. +5. [Bidding service] + * Bidding service calls into a TEE based Key-Value service to fetch real time signals. + * These real time signals are then fed to the buyer provided generateBid [UDF](#generatebid-udf) to get a single Protected App Signals ad with bid which is then returned to BFE. +6. [Buyer Frontend (BFE) Service] + * BFE handles GenerateProtectedAppSignalsBidsResponse and returns the ad and bid to the seller’s SFE service. If Protected Audience ads are also participating in the auction then BFE waits for bids for both Protected Audience and Protected App Signals ads before returning all the ads and bids to SFE. +7. [Seller Frontend (SFE) Service] SFE waits for all the buyer bids, collects scoring signals from seller’s KV service for all the render URLs in the bids received from BFEs and sends these ads and bids along with the scoring signals to the auction service. +8. [Auction service] + * Auction service [scores](#auction-service) all the ads / bids and chooses a winner. + * Note: Both Protected Audience and Protected App Signals bids are scored in this single auction. + * Each bid is scored in isolation. + * Seller and buyer's reporting urls and registered beacons are also generated in the auction service once a winning ad / bid is selected. Refer [here](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_event_level_reporting.md) for more information. + * Auction service returns information about winning ad / bid and reporting urls back to SFE. +9. [Seller Frontend (SFE) Service] + * SFE sends back the encrypted [AuctionResult](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L208) in SelectAdResponse to the Seller's Ad Service. This response can contain a winner or can contain a chaff (e.g. if no buyers ended up participating in the auction or if no buyers generated a bid for candidate ads). +10. Seller's ad service includes the [AuctionResult](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L208) ciphertext in ad response to the client. This payload is decrypted by the seller on the device. +11. The ad is rendered and then adtech's reporting endpoints are pinged from the client. + + +### API Updates + +The following sections provide the detailed design of the Protected App Signals flow with Bidding and Auction services. + + +#### [Android API](https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals#curate-protected) + +To support Protected App Signals, the seller's SDK on Android will send ad requests to the seller's ad service, where the request payload would include additional data in encrypted [ProtectedAuctionInput](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L53-L58) (which is a [new](#new-protectedauctioninput-message) message type). The Protected App Signals are treated as an opaque byte string by B&A. Refer [here](https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals#curate-protected) for more details about Protected App Signals curation. + + +#### Bidding and Auction services API + +Following sections go over the updates to the B&A API required to support Protected App Signals ad targeting. + + +##### SelectAd API Updates + +###### [New]` ProtectedAuctionInput` Message + +B&A supports `ProtectedAudienceInput` proto message to get the encrypted protected audience data in `SelectAdRequest`. This` ProtectedAudienceInput `message has now been deprecated. + +```protobuf +message ProtectedAudienceInput { + option deprecated = true; + + // ... +} +``` + +With Protected App Signals ad targeting use-case, we generalize the `ProtectedAudienceInput` message by introducing `ProtectedAuctionInput`. + +ProtectedAuctionInput message will be a superset that includes Protected App Signals as well as the Protected Audience data sent from the device. Android will migrate to support `ProtectedAuctionInput `from` ProtectedAudienceInput`. Seller's ad service must depend on the `ProtectedAuctionInput `message going forward. + +```protobuf +// ProtectedAuctionInput is generated and encrypted by the client, +// passed through the untrusted Seller service, and decrypted by the +// SellerFrontEnd service. +// It is the wrapper for all of BuyerInput and other information required +// for the Protected Audience auction. +message ProtectedAuctionInput { + // Input per buyer. + // The key in the map corresponds to IGOwner (Interest Group Owner) that + // is the Buyer / DSP. This string that can identify a + // buyer participating in the auction. The value corresponds to plaintext + // BuyerInput ingested by the buyer for bidding. + map buyer_input = 1; + + // Publisher website or app. + // This is required to construct browser signals for web. + // It will also be passed via GetBids to buyers for their Buyer KV lookup + // to fetch trusted bidding signals. + string publisher_name = 2; + + // A boolean value which indicates if event level debug reporting should be + // enabled or disabled for this request. + bool enable_debug_reporting = 3; + + // Globally unique identifier for the client request. + string generation_id = 4; + + // Consented debugging field. + ConsentedDebugConfiguration consented_debug_config = 5; +} +``` + +###### [New] `ProtectedAppSignals` Message + +A new proto message `ProtectedAppSignals` is added to `BuyerInput.BuyerInput `for each buyer is sent in encrypted [ProtectedAuctionInput](https://github.com/privacysandbox/bidding-auction-servers/blob/06bf6ed1bd98916d1a9b6887518936655b4d537b/api/bidding_auction_servers.proto#L64) from Android.` `There is an entry in `buyer_input` map for each buyer ad-tech with Protected App Signal data (and / or Protected Audience data) who wants to participate in a Protected Auction. + +```protobuf +// Grouping of data pertaining to app install ad selection flow. +message ProtectedAppSignals { + // Signals from device that can help generate a relevant app + // install ad for user. + bytes app_install_signals = 1; + + // Version of the encoding used for the Protected App Signals. + int32 encoding_version = 2; +} + +// A BuyerInput includes data that a buyer (DSP) requires to generate bids. +message BuyerInput { + // InterestGroup (a.k.a CustomAudience) information passed from the client. + message InterestGroup { + // ... + // ... + } + // The Interest Groups (a.k.a Custom Audiences) owned by the buyer. + repeated InterestGroup interest_groups = 1; + + // Signals from device that can help generate a relevant app install ad for + // users. + ProtectedAppSignals protected_app_signals = 2; +} +``` + +###### [New] `ContextualProtectedAppSignalsData` Message + +A new proto message is introduced in [AuctionConfig](https://github.com/privacysandbox/bidding-auction-servers/blob/64cf212087572e4f0c1eac56083e49489d85e06a/api/bidding_auction_servers.proto#L387) > [PerBuyerConfig](https://github.com/privacysandbox/bidding-auction-servers/blob/64cf212087572e4f0c1eac56083e49489d85e06a/api/bidding_auction_servers.proto#L418) to enable buyer RTB servers to specify the contextual ad render ids that should participate in the Protected App Signals auction. The new message definition is as follows: + +```protobuf +// Protected App Signals ad related data that Buyer adtechs can send via +// contextual path to control which PAS ads participate in the auction. +message ContextualProtectedAppSignalsData { + // Optional. + // Raw Ad ids that can be used to lookup signals from the KV server. + repeated string ad_render_ids = 1; + + // Optional. + // Bool indicating whether ads should also be fetched from ads retrieval + // service. + // + // If true, the Bidding server will send an Ad fetch request to the + // Ad retrieval service and the request will contain the list of ad_render_ids + // as an additional parameter. + // + // If false, the ad_render_ids will be sent to the TEE K/V server to fetch + // the ads related metadata. + bool fetch_ads_from_retrieval_service = 2; +} +``` + +##### Code Version Information + +UDF (user defined functions) for bidding, scoring and reporting developed by adtechs can be fetched from adtech hosted endpoints or cloud storage endpoints. Refer [here](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_system_design.md#code-blob-fetch-and-code-version) for more information. + +B&A plans to support multiple versions of UDF for bidding, scoring and reporting. With UDF multi versioning support, multiple versions of UDF will be prefetched off the critical path in Bidding and Auction service at service startup and periodically thereafter; refer [here](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_system_design.md#code-version-flow) for more information. + +To facilitate A/B experimentation of `UDF `logic, B&A API provides support for adtechs to specify the version of `UDF` they would like to use per ad selection request. + +_NOTE: Currently, only B&A API supports multiple code versions for bidding and scoring. However, the full support for multi code versioning in B&A (including that for reporting) to facilitate adtech's A/B experimentation will be available later._ + + + + +###### `generateBid` Code Version + +B&A's Protected App Signals support requires buyer ad-techs to provide a [new generateBid UDF](#generatebid-udf). + + + +* Note: This UDF is different from that of [Protected Audience](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#generatebid). + +Since ad techs may need to A/B test their `generateBid` logic, we provide API support for adtechs to specify what version of `generateBid` they would like to use per auction request in [SelectAdRequest.AuctionConfig.PerBuyerConfig](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L336) + + +```protobuf + // Per buyer configuration. + message PerBuyerConfig { + // ... Existing Fields here ... + + // Optional. + // Version of buyer's GenerateBid() code for Protected App Signals. + // The string must be an object name belonging to the + // Cloud Storage bucket specified at Bidding service startup. + // A buyer can pass this information to the Seller in RTB response. + // If a version is not specified, the default version + // (specified in the service startup config) will be used. + string protected_signals_generate_bid_code_version = 5; + + // Optional. + // Version of buyer's GenerateAdRetrievalUserMetadata() code for + // Protected App Signals. + // The string must be an object name belonging to the + // Cloud Storage bucket specified at Bidding service startup. + // A buyer can pass this information to the Seller in RTB response. + // If a version is not specified, the default version + // (specified in the service startup config) will be used. + string protected_signals_generate_embeddings_version = 6; + } +``` + +###### `scoreAd` Code Version + +B&A also provides a [field](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L408) in the [AuctionConfig](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L305) for seller’s to control the code version for score ads as well: + +```protobuf + message SellerCodeExperimentSpecification { + // The Id is specified by the seller to support coordinated experiments + // with the seller's Key/Value services. + int32 seller_kv_experiment_group_id = 1; + + // The code version of the score ad module provided by the seller. + // The string must be an object name belonging to the + // Cloud Storage bucket specified at Auction service startup. + // If a version is not specified, the default version + // (specified in the service startup config) will be used. + string score_ad_version = 2; + } +``` + +#### Buyer Frontend API Updates + + +##### `GetBidsRawRequest` API updates + +To solicit bids from buyer ad techs, SFE sends a `GetBidsRequest` to each buyer’s BFE. + +Similar to `SelectAdRequest`,` GetBidsRequest` also includes `BuyerInput` where `ProtectedAppSignals` _can_ be embedded. Protected App Signals are passed in the `protected_app_signals_buyer_input` field in `GetBidsRequest`. + +This field will be populated only for Android clients to solicit Protected App Signals ad / bids but not for any other type of targeting. The change is shown below: + +```protobuf +// Protected App Signals input per buyer. +message ProtectedAppSignalsBuyerInput { + ProtectedAppSignals protected_app_signals = 1; + + // Contextual data related to PAS ads. + ContextualProtectedAppSignalsData contextual_protected_app_signals_data = 2; + +} + + +// Unencrypted request. +message GetBidsRawRequest { + // ... Existing fields ... + + // Protected App Signals buyer input. + ProtectedAppSignalsBuyerInput protected_app_signals_buyer_input = 10; +} +``` + + +##### `GetBidsResponse` API updates + +BFE will wait for both Protected Audience ads with bids as well as Protected App Signals ad bid and return a result with both types of bids back to SFE. To support this, the response message is augmented with Protected App Signals bid as shown below: + + +```protobuf +// Response to GetBidsRequest. +message GetBidsResponse { + // Unencrypted response. + message GetBidsRawResponse { + // Includes ad_render_url and corresponding bid value pairs for each IG. + // Represents a JSON object. + repeated AdWithBid bids = 1; + + // Includes ad_render_url and corresponding bid value pairs for app install + // ads. Represents a JSON object. + repeated ProtectedAppSignalsAdWithBid app_install_bids = 2; + } + + // Encrypted GetBidsRawResponse. + bytes response_ciphertext = 1; +} +``` + +#### Bidding Service API updates + + +##### A new RPC Endpoint + +A new API for bidding service is exposed to generate bids for Protected App Signals ads. + +```protobuf +// Bidding service operated by buyer. +service Bidding { + // Generate bids for ads in Custom Audiences (a.k.a InterestGroups) and + // filters ads. + rpc GenerateBids(GenerateBidsRequest) returns (GenerateBidsResponse) {...} + + // Generate bids for app install ads. + rpc GenerateProtectedAppSignalsBids(GenerateProtectedAppSignalsBidsRequest) returns (GenerateProtectedAppSignalsBidsResponse) { + option (google.api.http) = { + post: "/v1/generateappinstallbids" + body: "*" + }; + } +} +``` + +##### `GenerateProtectedAppSignalsBidsRequest` Message + +Following new message will be added to be used as an input to the newly added RPC call above: + +```protobuf +// Generate bids for all app install ads. +message GenerateProtectedAppSignalsBidsRequest { + // Unencrypted request. + message GenerateProtectedAppSignalsBidsRawRequest { + /********************* Common inputs for bidding ***********************/ + // Information about auction (ad format, size) derived contextually. + // Represents a JSON object. Copied from Auction Config in SellerFrontEnd + // service. + // Represents a serialized string that is deserialized to a JSON object + // before passing to Adtech script. + string auction_signals = 1; + + // Buyer may provide additional contextual information that + // could help in generating bids. Not fetched real-time. + // Represents a serialized string that is deserialized to a JSON object + // before passing to Adtech script. + // + // Note: This is passed in encrypted BuyerInput, i.e. + // buyer_input_ciphertext field in GetBidsRequest. The BuyerInput is + // encrypted in the client and decrypted in `BuyerFrontEnd` Service. + // Note: This is passed in BuyerInput. + string buyer_signals = 2; + + + // Signals used to generate features needed to generate ads for app + // install. + ProtectedAppSignals protected_app_signals = 3; + + // Seller origin. + // Sent to generateBid script in device signals. + string seller = 4; + + // Publisher app that is part of Buyer KV lookup url. + // Sent to generateBid script in device signals. + string publisher_name = 5; + + // Helpful context for logging and tracing the request. + LogContext log_context = 6; + + // Consented debugging field. + server_common.ConsentedDebugConfiguration consented_debug_config = 7; + + // A boolean value which indicates if event level debug reporting should be + // enabled or disabled for this request. + bool enable_debug_reporting = 8; + + // Contextual data regarding PAS ads. + ContextualProtectedAppSignalsData contextual_protected_app_signals_data = 9; + } + + // Encrypted GenerateProtectedAppSignalsBidsRawRequest. + bytes request_ciphertext = 1; + + // Version of the public key used for request encryption. The service + // needs use private keys corresponding to same key_id to decrypt + // 'request_ciphertext'. + string key_id = 2; +} +``` + +##### New `ProtectedAppSignalsAdWithBid` Message + +This new message would include a bid for Protected App Signal ad. This is built in the Bidding server and returned to BFE in `GenerateProtectedAppSignalsBidsResponse` (see [here](#generateprotectedappsignalsbidsresponse-message)). + +```protobuf +// Bid for an app install ad candidate. +message ProtectedAppSignalsAdWithBid { + // Metadata of the ad, this will be passed to Seller's scoring function. + // Represents a serialized string that is deserialized to a JSON object + // before passing to Adtech script. + google.protobuf.Value ad = 1; + + // Bid price corresponding to an ad. + float bid = 2; + + // Ad render url that identifies an ad creative. + string render = 3; + + // A numerical value used to pass reporting advertiser click or conversion + // cost from generateBid to reportWin. The precision of this number is + // limited to an 8-bit mantissa and 8-bit exponent, with any rounding + // performed stochastically. + double ad_cost = 4; + + // A 12 bit integer signal used as input to win reporting url generation for + // the Buyer. + int32 modeling_signals = 5; + + // Indicates the currency used for the bid price. + string bid_currency = 6; + + reserved 7; + + // Optional field for debug report URLs. + DebugReportUrls debug_report_urls = 8; + + // Holds schema version as well as features related information that needs + // to be send back to the buyer who has the winning bid. + string egress_payload = 9; + + // Temporary unlimited size egress features. Meant for experimentation only. + string temporary_unlimited_egress_payload = 10; +} +``` + +Since B&A release 3.4.1, PAS support in B&A provides buyers _temporary_ egress data for training the models used during ad selection. More details about egress data can be found [here](https://developers.google.com/privacy-sandbox/relevance/protected-audience/android/protected-app-signals#reporting) and [here](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_protected_app_signals_egress.md). + + +##### `GenerateProtectedAppSignalsBidsResponse` Message + +This will be the output of the new RPC: + +```protobuf +// Encrypted response to GenerateProtectedAppSignalsBidsRequest with bid prices +// corresponding to all eligible app install Ad creatives. +message GenerateProtectedAppSignalsBidsResponse { + // Unencrypted response. + message GenerateProtectedAppSignalsBidsRawResponse { + // Bids corresponding to ads. + repeated ProtectedAppSignalsAdWithBid bids = 1; + } + + // Encrypted GenerateProtectedAppSignalsBidsRawResponse. + bytes response_ciphertext = 1; +} +``` + +For the definition of `ProtectedAppSignalsAdWithBid message` please refer back to [this](#new-protectedappsignalsadwithbid-message) section. + + +### Protected Audience & Protected App Signals Combined Auction Flow + +The following sequence diagram shows the Protected Audience and Protected App Signals ad selection as part of a single auction. Note that the BFE branches out the bid generation flow for Protected Audience and Protected App Signals). BFE waits for both types of bids before returning an aggregated response with all the ads + bids to SFE. SFE then forwards these bids from buyer for scoring to the Auction service (after SFE has fetched the scoring signals from KV) to conduct an auction and choose a single winning ad with bid. + + +
+ Protected App Signals Sequence Diagram +
Figure 2.Protected App Signals Sequence Diagram
+


+ +**Legend:** + +Red: Protected App Signals Flow + +Green: Protected Audience Flow + + +## Buyer B&A Services + + +### Buyer Frontend (BFE) Service + +BFE is the frontend service for bidding in the buyer stack. It inspects the buyer input provided by the SFE and decides whether or not to solicit a bid for Protected App Signals from the bidding service. If Protected Audience is also present in the incoming buyer input then BFE will also solicit Protected Audience related bids from the Bidding service and return ads and corresponding bids for both types of ad back to SFE. + + +#### Configuration + +Buyer ad techs need to ensure the following while bringing up a buyer stack: + + + +* Protected App Signals support is enabled (via a [flag](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/production/deploy/gcp/terraform/environment/demo/buyer/buyer.tf#L64)) in the stack. +* An Ad Retrieval Service endpoint is deployed and reachable by the bidding service. +* A `prepareDataForAdRetrieval` UDF is hosted on an endpoint by the buyer adtech and should be reachable by the bidding service. + * Note: `prepareDataForAdRetrieval `UDF can be fetched from the` prepareDataForAdRetrievalCodeUrl `key in Terraform config. +* A buyer bidding UDF (`generateBid`) for Protected App Signals is hosted on an endpoint and is reachable by the bidding service. +* `reportWin` UDF is hosted on an endpoint and is reachable by auction service (in the seller stack). + +Next section provides details about how the bidding service uses the app signals and generates an ad with bid for participation in an auction conducted by the seller ad tech. + + +### Bidding Service + +Bidding service is at the heart of the bidding process for Protected App Signals. Bidding service: 1) decodes signals into a useful form (to buyer adtechs) via buyer provided UDF, 2) retrieves relevant ads, and 3) generates a bid for the most relevant ad. + +These three distinct stages in the bidding process are summarized below (for an overview refer to this [doc](https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals#buy-side-ad)): + + +#### `prepareDataForAdRetrieval` UDF + +This is a buyer ad tech provided UDF and is aimed at preparing all the data needed to get relevant ads from the ads retrieval service. (**Note**: This UDF is only used in the TEE Ad Retrieval flow and bypassed in the contextual flow). This UDF is expected to: + +1. Decode the incoming encoded signals. +2. Retrieve any embeddings that may be required in the (next) retrieval stage. This step optionally may involve a call to an [inference service](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/inference_overview.md) to fetch embeddings from a ML model. + +Signature of this UDF is as follows: + +```javascript +/** + * Inputs + * ------ + * 1. `encodedOnDeviceSignals` is a Uint8Array and would contain + * the encoded app signals emanating from device. + * 2. `encodedOnDeviceSignalsVersion` is an integer that helps the buyer + * ad techs to decide which version of decoding logic to use. + * 3. `sellerAuctionSignals` has the auction related information (See here). + * 4. `contextualSignals` flow from the RTB path (and can potentially carry + * contextual embeddings needed for making a prediction during bid + * generation). More details are here. + * + * Output + * ------ + * Returns a JSON object. The exact contents are controlled by the adtech + * (e.g. it can contain the decoded signals and embeddings required for + * retrieval). + */ +function prepareDataForAdRetrieval(encodedOnDeviceSignals, + encodedOnDeviceSignalsVersion, + sellerAuctionSignals, + perBuyerSignals) { + return {}; // A JSON struct useful for ad retrieval and bidding. +} +``` + +Note: that the data returned by this UDF is fed to both the ad retrieval service as well as the `generateBid` UDF in the pipeline. + + +#### Fetching Ads from Ad Retrieval Service + +The [Ads Retrieval service][3] is invoked by the bidding service to retrieve the top-k ads which have the highest chance of conversion. This service needs to provide an endpoint that supports a `PUT` HTTP verb. + + +##### Input Format + +The input data required for fetching the ads is passed in the `partitions > arguments` block. Bidding service sends the following information in a sequence in the body of the `PUT` request to the ad retrieval service in order to get relevant ads: + + +* Data retrieved from the `prepareDataForAdRetrieval` UDF (which can include the decoded app signals and embeddings required for retrieval). +* Device metadata (including information about the client IP, user agent and language etc.). This data can help the ad retrieval service filter the relevant ads e.g. different ads may be suitable based on the user's geolocation. +* Contextual signals flowing retrieved via the buyer’s own RTB server (these can among other things include embeddings based on the contextual data that can be helpful for making a final prediction during bidding) + +```json +{ + "partitions" : [ + { + "arguments" : [ + { + "data" : "{\"key1\": \"1\", \"key2\": \"2\"}" + }, + { + "data" : { + "X-Accept-Language" : "en-US,en;q=0.9", + "X-BnA-Client-IP" : "1.2.3.4", + "X-User-Agent" : "Mozilla/5.0 (Linux; Android 15; SM-G960U) Mobile MyAmazingApp/1.2.3 (com.example.myapp; in-app-ad; SDK/1.0)" + } + }, + { + "data" : "{\"key3\": \"3\"}" + } + ] + } + ] +} +``` + + +##### Output Format + +Output from the ad retrieval service should contain a JSON object in the following format: + +```json +{"singlePartition":{"stringOutput": "arbitrary text string"}} +``` + +Adtechs can choose to embed any information regarding in the retrieved Protected App Signals ads as value of the `singlePartition` > `stringOutput` key as long as that information is meaningful to the adtech’s `generateBid` script since this data will be ingested by the ad tech provided `generateBid` UDF. The returned data here can contain e.g. ads and ads metadata (including ads embeddings that can be used in `generateBid` to make a final prediction). + + +#### `generateBid` UDF + +Once all ads (and associated data) are retrieved from the ads retrieval service, bidding service relays this data to the ad tech provided `generateBid` UDF. The signature of this method is as follows: + +```javascript +/* + * Inputs + * ----- + * 1. `ads` Contains the data returned by the ad retrieval service or KV + * lookup. This data includes ad metadata and optionally + * trustedBiddingSignals as well. + * 2. `sellerAuctionSignals` has the auction related information (See here). + * 3. `buyerSignals` flow from the RTB path (and can potentially carry + * contextual embeddings needed for making a prediction during bid + * generation). More details are here. + * 4. `preprocessedDataForRetrieval`: This is the data returned by + * `prepareDataForAdsRetrieval` UDF. Note: This will only be populated for + * the retrieval flow. + * 5. `encodedOnDeviceSignals` is a Uint8Array and would contain + * the encoded app signals emanating from device. + * 6. `encodedOnDeviceSignalsVersion` is an integer that helps the buyer + * ad techs to decide which version of decoding logic to use. + * + * Output + * ------ + * Returns a JSON equivalent to this proto message. + * Note: Only one bid is returned among all the input Protected App Signals + * ads. + */ +function generateBid(ads, sellerAuctionSignals, perBuyerSignals, + preparedDataForAdRetrieval, encodedOnDeviceSignals, + encodedOnDeviceSignalsVersion) { +return { "ad": , + "bid": , + "render": , + "adCost": }; +} +``` + +#### `reportWin` UDF + +When the buyer adtech wins an auction, the seller uses this buyer-provided UDF to create reporting URLs and to register ad beacons. Note that both buyer and seller reporting URLs are generated in the auction service once scoring has been completed. More details about event level reporting can be found in this [explainer](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_event_level_reporting.md). + +This UDF will have the following signature: + +```javascript +function reportWin(auctionSignals, perBuyerSignals, signalsForWinner, + buyerReportingSignals, egressPayload, + temporaryUnlimitedEgressPayload) { + // ... +} +``` + +Details about all the arguments (except `egressPayload*`) can be found [here](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#reportwin). Egress arguments are meant to help buyers egress data (_temporarily_ unlimited) for training their models that are used during ad selection. Refer [here](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_protected_app_signals_egress.md) for more details about the expected egress data format. + +## Seller B&A Services + + +### Seller Frontend (SFE) Service + +As mentioned in the [overview section](#overview), SFE determines which buyers can participate in the auction and fetches bids from these buyers. Once the bids are fetched from all the buyers, SFE calls Auction service to decide on the winning ad and generate reporting urls. The winning ad and reporting urls are returned back to the client in encrypted [AuctionResult](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L208) in [SelectAdResponse](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/api/bidding_auction_servers.proto#L457). + + +#### Configuration + +Buyer ad techs need to ensure the following while bringing up a buyer stack Protected App Signals support is enabled (via [this](https://github.com/privacysandbox/bidding-auction-servers/blob/b222e359f09de60f0994090f7a57aa796e927345/production/deploy/gcp/terraform/environment/demo/seller/seller.tf#L66) flag) in the stack. + + + +* `reportWin` UDF endpoint provided by the buyers is utilized in the terraform module to help generate the reports for buyers winning the auction. + + +### Auction Service + +Auction service is responsible for choosing a winning bid amongst all buyers participating in Protected Auction and the ad types. This service scores bid for each ad using the seller adtech provided logic as detailed below: + + +#### `scoreAd` UDF + +The seller adtech provided scoreAd [UDF](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#scoread) is used by the auction service to generate a desirability score for each ad. Once every ad is scored by this UDF, the auction service chooses the winner Ad with the highest positive desirability score. The scoring logic is agnostic of the type of ad being scored, therefore the winning ad can either be a Protected App Signals Ad or a Protected Audience ad. + + +#### `reportResult` UDF + +When a bid is chosen as a winner, seller adtech provided reportResult [UDF](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#reportresult) is invoked to generate signals for winner ad tech. This data is then fed to the buyer provided Protected App Signals specific reportWin [UDF](#reportwin-udf) to generate the reporting URLs. The reporting urls are pinged from the device after the ad is rendered (and or upon user interaction). + + +[1]: https://github.com/salmanmlk +[2]: https://github.com/chatterjee-priyanka +[3]: https://github.com/privacysandbox/protected-auction-key-value-service/blob/main/docs/protected_app_signals/ad_retrieval_overview.md diff --git a/bidding_auction_services_protected_app_signals_egress.md b/bidding_auction_services_protected_app_signals_egress.md new file mode 100644 index 0000000..1167939 --- /dev/null +++ b/bidding_auction_services_protected_app_signals_egress.md @@ -0,0 +1,437 @@ +**Author:**
+[Rashmi Rao](https://github.com/rashmijrao), [John Cox](https://github.com/johncox-google), [Salman Malik](https://github.com/salmanmlk), Google Privacy Sandbox + +# Protected App Signals egress in Bidding and Auction Services + +Protected App Signals provides ad-techs a way to egress data from inside the privacy boundary to their own servers for model training. See the [Protected App Signals Explainer](https://developers.google.com/privacy-sandbox/relevance/protected-audience/android/protected-app-signals), particularly the [Reporting](https://developers.google.com/privacy-sandbox/relevance/protected-audience/android/protected-app-signals#reporting) section, for background. Refer design for Protected App Signals with B&A [here](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_protected_app_signals.md). + +This document describes the following: + +1. The set of feature types available to egress information to adtech servers for model training for Protected App Signals. + +2. The wire format of each feature. + +3. The wire format of the `egressPayload` itself. + +The wire representation of the `egressPayload` will be noised per feature type. More details will be provided in a future explainer update. + +Based on the information in this document, adtechs can write parsers to prepare the `egressPayload` and transform the values it contains into features for use in their model training systems. + +# Feature types + +We support two kinds of feature types: primitives, which can contain a single feature value; and collections, which can contain multiple primitives. + +**Nullability** + +_(Nullability will be a fast-follow feature; it will not be available immediately upon the release of egress vector. Please see this addendum describing a schema modification so your de-serializer can be prepared for nullability support.)_ + +Every feature type can optionally be made nullable, in which case the value `null` can be represented. By default features are _not_ nullable. A null value for a type is indicated by adding a signaling bit, 0 (for null) or 1 (for non-null), to the least-significant bit of the serialized representation. Consequently, the wire representation for a feature with `nullable = true` is always 1 bit larger than with `nullable = false`. + +The contents of collections can be nullable and are handled as above. Collections themselves can also be nullable; handled the same way: a null value's wire representation is 0s for all value bits, with the 0 signaling bit. + +Because values are _not_ nullable by default, values mentioned in the examples below are not nullable unless stated. + +## Primitive types + +These types represent single values: booleans, unsigned integers, and signed integers + +### `boolean-feature-type` + +This represents a single boolean value. + +Expected value: `true` or `false`, or optionally `null` + +Parameters: + +* `nullable`: boolean indicating whether the `boolean-feature-type` can represent a null state or not. Defaults to `false`. + +**Wire format** + +Non-nullable: + +0 (`false`) or 1 (`true`) + +Nullable: + +01 (`false`) or 11 (`true`), or 00 (`null`) + +### `unsigned-integer-feature-type` + +This represents a single non-negative integer value. + +Parameters: + +* `size`: unsigned integer indicating the number of bits based on the range of values. + + * Expected value: non-negative integer value in the range [0,2^size-1] + + * For example, consider an `unsigned-integer-feature-type` of `size` = `3`. It can have a value: ` [0, 7]` and will occupy 3 bits on the wire (or four bits on the wire if nullable, see examples below). +* `nullable`: boolean indicating whether the `unsigned-integer-feature-type` can represent a null state or not. Defaults to `false`. + +**Wire format** + +Binary representation of the unsigned integer. The integer will be converted to wire format following little endian byte order. If nullable, will be prefixed by the indicator byte. + +Examples: + +*Non-nullable*: + +If `value` = `5` and `size `= `3`, wire format will be `101` + +*Nullable*: + +If `value` = `5` and `size `= `3`, wire format will be `1011`. + +If `value` = `0` and `size` = `3`, wire format will be `0001` + +If `value` = `null` and `size` = `3`, wire format will be `0000` + +### `signed-integer-feature-type` + +This type can be used to represent a single positive or negative integer value. + +Parameters: + +* `size`: unsigned integer indicating the number of bits based on the range of values. + + * Expected value: integer value in the range [-2^(size-1),2^(size-1)-1] + + * For example, consider a `signed-integer-feature-type` of `size` = `4`. It can have a value `[-8, 7]`, and will occupy 4 bits on the wire (or 5 bits on the wire if nullable, see examples below). + +* `nullable`: boolean indicating whether the `signed-integer-feature-type` can represent a null state or not. Defaults to `false`. + +**Wire format** + +2’s complement representation of the signed integer. The integer will be converted to wire format following little endian byte order. + +Examples: + +*Non-nullable*: + +If `value` = `-3` and `size`=`4`, wire format will be `1101` + +*Nullable*: + +If `value` = `-3` and `size`=`4`, wire format will be `11011` + +If `value` = `0` and `size` = `4`, wire format will be `00001` + +If `value` = `null` and `size` = `4`, wire format will be `00000` + + +## Collection types + +These feature types represent a collection of homogeneous or heterogeneous values. + +The wire representation of the values will be in the right-to-left order. + +### `bucket-feature-type` + +This type can be used to represent an ordered list of `boolean-feature-type` values. + +Expected value: list of boolean (`true` or `false`, or optionally `null`) values, or optionally `null` + +**Parameters** + +* `allow-multiple`: indicates whether the bucket can contain multiple `true` values. +* `size`: number of values in the bucket. +* `nullable`: boolean indicating whether the `bucket-feature-type` as a whole can represent a null state or not. Defaults to `false`. + +**Wire format** + +Sequential bit representation of `boolean-feature-type` values. + +Examples: + +*Non-nullable collection*: + +Consider a `bucket-feature-type` of `size` = `4`. If the values are [`true`, `false`, `true`, `false`] this will occupy `4` bits on the wire. Wire format will be `0101` + +Again for `size` = `4`, now consider the values all being `nullable = true`. If the values are [`true`, `false`, `true`, `null`] this will occupy `8` bits on the wire. Wire format will be `00 11 01 11` (spaces for readability). + +*Nullable collection* + +Consider a `bucket-feature-type` of `size` = `4`. If the values are [`true`, `false`, `true`, `false`] this will occupy `5` bits on the wire. Wire format will be `01011` + +### `histogram-feature-type` + +This type can be used to represent an ordered, heterogeneous list of `unsigned-integer-feature-type` and `signed-integer-feature-type` values. + +**Parameters** + +* `size`: unsigned integer indicating the number of fixed size values in the histogram. + +Expected values: list of `unsigned-integer-feature-type` and `signed-integer-feature-type` values + +**Wire format** + +In general the wire format for the histogram is the wire format of each contained value, right-to-left. + +Examples: + +*Non-nullable histogram*: + +If the histogram contains `2` elements, the first of which is an unsigned 3-bit integer with the value `5`, and the second is a signed `4`-bit integer with the value `-3`, then the wire format would be `1101 101` (spaces for readability). + +Consider the same example, except the second value is now nullable. Now the wire format requires eight bits instead of seven, and becomes `11011 101`. + +Consider the same example again, except now the second value is nullable and indeed `null`. The wire format still requires eight bits and is `00000 101`. + +*Nullable histogram*: + +Consider the original example, except the histogram is now itself nullable. Now the representation requires eight bits, becoming `1101 101 1`. + +Now consider if both the histogram itself is nullable, and the second value is nullable and indeed `null`. The wire format now requires nine bits and is `00000 101 1`. + +# Payload wire format + +The definition of the wire format of a payload is called its _protocol_. Below we describe the first wire format, or protocol version `1`. The protocol version included in the payload will be set by the platform. + +A payload is made up of two parts: a **header** containing metadata information used for serialization and deserialization; and a **body** containing serialized feature values. + +## Header + +The header itself has two parts: + +1. **Protocol version**: unsigned `5`-bit int indicating the version of the wire format specification used to encode the payload. + +2. **Schema version**: unsigned `3`-bit int. Version identifier for the schema that defines the payload. + +The wire format of the header is the protocol version, then the schema version, right-to-left. For example, if the protocol version is `1` (`00001` on the wire) and the schema version is `2` (`010` on the wire), the header will be `01000001`. + +## Body + +The body contains serialized feature values, with values as defined in each feature type above. The order of the features is the same as their order in the provided schema for the payload, right-to-left. + +The body is `0`-padded. Details of the padding are slightly different for `egressPayload` and `temporaryUnlimitedEgressPayload`: + +1. `egressPayload` is first `0`-padded to its maximum size in bits, then `0`-padded to the nearest byte. +2. `temporaryUnlimitedEgressPayload` is `0`-padded to the nearest byte. + +## Example + +* Protocol version : `1` +* Example schema version : `2` +* Example max wire size for `egressPayload`: `20` bits + +Consider this example schema, feature values and corresponding wire format for each feature type specified the schema: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Feature type +

+(Defined in the schema) +

Feature type in collection +

+(Defined in the schema) +

Parameters +

+(Defined in the schema) +

Corresponding value in Json + Wire representation +
histogram-feature-type with size = 2 + unsigned-int-feature-type + size = 3 + 5 + 101 +
signed-int-feature-type + size = 4 + -3 + 1101 +
boolean-feature-type + + + false + 0 +
bucket-feature-type + boolean-feature-type + size = 4, allow-multiple = true, + nullable = true + [true, false, true, false] + 01011 +
boolean-feature-type + + + nullable = true + null + 00 +
+ + +Wire format of the feature values would be: +Feature values in wire format + +#### Wire format for temporaryUnlimitedEgressPayload: + +Wire format of the feature values + padding would be: + +Wire format for temporaryUnlimitedEgressPayload with padding + +Wire format of the feature values + padding + header would be: + +Wire format for temporaryUnlimitedEgressPayload with padding and header + +#### Wire format for egressPayload: + +Wire format of the feature values + padding would be: + +Wire format for egressPayload with padding + +Wire format of the feature values + padding + header would be: + +Wire format for egressPayload with padding + +## Addendum: Workaround for implementing nullability before official support + +While nullability will not be supported immediately upon the release of egress vector support, AdTechs can write their schemas in such a way that the serialized representation of the egress vector is identical before and after nullability is supported. + +Recall that a null value for a type is indicated by adding a signaling bit to the least-significant bit of the serialized representation. AdTechs can simply add a `boolean-feature-type` before each value which they intend to make nullable, and set it to indicate whether the following value is null: 0 (for null) or 1 (for non-null). + +Because this approach is identical to the way nullability support will be implemented, it allows AdTechs to write de-serialziation code now. Upon the release of nullability support, AdTechs will remove the boolean nullability flags and mark the values to which they correspond as nullable in the schema, and update their code for building the egress vector accordingly. + +Consider the example schema from above, modified to show nullability flags in bold before nullability support: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Feature type +

+(Defined in the schema) +

Feature type in collection +

+(Defined in the schema) +

Parameters +

+(Defined in the schema) +

Corresponding value in Json + Wire representation +
histogram-feature-type with size = 2 + unsigned-int-feature-type + size = 3 + 5 + 101 +
signed-int-feature-type + size = 4 + -3 + 1101 +
boolean-feature-type + + + false + 0 +
boolean-feature-type + + + true + 1 +
bucket-feature-type + boolean-feature-type + size = 4, allow-multiple = true + [true, false, true, false] + 0101 +
boolean-feature-type + + + false + 0 +
boolean-feature-type + + + false + 0 +
+ +The wire format displayed in the diagram above still applies - which is why this workaround allows de-serialization code to be written before nullability support is released. diff --git a/bidding_auction_services_system_design.md b/bidding_auction_services_system_design.md new file mode 100644 index 0000000..e583cb1 --- /dev/null +++ b/bidding_auction_services_system_design.md @@ -0,0 +1,755 @@ +> FLEDGE has been renamed to Protected Audience API. To learn more about the name change, see the [blog post](https://privacysandbox.com/intl/en_us/news/protected-audience-api-our-new-name-for-fledge) + +**Authors:**
+[Priyanka Chatterjee][1], Google Privacy Sandbox
+[Daniel Kocoj][2], Google Privacy Sandbox
+[Alek Kundla][3], Google Privacy Sandbox + + +# Bidding and Auction Services System Design +The [Bidding and Auction Services][4] (B&A) outlines a way to allow Protected +Audience auctions to take place in [trusted execution environment][5] (TEE) based +servers on a [supported cloud platform][6]. + +This explainer describes the system design of Bidding and Auction services. +For [high level design][7], [Adtech specs][8], [API][9], refer to this [explainer][4]. + +_Note:_ + * Refer [here][10] for Chrome client side design explainer for Bidding and Auction + services integration. + * Refer [here][50] for Android's client side design explainer for Bidding and Auction services + integration. + +## Overview + +![Architecture diagram.](images/unified-contextual-remarketing-bidding-auction-services.png) + +Seller's code in the publisher web page (in browser) or publisher app +(in android) sends a unified contextual and Protected Audience auction HTTPS +request to the seller's ad service. The request includes contextual payload and +encrypted [_Protected Audience data_][11]. Seller's ad service sends Real Time Bidding +(RTB) requests to buyers for contextual bids, and conducts a contextual auction. +Then the seller ad service sends a request to TEE based [SellerFrontEnd service][12] +that includes encrypted _Protected Audience data_, contextual signals and other +data. Protected Audience bidding and auction kicks off in [Bidding and Auction +services][4]. The SellerFrontEnd service decrypts encrypted _Protected Audience data_ +using decryption keys prefetched from [Key Management System][13]. Then SellerFrontEnd +service orchestrates bidding requests in parallel to buyers participating in the +auction. Within each buyer system, TEE based [BuyerFrontEnd service][14] looks up +real-time bidding signals from [buyer's key/value service][16] and calls TEE based +[Bidding service][15] for bidding. Buyers would need to return bids within a timeout +set by the seller. After bids are received from buyers, SellerFrontEnd service +looks up real-time scoring signals from [seller's key/value service][17] and calls +TEE based [Auction service][18] for scoring the bids. SellerFrontEnd returns an +encrypted auction result to the seller ad service. Seller ad service sends the +encrypted auction result back to the client. + +Bidding and Auction services are based on the gRPC framework. The server code is +developed in C++ and configurations are based on [Terraform][19]. The server code and +configurations will be open sourced by Google Privacy Sandbox. + +Bidding and Auction services also allows execution of Adtech owned code for +generating bids, scoring ads, generating reporting urls. The hosting environment +would protect the confidentiality of the Adtech code and contextual signals (if +the execution happens only in the cloud). + +The communication between services in the [Bidding and Auction system][4] is +protected by TLS / SSL and additionally, request-response payloads are encrypted +by [Hybrid Public Key Encryption][20] (HPKE). Also, request/response payload sent over +the wire will be compressed by gzip. + +## Design + +### Sell-side platform (SSP) system +Following services will be operated by an SSP, also referred to as a seller. + +#### Seller's ad service +The seller's ad service is the existing stack of the seller that facilitates +real-time bidding auctions. + +The seller's ad service will receive a unified contextual and Protected Audience +auctions HTTPS request from the client for selecting an ad. The request would +include contextual request payload and encrypted [_Protected Audience data_][11] +from the client. The encrypted _Protected Audience data_ includes Interest +Group (Custom Audience) and other information on the user's device required for +bidding. + +* Seller's code in the publisher web page or publisher app calls the client + (browser or android) to get the encrypted _Protected Audience data_. + +* Seller's code sends a HTTPS request to the seller's ad service that includes + contextual and encrypted _Protected Audience data_ (unified request). + + * The encrypted data is uploaded in the body of the HTTPS request and [device + metadata][21] is added to the HTTP request header. + + * The seller's ad service will not be able to decrypt encrypted _Protected + Audience data_. + + * _Note: There will be 7 size buckets for Protected Audience data, with + the minimum size being very small. This would ensure that the request + payload size does not leak enough bits with iterative requests. This would + mitigate re-identification of the user._ + +* Seller's ad service conducts contextual auction. + + * Seller sends real time bidding (RTB) requests to partner buyers. Buyers + respond with bids for contextual auction and may express demand to participate + in Protected Audience auction by returning _buyer_signals_ in RTB response. + + * Based on different factors, seller can determine if there is incremental + value in Protected Audience auctions. If there is value, seller's ad service + calls Bidding and Auction services. + +* Seller's ad service calls Bidding and Auction services for Protected Audience + auction after conducting contextual auction, only if there is for demand for + Protected Audience auction. + + * The encrypted _Protected Audience data_ is passed in the request along with + other required data. + + * Seller's ad service can send HTTP(S) requests or gRPC requests; HTTPS + requests would be translated to gRPC using [proxy service][22] configured in front + of SellerFrontEnd service. + +* The response from seller ad service would include a contextual ad winner + and / or encrypted [Protected Audience response][23]. + + * If Bidding and Auction services is called by seller ad service, an encrypted + Protected Audience response will always be returned. + +* The seller code in the publisher web page or app returns the encrypted + Protected Audience response to the client (browser or android). + + * Only the client (browser or android) will be able to decrypt the Protected + Audience response. + +_Note: The size of Protected Audience data must be small to optimize latency + and reduce network cost for Adtechs. We will publish a separate explainer for + the payload optimization guide for DSPs._ + + #### SellerFrontEnd service + + The front-end service of the system that runs in the [TEE][5] on a supported cloud + platform. The service is based on the gRPC framework and provides a gRPC endpoint + ["SelectAd"][24] that receives requests from the seller's ad service to initiate + Protected Audience auction flow. + + * SellerFrontEnd server instance loads configured data in memory at server startup. + This includes the following, for more details refer [here][25]: + * Domain address of other services that SellerFrontEnd depends on. + * This includes address of BuyerFrontEnd services of partner buyers, address of + seller key/value service, address of Auction service and address of key hosting + services in key management systems. + + * Seller's origin (domain address). + + * Global timeout of buyer. + + * Map of buyer origin to BuyerFrontEnd domain address for all buyers that the seller + has partnered with. + + * At server startup, the gRPC or HTTP connections to other services (BuyerFrontEnd + services of partner buyers, Auction service, seller's key/value service, cryptographic + key hosting services) are prewarmed. + + * Seller's ad service would send a _SelectAd_ request to TEE based SellerFrontEnd service. + * The request payload includes encrypted [_Protected Audience data_][11]; and other data passed + by the seller as follows: + * [AuctionConfig][24] + * _seller_signals_ : This can include the contextual bid used to filter Protected + Audience bids during scoring. + * _auction_signals_ + * _per_buyer_config_ map : Map of buyer domain to per buyer information like + _per_buyer_signals_, _per_buyer_timeout_ and other optional information. + * _buyer_list_ : List of buyers participating in the auction. This includes the + domain address of the buyers, also known as buyer origin or InterestGroupOwner. + * Note: This may be a subset of the buyer domains in _per_buyer_config_ map. + * _seller_ origin + + * The communication between seller's ad service and SellerFrontEnd over the untrusted + network is protected by TLS / SSL. The TLS / SSL session will terminate at the load + balancer in front of SellerFrontEnd service. + + * SellerFrontEnd service decrypts encrypted Protected Audience data. This data includes + BuyerInput for the buyers whose protected audience data is available on the user's + device. + + * SellerFrontEnd validates that the seller's origin in service configuration is the same + as that passed by the seller's ad service in the SelectAd request. Otherwise, + SellerFrontEnd returns an error response to the seller's ad service. + + * SellerFrontEnd orchestrates GetBids requests in parallel to buyers participating in the + auction for bidding. + * SellerFrontEnd will not send the requests to all buyers the seller partnered with, + i.e. the information configured in service configuration. + + * The requests will be sent to only those buyers in buyer_list whose corresponding + BuyerInput is available in Protected Audience data. + + * After bids are generated, buyers return GetBids response to SellerFrontEnd. The + GetBids response includes AdWithBid objects corresponding to each bid. AdWithBid + object includes ad_render_url or ad_component_render_url corresponding to the bid. + + * SellerFrontEnd looks up real-time scoring signals required for the auction from the + seller's key/value service. + * All ad render urls and ad component render urls are batched for scoring signals + lookup. + + * The urls are encoded in SellerFrontEnd before scoring signals lookup. + + * SellerFrontEnd sends a [ScoreAds][26] request to the Auction service to score ads and + select a winner. The request payload includes ads with bids, scoring signals, + contextual signals and other information required for the auction. + * Auction service returns a Protected Audience winner to SellerFrontEnd. + + * When ads are scored, the contextual bid may filter all Protected Audience bids. + In this case, the Auction service doesn't return a Protected Audience winner to + SellerFrontEnd. + + * SellerFrontEnd returns an encrypted AuctionResult in SelectAd response to seller's + ad service. + * If there is no Protected Audience winner, that would be indicated with is_chaff + field set to true in AuctionResult. Even in this case, an encrypted AuctionResult + that is padded with fake data will be returned to the seller's ad service. + + * The size of fake / padded [AuctionResult][23] would be in the order of real + AuctionResult size. This would ensure that the size of the response doesn't leak + any information to the seller. + + ##### Throttling in SellerFrontEnd + The [_ProtectedAudienceInput_][11] in [SelectAd][24] request would include a generation_id + that the client (browser / android) will generate based on [UUID][27] (Universal Unique Identifier). + SellerFrontEnd will cache the generation_id distributedly. Seller's ad service will + receive an error from SellerFrontEnd if repeated / replayed SelectAd requests are sent + that include the same generation_id in ProtectedAudienceInput. + + ##### Fake requests / Chaffs to DSP + To prevent the seller from determining the buyers on the user's device by observing + network traffic outside the TEE with the help of cloud / network logs, SellerFrontEnd + service sends fake requests (chaff) to "m" buyers not participating in the auction. + The set of "m" buyers are randomly selected from the list of partner buyers of the + seller configured in SellerFrontEnd server configuration, except the buyers receiving + real requests. The number "m" itself will be variable so that buyers on the user's + device (or buyers receiving real requests) are not leaked with iterative requests. + Additionally, "m" will be small (less than 5) so that buyer servers are not overwhelmed + with fake traffic. + + To prevent the seller from filtering buyers receiving chaff by observing response time + outside the TEE with the help of cloud / network logs, the response time from + BuyerFrontEnd to SellerFrontEnd would add random jitter. To prevent the seller from + filtering buyers receiving chaff by observing response packet size outside the TEE + with the help of cloud / network logs, the response payload will be padded. + + #### Auction service + The Auction service runs in the TEE on a supported cloud platform. The service is + based on the gRPC framework and provides a gRPC endpoint [ScoreAds][26], that receives + requests from SellerFrontEnd service to initiate the auction. This service responds + to requests from the SellerFrontEnd service and has limited outbound access over the + public network. + + Auction service allows seller owned code to execute in a custom code execution engine + within the TEE. The custom code execution engine is based on V8 sandbox and has + tighter security restrictions; the sandbox can not log information and doesn't have + disk or network access. + + * Auction server instances load configured data in memory at server startup. This + includes the endpoint from where adtech code modules are prefetched and address + of key hosting services in key management systems; for more details [refer here][28]. + + * Server instances prefetch [code blobs][29] owned by sellers from a cloud storage instance + at server startup and periodically. The code modules are cached in memory by the + custom code execution sandbox. Refer to the [Adtech code execution][30] section for more + details. + * During [B&A Alpha testing][31], code blobs will be prefetched at startup and periodically + from an arbitrary endpoint that is configured in the Auction service configuration. + + * SellerFrontEnd service sends ScoreAds request Auction service for scoring ads and + generating reporting urls. The request payload includes bids from buyers, contextual + signals, scoring signals and other required data. + + * For privacy, Auction service deserializes and splits scoring signals (for all + ads / bids) such that ScoreAd() can only ingest scoring signals required to score the + ad. + + * Auction service dispatches the request to a custom code execution engine with all + inputs required for adtech code execution. + * Within the sandbox, adtech's ScoreAd() code is executed within a separate worker + thread for scoring an ad. However, execution within worker threads can happen in + parallel. Refer to the [Adtech code execution][30] section for more details. + + #### Seller's key/value service + A seller's service receives requests from the [SellerFrontEnd service][12] with lookup keys + available in buyers' bids (such as ad_render_urls or ad_component_render_urls), and + returns real-time scoring signals required for auction. + + _Note: The seller's key/value system may be BYOS key/value Service or TEE based key/value + service depending on the timeline._ + + ![seq_1](images/ba-system-seq-11.png) + + ### Demand-side platform (DSP) system + Following services will be operated by an DSP, also referred to as a buyer. + + #### BuyerFrontEnd service + The front-end service of the system that runs in the [TEE][5] on a supported cloud platform. + The service is based on the gRPC framework and provides a gRPC endpoint [GetBids][32] that + receives requests from SellerFrontEnd service (or partner seller) to initiate bidding flow. + + * BuyerFrontEnd server instance loads configured data in memory at server startup. This + includes the following, for more details [refer here][33]: + * Domain address of other services that BuyerFrontEnd depends on. This includes address + of buyer's key/value service, address of Bidding service and address of key hosting + services in [Key Management System][13]. + + * At server startup, the gRPC or HTTP connections to other services (Bidding service, + buyer's key/value service, cryptographic key hosting services) are prewarmed. + + * The BuyerFrontEnd service receives GetBids request from SellerFrontEnd service of + partner seller. + + * The BuyerFrontEnd service decrypts GetBids request using decryption keys prefetched + from Key Management System. + + * The BuyerFrontEnd service fetches real-time bidding signals from the buyer's key/value + service; this is required for generating bids. The bidding_signal_keys in different + interest groups are batched for the lookup. + * The bidding signal response from key/value service is split by bidding_signal_keys + in interest groups. + + * The BuyerFrontEnd service sends a GenerateBids request to the Bidding service. The + Bidding service returns ad candidates with bid(s) for each interest group. + + * The BuyerFrontEnd returns all bids ([AdWithBid][35]) to SellerFrontEnd service. + + #### Bidding service + The Bidding service runs in the TEE on a supported cloud platform. The service is based on + the gRPC framework and provides an endpoint [GenerateBids][34], that receives requests from + BuyerFrontEnd service to initiate the bidding flow. This service responds to requests + from the BuyerFrontEnd service and has limited outbound access over the public network. + + Bidding service allows buyer owned code to execute in a custom code execution engine + within the TEE. The custom code execution engine is based on V8 sandbox and has tighter + security restrictions; the sandbox can not log information and doesn't have disk or + network access. Refer to the [Adtech code execution Engine][30] section for more details. + + * Bidding server instances load configured data in memory at server startup. This includes + the endpoint from where adtech code modules are prefetched and address of key hosting + services in key management systems; for more details [refer here][36]. + + * Server instances prefetch [code blobs][29] owned by buyers from a cloud storage instance at + server startup and periodically. The code modules are cached in memory by the custom + code execution sandbox. Refer to the [Adtech code execution][30] section for more + details. + * During [B&A Alpha testing][31], code blobs will be prefetched at startup and periodically + from an arbitrary endpoint that is configured in the Bidding service configuration. + + * BuyerFrontEnd service sends GenerateBids request to Bidding service for generating bids. + The request payload includes interest groups, per buyer signals, real time bidding + signals and other required data. + + * For privacy, Bidding service deserializes and splits bidding signals such that + GenerateBid() can only ingest bidding signals required to generate bid(s) per Interest + Group. + + * Within the Bidding service, a request is dispatched to a custom code execution engine / sandbox + with all inputs required for adtech code execution. + * Within the sandbox, adtech's GenerateBid() code is executed within a separate worker + thread for generating bids for an Interest Group. The execution within worker threads + can happen in parallel. Refer to the [Adtech code execution][30] section for more details. + + #### Buyer’s key/value Service + A buyer's Key/Value service receives requests from the [BuyerFrontEnd service][14] and returns + real-time buyer data required for bidding, corresponding to lookup keys. + + _Note: The buyer’s key/value system may be BYOS key/value Service or TEE based key/value + service depending on timeline._ + + ![seq_2](images/ba-system-seq-22.png) + + ## Cryptographic protocol & key fetching + The cryptographic protocol is bidirectional [Hybrid Public Key Encryption][13](HPKE). In this + mechanism, public key and private key pairs are versioned. The private keys have sufficiently + longer validity than public keys for the same version. Public keys are fetched from public + key hosting service and private keys are fetched from private key hosting services in [Key + Management Systems][20]. Each private key is split and fetched from two private key hosting + endpoints in the Key Management Systems, where each system is operated by a [Coordinator][37]. + Refer [here][38] for more details. + + ### Key fetching in client + Client (browser, android) periodically fetches a set of public keys from the public key + hosting service in Key Management Systems every 7 days and are cached client-side with a + fixed TTL in order of days. New versions of public keys are prefetched before the expiration + time of the previous set of keys. + + ### Key fetching in server + Server instances running in TEE prefetch all valid private keys from the Key Management + Systems at service startup and periodically in the non critical path. The private keys are + cached in-memory and have a caching TTL in order of hours; therefore these keys are refreshed + periodically every few hours. This ensures if any private key is compromised for some reason, + that is not in-use for long. Private keys are granted to the TEE based servers only after + attestation. The binary hash of the service and guest operating system running on the virtual + machine is validated against a hash of the open source image through attestation; note that + the attestation mechanism is cloud platform specific. The server instances also prefetch a + set of public keys every 7 days and cache in-memory for days. Public keys are required for + encryption of outgoing requests to other TEE based servers. + + ### Client to server communication + Client to server communication is protected by TLS over the untrusted public network and + the [_ProtectedAudienceInput_][11] is encrypted by [Oblivious HTTP][39] which is a wrapper over + bidirectional Hybrid Public Key Encryption(HPKE). The _ProtectedAudienceInput_ payload is separately + encrypted because the TLS session terminates at the load balancer of front end services in B&A. The + client encrypts the _ProtectedAudienceInput_ payload with a public key and the key version is prefixed + in the ciphertext with Oblivious HTTP. + + ### Server to server communication + Server to server communication over a public unsecured network is protected by TLS and + request-response payload encryption is based on bidirectional [Hybrid Public Key Encryption][13] (HPKE). + Depending on the network topology of services that would communicate over a private network, + request-response payload encryption may be sufficient protection. For TEE to TEE encryption, + the client server would encrypt the request payload with a public key and send the key version + separately along with the request ciphertext. + + ### Request-response encryption and decryption + * Request payload will be encrypted using a public key. + * The key version will be prefixed in the request ciphertext with [Oblivious HTTP][39]. + * The key version will be sent separately along with the ciphertext with bidirectional [HPKE][13]. + * In a TEE based server, client requests will be decrypted using split private keys. + * The private keys must be of the same version as the public key used for encryption. + * Response will be encrypted using bidirectional HPKE, using key material from HPKE context + of the request. + * The HPKE context is used to export a secret key material. + + ## Adtech Code Execution Engine + ### Roma + ![roma](images/roma.png) + + The adtech code execution engine is called [Roma][40] that provides a sandbox environment for + untrusted, closed source, self contained code. Roma provides two backend engines for execution - + 1. [V8][41], Google's open source high-performance JavaScript & WebAssembly engine written in C++. + 2. [Bring-Your-Own-Binary][51], for executing standalone binaries compiled from languages such as C/C++ and Go. ROMA BYOB uses a single instance of a double-sandboxed Virtual Machine Monitor (VMM) called [gVisor][52]. This mode is currently only supported for the Bidding service right now. + + Bidding service and Auction service that executes adtech code, have a dependency on Roma. The frontend + services of B&A do not have a dependency on Roma. The backend mode for bidding server can be specified as a start up configuration. + + Roma is based on a multiprocessing model. When the service (Bidding / Auction) starts + up, a Roma dispatcher process is initialized that forks child processes; each child + process forks processes to spawn the workers. The workers are prewarmed at server + startup and remain alive to process workloads. + + For the V8 backend, there is IPC (inter process communication) + between the gRPC server handler and dispatcher to dispatch requests for code execution + in workers. There is also an IPC channel between dispatcher and workers. The BYOB execution mechanism is explained [here][51]. + + For privacy, it is important to ensure that there is isolation of workers. This means + that workers will not share any memory with each other. + + For reliability of the system, it is important to ensure the following: + * In case a request causes a crash, that would be failed gracefully without disrupting + Bidding / Auction service. + * In case a worker crashes, resources will be freed up and the service will not be + disrupted. + * In a very rare event, if the dispatcher process crashes, the service will be + restarted. + + #### Roma workers + Following is the flow for spawning Roma workers: + * When the dispatcher process starts, that allocates all shared memory segments. + * The dispatcher forks an intermediate child process that deallocates all shared + memory segments, except the memory segment for the target Roma worker. + * The intermediate child process forks a target Roma worker. + * The target Roma worker re-parent to dispatcher and the intermediate child process exits. + * If a worker crashes or the process exits, the dispatcher is notified to clean up + resources. New worker is re-forked. + + There is a shared memory block between the dispatcher and each worker. This would help + avoid data copies and input / output delays and that is important for a low latency + system. Inside the shared memory block, each worker has a message queue for reading + requests and writing responses. Each queue element contains both request and response. + For incoming requests, the dispatcher picks a worker queue that has the least number + of pending requests to put the new item (request) to the back of the queue. However, + the dispatcher follows a round robin placement policy when worker queues have the + same number of pending requests. The processed requests are retrieved from the front + of the queue. Each worker picks pending items from front to the back of the queue, + processes them and marks complete. + + The number of workers is configurable, but by default is set to the number of + vCPU (virtual Central Processing Unit) of the virtual machine where the + Bidding / Auction gRPC application is hosted. The per worker memory limit is also + configurable. Each worker's resource usage is controlled using Linux kernel control + groups. + + ### Security guarantees + * Adtech code executing in Roma or the sandbox code, will not have outbound network + access from the sandbox. + * Bidding / Auction services that depend on Roma, will have limited outbound access. + * These services can fetch cryptographic keys from Key Management Systems that are + part of the trust circle. + * These services can respond to TEE based frontend services in B&A. + * These services can fetch code blobs from cloud storage buckets. + * During B&A Alpha testing, these services can fetch code from an adtech provided + endpoint configured in Bidding / Auction service configuration. + * Input / output (logging, disk or file access) will be disabled within the Roma + sandbox. + * Memory core dump will be disabled in Roma to prevent sensitive data leak. + * Roma will protect the confidentiality of the adtech code and signals during the + server side execution in a trusted environment. + * Any potential vulnerability (like v8 vulnerability) may cause the Adtech code to + escape the isolated worker. To mitigate this, the worker is further restricted with + Linux kernel partition, resource isolation and security mechanisms (i.e. Linux + namespaces, control groups and secure computing mode). + + ### Code properties + Following are the properties of code executed in Roma. + + * Code is adtech owned, not open-source. + * Code is not encrypted. + * The code is self contained and will be executed within an isolated worker thread. + * The output of the code can only be part of the encrypted response sent to frontend + services in B&A. + * The code blob size can vary per adtech. + * For V8 backend - The code can be in Javascript or WASM instantiated with Javascript. + * Most languages like Java, C++, C# can be compiled to WASM bytecode. + * For BYOB backend - The code can be a standalone binary built from C/C++ and Go. + + ### Code Execution + The code modules are fetched by the Bidding / Auction service in the non-request path, + then dispatched to Roma along with the code version (in case of multiple code versions). + The code is cached in Roma. In the request path, the code is executed in a Roma worker + in isolation. The code is executed in parallel in different Roma workers. There is no + state saved between two executions in the same worker. There is no shared memory resource + between executions in different workers. + + **Every worker has its own copy of code. However, the number of copies of code at any point + is bound by the number of Roma workers in Bidding / Auction service**. + + Following modes of code execution are supported: + + #### Javascript Execution + Javascript can be executed in a Roma worker with the V8 backend. The Javscript code will be prefetched by Bidding / Auction + service at server startup and periodically in the non critical path. + + Javascript code is preloaded, cached, interpreted and a snapshot is created. A new context is + created from the snapshot before execution in a Roma worker. + + #### Javascript + WASM Execution + WASM bytecode can be executed in a Roma worker along with Javascript with the V8 backend. The Javscript and WASM binary will + be prefetched by Bidding / Auction service at server startup and periodically in the non critical path. + The WASM binary must be delivered with an application/wasm MIME type (Multipurpose Internet Mail Extensions). + + When Javascript and WASM are provided for an execution, it is recommended that the adtechs inline + WASM in the Javascript code. However, if the WASM code is not inlined, Bidding service would + take care of that with a Javascript wrapper that inlines the WASM. + + Javascript code is preloaded, cached, interpreted and a snapshot is created. The WASM is not included in + the Javascript snapshot. + + The [WASM module][42] will be instantiated synchronously using [WebAssembly.Instance][43]. + A WebAssembly.Instance object is a stateful, executable instance of a WASM module that + contains all the [Exported WebAssembly functions][44] that allow invoking WebAssembly code + from JavaScript. + + #### WASM Execution + In this case, no Javascript is required. + + WASM bytecode can be executed in a Roma worker with the V8 backend. The WASM binary will be prefetched by Bidding / Auction + service at server startup and periodically in the non critical path. The WASM binary must be delivered with + an application/wasm MIME type. + + The WASM module is preloaded and cached in Roma in the non request path. WASM is parsed for every execution + within a Roma worker. + + #### Binary Execution + A standalone binary compiled as per [specifications][55] from languages such as C/C++ and Go can be executed in a Roma worker with the BYOB backend. The binary will be prefetched by Bidding / Auction + service at server startup and periodically in the non critical path. The binary must be developed as per the [specifications][53] for receiving input and providing output ([Example][54]). + + ### Execution in Bidding / Auction services + + #### Execution within Bidding service + [GenerateBid()][45] will be executed per interest group in parallel and in isolation. For + every [GenerateBids][34] request, a dispatch request is sent to Roma that batches all the + inputs. Roma will execute pre-initialized GenerateBid() per interest group in separate + worker processes in parallel. + + ReportWin() will be executed in a Roma worker to generate reporting url for the buyer. + + #### Execution within Auction service + [ScoreAd()][46] will be executed per ad (bid) in parallel and in isolation. For every + [ScoreAds][26] request, a dispatch request is sent to Roma that batches all the inputs. + Roma will execute pre-initialized ScoreAd() per ad (bid) in separate worker processes + in parallel. + + ReportResult() will be executed in a Roma worker in the Auction service to generate a + reporting url for the seller. + + ### Code blob fetch and code version + #### Overview + Adtech will need to provide code modules for bidding, scoring and reporting url + generation. The Bidding / Auction services will prefetch code modules in the + non-request path periodically from AWS S3 and GCP GCS buckets, configured by + adtechs. The code modules will be loaded in Roma and cached. + + The following is the layout for the Auction service for scoring ads, an identical + layout will exist for the Bidding service. + + ![code_blob1](images/blob_fetch21.png) + + #### Lifecycle of the code storage bucket + The following is an overview of the lifecycle of a bucket update. The Bidding or Auction + service queries the bucket for its objects (code modules) and then updates Roma with code + module additions and code module deletions. + + ![code_blob2](images/blob_fetch22.png) + + #### Code module fetch + *For [Bidding and Auction services alpha testing][31], the code modules can be prefetched from + arbitrary endpoints provided by the adtechs. The url endpoints must be configured in Bidding / + Auction server configuration. If an adtech requires both Javascript and WASM, they must provide + two different url endpoints that will be set in Bidding / Auction service configuration. The + Bidding / Auction services would prefetch adtech code modules at server startup and periodically + every few hours.* + + Beyond alpha testing, a publisher-subscriber based messaging system will be supported + so that any time the adtech updates the bucket, the service will receive a notification + and load the new code modules. + + The following portrays the cloud architecture of a pub/sub system in [AWS][47] (a similar + system will be supported for [GCP][48]). + + In AWS, an EC2 instance has its own notification queue (SQS) and changes to the S3 bucket + are what trigger the notification system (SNS). The adtech will add or remove a code module + in the storage bucket, and Bidding / Auction services will eventually receive a notification + to check the bucket; they will subsequently load any updates so that the running Roma + instances match the state of the modules in the bucket. + + ![code_blob3](images/blob_fetch23.png) + + #### Code version flow + * Adtechs can store multiple versions of code modules in Cloud Storage. + * Adtechs can pass the code version used by code modules in the [SelectAd][24] request. + * Buyers can return the code version used by code module for the request + * Bidding / Auction service fetches code objects from the Cloud Storage instance (bucket) + at service startup and periodically fetches the code objects every n hours. + * Roma dispatcher API would expose a Load() to accept a code object along with + _code_version_id_. + * In non critical path: + * Every n hrs, the service compares the list of current object names (code versions) + in the bucket to what it has loaded into Roma. + * Based on the difference, the service adds new code objects to Roma and evicts + non-existent objects from Roma. + * In their request, the adtech may specify the code version (object name such as v1, v2 + and so on). Roma will be able to service parallel requests specifying different code + versions. + * In the request path, grpc service would dispatch the workload with inputs required for + execution of cached code. + * For Bidding / Auction services crashes / loses state, the latest code can be fetched + when the service restarts. + + ## Reporting + Event level win reporting urls will be generated on the server side and sent to the + client (browser / android), so that the client can ping the reporting endpoints of the + Adtech. + + More details about reporting will be published in a different explainer. + + ## K-Anonymity Integration + An explainer will be published to cover details on K-Anonymity Integration. + + ## Concurrency + The concurrency model is based on asynchronous thread pool and callbacks. The thread pool + library is based on [EventEngine][49]. The Input / Output (IO) processing model is asynchronous IO. + + ## Cloud Deployment + + ### Network Topology + * Seller and Buyer may operate services in different cloud platforms. + * Services within the Seller system (i.e. SellerFrontEnd, Auction) will be co-located and + share a VPC network. The communication between SellerFrontEnd service and Auction service + is over a private network. + * Services within the Seller system (i.e. BuyerFrontEnd, Bidding) will be co-located and + share a VPC network. The communication between BuyerFrontEnd service and Bidding service is + over a private network. + + ### Availability + Bidding and Auction services will support multi regional replication. An adtech deploying the + server instances to a supported cloud platform, would decide the availability of these services. + + ## Related material + * [Bidding and Auction services high-level design and API explainer][4] + * [Bidding and Auction Services Multi Seller Auctions](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_multi_seller_auctions.md) + * [Bidding and Auction services AWS cloud support and deployment guide](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_aws_guide.md) + * [Protected Audience services](https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md) + * [Chrome client design for Bidding and Auction services integration](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md) + + +[1]: https://github.com/chatterjee-priyanka +[2]: https://github.com/dankocoj-google +[3]: https://github.com/akundla-google +[4]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md +[5]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trusted-execution-environment +[6]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#supported-public-cloud-platforms +[7]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#unified-contextual-and-fledge-auction-flow-with-bidding-and-auction-services +[8]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#specifications-for-adtechs +[9]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#service-apis +[10]: https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md +[11]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#protectedaudienceinput +[12]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#sellerfrontend-service +[13]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#key-management-systems +[14]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#buyerfrontend-service +[15]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#bidding-service +[16]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#buyers-keyvalue-service +[17]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#auction-service +[18]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#sellers-keyvalue-service +[19]: https://www.terraform.io/ +[20]: https://datatracker.ietf.org/doc/rfc9180/ +[21]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#metadata-for-filtering-in-buyer-keyvalue-service +[22]: https://www.envoyproxy.io/ +[23]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#auctionresult +[24]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#sellerfrontend-service-and-api-endpoints +[25]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#sellerfrontend-service-1 +[26]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#auction-service-and-api-endpoints +[27]: https://en.wikipedia.org/wiki/Universally_unique_identifier +[28]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#auction-service-1 +[29]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#adtech-code +[30]: #adtech-code-execution-engine +[31]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#alpha-testing +[32]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#buyerfrontend-service-and-api-endpoints +[33]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#buyerfrontend-service-1 +[34]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#bidding-service-and-api-endpoints +[35]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#adwithbid +[36]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#bidding-service-1 +[37]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#deployment-by-coordinators +[38]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#system-overview +[39]: https://datatracker.ietf.org/wg/ohttp/about/ +[40]: https://github.com/privacysandbox/control-plane-shared-libraries/tree/main/cc/roma +[41]: https://v8.dev/ +[42]: https://developer.mozilla.org/en-US/docs/WebAssembly/JavaScript_interface/Module +[43]: https://developer.mozilla.org/en-US/docs/WebAssembly/JavaScript_interface/Instance +[44]: https://developer.mozilla.org/en-US/docs/WebAssembly/Exported_functions +[45]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#generatebid +[46]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#scoread +[47]: https://aws.amazon.com/what-is/pub-sub-messaging/ +[48]: https://cloud.google.com/pubsub +[49]: https://grpc.github.io/grpc/core/classgrpc__event__engine_1_1experimental_1_1_event_engine.html +[50]: https://developers.google.com/privacy-sandbox/relevance/protected-audience/android/bidding-and-auction-services +[51]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/roma_bring_your_own_binary.md +[52]: https://gvisor.dev/ +[53]: https://github.com/privacysandbox/data-plane-shared-libraries/blob/619fc5d4b6383422e54a3624d49a574e56313bc8/docs/roma/byob/sdk/docs/udf/Communication%20Interface.md +[54]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/roma_bring_your_own_binary.md#example +[55]: https://github.com/privacysandbox/data-plane-shared-libraries/blob/619fc5d4b6383422e54a3624d49a574e56313bc8/docs/roma/byob/sdk/docs/udf/Execution%20Environment%20and%20Interface.md + + + + + + + + + + + + + + + + diff --git a/cross_cloud_coordinators.md b/cross_cloud_coordinators.md new file mode 100644 index 0000000..bdf79d2 --- /dev/null +++ b/cross_cloud_coordinators.md @@ -0,0 +1,196 @@ +_Note: the explainer is posted in the Protected Auction repository, but applies to all Privacy Sandbox services._ + +# Cross Cloud Coordinators + +## Background + +The Privacy Sandbox Coordinator model has been introduced as part of [processing aggregatable reports](https://github.com/WICG/attribution-reporting-api/blob/main/AGGREGATION_SERVICE_TEE.md) in a privacy preserving way; the aggregatable reports are generated by the [Chrome Attribution Reporting API](https://github.com/WICG/attribution-reporting-api/blob/main/AGGREGATE.md). The use of Coordinator Services in Privacy Sandbox has since been extended to include [Protected Auction Services](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/trusted_services_overview.md). + +The current Coordinator Services consist of several components: (1) private/public encryption key generation, (2) workload attestation and private/public encryption key distribution, (3) authentication and authorization management, and (4) aggregatable report accounting (used by [aggregation service](https://github.com/privacysandbox/aggregation-service). + +A split trust model has been implemented where decryption keys are split across two independent deployments of the coordinator services where each deployment is operated by an independent party ([Aggregation Service AWS Coordinators](https://www.accenture.com/us-en/support/google-privacy-sandbox)). Currently, the coordinator services serving workloads on a specific public cloud (e.g. AWS) are operated in the same public cloud. + +## Key Terms + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Coordinator

Operator of coordinator services. Each coordinator deploys and operates the necessary components (Type-1 or Type-2) to support ad tech workloads.

Coordinator pair

An operated pair of coordinator services (Type-1, Type-2) in the same public cloud.

Ad tech

A company that provides services to deliver and measure ads. In this explainer it is used to describe the entity running a Privacy Sandbox trusted service workload compiled from open source code and interacting with Coordinator Services.

Attestation

A mechanism to authenticate software identity, usually with cryptographic hashes or signatures. In this explainer, attestation matches the code running in the coordinator-operated key-generation service or adtech-operated workload service with open source code.

Note: this is not the Privacy Sandbox Enrollment Attestation.

Trusted execution environment (TEE)

A secure, private, and isolated execution context. The TEE's contents are protected from observation and tampering by unauthorized parties, including the root user. The requirements for supported public TEE implementations can be found here.

Multi-party key generation and distribution service (MPKGDS)

Formerly known as Key Management System (KMS).
Service responsible to

+
    +
  1. securely generate public/private key pairs and securely distribute the private key splits among the two independently operated coordinators.
  2. +
  3. securely distribute public keys to user and workload clients.
  4. +
  5. securely distribute private key splits to authenticated, authorized and attested workloads running in a TEE.
  6. +
+

Aggregatable Report Accounting

A distributed ledger that tracks and limits the number of times a given aggregatable report sharedID can be processed to generate noised summary reports. The number of times allowed is referred to as the "budget" for the given sharedID.

The aggregatable report accounting budget is maintained in both Type-1 and Type-2 coordinator deployments.

Cloud KMS

Key Management Service offered by a public cloud provider as a managed service for hardware or software backed symmetric and asymmetric encryption keys

+ +## Current Coordinator Services + +**The Type-1 coordinator** deploys the following components: (1) private/public encryption key generation and public key distribution, (2) workload attestation and private encryption key distribution, (3) authentication and authorization management, and (4 - use-case dependent) aggregatable report accounting (used by [Aggregation Service](https://github.com/privacysandbox/aggregation-service)). + +**The Type-2 coordinator** deploys the following components: (1) workload attestation and private encryption key distribution, (2) authentication and authorization management, and (3 - use-case dependent) aggregatable report accounting (used by [Aggregation Service](https://github.com/privacysandbox/aggregation-service)). + +### Multi-Party Key Generation + +The multi-party key generation service generates a [set of public/private](https://github.com/WICG/attribution-reporting-api/blob/main/AGGREGATE.md#encrypted-payload) asymmetric [HPKE](https://datatracker.ietf.org/doc/rfc9180/) keys. The private keys are split and stored independently in the respective Type-1 and Type-2 coordinator deployment. To ensure keys are generated, distributed and stored securely, the key generation service running on the Type-1 coordinator is (1) running inside a TEE, (2) attested by the Type-2 coordinator to run known open sourced code, and (3) does not store the Type-2 private key split in the Type-1 operated storage systems. The TEE based key generation creates a configurable count of public / private key pairs (in current deployed systems, 5 key pairs are generated) which are rotated within a configurable time window (in current deployed systems public keys are valid for 7 days, private keys are served between 90 and 365 days if not compromised (revoked)). Public keys and the Type-1 private key split are stored in the Type-1 coordinator managed storage systems, the private key split is encrypted with the cloud provider managed key management service (cloud KMS) before storage. The Type-2 private key split is securely sent to the Type-2 coordinator from within the TEE before it is discarded from the TEE memory. The Type-2 coordinator encrypts the Type-2 private key split with the cloud provider managed key management service before storage. + +### Public Encryption Key Distribution + +The Type-1 coordinator distributes the current valid public keys through a CDN (content delivery network) backed endpoint. Caching Headers are dynamically set to limit client caching beyond the key's validity and allow for cache invalidation in the case of compromised keys and out-of-cycle key rotations. + +Serving endpoints follow the following format: https://publickeyservice.[use-case-area].[workload_cloud].privacysandboxservices.com/.well-known/[workload-use-case]/v1/public-keys + +E.g. [https://publickeyservice.msmt.aws.privacysandboxservices.com/.well-known/aggregation-service/v1/public-keys](https://publickeyservice.msmt.aws.privacysandboxservices.com/.well-known/aggregation-service/v1/public-keys) + +With the following content format: +```json +{ + "keys": [ + { + "id": "[arbitrary string identifying the key (up to 128 characters)]", + "key": "[base64 encoded public key]" + }, + // Optionally, more keys. + ] +} +``` + +### Private Encryption Key Split Distribution + +Private decryption keys are required for workloads to process the data encrypted with public encryption keys for the specific workload privacy sandbox APIs. + +Private decryption key splits are encrypted with a respective coordinator managed cloud KMS key. Type-1 coordinator manages a symmetric encryption key to encrypt the Type-1 private key split, Type-2 coordinator manages a symmetric encryption key to encrypt the Type-2 private key split. Onboarded ad tech credentials can fetch encrypted key splits from the authenticated private key distribution endpoint operated by each coordinator (Type-1 and Type-2). + +### Workload Attestation and Private Decryption Key reconstruction + +Decryption of the encrypted private key splits is only authorized within TEE workload images whose cryptographic hash matches one of the hashes on the coordinator maintained allowlist and running on supported TEE hardware/software stack verified through workload attestation. + +Once a workload attests successfully, it will be granted permission to use the respective coordinator managed cloud KMS key to decrypt ("unwrap") the Type-1 and Type-2 private key splits. Once both private key splits are successfully decrypted, they can be combined to reconstruct the full private HPKE decryption key. + +### Aggregatable Report Accounting + +The Aggregatable Report Accounting ledger is used for accounting to ensure encrypted payloads of aggregatable reports are not processed more often than the applicable budget limit. To do this, it uses privacy budget keys as received from the client that needs to consume the budget. + +If the budget has not already been exhausted for the specified privacy budget keys, the transaction request is accepted, budget consumption is recorded in the ledger and a success code is returned to the client. If both transaction requests to each of the coordinators (Type-1 and Type-2) return successfully, the client can proceed with using the data to generate results.. If at least one of the transaction requests to the coordinators fails, the client fails the job, without generating a result. + +The Aggregatable Report Accounting ledger transaction endpoint is authenticated with onboarded ad tech credentials. Each transaction request carries an authentication header, which is used for validating authentication and verifying authorization matching the credentials to only allow consuming budget for data the client owns. + +### Authentication / Authorization for Ad Tech workloads + +Each coordinator deployment (Type-1 and Type-2) maintains per ad tech workload authentication credentials and permissions independently. + +#### Authentication + +Credential management depends on the credential management system supported by the respective public cloud provider. Each ad tech requires onboarded credentials on both coordinator deployments (Type-1 and Type-2) to have their deployed workloads be able to interact with the coordinators. + +#### Authorization + +The following endpoints are authenticated / authorized to onboarded ad techs: + +- Encrypted private key split endpoint +- Cloud KMS key decryption - this endpoint additionally requires successful workload attestation for permission to use decryption keys +- (use-case dependent) aggregatable report accounting endpoint + +### Attestation + +Workload attestation is integrated with the respective public cloud provider's TEE platform and enforces access to cloud KMS decryption keys to allowlisted workload image hashes only. Access to the cloud KMS decryption keys is required to decrypt the respective coordinator private key splits. + +## Proposed Changes to support cross cloud coordinators + +As outlined in the [call for feedback](https://github.com/WICG/protected-auction-services-discussion/issues/69) to consolidate coordinator services in one cloud, the key changes of this proposal are to operate the following services, consolidated in one cloud and serving workloads running in any of the [supported workload public clouds](./public_cloud_tees.md): + +1. private/public encryption key generation and public key distribution +1. wrapped private encryption key distribution +1. authentication and authorization management +1. (use-case dependent) aggregatable report accounting service + +Workload attestation, private key split decryption with cloud KMS and therefore required authentication / authorization management will stay in the respective public cloud provider where the workload is running. + +If the ad tech workload is operated in a public cloud different from where the coordinator services are deployed, a cross-cloud dependency will be added to enable communication with the consolidated coordinator components, and identity federation will be used for cross cloud authentication to obtain coordinator cloud credentials. Authorization permissions will be set on the federated identity. + +### Multi-Party Key Generation + +The multi-party key generation will run in the coordinator public cloud and attestation of the key generation binary will be facilitated in the same public cloud the key generation is running in. The private key splits will be encrypted with the supported workload public cloud KMS service. E.g., a coordinator pair running on _public-cloud-a_ serving workloads running in _public-cloud-b_ will encrypt the private key splits with keys managed in _public-cloud-b_ KMS. Public keys and encrypted private key splits will be stored in the same public cloud the key generation runs in. + +### Public Encryption Key Distribution + +Encryption keys for any supported public cloud will be served from the public cloud where the coordinator services run in. + +### Private Encryption Key Split Distribution + +Private decryption key splits encrypted with the respective cloud KMS of the public cloud where the workload is running in will be distributed to authenticated (through identity federation if applicable) workload clients of onboarded ad techs. + +### Workload Attestation and Private Decryption Key reconstruction + +Workload attestation and private key split decryption will stay in the respective cloud where the workloads are running. + +### Aggregatable Report Accounting + +The aggregatable report accounting service will be operated from the coordinator public cloud. The flow to consume privacy budget will stay the same. Federated credentials will be used for authentication and authorization when the workload is running on a public cloud that is different from that of the coordinator services. + +### Authentication / Authorization for Ad Tech workloads + +Authentication of ad tech workload credentials will be done through identity federation where credentials can be exchanged for federated credentials in the public cloud where the coordinator services are operated. Coordinators will configure Identity Provider (IdP) mappings for supported IdPs into federated credentials for onboarded ad techs. + +Ad tech onboarding information will be maintained in both clouds, (i) the public cloud where the coordinator endpoints are running for identity federation and authorization and (ii) in the workload cloud for authentication and authorization to the workload cloud KMS. + +#### Authentication + +Authentication against coordinator provided endpoints and public cloud endpoints where the coordinator is operated in will use federated credentials where applicable (for all workloads running in a different public cloud to the coordinator public cloud). + +#### Authorization + +Authorization will be based on federated credentials and permissions will be configured on the on the federated credentials for all onboarded ad techs + +## Privacy Considerations + +Ad techs will continue to operate on their choice of supported cloud (AWS workloads will stay on AWS, GCP workloads will stay on GCP). This change will impact the coordinator services, where the cryptographic keys are held and the aggregatable report accounting ledger (relevant only for Aggregation Service) is operated. + +We will continue to split the operation of the coordinator services between Google and an independent third party. + +Ad tech workloads that are running in a supported public cloud that is different from the public cloud the coordinator is running in will have an added cross cloud dependency and metadata (onboarding data - account / credential id, reporting sites, aggregatable report accounting ledger data) maintained in the coordinator's public cloud. + +## Security Considerations + +With identity federation, the coordinators will configure Identity Provider (IdP) mappings to credentials in the public cloud the coordinators are operated in. Initially, the IdPs of supported public clouds will be trusted. Support for other trusted OIDC compliant IdPs may be added at a later date. Supported IdPs are expected to perform secure credential verification to obtain the respective IdPs credential tokens. These tokens can then be exchanged for federated credential tokens in the coordinator public cloud for onboarded credential mappings. Further requests to endpoints in the coordinator public cloud will use the federated credential tokens. + +## Latency / Cost Considerations + +We expect minimal impact on ad techs running on any supported public cloud other than coordinator public cloud. + +Calls to the coordinator services are not in a latency critical path. For [Protected Auction Services](./trusted_services_overview.md) keys are fetched during startup and refreshes are done asynchronously, for [Aggregation Service](https://github.com/privacysandbox/aggregation-service) batch processing will be minimally affected by small increases of latency on key fetches (keys are cached for 8hrs) and PBS calls. + +Cost increases are expected to be minimal as calls to the coordinator services are infrequent and low data volume. + +## Availability / Disaster Recovery Considerations + +We expect to improve availability and disaster recovery measures through increased focus and resources to address these concerns. Resources that previously were split to maintain code bases to support deployment and operation of coordinator services in several clouds will now be able to concentrate on improving the reliability, availability, and resiliency of coordinator services focused on supporting deployment and operation in one public cloud. diff --git a/debugging_protected_audience_api_services.md b/debugging_protected_audience_api_services.md new file mode 100644 index 0000000..ff343d3 --- /dev/null +++ b/debugging_protected_audience_api_services.md @@ -0,0 +1,278 @@ +## Debugging Protected Audience API Services + +**Authors:**
+[Akshay Pundle][11], Google Privacy Sandbox
+[Michael Mihn-Jong Lee][12], Google Privacy Sandbox
+[Xing Gao][13], Google Privacy Sandbox + +The Protected Audience API ([Android][1], [Chrome][2]) proposes multiple +real-time services (e.g. [Bidding and Auction][3] and [Key/Value][4] services) +running in a [trusted execution environment][5] (TEE). These are isolated +environments for securely processing sensitive data with limited data egress. +Debugging is essential for running a production system. Reproducing issues and +finding root causes is essential to the iterative process of making a high +quality robust software. An isolated system running in a TEE poses unique +debugging challenges due to limited data inspection. + +In this document, we propose various methods of debugging TEE servers while +retaining the privacy characteristics. We introduce three ways for AdTechs to +debug the system: + +- **AdTech consented debugging:** Provides a way for AdTechs to gain access to + debugging data and requests that can be replayed on local systems +- **Local, Debug mode(`non_prod`):** Provides a way for AdTechs to use standard debugging + tools. This can be used along with AdTech consented debugging to replay and + debug requests. +- **Aggregate error reporting:** Provides aggregate error counts for tracking + errors in production systems. + +## AdTech consented debugging + +AdTech consented debugging enables AdTechs to get plaintext request and response +pairs, and other debugging data (e.g., verbose logging messages) from their +servers. The Protected Audience API normally protects this data and it is not +available to the AdTechs, or anyone else other than the user who owns the data. +This feature is aimed at AdTech developers looking to debug the system, and not +regular users of the system or API. + +The main use case here is when an AdTech wants to debug issues, they (the AdTech +developer) act as the user, and grant consent on their client by initiating a +special mode in Chrome or Android, thereby making their request and associated +data available for debugging. A user (in this case, the AdTech developer) is +fully in control of their Protected Audience API data, and chooses to make it +available for debugging; debug reporting will not be provided for requests from +any users besides those who have enabled this special mode. This consent allows +the system to collect the request and response pair in plaintext and the debug +data, and provide it to the AdTech. + +The production system collects logs, metrics, plaintext requests and other +details for such consented requests and makes them available to the AdTech. +Privacy impacting metrics (see [Monitoring Protected Audience API Services][6]) +that are normally noised to protect privacy, will be collected without noise for +these consented requests. + +With logs and metrics, AdTechs have more information to find the source of the +problem. They will also have the plaintext request that their server was called +with. This means they can replay this request on a local (or debug) instance of +the server, examine logs and use other standard debugging tools for finding +misbehaving code and performance problems. This also means there is no need to +replicate the entire set of servers involved in serving the Client request, +since AdTech already has the request their servers were called with. + +### Data flow + +When an AdTech wants to reproduce a problem, they will start Chrome or Android +and give consent via the UI. This will enable the system to collect debug data. +While giving consent, the AdTech developer will also need to provide a secret +debug-token known only to the AdTech. This will ensure that the set of consented +requests is only available for debugging on that particular AdTech's servers. + +![Data flow](images/debugging_protected_audience_api_services_data_flow.png) + +Subsequent Protected Audience API requests from this client will include the +consent and the secret debug-token in request to Protected Audience API servers. +Consent and the secret debug-token will be encrypted as part of the request +(request internals are not visible outside the TEE servers). If the server calls +other servers, the consent information will propagate to all Protected Audience +API TEE servers involved. + +The AdTech will provide each Protected Audience API server the secret +debug-token at startup. Upon receiving the request, the server will match the +secret debug-token from the request with the secret debug-token it was started +with. If the tokens match, the server will treat the request as a consented +request. Otherwise the request will be treated as a normal request. The consent +will be propagated to other Protected Audience API TEE servers in either case +(whether the request is treated as consented or otherwise) + +When a server determines a request to be consented, it will collect logs, +granular metrics and the plaintext request and response. These will be published +to the AdTech's cloud store, where they can access this information. Publishing +this will happen through [OpenTelemetry][7] and will use the same channels as +mentioned in the [Monitoring Protected Audience API Services][6] document. +Privacy impacting metrics that are usually exported with noise will be published +without noise for consented requests. + +### Reasons for using a secret debug-token + +- A secret debug-token helps filter out requests originating from other + AdTechs who may be debugging at the same time. This prevents intentional or + unintentional spam that would have made it difficult to isolate requests + that were sent by that particular AdTech. +- Without effective filtering, the associated cloud storage for an AdTech may + get filled with requests from other AdTechs, even though they didn't do any + debugging. + +### Provide Consent + +AdTech developers can give consent in Chrome and Android to make data from their +client available for debugging. Care will be taken to prevent abuse of the +setting. Note that this feature is targeted at the AdTech developers, not the +general Chrome or Android user. + +- Chrome: AdTech can provide the consent at + `chrome://flags#protected-audience-debug-token` with a debug token. + +![Chrome UI](images/debugging_protected_audience_api_services_chrome_ui.png) + +- Android: To enable AdTech consented debugging on your Android device follow the instructions + [here](https://github.com/privacysandbox/dev-tools/blob/main/adservices_cli/README.md#adtech-consented-debugging). + +### Configure TEE Servers + +To enable the AdTech consented debugging, every Protected Audience API TEE +server should run with the following runtime flags: + +``` +ENABLE_OTEL_BASED_LOGGING = "true" +CONSENTED_DEBUG_TOKEN = "secret debug token" +# Enables verbose logging messages. Set to "0" to disable. +PS_VERBOSITY = "10" +``` + +The value of `CONSENTED_DEBUG_TOKEN` should match the debug token set in the +device. If set, the value must be a string of length 6 or more characters. + +The behavior of `CONSENTED_DEBUG_TOKEN` is as following: + +1. Setting to an empty string: This will turn consented debugging off for that server. +2. Setting to a string of length 6 or more characters: This causes consented debugging to be turned on, with the given value as the debug token for that server. +3. Setting to a non-empty string less than 6 characters will prevent the server from starting. + + +### Access Debug Data + +Verbose logs, Event Message, and Adtech UDF Log are available. See more in [Debugging data](#debugging-data). + +Metrics for consented request is exported without noise. + +## Local debugging and Debug mode(`non_prod`) + +The Protected Audience API services will be runnable on the local machine. This +means it will be possible to set up a local instance (e.g. [B&A][8], [K/V][9] +services) of a Protected Audience API server, send plaintext requests collected +using the debugging mechanism described above, and use standard debugging tools +(since this will not be running inside a TEE). This includes examining system +logs, stack traces etc. Such a locally running server will not be able to +decrypt or respond to production traffic. + +The above described system of AdTech consented debugging gives the AdTech access +to the requests in plaintext that their servers were called with. They can then +set up local instances of servers and use these requests for further debugging +and reproducing the problem. + +### Debug mode(`non_prod`) + +TEE servers provide a special execution mode for debugging called Debug mode(`non_prod`). In +this mode, the server runs inside the TEE, but cannot attest with the production +[key management systems][10], so they cannot decrypt any production traffic. +This mode provides some debug capability like console logs. Debug mode(`non_prod`) is useful +for testing a more realistically deployed system compared to local testing. The +above method of replay requests can also be used with servers launched in the +Debug mode(`non_prod`). + +### Access Debug Data + +Verbose logs, Event Message, Adtech UDF Log and Debug info are available. See more in [Debugging data](#debugging-data). + +## Aggregate error reporting + +We will provide cumulative error counts per server. These will include request +level and system errors. We will not support errors directly originating from +AdTech provided javascript execution due to security reasons. Aggregate errors +will also be published using [OpenTelemetry][7] as mentioned in +[Monitoring Protected Audience API Services][6]. Error counts may also be noised +depending on the source of the error. For consented debug requests, the true +error counts will be reported without noise. + +The following errors will be available for tracking: + +- **Number of Requests responded:** The number of requests that were not + processed successfully and resulted in error. +- **Number of Requests responded with error, partitioned by type of error:** + Similar to the above metric, but the data will be at the gRPC status + granularity. +- **Number of requests initiated by this server that resulted in an error, + partitioned by type of error:** This will count the number of errors that + happened in requests initiated by the server (available at gRPC status + granularity). E.g. for SFE, this would include errors that occurred when + calling BFE servers. +- **Number of requests initiated by this server that resulted in an error, + partitioned by destination:** Similar to the above, but partitioned by the + request destination. + +## Debugging data +Bidding and Auction servers can be built in 2 [build_flavor](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/BUILD#L163), `prod` and `non_prod`. Only `prod` images can pass attestation and work in production (i.e. decrypt live traffic requests). +production request, but `non_prod` have more debugging methods with requests with fake encryption. + +Debugging data consists of the following: +1. Verbose logs + + This is the verbose logs from the B&A code. + - Export channel + - In `prod`, verbose logs are exported through [Open Telemetry log](https://opentelemetry.io/docs/concepts/signals/logs/), which can be viewed in cloud log explorer. + - In `non_prod`, they are also printed in stderr. + + - Log types + - System logs: + These logs are not related to user device request, they are always exported in both `prod` and `non_prod` builds. + - Request logs: + These logs are the result of server processing encrypted request. In `prod` builds, these are only exported for [AdTech consented debugging](#adtech-consented-debugging) requests. In `non_prod` builds, they are always exported. + + Each server has `PS_VERBOSITY` flag, which sets the verbosity level of the output logs. A higher verbosity will give more logging data, but can also slow down request processing. This is especially relevant in `non_prod` builds because there can be large volume of request logs. At high QPS, we recommend setting `PS_VERBOSITY` `<=3`. + +2. Event message + + Request and response from the server can also be exported in a structured format. The following [structured proto message](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L1582) defines the proto for these. + It is written to cloud storage through Open Telemetry log collector([example config](https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/gcp/terraform/services/autoscaling/collector_startup.tftpl)) for [AdTech consented debugging](#adtech-consented-debugging). + +3. Adtech UDF Log + + This is the log from `console.log` from javascript including `generateBid` `ScoreAd` and `reportResult`. They are only exported for [AdTech consented debugging](#adtech-consented-debugging), through both Request logs and Event message. + +4. Debug info in server response + + This includes request logs and event message mentioned above. + It is available in `non_prod` only, when [debug_info bit](https://github.com/privacysandbox/data-plane-shared-libraries/blob/e852bce2b8a11a52154103dd25949556e8a4f09e/src/logger/logger.proto#L33) is turned on in server request, Debug Info will be included in [server response](https://github.com/privacysandbox/bidding-auction-servers/blob/e40a4fccdce168379189ab7b6b87b55b1e3f736d/api/bidding_auction_servers.proto#L748). + + To use this feature, [secure_invoke](https://github.com/privacysandbox/bidding-auction-servers/tree/main/tools/secure_invoke) + can be used to send request with debug_info bit. + +## FAQ + +### How to activate Adtech UDF logs + +For both buyer and seller: + +**On server** + +- telemetry_config is **not** `mode: OFF` +- enable_otel_based_logging=true +- consented_debug_token=$YOUR_TOKEN +- ps_verbosity>=3 if you want to see UDF log in Verbose logs( Request logs ) + +**In SFE request (Chrome)** + +- Set the token and set it to enable (chrome://flags#protected-audience-debug-token). Effectively you want this in request: + +``` +"raw_protected_audience_input": { + "consented_debug_config": { + "is_consented": true, + "token": "$YOUR_TOKEN" + }, +``` +`consented_debug_config` will be populated to all B&A servers, including from seller(SFE) to buyer(BFE). + +[1]: https://developer.android.com/design-for-safety/ads/fledge +[2]: https://developer.chrome.com/docs/privacy-sandbox/fledge/ +[3]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md +[4]: https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md +[5]: https://en.wikipedia.org/wiki/Trusted_execution_environment +[6]: https://github.com/privacysandbox/fledge-docs/blob/main/monitoring_protected_audience_api_services.md +[7]: https://opentelemetry.io/ +[8]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_aws_guide.md#local-testing +[9]: https://github.com/privacysandbox/fledge-key-value-service/blob/main/docs/deploying_locally.md +[10]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#key-management-systems +[11]: https://github.com/akshaypundle +[12]: https://github.com/mihnjong-l +[13]: https://github.com/xinggao01 diff --git a/images/Protected Audience Event Level Reporting with Bidding & Auction Services.svg b/images/Protected Audience Event Level Reporting with Bidding & Auction Services.svg new file mode 100644 index 0000000..f5821b0 --- /dev/null +++ b/images/Protected Audience Event Level Reporting with Bidding & Auction Services.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/images/The flow in buyer system.png b/images/The flow in buyer system.png new file mode 100644 index 0000000..4069d22 Binary files /dev/null and b/images/The flow in buyer system.png differ diff --git a/images/Unified Contextual and Protected Audience flow with Bidding and Auction services.png b/images/Unified Contextual and Protected Audience flow with Bidding and Auction services.png new file mode 100644 index 0000000..97db953 Binary files /dev/null and b/images/Unified Contextual and Protected Audience flow with Bidding and Auction services.png differ diff --git a/images/Win reporting design.svg b/images/Win reporting design.svg new file mode 100644 index 0000000..ce569cf --- /dev/null +++ b/images/Win reporting design.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/images/ba-optimized-payload.png b/images/ba-optimized-payload.png new file mode 100644 index 0000000..9548da9 Binary files /dev/null and b/images/ba-optimized-payload.png differ diff --git a/images/ba-system-seq-1.png b/images/ba-system-seq-1.png new file mode 100644 index 0000000..97db953 Binary files /dev/null and b/images/ba-system-seq-1.png differ diff --git a/images/ba-system-seq-11.png b/images/ba-system-seq-11.png new file mode 100644 index 0000000..df1e8c1 Binary files /dev/null and b/images/ba-system-seq-11.png differ diff --git a/images/ba-system-seq-2.png b/images/ba-system-seq-2.png new file mode 100644 index 0000000..4069d22 Binary files /dev/null and b/images/ba-system-seq-2.png differ diff --git a/images/ba-system-seq-21.png b/images/ba-system-seq-21.png new file mode 100644 index 0000000..922f35b Binary files /dev/null and b/images/ba-system-seq-21.png differ diff --git a/images/ba-system-seq-22.png b/images/ba-system-seq-22.png new file mode 100644 index 0000000..33ebd83 Binary files /dev/null and b/images/ba-system-seq-22.png differ diff --git a/images/ba_experiment.png b/images/ba_experiment.png new file mode 100644 index 0000000..9d0dc13 Binary files /dev/null and b/images/ba_experiment.png differ diff --git a/images/ba_experiment_traffic.png b/images/ba_experiment_traffic.png new file mode 100644 index 0000000..f795fab Binary files /dev/null and b/images/ba_experiment_traffic.png differ diff --git a/images/benchmarking_tool_output.png b/images/benchmarking_tool_output.png new file mode 100644 index 0000000..7dd9496 Binary files /dev/null and b/images/benchmarking_tool_output.png differ diff --git a/images/bid_cur_ad_pub_setup.png b/images/bid_cur_ad_pub_setup.png new file mode 100644 index 0000000..ed40130 Binary files /dev/null and b/images/bid_cur_ad_pub_setup.png differ diff --git a/images/bid_cur_buyer_flow.png b/images/bid_cur_buyer_flow.png new file mode 100644 index 0000000..bfe5463 Binary files /dev/null and b/images/bid_cur_buyer_flow.png differ diff --git a/images/bid_cur_seller_flow.png b/images/bid_cur_seller_flow.png new file mode 100644 index 0000000..87742dc Binary files /dev/null and b/images/bid_cur_seller_flow.png differ diff --git a/images/bidding_auction_cost_fig1.png b/images/bidding_auction_cost_fig1.png new file mode 100644 index 0000000..2342db0 Binary files /dev/null and b/images/bidding_auction_cost_fig1.png differ diff --git a/images/bidding_auction_cost_fig2.png b/images/bidding_auction_cost_fig2.png new file mode 100644 index 0000000..9fbd19a Binary files /dev/null and b/images/bidding_auction_cost_fig2.png differ diff --git a/images/bidding_auction_cost_fig3.png b/images/bidding_auction_cost_fig3.png new file mode 100644 index 0000000..60fcee3 Binary files /dev/null and b/images/bidding_auction_cost_fig3.png differ diff --git a/images/bidding_auction_cost_fig4.png b/images/bidding_auction_cost_fig4.png new file mode 100644 index 0000000..fae5eb2 Binary files /dev/null and b/images/bidding_auction_cost_fig4.png differ diff --git a/images/bidding_auction_services_packaging.svg b/images/bidding_auction_services_packaging.svg new file mode 100644 index 0000000..7728992 --- /dev/null +++ b/images/bidding_auction_services_packaging.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/images/blob_fetch1.png b/images/blob_fetch1.png new file mode 100644 index 0000000..ba89804 Binary files /dev/null and b/images/blob_fetch1.png differ diff --git a/images/blob_fetch2.png b/images/blob_fetch2.png new file mode 100644 index 0000000..4cceda3 Binary files /dev/null and b/images/blob_fetch2.png differ diff --git a/images/blob_fetch21.png b/images/blob_fetch21.png new file mode 100644 index 0000000..3895aeb Binary files /dev/null and b/images/blob_fetch21.png differ diff --git a/images/blob_fetch22.png b/images/blob_fetch22.png new file mode 100644 index 0000000..975aabf Binary files /dev/null and b/images/blob_fetch22.png differ diff --git a/images/blob_fetch23.png b/images/blob_fetch23.png new file mode 100644 index 0000000..78f0e90 Binary files /dev/null and b/images/blob_fetch23.png differ diff --git a/images/blob_fetch3.png b/images/blob_fetch3.png new file mode 100644 index 0000000..8714439 Binary files /dev/null and b/images/blob_fetch3.png differ diff --git a/images/buyer_aws_diagram.svg b/images/buyer_aws_diagram.svg new file mode 100644 index 0000000..a77bba0 --- /dev/null +++ b/images/buyer_aws_diagram.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/images/buyer_gcp_diagram.svg b/images/buyer_gcp_diagram.svg new file mode 100644 index 0000000..0dd50b2 --- /dev/null +++ b/images/buyer_gcp_diagram.svg @@ -0,0 +1,176 @@ + + + + + + + + + + + + + + + + + + + + + + + +Buyer Multi-Region GCP Architecture + + + + + + +Code Modules +Cloud Storage + + +Fetched periodically. + + + + + +Global External LB +Cloud Load Balancer + + +Supports gRPC. Automatically performs regional routing. + + + + + +Global VPC + + + + +us-central1 regional subnet + + + + +Buyer Frontend +Compute Engine + + +Confidential Space. Accepts gRPC. + + + + + +Cloud +Service +Mesh + + + + +Bidding Service +Compute Engine + + +Confidential Space. Accepts plaintext gRPC. + + + + + +europe-central2 regional subnet + + + + +Buyer Frontend +Compute Engine + + +Confidential Space. Accepts gRPC. + + + + + +Cloud +Service +Mesh + + + + +Bidding Service +Compute Engine + + +Confidential Space. Accepts plaintext gRPC. + + + + +External Services + + + + +Key/Value Services + + + + + +Key Management, Attestation + + + + + +Seller Frontends + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/images/chaffing_diagram.png b/images/chaffing_diagram.png new file mode 100644 index 0000000..ed671cc Binary files /dev/null and b/images/chaffing_diagram.png differ diff --git a/images/debugging_protected_audience_api_services_chrome_ui.png b/images/debugging_protected_audience_api_services_chrome_ui.png new file mode 100644 index 0000000..f51b7c0 Binary files /dev/null and b/images/debugging_protected_audience_api_services_chrome_ui.png differ diff --git a/images/debugging_protected_audience_api_services_data_flow.png b/images/debugging_protected_audience_api_services_data_flow.png new file mode 100644 index 0000000..d210bf2 Binary files /dev/null and b/images/debugging_protected_audience_api_services_data_flow.png differ diff --git a/images/debugging_protected_audience_api_services_gcp_logs_explorer.png b/images/debugging_protected_audience_api_services_gcp_logs_explorer.png new file mode 100644 index 0000000..c47dc94 Binary files /dev/null and b/images/debugging_protected_audience_api_services_gcp_logs_explorer.png differ diff --git a/images/device-orchestrated-comp-auction-data-flow.png b/images/device-orchestrated-comp-auction-data-flow.png new file mode 100644 index 0000000..f199262 Binary files /dev/null and b/images/device-orchestrated-comp-auction-data-flow.png differ diff --git a/images/device-orchestrated-comp-auction-data-flow1.png b/images/device-orchestrated-comp-auction-data-flow1.png new file mode 100644 index 0000000..e13f625 Binary files /dev/null and b/images/device-orchestrated-comp-auction-data-flow1.png differ diff --git a/images/device-orchestrated-comp-auction-seq.png b/images/device-orchestrated-comp-auction-seq.png new file mode 100644 index 0000000..24cacbc Binary files /dev/null and b/images/device-orchestrated-comp-auction-seq.png differ diff --git a/images/device-orchestrated-comp-auction-seq1.png b/images/device-orchestrated-comp-auction-seq1.png new file mode 100644 index 0000000..8313b43 Binary files /dev/null and b/images/device-orchestrated-comp-auction-seq1.png differ diff --git a/images/device-orchestrated-comp-auction-seq2.png b/images/device-orchestrated-comp-auction-seq2.png new file mode 100644 index 0000000..15af6ad Binary files /dev/null and b/images/device-orchestrated-comp-auction-seq2.png differ diff --git a/images/device-orchestrated-comp-auction.png b/images/device-orchestrated-comp-auction.png new file mode 100644 index 0000000..812d286 Binary files /dev/null and b/images/device-orchestrated-comp-auction.png differ diff --git a/images/device-orchestrated-comp-auction1.png b/images/device-orchestrated-comp-auction1.png new file mode 100644 index 0000000..da1f71c Binary files /dev/null and b/images/device-orchestrated-comp-auction1.png differ diff --git a/images/egress-payload-with-padding.png b/images/egress-payload-with-padding.png new file mode 100644 index 0000000..8083e43 Binary files /dev/null and b/images/egress-payload-with-padding.png differ diff --git a/images/egress-payload.png b/images/egress-payload.png new file mode 100644 index 0000000..74c85ab Binary files /dev/null and b/images/egress-payload.png differ diff --git a/images/feature-values-wire-format.png b/images/feature-values-wire-format.png new file mode 100644 index 0000000..ac98a22 Binary files /dev/null and b/images/feature-values-wire-format.png differ diff --git a/images/fledge_kv_udf_flow.png b/images/fledge_kv_udf_flow.png new file mode 100644 index 0000000..576a476 Binary files /dev/null and b/images/fledge_kv_udf_flow.png differ diff --git a/images/inference_overview_fig1.png b/images/inference_overview_fig1.png new file mode 100644 index 0000000..d35dc4d Binary files /dev/null and b/images/inference_overview_fig1.png differ diff --git a/images/inference_overview_fig2.png b/images/inference_overview_fig2.png new file mode 100644 index 0000000..3907953 Binary files /dev/null and b/images/inference_overview_fig2.png differ diff --git a/images/inference_overview_fig3-2.png b/images/inference_overview_fig3-2.png new file mode 100644 index 0000000..8072e1b Binary files /dev/null and b/images/inference_overview_fig3-2.png differ diff --git a/images/inference_overview_fig3.png b/images/inference_overview_fig3.png new file mode 100644 index 0000000..b268c08 Binary files /dev/null and b/images/inference_overview_fig3.png differ diff --git a/images/inference_overview_fig4.png b/images/inference_overview_fig4.png new file mode 100644 index 0000000..9bce448 Binary files /dev/null and b/images/inference_overview_fig4.png differ diff --git a/images/kv_cost_example_aws.png b/images/kv_cost_example_aws.png new file mode 100644 index 0000000..7bf4ca2 Binary files /dev/null and b/images/kv_cost_example_aws.png differ diff --git a/images/kv_cost_example_gcp.png b/images/kv_cost_example_gcp.png new file mode 100644 index 0000000..ab7908c Binary files /dev/null and b/images/kv_cost_example_gcp.png differ diff --git a/images/kv_cost_figure.png b/images/kv_cost_figure.png new file mode 100644 index 0000000..f080e87 Binary files /dev/null and b/images/kv_cost_figure.png differ diff --git a/images/monitoring_protected_audience_api_services_fig_1.png b/images/monitoring_protected_audience_api_services_fig_1.png new file mode 100644 index 0000000..a437dea Binary files /dev/null and b/images/monitoring_protected_audience_api_services_fig_1.png differ diff --git a/images/monitoring_protected_audience_api_services_fig_10.png b/images/monitoring_protected_audience_api_services_fig_10.png new file mode 100644 index 0000000..3b53b94 Binary files /dev/null and b/images/monitoring_protected_audience_api_services_fig_10.png differ diff --git a/images/monitoring_protected_audience_api_services_fig_2.png b/images/monitoring_protected_audience_api_services_fig_2.png new file mode 100644 index 0000000..e251201 Binary files /dev/null and b/images/monitoring_protected_audience_api_services_fig_2.png differ diff --git a/images/monitoring_protected_audience_api_services_fig_3.png b/images/monitoring_protected_audience_api_services_fig_3.png new file mode 100644 index 0000000..ccd0483 Binary files /dev/null and b/images/monitoring_protected_audience_api_services_fig_3.png differ diff --git a/images/monitoring_protected_audience_api_services_fig_4.png b/images/monitoring_protected_audience_api_services_fig_4.png new file mode 100644 index 0000000..6de752f Binary files /dev/null and b/images/monitoring_protected_audience_api_services_fig_4.png differ diff --git a/images/monitoring_protected_audience_api_services_fig_5.png b/images/monitoring_protected_audience_api_services_fig_5.png new file mode 100644 index 0000000..e700608 Binary files /dev/null and b/images/monitoring_protected_audience_api_services_fig_5.png differ diff --git a/images/monitoring_protected_audience_api_services_fig_6.png b/images/monitoring_protected_audience_api_services_fig_6.png new file mode 100644 index 0000000..a723c06 Binary files /dev/null and b/images/monitoring_protected_audience_api_services_fig_6.png differ diff --git a/images/monitoring_protected_audience_api_services_fig_7.png b/images/monitoring_protected_audience_api_services_fig_7.png new file mode 100644 index 0000000..535b7af Binary files /dev/null and b/images/monitoring_protected_audience_api_services_fig_7.png differ diff --git a/images/monitoring_protected_audience_api_services_fig_8.png b/images/monitoring_protected_audience_api_services_fig_8.png new file mode 100644 index 0000000..83003db Binary files /dev/null and b/images/monitoring_protected_audience_api_services_fig_8.png differ diff --git a/images/monitoring_protected_audience_api_services_fig_9.png b/images/monitoring_protected_audience_api_services_fig_9.png new file mode 100644 index 0000000..4973d8d Binary files /dev/null and b/images/monitoring_protected_audience_api_services_fig_9.png differ diff --git a/images/multi-seller-reporting-fig1.png b/images/multi-seller-reporting-fig1.png new file mode 100644 index 0000000..55bf886 Binary files /dev/null and b/images/multi-seller-reporting-fig1.png differ diff --git a/images/multi-seller-reporting-fig2.jpg b/images/multi-seller-reporting-fig2.jpg new file mode 100644 index 0000000..437743a Binary files /dev/null and b/images/multi-seller-reporting-fig2.jpg differ diff --git a/images/multi-seller-reporting-fig5.png b/images/multi-seller-reporting-fig5.png new file mode 100644 index 0000000..1ab752c Binary files /dev/null and b/images/multi-seller-reporting-fig5.png differ diff --git a/images/multi-seller-reporting-fig6.png b/images/multi-seller-reporting-fig6.png new file mode 100644 index 0000000..d403609 Binary files /dev/null and b/images/multi-seller-reporting-fig6.png differ diff --git a/images/pas-cost-elements-diagram.png b/images/pas-cost-elements-diagram.png new file mode 100644 index 0000000..20b054f Binary files /dev/null and b/images/pas-cost-elements-diagram.png differ diff --git a/images/prebid_pa_component_1.png b/images/prebid_pa_component_1.png new file mode 100644 index 0000000..56021af Binary files /dev/null and b/images/prebid_pa_component_1.png differ diff --git a/images/prebid_pa_component_2.png b/images/prebid_pa_component_2.png new file mode 100644 index 0000000..681ed3e Binary files /dev/null and b/images/prebid_pa_component_2.png differ diff --git a/images/prebid_pa_on_device.png b/images/prebid_pa_on_device.png new file mode 100644 index 0000000..54aa377 Binary files /dev/null and b/images/prebid_pa_on_device.png differ diff --git a/images/protected-audience-mixed-mode-single-seller-parallelized.png b/images/protected-audience-mixed-mode-single-seller-parallelized.png new file mode 100644 index 0000000..c4122f8 Binary files /dev/null and b/images/protected-audience-mixed-mode-single-seller-parallelized.png differ diff --git a/images/protected-audience-multiseller-mixed-mode.png b/images/protected-audience-multiseller-mixed-mode.png new file mode 100644 index 0000000..9885413 Binary files /dev/null and b/images/protected-audience-multiseller-mixed-mode.png differ diff --git a/images/protected_app_signals_architecture_contextual_diagram.svg b/images/protected_app_signals_architecture_contextual_diagram.svg new file mode 100644 index 0000000..3c50ee3 --- /dev/null +++ b/images/protected_app_signals_architecture_contextual_diagram.svg @@ -0,0 +1,4 @@ + + + +
Client
Client
Seller Ad Tech
Seller Ad Tech
Buyer Ad Tech
Buyer Ad Tech
Device
Device
Figure 2: Protected App Signals Ad Selection with B&A using a Key-Value Service.
Figure 2: Protected App Signals Ad Selection with B&A using...
C. GenerateProtectedAppSignalsBidsRequest
C. GenerateProtectedAppSignalsBidsRequest
E. Ad with Bid
returned
E. Ad with Bid...
Buyer Frontend Service
Buyer Frontend Service
D. (c) Ad With Bid
returned

D. (c) Ad With Bid...
   Bidding
   Bidding
A. SelectAdRequest
A. SelectAdRequest
Seller Ad Service
Seller Ad Service
Key Value Service
Key Value Servi...
B. GetBidsRequest
B. GetBidsRequest
G. ScoreAdsRequest
G. ScoreAdsRequest
I. Auction Result
I. Auction Result
Seller Frontend Service
Seller Frontend S...
H. Winner returned
H. Winner returned
Auction Service
Auction Service
generateBid
generateBid
scoreAd
scoreAd
Select Winner
Select Winner
reportResult
reportResult
reportWin
reportWin
Fetch Ads Metadata
Fetch Ads Metadata
D. (a) Bid for a PAS Ad 
D. (a) Bid for a PAS Ad 
Legend
Legend
Trusted Service Under TEE
Trusted Service Under...
Untrusted Service
Untrusted Service
Seller KV
Service
Seller KV...
F. Get Scoring Signals
F. Get Scoring Signals
RTB Servers
RTB Servers
Contextual Ad Render Ids
Contextual Ad Render Ids
D. (b) Bid for a PAS Ad 
D. (b) Bid for a PAS Ad 
Text is not SVG - cannot display
\ No newline at end of file diff --git a/images/protected_app_signals_architecture_diagram.svg b/images/protected_app_signals_architecture_diagram.svg new file mode 100644 index 0000000..fa409b5 --- /dev/null +++ b/images/protected_app_signals_architecture_diagram.svg @@ -0,0 +1,4 @@ + + + +
Client
Client
Seller Ad Tech
Seller Ad Tech
Buyer Ad Tech
Buyer Ad Tech
Device
Device
Figure 1: Protected App Signals Ad Selection with B&A.
Figure 1: Protected App Signals Ad Selection with B&A.
C. GenerateProtectedAppSignalsBidsRequest
C. GenerateProtectedAppSignalsBidsRequest
E. Ad with Bid
returned
E. Ad with Bid...
Buyer Frontend Service
Buyer Frontend Service
D. (d) Ad With Bid
returned

D. (d) Ad With Bid...
   Bidding
   Bidding
A. SelectAdRequest
A. SelectAdRequest
Seller Ad Service
Seller Ad Service
Ad Retrieval Service
Ad Retrieval Se...
B. GetBidsRequest
B. GetBidsRequest
G. ScoreAdsRequest
G. ScoreAdsRequest
I. Auction Result
I. Auction Result
Seller Frontend Service
Seller Frontend S...
H. Winner returned
H. Winner returned
Auction Service
Auction Service
prepareDataForAdRetrieval
prepareDataForAdRetrieval
generateBid
generateBid
scoreAd
scoreAd
Select Winner
Select Winner
reportResult
reportResult
reportWin
reportWin
D. (a) Decode Signals 
D. (a) Decode Signals 
Fetch Ads Metadata
Fetch Ads Metadata
D. (b)Fetch Ads 
D. (b)Fetch Ads 
D. (c) Bid for a PAS Ad 
D. (c) Bid for a PAS Ad 
Legend
Legend
Trusted Service Under TEE
Trusted Service Under...
Untrusted Service
Untrusted Service
Seller KV
Service
Seller KV...
F. Get Scoring Signals
F. Get Scoring Signals
Text is not SVG - cannot display
\ No newline at end of file diff --git a/images/protected_app_signals_sequence_diagram.svg b/images/protected_app_signals_sequence_diagram.svg new file mode 100644 index 0000000..93e6706 --- /dev/null +++ b/images/protected_app_signals_sequence_diagram.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/images/roma.png b/images/roma.png new file mode 100644 index 0000000..01c7c68 Binary files /dev/null and b/images/roma.png differ diff --git a/images/seller_aws_diagram.svg b/images/seller_aws_diagram.svg new file mode 100644 index 0000000..bc3ff56 --- /dev/null +++ b/images/seller_aws_diagram.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/images/seller_gcp_diagram.svg b/images/seller_gcp_diagram.svg new file mode 100644 index 0000000..ff03b9e --- /dev/null +++ b/images/seller_gcp_diagram.svg @@ -0,0 +1,182 @@ + + + + + + + + + + + + + + + + + + + + + + + +Seller Multi-Region GCP Architecture + + + + + + +Code Modules +Cloud Storage + + +Fetched periodically. + + + + + +Global External LB +Cloud Load Balancer + + +Supports gRPC and HTTPS. Automatically performs regional routing. + + + + + +Global VPC + + + + +us-central1 regional subnet + + + + +Seller Frontend +Compute Engine + + +Confidential Space. Accepts gRPC and HTTPS. + + + + + +Cloud +Service +Mesh + + + + +Auction Service +Compute Engine + + +Confidential Space. Accepts plaintext gRPC. + + + + + +europe-central2 regional subnet + + + + +Seller Frontend +Compute Engine + + +Confidential Space. Accepts gRPC and HTTPS. + + + + + +Cloud +Service +Mesh + + + + +Auction Service +Compute Engine + + +Confidential Space. Accepts plaintext gRPC. + + + + +External Services + + + + +Buyer Frontends + + + + + +Key/Value Services + + + + + +Key Management, Attestation + + + + + +Seller Ad Services + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/images/server-orchestrated-comp-auction-data-flow.png b/images/server-orchestrated-comp-auction-data-flow.png new file mode 100644 index 0000000..6a6454c Binary files /dev/null and b/images/server-orchestrated-comp-auction-data-flow.png differ diff --git a/images/server-orchestrated-comp-auction-data-flow1.png b/images/server-orchestrated-comp-auction-data-flow1.png new file mode 100644 index 0000000..f754ea1 Binary files /dev/null and b/images/server-orchestrated-comp-auction-data-flow1.png differ diff --git a/images/server-orchestrated-comp-auction-data-flow2.png b/images/server-orchestrated-comp-auction-data-flow2.png new file mode 100644 index 0000000..d64ed93 Binary files /dev/null and b/images/server-orchestrated-comp-auction-data-flow2.png differ diff --git a/images/server-orchestrated-comp-auction-seq.png b/images/server-orchestrated-comp-auction-seq.png new file mode 100644 index 0000000..aa4e96b Binary files /dev/null and b/images/server-orchestrated-comp-auction-seq.png differ diff --git a/images/server-orchestrated-comp-auction-seq1.png b/images/server-orchestrated-comp-auction-seq1.png new file mode 100644 index 0000000..14d7cf8 Binary files /dev/null and b/images/server-orchestrated-comp-auction-seq1.png differ diff --git a/images/server-orchestrated-comp-auction-seq2.png b/images/server-orchestrated-comp-auction-seq2.png new file mode 100644 index 0000000..e24e275 Binary files /dev/null and b/images/server-orchestrated-comp-auction-seq2.png differ diff --git a/images/server-orchestrated-comp-auction.png b/images/server-orchestrated-comp-auction.png new file mode 100644 index 0000000..da1f71c Binary files /dev/null and b/images/server-orchestrated-comp-auction.png differ diff --git a/images/server-orchestrated-comp-auction1.png b/images/server-orchestrated-comp-auction1.png new file mode 100644 index 0000000..812d286 Binary files /dev/null and b/images/server-orchestrated-comp-auction1.png differ diff --git a/images/temporary-unlimited-egress-payload.png b/images/temporary-unlimited-egress-payload.png new file mode 100644 index 0000000..933c01a Binary files /dev/null and b/images/temporary-unlimited-egress-payload.png differ diff --git a/images/temporary-unlimited-egress-with-padding.png b/images/temporary-unlimited-egress-with-padding.png new file mode 100644 index 0000000..07b275a Binary files /dev/null and b/images/temporary-unlimited-egress-with-padding.png differ diff --git a/images/unified-contextual-remarketing-bidding-auction-services.png b/images/unified-contextual-remarketing-bidding-auction-services.png new file mode 100644 index 0000000..34312bf Binary files /dev/null and b/images/unified-contextual-remarketing-bidding-auction-services.png differ diff --git a/images/win-reporting.svg b/images/win-reporting.svg new file mode 100644 index 0000000..1bc099a --- /dev/null +++ b/images/win-reporting.svg @@ -0,0 +1 @@ +Created with Raphaël 2.2.0Win reporting flow for Single Seller AuctionClientClientAdExchangeAdExchangeBFEBFEBidding ServiceBidding ServiceBuyer's Repoting EndpointBuyer's Repoting EndpointSFESFEAuctionServiceAuctionServiceSeller's Reporting EndpointSeller's Reporting EndpointRemarketingInputSelectAdRequestGetBidsRequestGetBidsResponseScoreAdRequestreportResult()reportWin()ScoreAdResponsereportResultUrlreportWinUrlSelectAdResponseRemarketing/Contextual responsereportResultUrlreportWinUrl \ No newline at end of file diff --git a/inference-explainers/inference_api.md b/inference-explainers/inference_api.md new file mode 100644 index 0000000..51a3948 --- /dev/null +++ b/inference-explainers/inference_api.md @@ -0,0 +1,370 @@ +# Inference API + +**Authors:**
+[Michael Mihn-Jong Lee][6], Google Privacy Sandbox
+[Steve Gan][5], Google Privacy Sandbox
+ +## Introduction + +This document provides a comprehensive overview of the Bidding and Auction (B&A) Inference service API with externally supplied models ([B&A inference overview][1]), including supported machine learning (ML) frameworks, model formats, model management processes, and the JavaScript API for executing inference tasks. + +## Supported Machine Learning Frameworks and Model Formats + +The Tensorflow and PyTorch runtimes are supported, each with a dedicated inference service sidecar binary. Currently, we offer the Tensorflow v2.14.0 sidecar binary and the PyTorch v2.1.1 sidecar binary. + +For Tensorflow, the SavedModel format is used while PyTorch uses the TorchScript format. We recommend that ad techs develop and train models in an environment in which the ML runtime version aligns with the intended sidecar. + +The maximum size for a single ML model is 2GB. + +**Important**: Ad techs must ensure their ML models are **freezable** for successful loading. Unfreezable models will fail to load. Refer to [the following section](#model-freezing) for details. + +## Model Freezing + +During ML model registration, the inference sidecar automatically freezes ad techs’ ML models. For successful registration, ad techs must ensure their models are **freezable**. If the models cannot be frozen, the registration process will fail. + +We recommend freezing models to prevent unexpected issues, especially for TensorFlow models. + +### Freezing TensorFlow Models + +During model registration, ad tech’s model is frozen using [tensorflow::FreezeSavedModel][8]. We highly recommend pre-freezing TensorFlow models, especially those built with [Keras][11]. + +Use [convert_variables_to_constants_v2_as_graph][9] function to freeze TensorFlow models. Here’s a Python code example to freeze a Keras model: + +```python +import tensorflow as tf +from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2_as_graph + +model = tf.keras.models.load_model('keras_model_path') + +inputs = [ + tf.TensorSpec(input.shape, input.dtype, input.name) + for input in model.inputs +] +real_model = tf.function(model).get_concrete_function(inputs) + +frozen_func, _ = convert_variables_to_constants_v2_as_graph(real_model) +tf.saved_model.save(frozen_func, 'frozen_model_path', signatures=frozen_func) +``` + +To confirm a TensorFlow model will load correctly, use the [freeze_model tool][10]. + +### Freezing PyTorch Models + +During model registration, ad tech’s model is frozen using [torch.jit.freeze][7]. Please make sure that a PyTorch model is compatible with [torch.jit.freeze][7]. Ad Techs can pre-freeze their model or confirm a model is freezable by running the following code: + +```python +import torch + +model = torch.jit.load('torch_model.pt') +frozen_model = torch.jit.freeze(model.eval()) +frozen_model.save('frozen_model.pt') +``` + +## Model Management + +ML models used by the inference service are fetched periodically from a linked cloud storage (e.g., [Google Cloud Storage buckets][2], [Amazon S3 buckets][3]) and made available for serving. Each model is uniquely identified by its path within the cloud storage. + +### Model Loading + +To determine the models to load, the inference service looks for a JSON model configuration file stored in the same bucket as the models. Ad techs are responsible for maintaining and updating this model configuration file. The path to the configuration file is exposed as a Terraform parameter `INFERENCE_MODEL_CONFIG_PATH`. The B&A service periodically checks the specified configuration file (at intervals configurable by ad techs) for any changes and triggers the loading of new models into the inference service sidecar’s memory and execution of warm-up requests, which will be explained later in this document. + +Ad techs can implement model versioning by structuring model storage paths to include version identifiers. For example, storing models under the directories such as `pcvr_v1/` and `pcvr_v2/` distinguishes between the two versions. + +For example: + +```json +{ + "model_metadata": [ + { + "model_path": "pcvr_v1/", + "checksum": "dd94b3b08ea19b4240aa5e5f68ff0447b40e37ebdd06cc76fa1a2cf61143a10c", + "warm_up_batch_request_json": "..." + }, + { + "model_path": "pcvr_v2/", + "checksum": "...", + "warm_up_batch_request_json": "..." + } + ] +} +``` + +This model configuration file features a top-level array with each entry containing the metadata of the fetched models. The mandatory `model_path` field specifies the path to the model in the cloud storage. If the specified model path points to a directory, it needs to end with a "/" suffix. This suffix indicates that the entire directory will be used for model registration. Without the "/" suffix, an exact path match is expected, and only the specified file will be registered as a model. In the example above, both model paths refer to directories. + +In TensorFlow, models are typically stored as directories containing multiple files. For example, a `pcvr/` model directory might have the following structure: + +``` +pcvr/saved_model.pb +pcvr/variables/variables.data-00000-of-00001 +pcvr/variables/variables.index +``` +To register the entire model including all three files, the `model_path` field needs to be set to `pcvr/`. + +For PyTorch, models are stored as single files, such as `pcvr/model.pt`. For registration, you can specify either the directory path (`pcvr/`) or the file path (`pcvr/model.pt`) as the `model_path`. + +The optional `checksum` field is the SHA256 checksum of the model represented as a hexadecimal string. Model checksums are computed using the following steps: + +1. Compute the SHA256 checksum for each individual model file. +2. Arrange the file checksums in ascending order based on their file paths. +3. Concatenate all the ordered file checksums into a single string. +4. Compute the final SHA256 checksum on the concatenated checksum string. + +The listed steps are equivalent to the following bash command. This command works for both a directory model path and a single-file model path: + +``` +find -type f -exec sha256sum {} \; | sort -k 2 | awk '{print $1}' | tr -d '\n' | sha256sum | awk '{print $1}' +``` + +The optional `warm_up_batch_request_json` is a JSON batch request used to warm up the model before any traffic is served. This request is parsed and sent to the model to trigger initialization during the loading phase, reducing the latency impact of model lazy initialization during the first request. The format of this warm-up request uses the same format as the input to the runInference JavaScript function, which is described later in this document. + +Refer to the [B&A Inference Onboarding Guide][4] for more details about using the model configuration file with the Terraform configuration. + +Note that after a model is loaded, its original metadata entry must be retained to keep the model. Removing the metadata will be treated as a model deletion, as explained in the next section. + +### Model Deletion + +Ad techs may want to delete stale models to free up memory for the inference sidecar. This can be accomplished by removing stale model entries from the model configuration file. During the periodic fetching process, the inference service checks whether any loaded model is absent from the configuration file. If such models are found, they are deleted from the inference sidecar. + +For example, consider the inference service receiving its first model configuration file at time A, which instructs it to load `pcvr_v1/` into the inference sidecar. At time B, if `pcvr_v1/` is no longer needed and `pcvr_v2/` is required instead, ad techs can upload a new model configuration file to the cloud bucket. This operation will remove `pcvr_v1/` from the inference sidecar (since it is no longer listed in the configuration file) and attempt to load `pcvr_v2/` from the cloud bucket. In other words, the inference sidecar synchronizes its internal model storage with the updated configuration file. + +Time A + +```json +{ + "model_metadata": [ + { + "model_path": "pcvr_v1/", + "checksum": "..." + } + ] +} +``` + +Time B + +```json +{ + "model_metadata": [ + { + "model_path": "pcvr_v2/", + "checksum": "..." + } + ] +} +``` + +When switching model versions, any remaining traffic for the previous model version may need to be processed before fully transitioning to the new version. To avoid traffic disruptions during this process, the inference service can postpone the removal of the older model version. This is achieved by configuring the `eviction_grace_period_in_ms` parameter in the configuration file, which specifies the delay (in milliseconds) before the old model is evicted. For instance, assume that at time A, the following configuration is supplied, upon deletion, the `pcvr_v1/` model will be retained for an additional 60 seconds even though `pcvr_v2/` is loaded into the inference service. During this grace period, traffic directed to `pcvr_v1/` will still be processed. + +```json +{ + "model_metadata": [ + { + "model_path": "pcvr_v1/", + "checksum": "...", + "eviction_grace_period_in_ms": 60000 + }, + ] +} +``` + +### Model In-Place Update + +It is possible to update a model in place by overwriting it at the same model path in the model configuration file. An in-place update is treated the same as a model deletion followed by a model addition with the previous `eviction_grace_period_in_ms` applied. This blocks the loading of new model content until the grace period expires, after which the new model content is loaded during the next polling cycle. The inference service detects this update by recognizing changes to the model checksum. When a model is updated in this way, all associated metadata, including `warm_up_batch_request_json` and `eviction_grace_period_in_ms`, is modified with the new model entry from the configuration file after the previous grace period has expired and that version of the model is deleted. If the model's checksum remains unchanged between pollings, no update will be triggered. The model along with its original metadata remains unaffected. + +## JavaScript API + +We expose the Inference service capabilities as JavaScript functions in ad tech code modules. Currently, the Inference service can be executed as a part of the proprietary bidding logic. Ad techs can query available models and issue batched inference service requests to a selected model set. + +### JavaScript functions + +#### getModelPaths + +The `getModelPaths` function doesn’t accept arguments and returns the available models in the inference service sidecar. The returned result is represented in a serialized JSON array of model paths. For example, `[“pcvr_v1/”, “pcvr_v2/”]`. + +#### runInference + +The `runInference` function accepts batch requests and returns batch responses. + +#### Batch request + +The batch inference service request is represented as a JSON object containing a top-level array. Each element in the array represents an individual request and is represented as a JSON object. + +#### Individual request mandatory fields + +- `model_path`: Specifies the target model for the request +- `tensors`: An array of JSON objects representing the input tensors to the model. + +#### Tensor mandatory fields + +- `data_type`: Specifies the data type of the tensor, chosen from DOUBLE, FLOAT, INT8, INT16, INT32, INT64. +- `tensor_shape`: An array specifying the shape of the tensor. +- `tensor_content`: A flattened row-major representation of the tensor values. +- `tensor_name`: Specifies the tensor's name to match the model's signature. This value is ignored for PyTorch models. + +`tensor_content` values are represented as strings, with plans to eventually enable storage as numeric types, as numeric value support is currently under development. + +The following illustrates a batch request involving two models, `pcvr_v1/` and `pcvr_v2/`: + +```json +{ + "request": [ + { + "model_path": "pcvr_v1/", + "tensors": [ + { + "tensor_name": "feature1", + "data_type": "DOUBLE", + "tensor_shape": [2, 1], + "tensor_content": ["0.454920", "-0.25752"] + } + ] + }, + { + "model_path": "pcvr_v2/", + "tensors": [ + { + "tensor_name": "feature1", + "data_type": "INT32", + "tensor_shape": [2, 1], + "tensor_content": ["5", "6"] + }, + { + "tensor_name": "feature2", + "data_type": "FLOAT", + "tensor_shape": [2, 2], + "tensor_content": ["0.5", "0.6", "0.7", "0.8"] + } + ] + } + ] +} +``` + +#### Batch response + +The batch inference request is structured as a JSON object containing a top-level array. Each element within this array represents an individual response represented as a JSON object. This array consists of at least one element. If a batch inference fails, this one element represents the batch error. + +Each individual response object signifies either successful inference output or an error that was encountered during inference. + +- For a success inference, each tensor JSON object follows the same format as in the request case except that `tensor_content` directly contains numericals instead of strings. + +- In the failure case, the `model_path` field is optional. It is populated when the inference directed towards a specific model fails and is left empty when an entire batch request fails. The response has a mandatory `error` field with each error composed of a JSON object with `error_type` and `description` string fields. + We currently report the following error types: + + - UNKNOWN + - INPUT_PARSING + - MODEL_NOT_FOUND + - MODEL_EXECUTION + - OUTPUT_PARSING + - GRPC + +The `description` field propagates back the original Abseil error message from the inference sidecar C++ program. + +The following are some illustrative scenarios for reference: + +- **Batch with all successful outputs:** The top-level array contains multiple response objects, each with the `model_path` and `tensor` fields populated which indicates successful inference for each input in the batch: + +```json +{ + "response": [ + { + "model_path": "pcvr_1/", + "tensors": [ + { + "tensor_name": "PartitionedCall:0", + "data_type": "FLOAT", + "tensor_shape": [2, 1], + "tensor_content": [0.1167, 0.1782] + } + ] + }, + { + "model_path": "pcvr_2/", + "tensors": [ + { + "tensor_name": "PartitionedCall:0", + "data_type": "FLOAT", + "tensor_shape": [2, 1], + "tensor_content": [0.3422, 0.2346] + } + ] + } + ] +} +``` + +- **Batch with mixed successes and failures:** The top-level array contains a mix of response objects. Some have `model_path` and `tensors`, while others have populated the `error` field, indicating that some inferences succeeded while others failed. + +```json +{ + "response": [ + { + "model_path": "pcvr_1/", + "tensors": [ + { + "tensor_name": "PartitionedCall:0", + "data_type": "FLOAT", + "tensor_shape": [2, 1], + "tensor_content": [0.1167, 0.1782] + } + ] + }, + { + "model_path": "pcvr_2", + "error": { + "error_type": "MODEL_NOT_FOUND", + "description": "NOT_FOUND: Requested model pcvr_2 not found" + } + } + ] +} +``` + +- **Complete batch failure:** The top-level array contains a single response object with a populated `error` field. The `model_path` field is absent in this scenario, as the failure happens at the batch level rather than at the model specific level. + +```json +{ + "response": [ + { + "error": { + "error_type": "GRPC", + "description": "transport closed" + } + } + ] +} +``` + +#### Inference JavaScript API code example + +The following is an example of how ad techs can use `getModelPaths` with `runInference` functions together to execute an inference task in JavaScript: + +```javascript +const modelPath = getLatestModel(getModelPaths()); + +const batchRequest = JSON.stringify({ + request: [{ model_path: modelPath, tensors: [input1, input2 /*...*/] }], +}); + +const inference_result = JSON.parse(runInference(batchRequest)); + +for (const response of inference_result.response) { + if (response.error) { + handleError(response.error.error_type); + } else { + console.log("Use inference output:", response.tensors); + } +} +``` + +[1]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/inference_overview.md#inference-with-externally-provided-models +[2]: https://cloud.google.com/storage/docs/buckets +[3]: https://aws.amazon.com/s3 +[4]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/inference-explainers/inference_onboarding_guide.md +[5]: https://github.com/steveganzy +[6]: https://github.com/mihnjong-l +[7]: https://pytorch.org/docs/stable/generated/torch.jit.freeze.html +[8]: https://github.com/tensorflow/tensorflow/blob/r2.14/tensorflow/cc/tools/freeze_saved_model.h#L36 +[9]: https://github.com/tensorflow/tensorflow/blob/r2.14/tensorflow/python/framework/convert_to_constants.py#L1223 +[10]: https://github.com/privacysandbox/bidding-auction-servers/blob/release-4.5/services/inference_sidecar/modules/tensorflow_v2_14_0/tools/freeze_model.cc +[11]: https://keras.io/ diff --git a/inference-explainers/inference_onboarding_guide.md b/inference-explainers/inference_onboarding_guide.md new file mode 100644 index 0000000..28d3528 --- /dev/null +++ b/inference-explainers/inference_onboarding_guide.md @@ -0,0 +1,379 @@ +# B&A Inference Basic Setup and Testing Guide + +**Authors:**
[Steve Gan][17], Google Privacy Sandbox
+ +## Introduction + +This document describes how to package and deploy Bidding and Auction services ([B&A system +design][1]) with machine learning (ML) inference capabilities ([B&A inference overview][2]) in cloud +and local environments. This guide assumes you have administrative access to your cloud provider +accounts and are familiar with the deployment process for Bidding and Auction servers and using ML +models for inference. + +## Cloud Deployment + +The Bidding and Auction services can be deployed on Google Cloud Platform (GCP) or Amazon Web +Services (AWS). We provide GCP and AWS support for using Bidding and Auction services with inference +capabilities. Documentation for packaging and deploying Bidding and Auction services is available in +the [Bidding and Auction services GCP cloud support and deployment guide][3] (GCP) and the [Bidding +and Auction services AWS cloud support and deployment guide][4] (AWS). + +The inference capabilities require additional steps to configure cloud storage and Terraform-based +services, and are described in this document. While inference for Bidding and Auction services is +designed in a generic way, inference support is currently limited to the Bidding service. + +To deploy and use the inference capabilities, you will: + +- Build and package the Bidding and Auction servers images that includes the inference + capabilities. +- Upload ML models to a cloud bucket and generate a metadata file summarizing all models you are + using. +- Update UDF code modules to invoke the inference JavaScript functions. +- Supply additional inference-specific Terraform configurations to deploy the Bidding and Auction + servers with inference enabled. + +The following sections provide a step-by-step breakdown of the cloud deployment process for each +supported cloud. + +### GCP Guide: Package, deploy, and run a service + +This section describes how to package and deploy B&A services on GCP. + +The process for creating a functioning service with inference on GCP has two major steps: + +1. **Packaging:** Create a Docker image containing the services’ code. +2. **Deployment:** Run Terraform to bring up the Bidding and Auction services with inference. + +#### Step 0: Prerequisites + +1. Refer to the [Bidding and Auction services GCP cloud support and deployment guide][3] to + familiarize yourself with the packaging and deployment process for GCP. This document assumes you + are able to create a functioning service stack (following Step 0 to Step 2.5 in the Package, + Deploy, and Run a Service Guide section of the above guide). It highlights the additional steps + required to enable inference capabilities. +2. Running Bidding and Auction services with inference requires functioning machine learning models. + Choose Tensorflow or PyTorch as your ML platform, and then familiarize yourself with the + [Tensorflow][5] or [PyTorch][6] ML framework. This document assumes you are able to create and + save models in your chosen framework. + +#### Step 1: Packaging + +**Build the GCP Confidential Space Docker image:** Follow the instructions in the aforementioned +[GCP Guide][3] to build a docker image for all B&A services. The inference artifacts, including +both the “tensorflow_v2_14_0” and “pytorch_v2_1_1” sidecar binaries (ML runtimes and versions +that we currently support), will be packaged within the Bidding service image. +> **_Note:_** Other versions of Tensorflow and PyTorch will be supported in the future. + +#### Step 2: Deployment + +1. **Upload ML models:** Use a GCS bucket to host proprietary ML models. The Bidding server fetches + the uploaded models during runtime and forwards them to the inference sidecar for handling + requests. The bucket name is required by the Terraform configuration. Additionally, you must + upload a model configuration file that specifies the models being fetched to the cloud bucket and + include its path relative to the cloud bucket in the Terraform configuration. The model + configuration file has the following format: + + ``` + { + "model_metadata": [ + { + "model_path": "model_1", + "checksum": "dd94b3b08ea19b4240aa5e5f68ff0447b40e37ebdd06cc76fa1a2cf61143a10c" + }, + { + "model_path": "model_2", + "checksum": "" + } + ] + } + ``` + + Each model is specified by its path relative to the cloud bucket and may be accompanied by an + optional SHA-256 checksum of the model's content. The checksum can be computed by using the + following bash command: + + ``` + find -type f -exec sha256sum {} \; | sort -k 2 | awk '{print $1}' | tr -d '\n' | sha256sum | awk '{print $1}' + ``` + + Store the metadata file in the same bucket as the models. You can update it after server + deployment to fetch additional models. Note that you must grant READ permission for your + proprietary endpoints to access your models and use them with your service. + +2. **Upload Code Modules:** The inference capabilities of your proprietary JavaScript bidding code + modules are exposed with the `runInference` and `getModelPaths` inference callbacks. To use + inference capabilities, update your existing code modules to invoke these functions. Refer to + [this section][7] of the B&A Inference Overview Explainer for more details about these two + functions, and to [this section][8] of the GCP Guide for more information about uploading code + modules. + +3. **Configure Terraform variables:** Configure the inference capabilities using inference Terraform + flags. The names of all inference-related Terraform flags have an `INFERENCE` prefix. Include + them as part of the `runtime_flags` in the “buyer” module. + + 1. Use `INFERENCE_SIDECAR_BINARY_PATH` to expose your ML runtime selection. To enable + inference and select the ML runtime, set this flag to + `/server/bin/inference_sidecar_`, where `` is either + tensorflow_v2_14_0 for a Tensorflow runtime with the version 2.14.0, or pytorch_v2_1_1 for a + PyTorch runtime with the version 2.1.1 . + + > **_Note:_** Inference is disabled if this flag is not set. + + If inference is disabled, all inference-related Terraform flags are ignored and invocations of + inference callbacks in code modules will fail. + + 1. For recommended deployment configurations for Bidding and Auction servers with inference, + see [this configuration file][9]. You can find example values for the following flags in the + same configuration file. + + * Set `INFERENCE_MODEL_BUCKET_NAME` to the name of the GCS bucket used to + store models. + * Set `INFERENCE_MODEL_CONFIG_PATH` to the path to the model configuration file + relative to the model GCS bucket. This metadata file lists the models to be fetched. + * Set `INFERENCE_MODEL_FETCH_PERIOD_MS` to the period that the bidding server checks updates in the + model configuration file and fetches models. + * Set `INFERENCE_SIDECAR_RUNTIME_CONFIG` to a JSON + config that sets runtime configurations for the inference sidecar binary. It has the following + format: + + ``` + { + "num_interop_threads": , + "num_intraop_threads": , + "cpuset": + } + ``` + + These are performance-related flags for tuning the sidecar's performance. You can find more + details about these flags [here][18] + +5. **Apply Terraform:** There is no change in applying the Terraform step from [here][10]. + +6. **Test the Service:** To test your Bidding and Auction services with inference capabilities, you + can follow the same steps as described in [this section][11]. The Bidding and Auction flow will + use your code modules that invoke inference callbacks. You can verify that inference is being + invoked by monitoring the “inference.request.count” metric exported from the bidding server in + your GCP monitoring dashboard to ensure it is incrementing as expected. + +### AWS Guide: Package, deploy, and run a service + +This section describes how to package and deploy B&A services on AWS. + +The process for creating a functioning service with inference on AWS has two major steps: + +1. **Packaging:** Create an [Amazon Machine Image][12] (AMI) containing the services’ code in a + Nitro Enclave. +2. **Deployment:** Run Terraform to bring up the Bidding and Auction services with inference. + +#### Step 0: Prerequisites + +1. Refer to the [Bidding and Auction services AWS cloud support and deployment guide][4] to + familiarize yourself with the packaging and deployment process for running Bidding and Auction + services on AWS. This document assumes you are able to create a functioning service stack + (following Step 0 to Step 2.5 in the Package, Deploy, and Run a Service Guide section of the + above guide). It highlights the additional steps required to enable inference capabilities. + +2. Running Bidding and Auction services with inference requires functioning machine learning models. + Choose Tensorflow or PyTorch as your ML platform, and then familiarize yourself with the + [Tensorflow][5] or [PyTorch][6] ML framework. This document assumes you are able to create and + save models in your chosen framework. + +#### Step 1: Packaging + +**Build the Amazon Machine Image (AMI):** Follow the instructions in the aforementioned [AWS +Guide][4] to build a docker image for all B&A services. The inference artifacts, including both +the “tensorflow_v2_14_0” and “pytorch_v2_1_1” sidecar binaries (ML runtimes and versions that +we currently support), will be packaged within the Bidding service image. +> **_Note:_** Other versions of Tensorflow and PyTorch will be supported in the future. + +#### Step 2: Deployment + +1. **Upload ML models:** Use an S3 bucket to host your proprietary ML models. The Bidding server + fetches the uploaded models during runtime and forwards them to the inference sidecar for + production. The bucket name is required by the Terraform configuration. Additionally, you must + upload a model configuration file that specifies the models being fetched to the cloud bucket and + include its path relative to the cloud bucket in the Terraform configuration. The model + configuration file has the following format. + + ``` + { + "model_metadata": [ + { + "model_path": "model_1", + "checksum": "dd94b3b08ea19b4240aa5e5f68ff0447b40e37ebdd06cc76fa1a2cf61143a10c" + }, + { + "model_path": "model_2", + "checksum": "" + } + ] + } + ``` + + Each model is specified by its path relative to the cloud bucket and may be accompanied by an + optional SHA-256 checksum of the model's content. The checksum can be computed by using the + following bash command: + + ``` + find -type f -exec sha256sum {} \; | sort -k 2 | awk '{print $1}' | tr -d '\n' | sha256sum | awk '{print $1}' + ``` + + Store the metadata file in the same bucket as the models. You can update it after server + deployment to fetch additional models. You must grant READ permission for your EC2 host IAM role + to access your models and use them with your service. + +2. **Upload Code Modules:** The inference capabilities of your proprietary JavaScript bidding code + modules are exposed with the `runInference` and `getModelPaths` inference callbacks. To use + inference capabilities, update your existing code modules to invoke these functions. Refer to + [this section][7] of the B&A Inference Overview Explainer for more details about these two + functions, and to [this section][13] of the AWS Guide for more information about uploading code + modules. + +3. **Configure Terraform variables:** Configure the inference capabilities using inference Terraform + flags. The names of all inference-related Terraform flags have an `INFERENCE` prefix. Include + them as part of the `runtime_flags` in the “buyer” module. + + 1. Use `INFERENCE_SIDECAR_BINARY_PATH` to expose your ML runtime selection. To enable + inference and select the ML runtime, set this flag to + `/server/bin/inference_sidecar_`, where `` is either + tensorflow_v2_14_0 for a Tensorflow runtime with the version 2.14.0, or pytorch_v2_1_1 for a + PyTorch runtime with the version 2.1.1 . + + > **_Note:_** Inference is disabled if this flag is not set. + + If inference is disabled, all inference-related Terraform flags are ignored and invocations of + inference callbacks in code modules will fail. + + 1. For recommended deployment configurations for Bidding and Auction servers with inference, + see [this configuration file][14]. You can find example values for the following flags in the + same configuration file. + + * Set `INFERENCE_MODEL_BUCKET_NAME` to the name of the S3 bucket used to store models. + * Set `INFERENCE_MODEL_CONFIG_PATH` to the path to the model configuration file + relative to the model S3 bucket. This metadata file lists the models to be fetched. + * Set `INFERENCE_MODEL_FETCH_PERIOD_MS` to the period that the bidding server checks updates in the + model configuration file and fetches models. + * Set `INFERENCE_SIDECAR_RUNTIME_CONFIG` to a JSON + config that sets runtime configurations for the inference sidecar binary. It has the following + format: + + ``` + { + "num_interop_threads": , + "num_intraop_threads": , + "cpuset": + } + ``` + + These are performance-related flags for tuning the sidecar's performance. You can find more + details about these flags [here][18]. + +4. **Apply Terraform:** There is no change in applying the Terraform step from [here][15]. + +5. **Test the Service:** To test your Bidding and Auction services with inference capabilities, you + can follow the same steps as described in [this section][16]. The Bidding and Auction flow will + execute your code modules that invoke inference callbacks. You can verify that inference is being + invoked by monitoring the “inference.request.count” metric within the AWS CloudWatch Log Group + /metrics/bidding to ensure it increments as expected. + +## Local Testing Support + +You can test the B&A server stack with inference locally, before deploying to the cloud, using the +steps outlined below. + +1. Run `builders/tools/bazel-debian build` to build each server. + + - To use debug logging, pass the `build_flavor` directly to this tool. + - Build the BuyerFrontEnd, Bidding, SellerFrontEnd, and Auction servers. + + ``` + cd $(git rev-parse --show-toplevel) + # For debug build with logging + builders/tools/bazel-debian build //services/xxx_service:server --config=non_prod + + # For prod build without logging + builders/tools/bazel-debian build //services/xxx_service:server --config=prod + ``` + +2. Generate the inference sidecar. + + - For local testing, AdTech can generate either the “tensorflow_v2_14_0” sidecar for a + Tensorflow runtime with the version 2.14.0, the “pytorch_v2_1_1” sidecar for a PyTorch runtime + with the version 2.1.1 individually, or both, depending on your needs. (For local testing, you + can choose to build one or the other. For cloud deployment, running the tool automatically + generates both sidecars.) + - Note that inference sidecars are built within different Bazel workspaces than the servers even + though the source code resides in the same Github repository. + + ``` + # To generate the Tensorflow sidecar + cd services/inference_sidecar/modules/tensorflow_v2_14_0 + builders/tools/bazel-debian build //:generate_artifacts + + # To generate the PyTorch sidecar + cd services/inference_sidecar/modules/pytorch_v2_1_1 + builders/tools/bazel-debian build //:generate_artifacts + ``` + +3. Use debug scripts to start the servers. + + ``` + cd $(git rev-parse --show-toplevel) + # To start the buyer frontend server + ./tools/debug/start_bfe + + # To start the bidding server with inference + ./tools/debug/start_bidding --enable_inference + + # To start the seller frontend server + ./tools/debug/start_sfe + + # To start the auction server + ./tools/debug/start_auction + ``` + +4. You may adjust the startup arguments in the local debug scripts to experiment with different + server startup configurations as you see fit. + +5. To test inference is being invoked: + - Refer to the information outlined [here][11] to send a request to your local Bidding and + Auction servers. + - The Bidding and Auction flow will execute the default local debug code module at + [services/inference_sidecar/common/tools/debug/generateBidRunInference.js][19] , which prints + inference related logs. You may mount a different code module in the `tools/debug/start_bidding` + script and use console logging to verify inference is being invoked. + +[1]: + https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_system_design.md +[2]: + https://github.com/privacysandbox/protected-auction-services-docs/blob/main/inference_overview.md +[3]: + https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_gcp_guide.md +[4]: + https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_aws_guide.md +[5]: https://www.tensorflow.org +[6]: https://pytorch.org +[7]: + https://github.com/privacysandbox/protected-auction-services-docs/blob/main/inference_overview.md#using-versions-in-ba +[8]: + https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_gcp_guide.md#step-24-upload-code-modules +[9]: + https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/gcp/terraform/environment/demo/buyer/buyer.tf +[10]: + https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/gcp/terraform/environment/demo/README.md +[11]: + https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_gcp_guide.md#step-25-test-the-service +[12]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html +[13]: + https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_aws_guide.md#step-24-upload-code-modules +[14]: + https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/aws/terraform/environment/demo/buyer/buyer.tf +[15]: + https://github.com/privacysandbox/bidding-auction-servers/tree/main/production/deploy/aws/terraform/environment/demo/README.md +[16]: + https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_aws_guide.md#step-25-test-the-service +[17]: https://github.com/steveganzy +[18]: + https://github.com/privacysandbox/bidding-auction-servers/blob/release-4.3/services/inference_sidecar/common/proto/inference_sidecar.proto#L102 +[19]: + https://github.com/privacysandbox/bidding-auction-servers/blob/release-4.3/services/inference_sidecar/common/tools/debug/generateBidRunInference.js \ No newline at end of file diff --git a/inference_overview.md b/inference_overview.md new file mode 100644 index 0000000..3b72602 --- /dev/null +++ b/inference_overview.md @@ -0,0 +1,552 @@ +# B&A Inference Overview + +**Authors:**
+[Akshay Pundle][4], Google Privacy Sandbox
+Trenton Starkey, Google Privacy Sandbox
+ +## Background + +This document describes the proposed inference capabilities on [Bidding and +Auction Services][5] (B&A). We examine the building blocks that B&A will provide +for running inference. + +B&A enables real-time bidding and auctions for the [Protected App Signals][6] +(PAS) and [Protected Audience][7] (PA) APIs. Real-time bidding allows +advertisers to bid on each impression individually. Calculating the bid is a +core function in B&A. The B&A servers run in the cloud in a [Trusted Execution +Environment][8] (TEE) and apply technical data protection measures through +different phases of the auction including when bids are computed. + +During bid calculation, it is often beneficial to make inference predictions. +This is the process of using a pre-trained ML model supplied by the Ad Tech to +make predictions using the given input. For example, predicting click-through +(pCTR) and conversion rates (pCVR) can help to produce the right bid. To this +end, we aim to enable the running of such inferences against a variety of +models. + +## Inference on Bidding and Auction services + +Bidding and Auction (B&A) will be expanded to include inference capabilities. +This means B&A will have the ability to run pre-trained ML models supplied by +the Ad tech to make predictions. Inference service will be supported on +[Protected Audience][7] (PA) API and [Protected App Signals][6] (PAS) flows. + +### Protected App Signals +Inference will be supported from the PAS user-defined functions (UDFs) +[generateBid()][9] and [prepareDataForAdRetrieval()][10]. For more details on +these functions see [Protected App Signals][11] (PAS). + +### Protected Audience API +Inference service will be supported from the PA [generateBid()][20] UDF. + +### Note on UDF nomenclature +Throughout this document we will use `generateBid()` as our example. `generateBid()` +is a UDF defined by the ad tech for each of the PAS and PA flows, that includes +ad tech business logic and encapsulates the logic for calculating bids. The same +inference capabilities will be available in `prepareDataForAdRetrieval()` for PAS +deployments, and may be extended to other ad tech UDFs in the future for PAS and +PA deployments. + +We will support two main ways of running inference: + +1. **Inference with embedded models**: Models are encapsulated in ad tech + provided JavaScript code +1. **Inference with externally provided models**: Models are external to the ad + tech provided JavaScript code and APIs are provided to access them + +### Inference with embedded models + +Models can be embedded in the ad tech's implementation of `generateBid()`. The +ad tech can embed simpler models in their code, either implemented in JavaScript +or with [WebAssembly][12] (Wasm). + +With this method, the model's binary data and all code required to use it for +inference are bundled in the ad tech's provided code. This method does not +require fetching the model binary data (such as weights) externally. Instead, it +relies on all data being bundled along with the code. + +This method can support fully custom models. Since the model and code are fully +controlled by the ad tech, any compatible model format and code for inference +can be used: code written in JavaScript and code that can be compiled into Wasm. + +This method works better for smaller and simpler models. + +- Complex models often require large frameworks or libraries, and other + initialization, which may incur a higher cost to run inside the JavaScript + sandbox. +- Not all model code can be translated into Wasm or JavaScript in a + straightforward way. It may be more difficult to translate complex models and + supporting code. There is a tradeoff between model size and memory cost. + +### Inference with externally provided models + +In this method, we will support loading binary model data from external sources. +The models will be fetched periodically from a linked source (e.g. [Google Cloud +Storage buckets][13], [Amazon S3 buckets][14]) and loaded into memory. An API +will be provided to `generateBid()` to run inference against the loaded models. +The whole operation of invoking `runInference()`, running predictions using the +loaded models and returning the values to the caller will take place inside a +single TEE. Data will not leave the TEE, and will not be observable externally +either. Outside observers will not be directly able to observe how many times +inference was invoked, if at all. + +We will initially support [Tensorflow][15] and [PyTorch][16] based models. + +We will provide the following APIs, which will be usable from ad tech code: + +- **Run inference using the external models**: This is the main way to use + inference. A batch of inputs can be constructed in ad tech code and passed to + the API. Inference will run for each input on the given pre-loaded model and + the responses will be returned. The call, for illustrative purposes, may look + like the following: + ``` + runInference(modelPath, [input1, input2...]) + ``` +- **Get the list of paths of models available for use**: This would get the list + of models currently loaded and available for use in the server. The paths of + models would typically encapsulate version information (see [Versions in + B&A][17]). Along with the versioning strategy, this API gives the ad tech + precise control in choosing versions of models to run inference against. The + call, for illustrative purposes, may look like the following: + ``` + getModelPaths(...) + ``` + +Note that these APIs are for clarity only and may not reflect the actual +specification. We will define the actual API in more detail in future iterations +of this explainer. These APIs will also be callable from Wasm. + +## Factorized and Unified models + +Model inputs can use private features (e.g. user data) and non-private features +(e.g. ad data). Private features are available only inside the TEE and so, such +inference must run inside the TEE. This is because the private data should not +leave the privacy preserving boundary. Non-private features can be available on +the contextual path ([see B&A system design][18]), and models relying only on +such data can run outside the TEE, on ad tech hardware, and may also pre-compute +inferences. + +Common bid prediction models use contextual, ad and user features for +prediction. Out of these, only the user features are private. The machine +learning models using such private features are the only ones that need to run +inside the TEE. Contextual and ad features are not private, so pre-computation +on these can be done outside the TEE. + +For example, a model that relies only on contextual features can be run in the +contextual path during the RTB request. A model based on only ad features can +pre-compute results per ad and upload them to the K/V server. Such results can +be queried during request processing and made available to `generateBid()`. + +On the other hand, a model that uses user features, or one that uses user and ad +features will need to run inside the TEE, and cannot run outside the TEE +boundary. + +### Unified models + +A **unified** approach to such a prediction could run a model that takes +contextual, ad and user features as input, and makes a prediction, running the +entire model inside the TEE. B&A will support unified models directly through +the [two ways][19] of running inference (embedded and external models) described +above. In the unified approach, the entire model will need to run inside the +TEE, since it uses private features. These models will tend to be larger. Both +these factors can increase the cost and latency of running such models. + +### Factorized models + +**Model factorization** is a technique that makes it possible to break a single +model into multiple pieces, and then combine those pieces into a prediction. In +B&A-supported use cases, models often make use of three kinds of data: user +data, contextual data, and ad data. + +As described above, in the unified case, a single model is trained on all three +kinds of data. In the factorized case, you could break the model up into +multiple pieces. The model that includes the sensitive user data needs to be +executed within the trust boundary, on the buyer's bidding services. + +This makes the following design possible: + +1. Break the model up into a private piece (the sensitive user data) and one or + more non-private pieces (the contextual and ad data). +1. Optionally, inferences from some or all of the non-private pieces can be + passed in as arguments to a [User Defined Function][1] (UDF), e.g. + `generateBid()`. For example, contextual embeddings can be computed along the + contextual path, returned to the seller's frontend in the contextual response, + and then can be passed into the B&A auction as part of the `buyer_signals` + constructed by the seller when they initiate the Auction. +1. Optionally, ad techs can create models for non-private pieces (such as ads + features) ahead of time, then materialize embeddings from those models into + the ad retrieval server (for PAS) or [Key/Value server][21] (for PA) and fetch + these embeddings at runtime. +1. To make a prediction within a UDF, combine private embeddings (from + inference within the TEE) with non-private embeddings (from UDF + arguments or those fetched from the ad retrieval / [key/value servers][21]) + with an operation like a dot product. This is the final prediction. + +The following figure shows a factorized pCTR model broken into three parts, +contextual, user and ads towers. Only the User tower needs to run inside the +TEE. The contextual and ad towers can run outside the TEE since they do not rely +on private features. + +
+ Factorized pCTR model +
Figure 1. Factorized pCTR Model
+


+ +Factorized models are useful because loading and executing large models inside +the TEE may be expensive. Oftentimes, only a small part of the model relies on +private data (like user data), and the rest needs non-private (e.g. contextual +or ad) data. With factorization, parts that depend on non-private features can +be outside the TEE where they can be executed on ad tech hardware, while keeping +the models requiring private features inside the TEE. + +Factorization can be utilized only when the model can be broken down into its +constituents (as described above) fairly easily. If models rely on a lot of +cross features (e.g. ad x contextual information), factorization may not be an +effective strategy. + +We will support data flows that allow querying and passing such embeddings to +`generateBid()`, where they can be combined with user (private) embeddings to make the final prediction. The next section describes this data flow. + +## Overall flow + +### Loading external models + +B&A will support loading models inside the TEE. B&A servers will fetch and load +ad tech models periodically from a connected cloud bucket and make them +available for inference in the ad tech provided code (e.g. `generateBid()`). The +loading will be performed in the background, separate from the execution of +`generateBid()`. After loading, the models will also be pre-warmed in the +background, so that when the models are used in the request flow, they are +already warmed. + +
+ Loading external models +
Figure 2. Loading external models
+


+ +The loaded models will be executed in the supported frameworks (e.g. Tensorflow, +PyTorch). When `runInference()` is invoked from within `generateBid()`, the inference request will be forwarded to the corresponding executor for processing, and the results will be returned back to the calling function. These calls occur within the confines of a single TEE host, and data never leaves the TEE. + +Note that `runInference()` will not call out to the cloud buckets, or trigger +loading of the model. It will only execute requests against models that have +already been loaded. + +### Flows to support factorized models + +To support factorized models, B&A will support data flows that enable +transferring embeddings produced from the model parts to `generateBid()`, so +that they can be combined together with the prediction made through the +`runInference()` call. These flows are based on the existing [request +processing][2] flows in B&A. + +B&A will support these flows as following: + +- Support passing embeddings generated as a part of the Contextual RTB auction + to `generateBid()` through `buyer_signals`. These embeddings can be produced + in real time, or can be looked up from ad tech servers during the contextual + RTB request processing. +- For PAS workflows + - Support model loading and inference from `prepareDataForAdRetrieval()` + through the `runInference()` call. `prepareDataForAdRetrieval()` can be + used to generate private embeddings that are passed to the [Ad Retrieval + service][11] to filter and fetch Ads. + - Support querying embeddings from the Ad Retrieval service (runs inside + TEE) and passing them to `generateBid()`. These embeddings will be + produced offline and uploaded to the Ad retrieval service. The keys will + define what the embeddings are for, and appropriate keys can be looked up + during [retrieval request][10] to fetch the pre-computed embeddings +- For PA workflows, + - Support querying embeddings from the Key/Value service and passing them to + generateBid as a part of the regular key/value lookup. These embeddings + will be produced offline and uploaded to the ad techs key/value service. + The keys should be defined by the ad tech and will be fetched as a part of + the regular key/value fetch and passed on to `generateBid() +- Support model loading and inference in `generateBid()` through the + `runInference()` call, as described above. Here, inference runs inside TEE and + can use private features based on the inputs of `generateBid()`. This produces + an embedding that can be combined with the embeddings from the contextual + path and ad retrieval service, as described above. The data flows make + these embeddings available in `generateBid()` so they can be combined with + the private embedding to make a final prediction. + +The entire flow is encapsulated in the diagrams below along with an explanation +of each step. + +### PAS flow + +This flow is based on the [Protected App Signals flow][11] which +fits into the [B&A request processing flow][2]. + +
+ Sample flow for PAS factorized models +
Figure 3.1 Sample flow for PAS factorized models
+

+ +1. Client sends a request to the seller's untrusted server. +1. As part of real-time bidding (RTB) ad request, the seller sends contextual + data (publisher provided data) to various buyers. At this stage, buyers run an + ML model that requires only contextual features. This request occurs on the + contextual path not running on the TEEs. It runs on ad tech-owned hardware of + their choosing. +1. Once the contextual requests are processed, the contextual ad with bid is + sent back to the seller. Along with this, the buyer can send `buyer_signals`, + which the seller will forward to the appropriate buyer front end (BFE). The + `buyer_signals` is an arbitrary string, so the buyer can send the contextual + embedding and the version of the models being used (that `generateBid()` + should use) as a part of this. +1. The seller server forwards the request to the seller front end (SFE), which + runs inside the TEE. +1. The SFE sends requests to the buyer's BFE. It will include the + `buyer_signals` as a part of the request. +1. The BFE sends a request to the Bidding service. The Bidding service runs + `prepareDataForAdRetrieval()` that can optionally invoke inference. This UDF + prepares the request to send to the Ad retrieval server. +1. The request prepared above is sent to the Ad retrieval server, that will run + logic to fetch Ads. +1. The retrieved ads are sent back to the bidding service, where they are given + as input to `generateBid()`. Ad embeddings can be included as a part of this + response. The Bidding service then invokes `generateBid()`. This call includes + the `buyer_signals` and data from the Ad retrieval server. Thus, contextual + embedding and version (part of `buyer_signals`) and the ad embeddings (part of + data looked up from the Ad retrieval server) are sent to the `generateBid()` + invocation. +1. The calculated bid is returned. To compute the bid, `generateBid()` can run + models through `runInference()calls` and combine the output embedding with + other embeddings that were fetched above to make a final prediction. + +In the above flow, `prepareDataForAdRetrieval()` and `generateBid()` can use +external models through the `runInference()` API call. This runs inside the TEE +and can perform inference with models using private features (e.g. the user +tower). Alternatively (not shown in figure), each of the UDFs can use embedded +models to perform inference. + +### PA flow +The following diagram is a sample flow based on the [Protected Audience API][7] which +fits into the [B&A request processing flow][2]. + +
+ Sample flow for PA factorized models +
Figure 3.2 Sample flow for PA factorized models
+

+ +A flow for a Protected audience deployment will be similar to the above. + +1. Client sends a request to the seller's untrusted server. +1. As part of real-time bidding (RTB) ad request, the seller sends contextual + data (publisher provided data) to various buyers. At this stage, buyers run + an ML model that requires only contextual features. This request occurs on + the contextual path not running on the TEEs. It runs on ad tech-owned + hardware of their choosing. +1. Once the contextual requests are processed, the contextual ad with bid is + sent back to the seller. Along with this, the buyer can send `buyer_signals`, + which the seller will forward to the appropriate buyer front end (BFE). The + `buyer_signals` is an arbitrary string, so the buyer can send the contextual + embedding and the version of the models being used (that `generateBid()` + should use) as a part of this. +1. The seller server forwards the request to the seller front end (SFE), which + runs inside the TEE. +1. The SFE sends requests to the buyer’s BFE. It will include the + `buyer_signals` as a part of the request. +1. The BFE performs K/V lookup from the BYOS K/V server (or TEE-KV server, + depending on ad tech deployment). +1. The retrieved keys are sent to the bidding service, where they are given as + input to `generateBid()`. This can include ad embeddings that were queried from + the K/V server. The Bidding service then invokes ``generateBid()``. This call + includes the buyer_signals and data from the K/V server. Thus, contextual + embedding and version (part of `buyer_signals`) and the ad embeddings (part of + data looked up from the K/V server) are sent to the generateBid() invocation. +1. `generateBid()` logic is executed for each of the [Interest Groups][23] (IGs). This + executes the logic and each invocation can make calls to `runInference()` to + perform inference against any of the loaded models. The `generateBid()` logic + can also combine the output embedding with other embeddings (like ad and + contextual embeddings) that were fetched above to make a final prediction. +1. The final prediction is returned from `generateBid()`. + +In the above flow, `generateBid()` can use external models through the +`runInference()` API call. This runs inside the TEE and can perform inference +with models using private features (e.g. the user tower). Alternatively (not +shown in figure), each of the UDFs can use embedded models to perform inference. + +## Versioning + +Inference on B&A will support factorized models. There are multiple models and +embeddings that contribute towards the final prediction, some of which may be +pre-computed. It is important to make sure that model and embedding versions +used are consistent. + +For facilitating this, we will provide the buyer a way to choose the version of +models to run per request. As a part of contextual request processing, a version +for inference can be decided by the buyer. The model version information can be +passed through the `buyer_signals`. This field is a string and can be used to +pass arbitrary information from the contextual RTB request to `generateBid(). +buyer_signals` will also be passed to the K/V server where this model version +information can be used to look up embeddings. The ad tech is free to choose the +structure of the `buyer_signals` including how the model information is passed +and what it contains. For example, something like the following could be used: +``` +buyer_signals = { + ... + "model_version": { "pCVR_version": 1, "pCTR_version": 2 } + ... +} +``` +### Using versions in B&A + +The `runInference()` API runs inferences against a named model. `generateBid()` +is responsible for interpreting the version included in `buyer_signals` and +translating it to an actual path that corresponds to an entity in the cloud +bucket. A simple translation scheme could be something like +`${model_name}-${model_version`}. This name will be translated into a cloud +bucket folder from which the model was loaded as shown in the figure below. This +figure is for illustrative purposes only. It shows the mapping from model path +to an actual path in the cloud bucket. The cloud bucket will only be synced +periodically, so each `runInference()` call will not call out to the cloud +bucket. Thus, there will be some delay between a new model being available in +the cloud bucket, and it being made available to `runInference()`. We will cover +such details in future updates to this explainer. + +
+ Model name translation +
Figure 4. Model name translation
+


+ +The `getModelPaths()` function described [earlier][3] can be used along with +the version information to implement a versioning strategy by calling +`runInference()` with the chosen version. Here are some examples how this +could be done: + +``` +// run inference against a hardcoded version of a model +modelPath = translateToPath("pcvr", 1.0) // custom ad tech logic +runInference(modelPath, [inputs...]) + +// run inference against version specified in buyer signals +modelPath = translateToPath("pcvr", buyer_signals.model_version.pCVR_version); +runInference(modelPath, [inputs...]) + +// run inference against the latest version of a model +models = getModelPaths() +latestPcvrVersion = findLatestPcvr(models) // custom ad tech logic +modelPath = translateToPath("pcvr", latestPcvrVersion); +runInference(modelPath, [inputs...]) +``` + +## Supported platforms + +### Embedded models + +Ad tech will have full control over what platforms and model types are used +here. The models can be encapsulated in the ad tech provided JavaScript or Wasm +code, thus JavaScript and any language that compiles to Wasm will be supported. +Ad tech can bundle any libraries of their choice along with their code, so any +frameworks or custom written code can be supported as long as it is capable of +executing in the JavaScript sandbox. + +### ML execution for External models + +We will initially support 2 popular ML platforms: Tensorflow and PyTorch. More +detail to come in future updates on what model formats we will support based on +ecosystem feedback. + +### HW Acceleration + +Accelerator hardware availability is dependent on the cloud platforms. As of +this writing, confidential processing within TEEs takes place on CPUs. If or +when additional hardware like GPUs becomes available in cloud provider TEE +environments, we look forward to supporting them. + +Note that this pertains only to models executed inside the TEE (i.e. on B&A +servers). For factorized models with components outside the TEE , the ad tech is +free to choose whatever hardware they prefer. + +## Model size limits + +Larger models can help ad techs get higher utility while trading off cost. +Larger models have more memory and computation cost, and thus can be more +expensive to run. They can also cause greater latency in predictions. Thus, the +size of models that are reasonable to run depends on ad tech setup and the +tradeoffs made. + +We will not have any particular model size limits, and it will be up to the ad +tech to determine the cost vs utility curve to see what sizes of models are +sufficient, and how many inferences are made per request. + +For external models, based on our experiments, models up to 100Mb should have a +reasonable cost and latency. Ad techs should verify performance against their +own models to make sure the cost, utility and latency is in line with their +expectations. + +Embedded models are much more custom, and ad techs should experiment with their +setup to determine reasonable limits based on their cost and latency thresholds. + +## Sources of cost + +For inference, the cost predominantly comes from the resource usage to perform +inference (CPU costs, in our case). There can be some overhead from various +factors (e.g. translating inputs during request dispatch, fetching and storing +models etc.) Since this translates to cost, we will try to minimize such +overhead. + +We have seen that CPU utilization for inference scales with model size and +number of inference requests. Larger models consume more resources, as do more +requests. This can be a factor in considering how you factorize your models. + +For example, if the model running inside the TEE uses only **user** features, +then the results from this model can be shared across all ads being processed in +that invocation of `generateBid()`. Thus, inference needs to be called only once +per model class (i.e. once for pCVR, once for pCTR, etc.). This cost will scale +with the number of model classes. + +On the other hand, if the model running inside the TEE uses **user x ad** +features, then the predictions will be different per ad. Thus, inference will +need to be called for each ad that is being processed in `generateBid()`. So, +for each ad, pCVR and pCTR models will need to be called. Here the cost will +scale with the (number of model classes) x (number of ads). + +We will provide more guidance around cost aspects in future updates, or in a +separate explainer. + +## Seeking feedback + +We are actively engaging with the ecosystem to gather feedback as we develop our +inference solution. PyTorch or TensorFlow were the two most commonly mentioned +frameworks during our outreach. We are starting with these, and welcome +additional feedback. + +In addition, ad tech should also consider embedded models that can support +JavaScript and any language that can be compiled down to WebAssembly (see [list +here][22]). You can use this option for custom code, custom model formats, etc. + +[1]: https://cloud.google.com/bigquery/docs/user-defined-functions +[2]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#overview +[3]: #inference-with-externally-provided-models +[4]: https://github.com/akshaypundle +[5]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md +[6]: https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals +[7]: https://developer.chrome.com/docs/privacy-sandbox/protected-audience/ +[8]: https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trusted-execution-environment +[9]: https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals#generatebid +[10]: https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals#preparedataforadretrieval +[11]: https://developer.android.com/design-for-safety/privacy-sandbox/protected-app-signals#buy-side-ad +[12]: https://webassembly.org/ +[13]: https://cloud.google.com/storage/docs/buckets +[14]: https://aws.amazon.com/s3/ +[15]: https://www.tensorflow.org/ +[16]: https://pytorch.org/ +[17]: #using-versions-in-ba +[18]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_system_design.md#sellers-ad-service +[19]: #inference-on-bidding-and-auction-services +[20]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#generatebid +[21]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#buyer-byos-keyvalue-service +[22]: https://webassembly.org/getting-started/developers-guide/ +[23]: https://developers.google.com/privacy-sandbox/private-advertising/protected-audience#interest-group-detail diff --git a/key_value_service_contextual_signals_propagation.md b/key_value_service_contextual_signals_propagation.md new file mode 100644 index 0000000..acb8a05 --- /dev/null +++ b/key_value_service_contextual_signals_propagation.md @@ -0,0 +1,72 @@ +# Passing `buyer signals` from B&A to TKV for PA flow + + +# Overview + +For a Protected Audience (PA) auction, we are passing an existing input signal [Buyer signals](https://github.com/privacysandbox/bidding-auction-servers/blob/09e0c1fcd1aa4acd4fbd3ae8285eda882b197314/api/bidding_auction_servers.proto#L675) to TEE based [Key Value service (TKV)](https://github.com/privacysandbox/protected-auction-services-docs/blob/81eaa0b471c1ea0d8299816feea78318b004e57e/key_value_service_use_cases.md) with [Bidding and Auction (B&A)](https://github.com/privacysandbox/protected-auction-services-docs/blob/81eaa0b471c1ea0d8299816feea78318b004e57e/bidding_auction_services_api.md) version 4.5.0. Starting with that version, `Buyer signals` can be optionally propagated to the buyer’s TKV in the encrypted payload, i.e. never through cleartext. + +The purpose of this document is to explain how this works and how to enable it. + + +# Background + +During a PA auction performed using B&A servers, the Buyer Front End (BFE) calls out to the buyer’s TKV. Signals sent to TKV are [limited](https://github.com/WICG/turtledove/blob/fab196ac40402a1c537fd328aa4549c492ad314b/FLEDGE_Key_Value_Server_API.md#schema-of-the-request) because historically the TKV implementation matched Bring-Your-Own-Server (BYOS) functionality. The signals sent to BYOS are limited to reduce the impact if the service operator uses them to create a user’s cross-site profile. + +Now that the TKV implementation can be trusted to protect the user’s data, we are considering what additional signals can be made available in TKV. + +BFE has had access to [buyer signals](https://github.com/privacysandbox/bidding-auction-servers/blob/09e0c1fcd1aa4acd4fbd3ae8285eda882b197314/api/bidding_auction_servers.proto#L675), but previously TKV did not. More context on what buyer signals are is available [here](?tab=t.0#heading=h.8v5dxea5zu67). See this [section](?tab=t.0#heading=h.nvk9u9iec1k5) for more context on where the buyer signals are coming from. See this section talking about [motivation](?tab=t.0#heading=h.otnyz0js7au8). + + +# Motivation + +To select the best ad to show to a user, adtechs can utilize all available data. They also want to make their ad selection decision in the environment that not only has compute, e.g. B&A, but has access to data, i.e. TKV. + +To that end we can consider sending more contextual data to the TKV, where adtechs can use that data in their user defined functions (UDFs) along with data from advertiser sites to make improved ad retrieval decisions in an environment that has access to the AdTech's loaded data. + + +# What are `buyer signals` + +![Architecture diagram.](images/unified-contextual-remarketing-bidding-auction-services.png) + +The `buyer signals` field is a free form string. That string has serialized _contextual_ signals corresponding to each buyer in an auction that could help in generating bids. Each buyer is aware of the format in which that string is serialized and can take advantage of it in respective UDFs. + +Buyer signals are [set](https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L623) by an untrusted Seller Ad Service based on the contextual information available at the time of the B&A `SelectAdReqest`request creation. \ +`buyer signals` have a parameter name of `perBuyerSignals` in [generateBid](https://github.com/privacysandbox/protected-auction-services-docs/blob/81eaa0b471c1ea0d8299816feea78318b004e57e/bidding_auction_services_api.md#generatebid-javascriptwasm-spec). + + +# Propagation details + + +## Seller's ad server to B&A + +As mentioned above, buyer signals are [set](https://github.com/privacysandbox/bidding-auction-servers/blob/8b1a9808c279ebc66dc360d14b5ad9d35e34965c/api/bidding_auction_servers.proto#L724) by an untrusted Seller Ad Service based on the contextual information available at the time of the B&A `SelectAdReqest`that is sent to [SellerFrontEnd service (SFE).](https://github.com/privacysandbox/protected-auction-services-docs/blob/81eaa0b471c1ea0d8299816feea78318b004e57e/bidding_auction_services_api.md#sellerfrontend-service) + +That means that the perBuyerSignals (contextual signals) need to be available before SFE is called. + +SFE then [sends](https://github.com/privacysandbox/bidding-auction-servers/blob/722e1542c262dddc3aaf41be7b6c159a38cefd0a/api/bidding_auction_servers.proto#L854) that info to BFE. + + +## B&A to buyer's TEE Key/value service + +BFE sends that info in the [metadata](https://github.com/privacysandbox/protected-auction-key-value-service/blob/5d586e0046e7b482e70c1b97bf322a923340bfab/public/query/v2/get_values_v2.proto#L109) field `buyer_signals` request. That field is later available in each UDF execution. See the UDF API section in this [doc](https://github.com/privacysandbox/protected-auction-key-value-service/blob/6f702e02833a06c1bfd6a31f7d9c8f0ded98536d/docs/APIs.md) and this [proto](https://github.com/privacysandbox/protected-auction-key-value-service/blob/6f702e02833a06c1bfd6a31f7d9c8f0ded98536d/public/api_schema.proto#L30). + + +### Enabling signals propagation + +These signals won’t be propagated by default. By default there is no extra bandwidth or change in behavior. \ +An Adtech can choose to propagate by enabling a feature flag `propagate_buyer_signals_to_tkv` in BFE. + +Adding buyer signals to the payload is extra bandwidth. + + +# On device + +We will add buyer signals propagation support for on device shortly after. We’re planning to update this document with additional details once they are finalized and are ready to be shared. + + +# Feedback + +Feedback on this feature is welcome. Please provide it by opening an issue on github. + + +[1]: https://github.com/lx3-g diff --git a/key_value_service_cost.md b/key_value_service_cost.md new file mode 100644 index 0000000..2d08445 --- /dev/null +++ b/key_value_service_cost.md @@ -0,0 +1,310 @@ +# Key/Value Service Cost + +**Authors:** +
[Leiming Zhang][1], Google Privacy Sandbox +
[Lusa Zhan][2], Google Privacy Sandbox + +# Introduction + +[The Key Value service](https://github.com/privacysandbox/protected-auction-key-value-service/tree/main) is developed with the same underlying technologies as the Bidding and Auction services. Both systems have similar deployment and trust models and the general cost structures of the two are therefore similar. This explainer begins by quoting the [background section](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_cost.md#background) of the Bidding and Auction services. Various components and cost factors may also overlap with the Bidding and Auction services and similar quotations may be used to reflect the consistency. Quoted texts use italic style. Differences between the Key Value Server and Bidding and Auction Services are highlighted in red in quoted texts. + +# Background + +“ ${\color{red}The}$ ${\color{red}Key}$ ${\color{red}Value}$ ${\color{red}service}$ _runs in supported [Trusted Execution Environments (TEE)](https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trusted-execution-environment) provided by AWS and GCP. In this document, the Privacy Sandbox team explores the various factors that affect the cost of such a deployment and how these can be mitigated._ + +_There are different tradeoffs when provisioning this system like [QPS](https://en.wikipedia.org/wiki/Queries_per_second), acceptable latency, regional failover etc. Providing estimates for all combinations is impossible, and a lot depends upon the choice the ad tech makes, the tradeoffs that are considered etc._ + +_A framework has been developed to understand the costs of operating a privacy-preserving advertising system. This framework will help ad tech companies understand the source of costs and make informed decisions about how to operate their businesses._ + +_The cost components described depend on the underlying cost structures provided by cloud providers. The Privacy Sandbox team takes into account our current understanding of these, but the underlying cost structures can change over time, which would also change the cost components described in this document._ + +_The cost components described below scale with the throughput of the system. Provisioned costs (e.g. number of VMs) depend on how many resources are provisioned. Costs like network and load balancer cost scale with actual usage and are proportional to the volume of traffic processed by the ad tech._ + +_The cost depends on system performance which will need to be empirically measured. Ad techs can follow instructions in the [Cost Estimation](#cost-estimation) section or_ +${\color{red}deploy}$ +${\color{red}our}$ +${\color{red}system}$ +${\color{red}for}$ +${\color{red}Protected}$ +${\color{red}Audience}$ +${\color{red}/}$ +${\color{red}Protected}$ +${\color{red}App}$ +${\color{red}Signals}$ +${\color{red}testing}$ +_to collect such data and create detailed cost estimates._ + +_While it is impossible for the Privacy Sandbox team to collect representative data since it does not operate a production system with live traffic, it can measure resource consumption for a parameterized system with synthetic requests and setup. Ad tech can use these to arrive at ballpark cost estimates._ + +_Typical cloud costs fall into the following categories:_” + +## Network + +_“Network costs can be a significant component of operating costs. Services are usually operated inside a Virtual Private Cloud (VPC). Typically, network egress from a VPC is charged. This means that you would pay for outbound traffic from the services that go outside the VPC boundary, like going to the internet or to another region but not for traffic inside the VPC. Traffic inside VPC is usually free. Internal network components (like load balancers) that charge for data processing will still incur cost on internal traffic. For this reason, the system uses_ +${\color{red}direct}$ +${\color{red}calls}$ +${\color{red}with}$ +${\color{red}internal}$ +${\color{red}IP}$ +${\color{red}addresses}$ +${\color{red}for}$ +${\color{red}internal}$ +${\color{red}sharding}$ +${\color{red}traffic}$ +. +_This does not incur cost for traffic inside the VPC. Some network components like NAT (AWS, GCP) have hourly charges, and ingress and egress data charges based on usage.”_ + +## Infrastructure + +_"Infrastructure costs include Virtual Machine (VM) costs and other services used. VM cost depends on CPU, memory usage etc. These costs typically balance QPS, application latency and availability needs."_ + +## Data loading + +There are two ways to load data into the server: cloud storage and realtime update. Cloud storage cost depends on data size and metadata operations while realtime update cost depends mostly on data throughput. + +## Load balancer + +_“Load balancers typically charge for data processed. This includes inbound and outbound data. If a load balancer is used on internal traffic, this processing is still usually chargeable._ + +_NAT gateways are also chargeable. These are used to make outbound connections out of the VPC and typically also charge for processed data.”_ + +## TEE overhead + +_“Cloud operators may charge an overhead for TEE enclave usage. The way these charges are levied is different and operators should use their setup to determine how TEE costs affect the overall cost._ + +_GCP: [Confidential Compute charges](https://cloud.google.com/compute/confidential-vm/pricing)_ + +_AWS: [Enclave pricing](https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave.html#nitro-enclave-pricing)_” + +# High level cost breakdown + +
+ Fig. 1 - Key/Value server cost +
Figure. 1 Key/Value server cost
+


+ +Key/Value server costs breakdown in the following way, as shown in the figure above. + +**C1: _Load balancer:_** “_A load balancer (LB) forwards incoming requests to the_ +${\color{red}key/value}$ +${\color{red}server}$ +_instances. This cost depends on the total processed bytes by the LB and may also include components based on usage time.”_ + +**C2: Key/Value server:** Key/Value service processing consumes CPU and memory. + +**C3: Storage API requests and data transfer:** API requests like listing and fetching objects incur cost. Depending on the regions of your cloud storage and server instances, data transfer may incur cost. + +**C4: Cloud data storage:** Data storage for snapshots and delta files will incur cost. Metadata operations (e.g., listing objects) and inter-region data transfer may also incur cost. + +**C5: Realtime updates:** This is a low latency data loading pathway. The cost here mostly depends on data throughput. + +**C6: Response to external client:** This is the network egress cost of sending the responses back to the external client. NAT costs may also be accrued since a connection across the VPC boundary will be needed. + +**C7: Server monitor:** Server monitoring services will incur cost. This includes OpenTelematry (Otel) collector and monitoring services from cloud providers. + +# Details + +## Load balancer and Network address translation (NAT) + +### Load balancer + +_“The_ +${\color{red}key/value}$ +${\color{red}server}$ +_will have a load balancer for incoming requests. These will incur processing costs (_ +${\color{red}C1}$ +${\color{red}in}$ +_Fig. 1) depending on the total bytes routed. This will depend on QPS and request payload size.”_ + +However, if the key/value server only needs to handle traffic within the same VPC (e.g., traffic from Bidding and Auction services deployed under the same VPC. In Fig. 1, it’s shown as the internal client), you can disable the load balancer and no related cost will incur. Note that this feature (optional load balancer) is only available on GCP at the moment and support for other clouds (e.g., AWS) will be available later. + +### NAT + +_“A NAT is used to make outbound connections to the internet from within the VPC. NAT traffic is typically charged for bytes processed. The cost can be estimated by estimating the traffic, and can be controlled by limiting it.”_ + +### Traffic director (GCP) + +On GCP, Traffic director is used to handle internal traffic (e.g., traffic from an internal client). This cost depends on the total number of endpoints (instances). + +## VM costs + +The key/value server requires CPU and memory. From [Fig. 1](?tab=t.0#bookmark=id.oa1u44lgx1im), the costs for these are C2. These costs depend on factors like the total shards/replicas required, the size of dataset in a single shard, total throughput required, throughput per vCPU of each of the instances, how the servers are provisioned, what latency is acceptable, complexity of UDF etc. + +**Total shards/replicas required**: Sharding allows splitting data and having each server store only part of the data. Replicas for each shard provide more throughput bandwidth and extra reliability. [More about sharding and replica](https://github.com/privacysandbox/protected-auction-key-value-service/blob/release-0.15/docs/sharding/sharding.md). + +**Size of dataset per VM**: Data needs to be loaded to a machine’s RAM. Larger dataset requires machines with more RAM. + +**_Total throughput required_**: _“Desired throughput of request being processed. The required resources scale proportionally to this.”_ + +**_Throughput per vCPU:_**: “_Needs to be experimentally determined”_ + +**_How the servers are provisioned_**: _“Peak QPS may be much more than avg. QPS. Provisioning servers for peak QPS and extra redundancy would require more cost.”_ + +**_Latency_**: _“If degraded latency is acceptable, servers may be underprovisioned. If this tradeoff is made, at high QPS, the servers could start to lag and latency could be higher.”_ + +**Complexity of UDF**: Running User-Defined Functions (UDF) requires CPU and memory. The more time they run for, the more resources they consume. [More about UDF](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/key_value_service_user_defined_functions.md). + +## Network costs + +**Response to external client**: The response to external client will incur network egress/NAT costs. These will depend on the payload sent back, which will contain the query result. This cost is C6 in Fig. 1. + +**Response to internal client**: This will not incur any network cost as both the key/value server and the internal client are deployed in the same private network. + +## Data loading costs + +### Cloud storage + +Storage cost mostly depends on how much data (i.e., snapshot/delta files) you need to store and how frequently you need to do API requests (e.g., Get, List) to cloud storage. + +**Data storage**: The cloud storage for snapshot and delta files will incur cost, which depends on the total data storage size and their timespan in storage. Your choice of storage location also has an influence on the cost. This cost is C4 in Fig. 1. + +**Storage API requests**: Storage API requests like Get, List will incur cost. As for data reading/access, although the cost is not directly related to data size, the system may need to make multiple API requests depending on the buffer size. + +**Data transfer:** If the cloud storage bucket resides in a different region than server instances, additional data transfer cost will incur. + +### Realtime updates + +Realtime updates enable low-latency data loading via message queues. The system leverages AWS SQS/SNS and GCP Pub/Sub for realtime updates, respectively. Pub/Sub costs are based on data throughput. For SQS/SNS, the costs are based on the number of requests while each request is 64KB at max and any request with larger size is billed as multiple requests. Data transfer within the same region is typically free for both services. However, inter-region transfers may incur additional charges. This cost is C5 in Fig. 1. + +AWS: [SNS pricing](https://aws.amazon.com/sns/pricing/), [SQS pricing](https://aws.amazon.com/sqs/pricing/) + +GCP: [Pub/Sub pricing](https://cloud.google.com/pubsub/pricing) + +## Monitoring costs + +Monitoring related costs are C7 in Fig. 1. Note that the monitoring solution is provided as a reference implementation and can be customized. + +### Otel Collector + +Otel collector runs as a standalone service to collect key/value server metrics. On AWS, it’s running in the host machine, while, on GCP, it’s running in a dedicated VM. Regardless of platforms, VM related costs will incur for Otel collectors. + +### Cloud Monitoring + +Metrics collected by the Otel collector will be exported to Cloudwatch/Prometheus on AWS and Metrics Management / Logs explorer on GCP. All these cloud provided services will incur costs. + +# Cost estimation + +“_Both GCP and AWS itemize all costs needed for a key/value server deployment, and this can be used to build a quite accurate estimate of running costs. Estimating cost for a range of ad techs running these servers proves difficult due to the sheer number of factors that affect cost, including but not limited to:_ + +- _Deployment configuration for each ad tech_ +- _Estimation of compute capacity_” +- Estimation of payload size, including for sharded servers and [UDF read APIs](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/key_value_service_user_defined_functions.md#datastore-read-api) +- _“Estimation of response size”_ +- _“Identifying where pricing applies and the correct line item that should be applied.”_ +- CPU time taken up by UDFs + +## Cost test environment + +_“To overcome these obstacles, running the key/value server in a test environment over a short period of time may be used to extrapolate monthly or annual costs (or to verify calculated estimates). Guides are available for deploying an environment in [GCP](https://github.com/privacysandbox/protected-auction-key-value-service/blob/release-0.15/docs/deployment/deploying_on_gcp.md) and [AWS](https://github.com/privacysandbox/protected-auction-key-value-service/blob/release-0.15/docs/deployment/deploying_on_aws.md). The test environment should replicate, as closely as possible, the architecture that will be used in production. This includes:”_ + +- Deployment related factors: + - _“Network configuration and load balancers”_ + - Correct sizing of the instances + - Sharding configuration +- Runtime related factors: + - _“Request payloads and header information that are similar in size and composition to production payloads”_ + - User-defined functions implementation with similar latency and response size. In addition, they should mimic the number, request payload size, and request composition of [UDF read API](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/key_value_service_user_defined_functions.md#datastore-read-api) calls + - Overall data set size and individual record sizes + - Data update frequency and size while the server is running. This includes realtime updates. + +_“Correct sizing and quantity of the host instances will be a significant factor in the total cost of the environment.”_ Load testing should be used to identify the optimal number of vCPUs required. This will heavily depend on the runtime related factors listed above. + +_“To facilitate identifying the billing costs attributed by the test environment, each component can be [labeled (GCP)](https://cloud.google.com/resource-manager/docs/creating-managing-labels) or [tagged (AWS)](https://docs.aws.amazon.com/tag-editor/latest/userguide/tagging.html) with unique key value sets. These are needed to isolate the costs involved in the test environment from those of any other resources in use within the same project (GCP) or account (AWS). The [reference terraform configurations](https://github.com/privacysandbox/protected-auction-key-value-service/tree/release-0.15/production/terraform) in the code base demonstrate how this can be done.”_ + +However, while many components support labeling and tagging, many others do not. This includes GCP CloudNAT and some monitoring related services. To further isolate costs for these services, consider placing the test environment in a region by itself. + +## Running a cost test + +The key/value server supports both reading data via the external API and writing data via data uploads to blob storage. Both reads and writes will affect each other’s latency. + +In order to estimate the cost for the key/value service accurately, test setups for both need to replicate a production payload as close as possible and be run at the same time. + +_“Each test run should cover a minimum of 24 hours within the daily billing period of the cloud service that is being used._ + +- _AWS: midnight UTC - midnight UTC_ +- _GCP: midnight PST - midnight PST”_ + +### Data loading setup (write path) + +The test setup for the write path should include delta/snapshot updates at the same frequency, record size, and record numbers as the expected production data. This also includes real time updates. Since data generation is not part of the key/value service, each ad tech will need to set up their own pipelines to replicate data updates. + +The cost components for data loading generally include data transfer costs and server cost for loading the data into memory. Data transfer costs depend on the frequency of updates and the data size. The server cost for data loading depends on the required instance size to hold the data in memory and CPU usage to process the data. While data transfer costs could be estimated, the server cost for data loading is harder to estimate without a full data setup. + +### Request API serving setup (read path) + +_“Tests on the read path may be performed with any load test utility that supports gRPC (e.g. [GHZ](https://ghz.sh/) and [wrk2](https://github.com/giltene/wrk2)). Tests should provide a constant stream of calls to the test environment at a steady rate for the period of the test.”_ The QPS used does not have to match production traffic, but it should match expected QPS for a single instance or a single set of sharded instances. + +The request payload should match the [key/value service’s API](https://github.com/privacysandbox/protected-auction-key-value-service/blob/release-0.15/docs/APIs.md). + +```sh +RATE=60 +HOURS=48 +TOTAL_CALLS=$(( HOURS*60*60*RATE )) +ghz \ + -r $RATE \ + --total=$TOTAL_CALLS \ + --call=kv_server.v2.KeyValueService/GetValuesHttp \ + --data-file ~/tmp/test-payload.txt \ + demo.kv-server.your-domain.example:8443 +``` + +_Example command line call to start a two-day test run at 60 QPS_ + +## Collecting cost data + +_“After the test is completed the results may be found in the billing explorer of the respective cloud provider ([GCP Billing](https://pantheon.corp.google.com/billing/), [AWS Cost Explorer](https://console.aws.amazon.com/cost-management/home#/cost-explorer)).”_ The components for data loading and request serving will need to be separated to extrapolate a monthly or annual cost. + +### AWS + +_“In AWS, billing data is collected from AWS Cost Explorer using a daily usage report. This breaks down cost by component and can be exported in csv-format. An example from one run is shown below.”_ + +
+ Fig. 2 - Cost example on AWS +
Figure. 2 Cost example on AWS
+


+ +To find the data loading cost, the report should be filtered to include the key/value server’s required data loading services (S3, SNS, SQS) and any other services used for data generation. + +To find the request serving costs, the report should be filtered to exclude the above services. Note that some usage types are billed by hour, which will not scale with the number of requests. These include VpcEndpoint-Hours, NatGateway-Hours, and LoadBalancerUsage. To extrapolate monthly costs, those should be excluded as well. + +### GCP + +_“A similar tool is provided by GCP in the Reports section of the Billing interface breaking out the costs into [individual SKUs](https://cloud.google.com/skus). To identify the test data, the correct date range and include filters for the desired GCP labels (for NAT costs, the region filter may be used when the test environment is the only NAT user in the region).”_ + +
+ Fig. 3 - Cost example on GCP +
Figure. 3 Cost example on GCP
+


+ +To find the data loading cost, the report should be filtered to include the key/value server’s required data loading services (Cloud Storage, Cloud Pub/Sub) and any other services used for data generation. + +To find the request serving costs, the report should be filtered to exclude the above services. + +### Extrapolating monthly or annual cost + +These captured billing costs can be extrapolated to find the expected monthly or annual operating costs. + +Given a similar data loading architecture between the test and production, the data loading costs from the test should give a rough estimate of costs in production. + +_“The request serving cost is (roughly) linearly representative of a single day of operation for the architecture in question computing the total number of inbound requests. Since production level traffic will be higher than the test run, the “scaled up” daily cost must be calculated for the expected QPS. In the simplest case, the cost fogr the test environment can be multiplied by a factor of production-to-test QPS.”_ + +$$Cost_{production} = Cost_{\textit{data loading test}} + Cost_{\textit{request serving test}} \times \frac{Rate_{\textit{API request production}}}{Rate_{\textit{request serving test}}}$$ + +_“This estimated daily production cost can then be multiplied to obtain the monthly or annual production costs._ + +_Note that this linearity exists since most of the cost is based on data transfer bytes or processing, both of which are roughly linear with the rate of requests. Some additional savings will be seen in actual production costs that are not seen here such as the NAT uptime in a GCP environment, however these costs tend to be relatively minor (NAT uptime is $0.0014 per hour)."_ + +Note: + +1. Since the Key Value server is stateful, as it requires in-RAM data to process requests, scaling the QPS to a single machine may not be linear to the resource cost of the machine. In order to apply the above formula, it’s better to fully utilize the test machines’ resources so scaling to the production volume would only require horizontal scaling to more machines, which removes the uncertainty mentioned above, and makes the relation more linear. +2. _"This assumes a production architecture similar to that under test. More complex architectures to handle higher loads, such as a fan-out load balancing scheme, may yield different results.”_ + +[1]: https://github.com/formgit +[2]: https://github.com/lusayaa diff --git a/key_value_service_instance_sizing_guide.md b/key_value_service_instance_sizing_guide.md new file mode 100644 index 0000000..4a9ae79 --- /dev/null +++ b/key_value_service_instance_sizing_guide.md @@ -0,0 +1,270 @@ +# K/V Server Instance Sizing Guide + +**Author:** +
[Lusa Zhan][1], Google Privacy Sandbox + +## Introduction + +Instance sizing is a crucial aspect of configuring a key-value server for optimal performance. If the instance is too small, the server may not be able to handle the desired requests per second (RPS). On the other hand, the server may not reap the benefits of an instance that is too large, resulting in unnecessary cost to the ad tech. + +Multiple factors play a role in determining the most suitable instance size, including desired RPS, dataset size, request payload, [user-defined functions](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/key_value_service_user_defined_functions.md) (UDF), and CPU usage. The most meaningful estimates will require narrowing down the possible combinations of different factors based on an ad tech’s use case. + +To aid in this decision process, the K/V server provides a [benchmarking tool](https://github.com/privacysandbox/protected-auction-key-value-service/tree/main/tools/benchmarking) that automates some manual deployment and benchmarking steps. The tool accepts different deployment and request parameters, runs benchmarks, and provides a summary with the RPS of each combination of parameters. + + +## Benchmarking Tool + + +### Overview + +The [deploy_and_benchmark](https://github.com/privacysandbox/protected-auction-key-value-service/blob/main/tools/benchmarking/deploy_and_benchmark) tool will deploy one or multiple terraform configurations one by one and run benchmarks against the deployed server. + +The tool takes multiple input parameters, such as + + + +* A file with sets of Terraform variables to override +* Directory with snapshot files or delta files with key value mutations of type `Update` +* List of number of keys (`n`) to include from a snapshot file +* (Optionally) a directory with UDFs to upload and benchmark against + +Given these input parameters, the benchmarking tool completes the following steps for each combination of parameters + + + +* Deploy with the given set of terraform variable overrides +* Generate requests with `n` keys from the given snapshot file +* Run benchmarks +* Collect results in a CSV + +The output CSV file contains metadata on each benchmark iteration and the RPS that can then be analyzed by the user. + +_Note: The tool does _not_ upload key-value delta/snapshot files. Please ensure those are already loaded into the blob storage bucket that the server will be reading from. Similarly, the tool will not be performing continuous data generation or upload key value delta files while the server is already running. This will need to be handled by a separate pipeline._ + + +### Reference + +For requirements and detailed usage of the tool, please refer to the [README](https://github.com/privacysandbox/protected-auction-key-value-service/blob/main/tools/benchmarking/README.md). The tool includes more flags than are outlined in this document. + + +### Choosing input parameters + +In its simplest form, the tool can be used to iterate through a set of deployments that override the instance only. + +However, there are many other factors besides instance size that affect the server’s RPS, such as + + + +* Request payload size +* Key size and value size +* Whether it is sharded +* CPU usage of UDFs +* Data loading + +The tool is designed to allow iterating through some of these parameters, such as request payload and deployment variables . The benchmarking results can inform decisions around different deployment models, dataset configurations, payloads, and more. + + +### A note on runtime and benchmark durations + +The tool’s runtime depends on the number of combinations of input parameters it needs to iterate through and can increase significantly just by adding more combinations of input parameters. + +By default, the benchmarking tool uses a duration of 60 seconds for each iteration of a benchmark. Short benchmark iterations can be useful for initial testing with multiple variables, for instance to measure RPS across different datasets and request payloads. + +To accurately reflect production environments, it is recommended to increase the benchmark durations to several hours once the variables have been narrowed down. These benchmarks should run parallel to a data update pipeline that mimics a production environment in terms of payload, data update size, and frequency. + + +## Tutorial + +This is a step-by-step example of how to use the tool with some parameters. The example goes through generating sample data that is loaded into the server at startup, setting input parameters for the script, and looking at the output. + +This example assumes that the dataset and the UDFs are fixed. The request payload may contain anywhere from 10 to 100 keys. Since the dataset is small, sharding will not be considered. + +It is mostly interested in which instance size to choose. + + +### Completing prerequisites + +Follow the [requirements](https://github.com/privacysandbox/protected-auction-key-value-service/blob/main/tools/benchmarking/README.md#requirements) for running the tool. Start from the workspace root. + + +#### Generating data and uploading data to be loaded into KV server + +Data that is loaded into the K/V server needs to be generated separately. More info in [data loading guides](https://github.com/privacysandbox/protected-auction-key-value-service/tree/main/docs/data_loading). + +Some initial data should be loaded into the data storage bucket before the server is deployed to make sure the lookup calls are not retrieving nonexistent data. + +Generate a delta file and upload it to the cloud storage bucket the server will be pulling data from. + + +``` +builders/tools/bazel-debian build tools/benchmarking/example/kv_data:generate_100_delta + +KV_DELTA=bazel-bin/tools/benchmarking/example/kv_data/DELTA_1000000000000004 + +// If uploading to AWS. +// AWS_S3_BUCKET should match the terraform's s3_delta_file_bucket_name +// Ex: AWS_S3_BUCKET=s3://my-delta-file-bucket +aws s3 cp $KV_DELTA $AWS_S3_BUCKET + +// If uploading to GCP +// GCLOUD_STORAGE_BUCKET should match the terraform's data_bucket_id +// Ex: GCLOUD_STORAGE_BUCKET=gs://my-delta-file-bucket +gcloud storage cp $KV_DELTA $GCLOUD_STORAGE_BUCKET +``` + + +When the server starts, it will load the data from the storage bucket. This initial data loading process will not affect the server’s RPS, since it should be completed before the server is ready to serve requests. + +This tutorial will not cover continuous data loading while the server is up. However, data loading does affect RPS. For a more comprehensive benchmarking test, it is recommended to update/load data as the server is running. The frequency and duration of data loading should be shorter than each benchmark iteration. For instance, if the benchmark duration is 1 hour, data loading intervals could be 20 minutes. + + +#### Generating data for requests + +As part of the benchmark, the tool sends requests to the server. The requests are generated using keys from snapshot files or delta files with key value mutations of type `Update`. The tool will iterate through each delta/snapshot file in a given directory, generate one request for each, and perform benchmark iteration for each request. + +The previously generated delta files from this tutorial include only `Update` key value mutations and can be used for request generation. + + +``` +mkdir example_kv_delta +SNAPSHOT_DIR=example_kv_delta +cp $KV_DELTA $SNAPSHOT_DIR +``` + + + +#### Writing and generating UDFs + +Custom UDFs should be generated and uploaded in advance. See the [guide for generating UDFs](https://github.com/privacysandbox/protected-auction-key-value-service/blob/main/docs/generating_udf_files.md). + +The provided [example UDF](https://github.com/privacysandbox/protected-auction-key-value-service/blob/main/tools/benchmarking/example/udf_code/benchmark_udf.js) is a passthrough JavaScript UDF that calls the lookup function on the input data. + + +``` +builders/tools/bazel-debian run //tools/benchmarking/example/udf_code:benchmark_udf_js_delta + +UDF_DELTA=dist/deltas/DELTA_2000000000000001 + +// If uploading to AWS +aws s3 cp $UDF_DELTA $AWS_S3_BUCKET + +// If uploading to GCP +gcloud storage cp $UDF_DELTA $GCLOUD_STORAGE_BUCKET +``` + + +This is an optional step. If no custom UDF is uploaded, the server will use the [default UDF](https://github.com/privacysandbox/protected-auction-key-value-service/blob/main/public/udf/constants.h). + + +### Running the tool + +Once the data is generated and uploaded, the input parameters for the tool can be defined. + +Deployment related arguments are as follows: + + +``` +# Set the cloud provider +CLOUD_PROVIDER="aws" +CLOUD_PROVIDER="gcp" + +# Set the terraform files the tools will use to deploy the server +TF_VAR_FILE=/path/to/my.tfvars.json +TF_BACKEND_CONFIG=/path/to/my.backend.conf + +# Set the URL of the server for the environment that will be deployed +# The URL will be shown as the output of the `terraform apply` command +SERVER_URL="https://demo.my-example.com" + +# Mininum amount of seconds to wait after terraform apply before checking +# server health. Sometimes terraform apply finishes, but an existing (old) +# server may still be online for a bit. +MINIMUM_SERVER_WAIT_SECS=300 +``` + + +You will need to iterate through different instance sizes, so create a file with terraform variables that should be overridden. Each line in the file is considered one set of terraform variable overrides and will be used for one deployment. + +For instance, for AWS, we may want to iterate through these instance types: + + +``` +instance_type=c5.4xlarge,enclave_cpu_count=12,enclave_memory_mib=24576,num_shards=1,udf_num_workers=12,instance_ami_id=ami-0c3ea8d5ff3ef70d2 +instance_type=c5.12xlarge,enclave_cpu_count=36,enclave_memory_mib=73728,num_shards=1,udf_num_workers=36,instance_ami_id=ami-0c3ea8d5ff3ef70d2 +instance_type=m5.4xlarge,enclave_cpu_count=12,enclave_memory_mib=49152,num_shards=1,udf_num_workers=12,instance_ami_id=ami-0c3ea8d5ff3ef70d2 +instance_type=m5.12xlarge,enclave_cpu_count=36,enclave_memory_mib=147456,num_shards=1,udf_num_workers=36,instance_ami_id=ami-0c3ea8d5ff3ef70d2 +``` + + +On GCP, the file may look like this: + + +``` +machine_type=n2d-standard-4,num_shards=1,udf_num_workers=4 +machine_type=n2d-standard-8,num_shards=1,udf_num_workers=8 +machine_type=n2d-standard-32,num_shards=1,udf_num_workers=32 + +machine_type=c2d-standard-4,num_shards=1,udf_num_workers=4 +machine_type=n2d-standard-8,num_shards=1,udf_num_workers=8 +machine_type=n2d-standard-32,num_shards=1,udf_num_workers=32 +``` + + +Save the file: + + +``` +TF_OVERRIDES=${PWD}/path/to/tf_overrides.txt +``` + + +Set the number of keys to choose to include in each benchmark iteration’s request. In this case, the tool will create two requests for the given snapshot file (in `$SNAPSHOT_DIR`)" one request with 10 keys, one request with 100. + + +``` +NUMBER_OF_LOOKUP_KEYS_LIST="10 100" +``` + + +Finally, choose where the summary CSV should be written: + + +``` +OUTPUT=${PWD}/example_benchmarking_results.csv +``` + + +Run the tool: + + +``` +./tools/benchmarking/deploy_and_benchmark \ +--cloud-provider ${CLOUD_PROVIDER} \ +--server-url ${SERVER_URL} \ +--tf-var-file ${TF_VAR_FILE} \ +--tf-backend-config ${TF_BACKEND_CONFIG} \ +--snapshot-dir ${SNAPSHOT_DIR} \ +--tf-overrides ${TF_OVERRIDES} \ +--ghz-tags "${GHZ_TAGS}" \ +--minimum-server-wait-secs "${MINIMUM_SERVER_WAIT_SECS}" \ +--csv-output ${OUTPUT} + +``` + + + +### Example output + +The output is a CSV file with one row per benchmark iteration. It includes the terraform variables that were overridden for the iteration, the files that were used to generate keys for the benchmark iteration’s request, and other metadata. The **rps** column contains the number of requests per second and can be used to determine which instance size to choose. + +
+ Fig. 1 - Sample benchmarking tool output +
Figure. 1 Sample benchmarking tool output
+


+ + + +[1]: https://github.com/lusayaa diff --git a/key_value_service_sharding.md b/key_value_service_sharding.md new file mode 100644 index 0000000..15097ab --- /dev/null +++ b/key_value_service_sharding.md @@ -0,0 +1,107 @@ +# **Sharding the Key Value Service** + + +# **Overview** + +The Protected Audience Key Value Service stores an in-memory data set of all key/value pairs needed to support end users queries. AdTechs have expressed that their data sets are larger than what can fit in a single machine's memory. We will need to support data sharding while maintaining user privacy. + + +# **Requirements** + +The following high level requirements are broken down into two sections: feature set and privacy. The utility section breaks down the work into everything needed to support a sharded peer to peer data set. The privacy section defines what is required to maintain user privacy while querying this sharded data set. + + +## **Feature set** + +The goal is to be able to scale our Key Value service to support arbitrarily large data sets without sacrificing any of the other requirements we have. + + +### **Functionality** + +The sharding concept is internal to the system. Devices and external clients should not know that the service is sharded. + + +##### **Static sharding** + +Shard numbers can be fixed and set at system startup. No dedicated hotspot detection. Service operator is responsible for determining hotspots by checking metrics like QPS, error rate, RAM usage, and etc. + +In the event that AdTech demands dynamic resharding of data, we can address this later. It is not currently considered a requirement. + + +### **Reliability** + + +#### _High availability_ + +We will offer the ability for the AdTech to operate shard replicas. The failure of a single replica should not be detrimental. The number of replicas is up to the AdTech. + + +#### _In-Flight Resharding_ + +The reason we need in-flight resharding is that, as the amount of data grows AdTechs will need to be able to add more shards. + +It should be possible to reshard without a spike in latency. An AdTech needs to be able to reshard without taking down the entire system. + + +#### _Consistency_ + +All peers should converge towards data consistency. They do not all need to share the same data freshness at the same time. + + +#### _Goal_ + +No central point of failures. Have a clear understanding of what effect a degradation of each component has on the system. + + +### **Minimal overhead** + + +#### _Infrastructure Complexity_ + +Avoid new cloud service requirements where possible. This helps with the cost, ease of support and debugging. All peer servers within the network should have the same specification. + + +#### _Latency_ + +Additional serving latency is constant with the launching of shards, but does not grow with the number of shards. Latency will grow as shards-queried-per-request goes up. + +Support batching of cross peer lookups to match the potential batching of lookups queried by the UDF. + + +#### _Optional_ + +Sharding is optional, in the event AdTech can fit all data within a single shard, it should work without any additional overhead. + + +#### _Cost_ + +Cost should be estimated and be predictable. + + +### **Scaling** + +The system owner can change the total shard number to scale horizontally as resources are added. It is a problem where you can throw additional peer instances at the problem. + + +## **Privacy** + +Adding sharding exposes the attack vectors that use the fact that an AdTech controls a UDF -- user defined function. All such attacks use the fact that when lookup requests are made to data servers, an AdTech can observe those encrypted downstream requests. Since an AdTech controls the load balancer, an AdTech could route only one request out of all requests to a dedicated EC2 box for which the downstream requests are observed. + +### The attacks we are currently aware of are: +#### 1. Number of lookup requests + +A UDF can be created that queries data servers once for key A, and twice for key B. The number of roundtrips per UDF can reveal information about the key. +#### 2. Timing of lookup requests +UDF allows an AdTech to wait for extra time between the requests. E.g. if a key is A wait for 20 ms, before making one more query to the data server, and for key B wait 40 ms. +#### 3. Payload size of lookup requests +UDF controls the payload size of downstream requests which is one more way to reveal what's being queried. +It would be possible to mitigate some of those attacks by creating dummy network requests to shards that do not contain relevant data, adding noise to the response time, or padding queries with dummy data. + +Such mitigations would be prohibitively expensive for server operators and may not fully address the issues. For instance, side-channel attacks on the bidding server itself might still be possible. In addition, they would not be meaningful [until TEE usage is required for K/V servers](https://developer.chrome.com/docs/privacy-sandbox/fledge-api/feature-status/#trusted-execution-environment-usage-for-keyvalue-service). We will continue to investigate potential mitigations for the known attacks until then. + +Note that because it will not be possible to do network level replay attacks, and there are certain limitations and additional cost and constraints to "client side" replay attacks (such as triggering an auction multiple times), the practicality and scalability of known attacks remains to be proven. + + +## **Future Plans** + +We are exploring options to make the data storage have lower cost in the future, by potentially allowing some data to be stored on disc. Currently, we’re trying to assess the feasibility of this approach and the associated tradeoffs, e.g. how much extra latency that would add. Please open a [GitHub issue](https://github.com/privacysandbox/fledge-key-value-service/issues) or reach out to us if you would like to provide us with feedback regarding your scale. diff --git a/key_value_service_trust_model.md b/key_value_service_trust_model.md index d4cc80d..7ebfe88 100644 --- a/key_value_service_trust_model.md +++ b/key_value_service_trust_model.md @@ -1,3 +1,5 @@ +> FLEDGE has been renamed to Protected Audience API. To learn more about the name change, see the [blog post](https://privacysandbox.com/intl/en_us/news/protected-audience-api-our-new-name-for-fledge) + # FLEDGE Key/Value service trust model Authors: @@ -6,7 +8,7 @@ Authors: ## Context -We are currently in an early requirements gathering and design phase and aim to test the FLEDGE key/value service either in 2H 2022 or 1H 2023. Use of this service is not currently part of the critical path for third-party cookie deprecation, however we welcome input on the design, privacy properties and timing from the ecosystem on both setting up their own server as well as leveraging one described here. +The Privacy Sandbox team is developing the FLEDGE key/value service and [an initial version](https://github.com/privacysandbox/fledge-key-value-service) is ready for Adtechs to test out. Use of this service is not currently part of the critical path for third-party cookie deprecation, but [will be required eventually](https://developer.chrome.com/docs/privacy-sandbox/fledge-api/feature-status/). We welcome input on the design, privacy properties and timing from the ecosystem on both setting up their own server as well as leveraging one described here. ## Introduction @@ -120,8 +122,8 @@ The encryption is bi-directional. Responses back to the client software are also The initial implementation strategy for the key/value service is as follows: -* A subsequent update of the [API explainer](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md) would incorporate new parameters to the Read API to facilitate secure communication, [Github issue here](https://github.com/WICG/turtledove/issues/294). -* The TEE based key/value service implementation would be deployed on cloud service(s) which support needed security features. We envision the key/value service being capable of deployment with multiple cloud providers. +* A newer version of the query API dedicated for the TEE-based service is presented in the [API explainer](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md). +* The TEE based key/value service implementation would be deployed on cloud service(s) which support needed security features. We envision the key/value service being capable of deployment with multiple cloud providers. Currently the implementation is [available for testing](https://github.com/privacysandbox/fledge-key-value-service) and its development to add more features and improve its production-readiness is in progress. ## Support for user-defined functions (UDFs) @@ -141,7 +143,7 @@ The following principles are necessary in order to preserve the trust model: * _Sandbox_ - the custom code will be executed inside a sandbox that limits what it is allowed to do. We're currently looking at the [Open Source V8 engine](https://v8.dev/) inside [Sandbox2](https://developers.google.com/code-sandboxing/sandbox2), which has support for both JavaScript and Web Assembly (WASM). Other suggestions are welcome! * _No network, disk access, timers, or logging_ - this will be enforced using the sandbox, above. This preserves the key/value service's principle of no side-effects and avoids leaking user data. Coarse timers may be allowed but fine-grained timers are disallowed to help prevent covert channels (e.g. SPECTRE). -* _Individual request handling_ - Per the [FLEDGE explainer](https://github.com/WICG/turtledove/blob/main/FLEDGE.md#31-fetching-real-time-data-from-a-trusted-server), a request to the key/value service may be for multiple keys. The UDF will be called separately for each individual key, rather than once for all of the keys. This ensures that each key is processed independently and prevents a group of keys from being used as a cross-site profile for a user. +* _Partitioned request handling_ - Per the [FLEDGE explainer](https://github.com/WICG/turtledove/blob/main/FLEDGE.md#31-fetching-real-time-data-from-a-trusted-server), a request to the key/value service may be for multiple keys. In the [key/value query API v2](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md#query-api-version-2), a request consists of multiple partitions. Each partition contains a collection of keys that can be processed together without privacy leakage. The UDF will be called for each partition. This prevents groups of keys from being used as a cross-site profile for a user. * _Data store APIs_ - The key/value service will expose an API to the UDF to read data from the data store. There will be no write APIs to the data store. * _Side-effect free_ - The UDF can read data from the Data store APIs but cannot write data to any location apart from returning it to the FLEDGE client. No state is shared between UDF executions. * _Limited request metadata access_ - Each request to the key/value service contains the keys to look up as well as some amount of request metadata. This includes the user IP address, request timestamp, and [experiment id](https://github.com/WICG/turtledove/issues/191). We expect to allow the UDF to have access to some of this metadata and will be updating this explainer with details of that once we work through how this will fit into the privacy model. @@ -149,7 +151,44 @@ The following principles are necessary in order to preserve the trust model: ### API -We'll update the [API explainer](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md) with the APIs that we plan to provide for UDFs. +For the APIs we plan to provide for UDFs, please see the [UDF explainer](https://github.com/privacysandbox/fledge-docs/blob/main/key_value_user_defined_functions.md). + +## Cross-site identifier handling + +### Protections + +> “IP addresses can be stable over periods of time, which can lead to user identification across first-parties.” + +– [IP protection proposal](https://developer.chrome.com/docs/privacy-sandbox/ip-protection/) + +Along with the key/value service requests, some metadata may be present that can be potentially considered cross-site identifiers, such as IP addresses, browser agent, etc. + +The [Trusted Execution Environment](https://github.com/privacysandbox/fledge-docs/blob/main/trusted_services_overview.md#trusted-execution-environment) (TEE)-based key/value service enforcement does not require IP protection to be in place to begin. But the IP protection will eventually enhance the privacy property of the service. + +With the key/value service trust model, the visibility of IP address and other metadata has the following properties: + +1. The operator of a trusted key/value service can observe the incoming traffic’s IP addresses. Even though the server itself runs inside TEE, a TEE (and the server that runs inside it) is only one of many components in the system. Other components in the serving path, such as the load balancer, can observe all the traffic and have information of the IP address of each request. +1. The [user-defined functions](https://github.com/privacysandbox/fledge-docs/blob/main/key_value_service_trust_model.md#support-for-user-defined-functions-udfs) (UDF) that is executed by the server inside TEE, is subject to the server code regarding whether or not it can see the IP address, and the server code is not modifiable by the server operator. +1. Requests from the [bidding & auction service](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md) may effectively contain these user metadata too. + +There are then primarily 2 questions around this: + +1: What user privacy improvements does the TEE-based key/value service have to protect against user profile building with IP addresses or other cross-site metadata? + +For IP addresses which will be protected by the [IP protection project](https://developer.chrome.com/docs/privacy-sandbox/ip-protection/), they will be hidden from the K/V service. + +For other IP addresses or metadata, there are still substantial constraints. Tracking the user requires information about the user's activities. As stated in previous sections, the key/value request and response content is only visible inside the TEE. From the service operator’s point of view, it only knows that a user with a certain IP address has sent an opaque request to the service and receives an opaque response. + +There is very limited information in this exchange that the operator can observe, such as the time or size of the request and the number of requests within a time period. This significantly reduces the amount of observable information compared to the BYOS key/value service model, and is an even greater privacy improvement compared to the 3rd party cookie model. + +2: Can the operator perform precise microtargeting based on the IP address or other metadata? + +From the [Protected Audience API explainer](https://github.com/WICG/turtledove/blob/main/FLEDGE.md#summary:~:text=protection%20against%20microtargeting): “The browser will provide protection against microtargeting, by only rendering an ad if the same rendering URL is being shown to a sufficiently large number of people.” This is achieved using the [k-anonymity server](https://github.com/WICG/turtledove/blob/main/FLEDGE_k_anonymity_server.md). + +### Engage and share feedback + +Depending on the use cases, the Geolocation could be calculated before the key/value look up time, or at a coarser level than a precise value. +If you have use cases that require IPGeo calculations at key/value lookup time with a specific level of granularity, please leave feedback by posting in the [key/value github service repo](https://github.com/privacysandbox/fledge-key-value-service) or via other channels. ## Open questions diff --git a/key_value_service_use_cases.md b/key_value_service_use_cases.md new file mode 100644 index 0000000..dc2fc1e --- /dev/null +++ b/key_value_service_use_cases.md @@ -0,0 +1,15 @@ +The Key/Value (KV) server is a TEE based Key/Value database that can be used to store and integrate user level and campaign level data into Protected Audiences Auctions. It can be used for a variety of use cases, including: + +**Real-time creative status retrieval:** SSPs can use the Key/Value server for lookup of a previous scan of a creative so they can block ads that violate their policies. When an SSP discovers that an ad violates their policy, they can write to the Key/Value server to indicate that the ad should not be presented. The SSP can then check the creative render URL against the key value in the auction code to identify if the ad is “allowed”, “blocked”, or “unknown status”. More generally, the Key/Value response for a creative could include a list of policy categories the ad falls into according to the scan, meaning the SSP's auction code could allow the creative on some publishers but not others. + +**Budgeting**: the Key Value server can be used to store campaign budget strategy such as {full-speed, slow, off} in near real time. When bidding takes place, an ad can check the budget strategy for the ad by doing a lookup of render URL to campaign ID and then using the campaign ID to check for the budget strategy. For example, when the DSP places an ad on the browser, it stores "campaign": "#123" in the ad's metadata in the IG, and also stores "spendrate#123" in the IG's set of keys. Then, when the computedBid() function is looking over the IG's ads, it checks the key "spendrate"+ad.campaign and finds the value. + +The ad server can also use geo information to return a value that is, say, country specific. For example, there might be different budgets or different policies in different countries, which will give different return values from the Key/Value server. + +**Campaign liveness check**: The Key/Value server can check whether an ad-tech has paused or unpaused a campaign. + +**Real-time bidding information**: The Key/Value server can bring real time bidding information. For example, if the advertiser is running a target cost-per-acquisition (CPA) campaign, the real-time CPA value used for bidding would come from the Key/Value server. + +**ML Model Evaluation**: The contents of the "key" sent to the server is your choice. It can even be some representation of the input data to an ML model. Using the Key/Value server's User-Defined Functions, you could perform inference from a model, using both the input data stored in the key and the hostname as input signals. + +The Key/Value server is a powerful tool that can be used to improve the efficiency and effectiveness of AdTech campaigns. If you have other use cases that involve real-time data in the auction, we can discuss your specific needs and determine the best combination of contextual and K/V requests to meet them. If you have any questions or would like to see additional use cases supported, please contact us via GitHub or your preferred communication channel. diff --git a/key_value_service_user_defined_functions.md b/key_value_service_user_defined_functions.md new file mode 100644 index 0000000..6c09b62 --- /dev/null +++ b/key_value_service_user_defined_functions.md @@ -0,0 +1,573 @@ +> FLEDGE has been renamed to Protected Audience API. To learn more about the name change, see the [blog post](https://privacysandbox.com/intl/en_us/news/protected-audience-api-our-new-name-for-fledge) + +# Key/Value service user-defined functions (UDFs) + +* Author: Lusa Zhan + +* Contributor: Peiwen Hu + + +## Overview + +The key/value service is adding support for executing user-defined functions +(UDFs). This explainer describes the planned critical user journey and API that +will need to be followed to use UDFs in the trusted key/value service. +An initial version has been added to the +[key/value service repo](https://github.com/privacysandbox/fledge-key-value-service) +in release 0.9.0. + +The UDF code snippet is loaded through a cloud file storage solution such as the +AWS S3 and will be executed when the server receives a +[query request](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md#query-api-version-2). +Once per +[partition](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md#request-version-20), +the request handler invokes the UDF, which can run proprietary computation logic +and in addition, has access to a read API provided by the key/value service to +query the server’s internal data store, populated by dataset from the service +operator. + +![Overview of UDF Flow](images/fledge_kv_udf_flow.png) + + +_The diagram above shows the basic UDF code loading and request flow: +At startup, the server loads untrusted code and data into memory. +When the client sends a request to the server, the server splits +the keys in the request into multiple groups of keys and +calls the UDF per group of keys. +The UDF can perform any number of lookup calls to the in-memory K/V data store. +The server collects the results of the UDF executions, aggregates them, +and returns them in the response to the client._ + +By default, we will load a simple pass-through UDF that will do an internal datastore lookup +for the keys and return the values. +To override the default UDF, see the +[UDF guide](https://github.com/privacysandbox/fledge-key-value-service/blob/main/docs/generating_udf_files.md). + +For details on the broader trust model, see the [trust model explainer](https://github.com/privacysandbox/fledge-docs/blob/main/key_value_service_trust_model.md#design-principles). + + +## UDF fact sheet + +(For the Initial phase only) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Characteristics + Description + Note +
Language supported + Javascript, WASM + WASM support is limited to “inline WASM” initially, where the WASM code must be invoked by a piece of javascript driver code, i.e., the UDF entrypoint must still be in javascript. +
Number of UDF implementations + One fixed code snippet can be loaded at a time. To override a code snippet, upload a UDF configuration. + +
Engine + Chrome V8 + +
Timeout + 1 second + Currently not configurable, but will be in future iterations. +
Visibility + Private + Code is only known to the service operator. The code is not part of the attestation process in the trust model that requires open sourcing. Privacy constraints of the code execution are upheld by the runtime environment. +
+ + +## Engage and share feedback + +1. Examine the APIs proposed in the sections below. Do your use cases require + more? Please leave feedback by posting github issues in the + [key/value service repo](https://github.com/privacysandbox/fledge-key-value-service/issues). + +2. Provide feedback on the overall capabilities and requirements, such as: + + * Are you interested in specific languages? + * What is your expected code snippet size? + * How often do you expect to update your UDF? What is the accepted latency + for the code update? + * Do you need multiple versions of UDFs to co-exist? What is the use case? + +3. Write your UDF and test it out by following the [steps + below](#running-the-kv-server-with-a-udf). + + The key/value service repo also includes a tool to check the compatibility of UDFs without having to spin up a K/V server (see [testing section of the UDF guide](https://github.com/privacysandbox/fledge-key-value-service/blob/main/docs/generating_udf_files.md#3-test-the-udf-delta-file)). + Note that the UDF feature does not yet include trust model enforcements. However, the principles are outlined in the [trust model explainer](https://github.com/privacysandbox/fledge-docs/blob/main/key_value_service_trust_model.md#design-principles) and should be followed to ensure compatibility with the final implementation. + + +## Running the K/V server with a UDF + +1. Write the UDF Code snippet + + The code snippet should have the following properties: + + * Language + * Javascript (if desired, Javascript with inline WASM) + * UDF handler + * The code snippet should contain a UDF handler function that serves + as the main entry point for execution. + * While function and parameter naming in the handler signature is up + to Adtech, the input and output are expected to be in the specified + format outlined in the [API](#user-defined-function-api-v1) (see + [input](#input) and + [output](#output)). + * Note that the server will pass the input as a *JSON object*, but + expects the output to be a *serialized JSON string*. + * (optional) Call the [datastore read API](#datastore-read-api) + * The UDF can optionally make one or more `getValues` calls to get + key-value pairs from the internal datastore. + +2. Load the code snippet + + At the moment, the UDF code snippet is loaded as a snapshot or delta file + through AWS and should be in the same S3 bucket as other delta files. + + The snapshot or delta file itself must contain a + [UserDefinedFunctionsConfig](https://github.com/privacysandbox/fledge-key-value-service/blob/main/public/data_loading/data_loading.fbs). + + See details on how to generate and upload UDF delta files in the + [UDF guide](https://github.com/privacysandbox/fledge-key-value-service/blob/main/docs/generating_udf_files.md). + +3. Deploy the server + + Deploy the server either on + [AWS](https://github.com/privacysandbox/fledge-key-value-service/blob/main/docs/deploying_on_aws.md) + or + [locally](https://github.com/privacysandbox/fledge-key-value-service/blob/main/docs/developing_the_server.md). + +4. Run a query + + Use + the[ Query API](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md#query-api-version-2) + (v2 or above) to test the UDF. + +## APIs + +### User-defined function API (v1) + +#### Generic API + +The base UDF API is generic and can be customized with API overlays for different use cases (see [API doc](https://github.com/privacysandbox/fledge-key-value-service/blob/main/docs/APIs.md#api-all-in-one-user-guide)). + +##### Input + +The generic signature is the following: + +``` +String myHandlerName(UDFExecutionMetadata arg1, [UDFArgument] args) +``` +or the equivalent Javascript: +```javascript +function myHandlerName(executionMetadata, ...udf_arguments) {} +``` + +However, if the number of arguments is known, such as in certain use cases, the UDF signature can also directly use them. For example, if it is known that the UDF will take 3 arguments: + +``` +String myHandlerName(UDFExecutionMetadata arg1, [String] some_input, [Object] some_other_input) +``` + +###### Schema definitions: + +* [UDFExecutionMetadata](https://github.com/privacysandbox/fledge-key-value-service/blob/main/public/api_schema.proto#L25) contains metadata from the [GetValuesRequest](https://github.com/privacysandbox/fledge-key-value-service/blob/main/public/query/v2/get_values_v2.proto). + +```json +{ + "title": "UDFExecutionMetadata", + "description": "Request metadata", + "type": "object", + "additionalProperties": false, + "properties" : { + "requestMetadata": { + "definition": "Metadata passed from the request. Specs can be defined by API overlay.", + "type": "object" + }, + "udfInterfaceVersion": { + "description": "UDF Interface version", + "type":"integer" + } + }, + "required": ["requestMetadata", "udfInterfaceVersion"], +} +``` + +* [UDFArguments](https://github.com/privacysandbox/fledge-key-value-service/blob/main/public/api_schema.proto) are also passed in from the GetValuesRequest. The server can pass in a different number of arguments of different types. + +```json +{ + "title": "UDFArgument", + "type": ["number","string","boolean","object","array", "null"], + "additionalProperties": false +} +``` + +##### Output + +The output can be any JSON value. + +#### Protected Audience API overlay + +Overlay for the [PA API](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md#query-api-version-2). + +##### Input + +###### Schema of the input + +* `UDFExecutionMetadata`: + +```json +{ + "title": "UDFExecutionMetadata", + "description": "Request metadata", + "type": "object", + "additionalProperties": false, + "properties" : { + "requestMetadata": { + "type": "object", + "properties": { + "hostname": { + "type": "string", + } + } + }, + "udfInterfaceVersion": { + "description": "UDF Interface version", + "type":"integer" + } + }, + "required": ["requestMetadata", "udfInterfaceVersion"], +} +``` + +Example: + +```json +{ + "requestMetadata": { + "hostname": "example.com" + }, + "udfInterfaceVersion": 1 +} +``` + + +* `UDFArgument` +```json +{ + "title": "UDFArgument", + "description": "Contains data for one partition", + "type": "object", + "additionalProperties": false, + "properties": { + "tags": { + "description": "List of tags describing argument's attributes", + "type": "array", + "items": { + "type": "string" + } + }, + "data": { + "type": "array", + "items": { + "type": "string" + } + } + } +} +``` + +Example: + +```json +{ + "tags": [ + "structured", + "groupNames" + ], + "data": [ + "IG1" + ] +} +``` + + + +##### Output + +The output of the UDF is a +JSON object with a `keyGroupOutputs` property. The server uses the output to +build its response to the client. + +###### Schema of the output + +```json +{ + "title": "UDF Output Object", + "description": "output of one UDF call", + "type": "object", + "additionalProperties": false, + "properties": { + "keyGroupOutputs": { + "type": "array", + "items": { + "$ref": "#/$defs/single_key_group_output" + } + }, + "udfOutputApiVersion": { + "description": "UDF Output API version", + "type": "integer" + } + }, + "required": ["keyGroupOutputs", "udfOutputApiVersion"], + "$defs": { + "single_key_group_output": { + "description": "Output for one key group", + "type": "object", + "additionalProperties": false, + "properties": { + "tags": { + "description": "List of tags describing this key group's attributes", + "type": "array", + "items": { + "type": "string" + } + }, + "keyValues": { + "description": "If a keyValues object exists, it must at least contain one key-value pair. If no key-value pair can be returned, the key group should not be in the response.", + "type": "object", + "patternProperties": { + ".*": { + "$ref": "#/$defs/single_value_output" + } + } + } + } + }, + "single_value_output": { + "description": "One value to be returned in response for one key", + "type": "object", + "additionalProperties": false, + "properties": { + "value": { + "type": [ + "string", + "number", + "integer", + "object", + "array", + "boolean" + ] + }, + "global_ttl_sec": { + "description": "Adtech-specified TTL for client-side caching, not dedicated to a specific subkey. In seconds. Unset means no caching.", + "type": "integer" + }, + "dedicated_ttl_sec": { + "description": "Adtech-specified TTL for client-side caching, specific to the subkey in the request. In seconds. Unset means no caching.", + "type": "integer" + } + }, + "required": [ + "value" + ] + } + } +} +``` + +Example output: + +```json +{ + "keyGroupOutputs": [ + { + "tags": ["structured", "groupNames"], + "keyValues": + { + "IG1": { + "value": { + "priorityVector": {"signal1": 1} + }, + "global_ttl_sec": 1 + } + } + }, + { + "tags": ["custom", "adComponentRenderUrls"], + "keyValues": { + "adcomponents.com/1": { + "value": "value1", "global_ttl_sec": 120 + }, + "adcomponents.com/2": { + "value": ["value2A", "value2B"], + "global_ttl_sec": 60 + } + } + } + ], + "udfOutputApiVersion": 1 +} +``` + +##### Example PA code snippet + +```javascript +function handleRequest(executionMetadata, ...udf_arguments){ + const keyGroupOutputs = getKeyGroupOutputs( + executionMetadata.requestMetadata.hostname, + udf_arguments + ); + return { keyGroupOutputs, udfOutputApiVersion: 1 }; +} +``` + +### Datastore Read API + +We expose a read-only API to the datastore that is accessible from within the UDF. A call to `getValues(keys)` returns the key value pairs from the datastore for the given input. It is registered as a regular Javascript function. + + +#### Input + +The input is an array of strings that should be looked up from the datastore. + + +##### Schema of the input + +```json +{ + "title": "getValues input", + "type": "array", + "items": { + "type": "string" + } +} +``` + +Example input: + +```json +["lookup_key1", "lookup_key2"] +``` + +#### Output + +The output is a ***serialized JSON string*** that needs to be parsed. + +The JSON contains an overall status and a map of requested keys and lookup results for each key. + +The top-level status indicates if there was an error with the overall lookup, e.g. a malformed request or an internal error that led to all lookups failing. + +Each key has its own key lookup result, which is either a string value or a status: +* If the key lookup is successful, the individual key lookup result will be a string value. +* If an error occurs during a specific key lookup (e.g. missing key), the individual lookup + result will be a status with a non-zero status code. + The UDF can decide which keys it wants to retry. + +##### Schema of the output + +```json +{ + "type": "object", + "properties": { + "kvPairs": { + "type": "object", + "patternProperties": { + ".*": { + "$ref": "#/$defs/single_kv_pair" + } + } + }, + "status": { + "description":"Status for getValues call. May still be ok even if some key lookups return errors.", + "$ref": "#/$defs/status" + } + }, + "required": [ + "kvPairs", "status" + ], + "$defs": { + "single_kv_pair": { + "description": "A single KV pair", + "type": "object", + "additionalProperties": false, + "properties": { + "value": { + "type": [ + "string" + ] + }, + "status": { + "description": "Status for specific key lookup. Set if error occurred.", + "$ref": "#/$defs/status" + } + } + }, + "status": { + "type": "object", + "additionalProperties": false, + "properties": { + "code": { + "type": "integer" // 0: OK, non-zero: Error + }, + "message": { + "type": "string" + } + }, + "required": [ + "message", + "code" + ] + } + } +} +``` + +Example output: + +```json +{ + "kvPairs": { + "key1": { + "value": "value1" + }, + "key2": { + "status": { + "code": 2, + "message": "Some error" + } + } + }, + "status": { + "code":0, + "message":"ok" + } +} +``` \ No newline at end of file diff --git a/monitoring_protected_audience_api_services.md b/monitoring_protected_audience_api_services.md new file mode 100644 index 0000000..7c8b1b9 --- /dev/null +++ b/monitoring_protected_audience_api_services.md @@ -0,0 +1,646 @@ +# Monitoring Protected Audience API Services + +**Authors:**
+[Akshay Pundle](https://github.com/akshaypundle), Google Privacy Sandbox
+[Brian Schneider](https://github.com/bjschnei), Google Privacy Sandbox
+[Xing Gao](https://github.com/xinggao01), Google Privacy Sandbox
+[Roopal Nahar](https://github.com/roopalna), Google Privacy Sandbox
+[Chau Huynh](https://github.com/chau-huynh), Google Privacy Sandbox
+ +Protected Audience API ([Android](https://developer.android.com/design-for-safety/ads/fledge), [Chrome](https://developer.chrome.com/docs/privacy-sandbox/fledge/)) proposes multiple real time services ([Bidding and Auction](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md) and [Key/Value](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md) services) that run in a [trusted execution environment](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/trusted_services_overview.md#trusted-execution-environment) (TEE). These are isolated environments for securely processing sensitive data, with very limited data egress. Due to the privacy guarantees of such a system, traditional ways of monitoring are not viable as they may leak sensitive information. Nevertheless, monitoring is a critical activity for operating such services. + +This proposal implements mechanisms to provide telemetry data using privacy-preserving technologies. Services running inside a TEE are now able to export common system and business metrics using these mechanisms. We use methods including [differential privacy](https://en.wikipedia.org/wiki/Differential_privacy) and data aggregation to maintain the privacy-preserving nature of Protected Audience API services while providing critical metrics that will help ad techs monitor their systems effectively. + +## High level Implementation + +* The servers have integrated with [OpenTelemetry](https://opentelemetry.io/) (OTel) so data can be exported to monitoring systems for creating dashboards, defining alerts and other uses. +* System metrics are supported, such as CPU resource utilization, and server specific business metrics. These metrics are predefined in the system, instrumented into the code by the privacy sandbox developers. +* Metrics are either non-noised or have noise added. +* There is a subset of metrics that are sensitive and may reveal information about user activity or which might give the ad tech running a server an unfair advantage. These metrics are noised using differential privacy. +* Other metrics do not have noise added, but are aggregated by OpenTelemetry for performance reasons, before being published. + + +## Code references + +The metrics implementation is split into two parts: + +* Common metrics are implemented in [definition.h](https://github.com/privacysandbox/data-plane-shared-libraries/blob/main/src/metric/definition.h) +* Metrics specific to a bidding and auction server are implemented in [server_definition.h](https://github.com/privacysandbox/bidding-auction-servers/blob/main/services/common/metric/server_definition.h). + + +## Properties of a metric + +This section describes the properties defined in the [list of metrics](#list-of-metrics). + + +### Metric name + +A name that uniquely identifies the metric among any Privacy Sandbox servers. + + +### Noising + +Metrics can either have noise added or be non-noised. The list of metrics differentiates between +metrics that will be exported without noise, and metrics that will have +noise added. + + +### Instrument + +In OpenTelemetry, metrics are recorded through [instruments](https://opentelemetry.io/docs/specs/otel/metrics/api/#instrument). We support three types of metric instruments for collection. + + +#### UpDownCounter + +A UpDownCounter ([OTel reference](https://opentelemetry.io/docs/specs/otel/metrics/api/#updowncounter)) sums the values it receives. For example, a counter may count the total number of bytes received, or total number of milliseconds taken for computation. These are aggregatable quantities, and the counter aggregates them. Counters can be used for noised and non-noised metrics. + + +#### Histogram + +A histogram ([OTel reference](https://opentelemetry.io/docs/specs/otel/metrics/api/#histogram)) is useful to record distributions of metrics. For example, it is useful to analyze various percentiles of the request duration or the size of the response payload. These can be recorded as a histogram. Histograms can be used for noised and non-noised metrics. + + +#### Gauge + +A gauge ([OTel reference](https://opentelemetry.io/docs/specs/otel/metrics/api/#gauge)) is useful for collecting a non-aggregatable value such as CPU usage. Gauges can only be used for non-noised metrics. + + +### Attribute + +Metrics can have associated attributes. Attributes common to all metrics are +listed in the [Common Attributes]( #common-attributes) section. In addition, metrics can have extra +attributes that are per-metric. These are recorded in the Attributes column in +the below table. + + +## List of metrics + + +### Common Metrics + +These are common metrics tracked across all B&A servers. + + +| Metric Name | Description | Instrument | Noising | Attributes | +|-----------------------------------------------------|---------------------------------------------------------------------------------|---------------|----------------|--------------------------------------------------------------------------------| +| request.count | Total number of requests received by the server | UpDownCounter | Not Noised | | +| request.duration_ms | Total time taken by the server to execute the request | Histogram | Not Noised | | +| request.failed_count_by_status | Total number of requests that resulted in failure partitioned by Error Code | UpDownCounter | Not Noised | Absl error code | +| request.size_bytes | Request size in bytes | Histogram | Not Noised | | +| response.size_bytes | Response size in bytes | Histogram | Not Noised | | +| system.cpu.percent | CPU usage | Gauge | Not Noised | total utilization, main process utilization, total load, total cpu cores | +| system.memory.usage_kb | Memory usage | Gauge | Not Noised | MemTotal, MemAvailable, main process | +| system.thread.count | Thread count | Gauge | Not Noised | | +| initiated_request.count_by_server | Total number of requests initiated by the server partitioned by outgoing server | UpDownCounter | Noised with DP | Server name | +| system.key_fetch.failure_count | Failure counts for fetching keys with the coordinator | Gauge | Not Noised | public key dispatch, public key async, private key dispatch, private key async | +| system.key_fetch.num_keys_parsed_on_recent_fetch | Number of keys parsed on the most recent key fetch | Gauge | Not Noised | public key GCP, public key AWS, private key | +| system.key_fetch.num_keys_cached_after_recent_fetch | Number of keys currently cached in memory after the most recent key fetch | Gauge | Not Noised | public key GCP, public key AWS, private key | + + +### SFE Metrics + +These are metrics tracked on the Seller front end B&A servers. + +| Metric | Description | Instrument | Noising | Attributes | +|--------------------------------------------------|----------------------------------------------------------------------------------|---------------|----------------|-----------------| +| initiated_request.auction.duration_ms | Total duration request takes to get response back from Auction server | Histogram | Noised with DP | | +| initiated_request.auction.errors_count_by_status | Initiated requests to auction that resulted in failure partitioned by Error Code | UpDownCounter | Noised with DP | Absl error code | +| initiated_request.auction.size_bytes | Size of the initiated Request to Auction server in Bytes | Histogram | Noised with DP | | +| initiated_request.bfe.errors_count_by_status | Initiated requests to BFE that resulted in failure, partitioned by Error Code | UpDownCounter | Noised with DP | Absl error code | +| initiated_request.kv.duration_ms | Total duration request takes to get response back from KV server | Histogram | Noised with DP | | +| initiated_request.kv.errors_count_by_status | Initiated requests to KV that resulted in failure partitioned by Error Code | UpDownCounter | Noised with DP | Absl error code | +| initiated_request.kv.size_bytes | Size of the Initiated Request to KV server in Bytes | Histogram | Noised with DP | | +| initiated_response.auction.size_bytes | Size of the initiated Response from Auction server in Bytes | Histogram | Noised with DP | | +| initiated_response.kv.size_bytes | Size of the Initiated Response from KV server in Bytes | Histogram | Noised with DP | | +| sfe.error_code | Number of errors in the SFE server by error code | UpDownCounter | Noised with DP | Error code | +| sfe.initiated_request.count_by_buyer | Total number of initiated requests per buyer | UpDownCounter | Noised with DP | Buyer | +| sfe.initiated_request.duration_by_buyer | Initiated requests duration per buyer | UpDownCounter | Noised with DP | Buyer | +| sfe.initiated_request.errors_count_by_buyer | Total number of initiated requests failed per buyer | UpDownCounter | Noised with DP | Buyer | +| sfe.initiated_request.size_by_buyer | Initiated requests size per buyer | UpDownCounter | Noised with DP | Buyer | +| sfe.initiated_response.size_by_buyer | Initiated response size per buyer | UpDownCounter | Noised with DP | Buyer | + + + +### Buyer Frontend Metrics + +These are metrics tracked on the BFE B&A servers. + +| Metric | Description | Instrument | Noising | Attributes | +|--------------------------------------------------|-----------------------------------------------------------------------------------|---------------|----------------|-----------------| +| bfe.error_code | Number of errors in the BFE server by error code | UpDownCounter | Noised with DP | Error code | +| initiated_request.bidding.duration_ms | Total duration request takes to get response back from bidding server | Histogram | Noised with DP | | +| initiated_request.bidding.errors_count_by_status | Initiated requests to Bidding that resulted in failure, partitioned by Error Code | UpDownCounter | Noised with DP | Absl error code | +| initiated_request.bidding.size_bytes | Size of the Initiated Request to Bidding server in Bytes | Histogram | Noised with DP | | +| initiated_request.kv.duration_ms | Total duration request takes to get response back from KV server | Histogram | Noised with DP | | +| initiated_request.kv.errors_count_by_status | Initiated requests by KV that resulted in failure partitioned by Error Code | UpDownCounter | Noised with DP | Absl error code | +| initiated_request.kv.size_bytes | Size of the Initiated Request to KV server in Bytes | Histogram | Noised with DP | | +| initiated_response.bidding.size_bytes | Size of the Initiated Response from Bidding server in Bytes | Histogram | Noised with DP | | +| initiated_response.kv.size_bytes | Size of the Initiated Response from KV server in Bytes | Histogram | Noised with DP | | + +### Bidding Metrics + +These are metrics tracked on the Bidding B&A servers. + +| Metric | Description | Instrument | Noising | Attributes | +|-----------------------------------------|----------------------------------------------------------|---------------|----------------|------------| +| js_execution.duration_ms | Time taken to execute the JS dispatcher | Histogram | Noised with DP | | +| js_execution.error.count | No. of times js execution returned status != OK | UpDownCounter | Noised with DP | | +| business_logic.bidding.bids.count | Total number of bids generated by bidding service | UpDownCounter | Noised with DP | | +| business_logic.bidding.zero_bid.count | Total number of times bidding service returns a zero bid | UpDownCounter | Noised with DP | | +| business_logic.bidding.zero_bid.percent | Percentage of times bidding service returns a zero bid | UpDownCounter | Noised with DP | | +| bidding.error_code | Number of errors in the bidding server by error code | UpDownCounter | Noised with DP | Error code | + +### Auction Metrics + +These are metrics tracked on the Auction B&A servers. + +| Metric | Description | Instrument | Noising | Attributes | +|---------------------------------------------|------------------------------------------------------------------------------------------------|---------------|----------------|-------------------------| +| js_execution.duration_ms | Time taken to execute the JS dispatcher | Histogram | Noised with DP | | +| js_execution.error.count | No. of times js execution returned status != OK | UpDownCounter | Noised with DP | | +| business_logic.auction.bids.count | Total number of bids used to score in auction service | UpDownCounter | Noised with DP | | +| business_logic.auction.bid_rejected.count | Total number of times auction service rejects a bid partitioned by the seller rejection reason | UpDownCounter | Noised with DP | Seller_rejection_reason | +| business_logic.auction.bid_rejected.percent | Percentage of times auction service rejects a bid | Histogram | Noised with DP | | +| auction.error_code | Number of errors in the auction server by error code | UpDownCounter | Noised with DP | Error code | + + +## Common attributes + +These attributes are consistent across all metrics tracked on the B&A servers +and will not be subjected to noise addition. This is because they are either +constant throughout the server's lifetime (such as service version) or externally +available (such as timestamp). Once differential privacy is implemented, these +attributes will be appended to the data post-noising, just prior to their +release from the TEE. + +| Attribute name | Description | +|:----------------------:|:-------------------------------------------------------------------:| +| Time | Time the metric was released. | +| Service name | Name of the service that the metric was measured on | +| Server-id | The id of the machine the metric was measured on. | +| Task-id | Unique id of the replica index identifying the task within the job. | +| Deployment Environment | The environment in which the server is deployed on. | +| Server Release Version | Specifies the current version number of the server software in use | +| Zone/ Region | The GCP zone or AWS region the instances are deployed to | +| Operator | The Operator configured through terraform | + +## Integration with OpenTelemetry and monitoring systems + +[OpenTelemetry](https://opentelemetry.io/) provides a cloud-agnostic API for recording metrics, traces, +and logging. The open source code running in the secure enclaves will be +instrumented using this API. The code will also be responsible for defining how +these metrics are exported from within the TEEs to the untrusted outside world. +This includes the steps defined above, like aggregation and differential +privacy. Once this telemetry leaves the secure enclave, it is available to the +ad tech to use as they please. + +We expect that most ad techs will choose to use an off-the-shelf [OpenTelemetry](https://opentelemetry.io/docs/collector/) +[collector](https://opentelemetry.io/docs/collector/) to collect the data, filter or sample as desired, and then +transport the data to the monitoring system(s) of their choice. The collector +will not run in the trusted enclave, meaning the ad tech can freely change the +configuration, shaping and sending the data to systems like [AWS CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html), +[Google Cloud Monitoring](https://cloud.google.com/monitoring/docs) or any system that integrates with OpenTelemetry +(see [list](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter)). + +Data made available to monitoring systems can then be used to determine and +alert on system health. They can also provide valuable dashboards and debugging +information. The trusted server code guarantees that exported telemetry will not +compromise user privacy, and therefore any sort of querying of this data can be +done to benefit the ad tech that operates the system. + +The OpenTelemetry Collector receives [traces](https://opentelemetry.io/docs/concepts/signals/traces/), [metrics](https://opentelemetry.io/docs/concepts/signals/metrics/), and [logs](https://opentelemetry.io/docs/concepts/signals/logs/), processes the telemetry, and exports it to a wide variety of observability backends using its components. The collector runs outside of TEE, so Ad tech can modify and configure it as they wish without affecting the code running inside the TEE. + +There is a default setup for both [GCP](https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/deploy/gcp/terraform/services/autoscaling/collector_startup.tftpl) and [AWS](https://github.com/privacysandbox/bidding-auction-servers/blob/main/production/packaging/aws/common/ami/otel_collector_config.yaml) deployment. +* OTel metric is used to receive [metric](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/monitoring_protected_audience_api_services.md) exported from TEE. +* OTel log is used to receive [consented debugging log](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/debugging_protected_audience_api_services.md#adtech-consented-debugging). +* OTel trace is not used currently. + + +## GCP Cloud Monitoring Integration + + +### Configure the collector + +The collector runs in an [individual instance](https://github.com/privacysandbox/bidding-auction-servers/blob/b2cfda5f00bcfa5afe92ef178a367797d0707e80/production/deploy/gcp/terraform/services/autoscaling/main.tf#L341), which is deployed with server stacks by terraform. The [collector end point ](https://github.com/privacysandbox/bidding-auction-servers/blob/b2cfda5f00bcfa5afe92ef178a367797d0707e80/production/deploy/gcp/terraform/environment/demo/seller/seller.tf#L88)is set up to point to the instance during deployment. + +The preset services are shown below, they all receive `otlp` and export to `googlecloud`. + +* `receivers` should not be changed, since the servers export `otlp`. +* `processors` should have `batch`, which reduce system resource usage. Optionally other processers can be added, such as filter. +* `exporters `can be replaced by any other tool ad tech prefers, such as opencensus, prometheus. It is possible to export to metric to multiple places, such as `[googlecloud, prometheus]` + + +``` + service: + pipelines: + traces: + receivers: [otlp] + processors: [batch] + exporters: [googlecloud] + metrics: + receivers: [otlp] + processors: [batch] + exporters: [googlecloud] + logs: + receivers: [otlp] + processors: [batch] + exporters: [googlecloud] +``` + + + +### Dashboards + +[Deploying Bidding and Auction Servers on GCP](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_gcp_guide.md) includes functionality to +export metrics to [GCP cloud Monitoring](https://cloud.google.com/monitoring/docs). Dashboards with the metrics are +included for both buyer and seller deployments. To locate the dashboards, +navigate to the Cloud Monitoring section in your GCP console. Upon opening a +dashboard, you will find various widgets displaying the instrumented metrics. +The following figures show the location of the dashboards and a sample dashboard. + +![Figure 1. Locating the dashboard](images/monitoring_protected_audience_api_services_fig_1.png "image_tooltip") + +**Figure 1.** Locating the dashboard + +![Figure 2. Sample Dashboard](images/monitoring_protected_audience_api_services_fig_2.png "image_tooltip") + +**Figure 2.** Sample Dashboard + + +**Terraform Configuration for dashboards:** + +* The definitions of the dashboards can be found at [Dashboard terraform](https://github.com/privacysandbox/bidding-auction-servers/tree/b27547a55f20021eb91e1e61b0d2175b4aee02ea/production/deploy/gcp/terraform/services/dashboards). +* Customizing Metric Exporter can be done through the [metric exporter deployment config](https://github.com/privacysandbox/bidding-auction-servers/blob/b27547a55f20021eb91e1e61b0d2175b4aee02ea/production/deploy/gcp/terraform/services/autoscaling/collector_startup.tftpl#L63). + + +### Setting up alerts + +In GCP, [alerts](https://cloud.google.com/monitoring/alerts) can be added to monitor metrics with the following steps. + +1. Under cloud Monitoring - Alerting, create policy. +2. Select an existing metric. +3. Configure the trigger condition (that will generate incident) and notification from configured notification channels. +4. After being created, an alarm can be enabled/disabled. Snooze can be created to temporarily silence the alert. + +See an example below. + +![Figure 3. set up alerts](images/monitoring_protected_audience_api_services_fig_3.png "image_tooltip") + +**Figure 3.** set up alerts + + +### Setting up Service-Level Objective + +In GCP, Service-Level Objective (SLO) can be set up with existing metrics with following steps. + + + +1. Define a custom service under cloud Monitoring - Detect - SLOs +2. Under defined service, create SLO with Service Level Indicator (SLI) that monitors an existing metric. +3. Define a performance goal, such as 99%. +4. Once SLO defined, SLO alert can be set with lookback duration and burn rate. Notification channels can be added. + +See an example below. + +![Figure 4. set up SLO](images/monitoring_protected_audience_api_services_fig_4.png "image_tooltip") + +**Figure 4.** set up SLO + + +## AWS Monitoring Integration + + +### Configuring the collector + +The collector runs on the host where the trusted server runs in the [Nitro Enclave](https://aws.amazon.com/ec2/nitro/nitro-enclaves/). As a result, the collector endpoint is set to point to the localhost port. + +The preset services are shown below, they all receive `otlp` and export to `aws`. + + + +* `receivers` and `processors` should not be changed. +* `exporters `can be replaced by any other tool ad tech prefers, such as opencensus, prometheus. It is possible to export to metric to multiple places, such as `[awsemf, prometheus]` + + +``` +service: + pipelines: + traces: + receivers: [otlp,awsxray] + processors: [batch/traces] + exporters: [awsxray] + metrics: + receivers: [otlp] + processors: [batch/metrics] + exporters: [awsemf] + logs: + receivers: [otlp] + processors: [batch/logs] + exporters: [awscloudwatchlogs] + +``` + + + +### Dashboards + +[Deploying Bidding and Auction Servers on AWS](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_aws_guide.md) includes functionality to +export metrics to [AWS CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). Dashboards with the metrics are +included for both buyer and seller deployments. To locate the dashboards, +navigate to the CloudWatch section in your AWS console. Upon opening a +dashboard, you will find various widgets displaying the instrumented metrics. +The following figures show the location of the dashboards and a sample dashboard. + +![Figure 5. Locating aws dashboard](images/monitoring_protected_audience_api_services_fig_5.png "image_tooltip") + +**Figure 5.** Locating aws dashboard + + +![Figure 6. Sample aws Dashboard](images/monitoring_protected_audience_api_services_fig_6.png "image_tooltip") + +**Figure 6.** Sample aws Dashboard + +**Terraform Configuration for dashboards:** + + + +* The definitions of the dashboards can be found at [Dashboard terraform](https://github.com/privacysandbox/bidding-auction-servers/tree/b27547a55f20021eb91e1e61b0d2175b4aee02ea/production/deploy/aws/terraform/services/dashboards). +* Customizing Metric Exporter can be done through the [metric exporter deployment config](https://github.com/privacysandbox/bidding-auction-servers/blob/b27547a55f20021eb91e1e61b0d2175b4aee02ea/production/packaging/aws/common/ami/otel_collector_config.yaml#L52). + +> [!NOTE] +> AWS CloudWatch dashboard doesn't support the [Otel histogram data model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#histogram), Consequently, when visualizing metrics such as latency, the dashboards can display only basic statistics like the mean (average) value, rather than more detailed percentile metrics (e.g., p95 or p99). Adtech can [configure the collector](#AWS-Monitoring-Integration) to send metric to a 3rd party dashboard that support the Otel histogram. An example configuration for this setup will be published in the future. + +### Configuring alerts + +In AWS, alarms can be added to monitor metrics with following steps. + + + +1. Under CloudWatch: Alarms, Create alarm +2. Select an existing metric +3. Configure the trigger condition and notification +4. After being created, alarm can be enable/disabled + +See an example below. + + +![Figure 7. set up aws alerts](images/monitoring_protected_audience_api_services_fig_7.png "image_tooltip") + +**Figure 7.** set up aws alerts + + + +### Setting up SLOs + +AWS SLO is still in [Preview release](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Monitoring-Sections.html). + +More details will be added for AWS in future + + +## Differential privacy and noising + +To uphold the privacy guarantees of the Protected Audience API, we use [differential privacy](https://github.com/google/differential-privacy) (DP) to add noise to sensitive metrics before they leave the trusted execution environment (TEE). Ad tech partners can select the specific metrics they wish to track from a list of [pre-instrumented metrics](#list-of-metrics). All servers within the same service will track this identical set of metrics. + +In DP, the level of noise added depends on several factors, notably the available privacy budget. A higher budget allows for less noise, while a lower budget necessitates more. Our system establishes a total privacy budget encompassing all tracked privacy-sensitive metrics. This budget is then allocated proportionally across the metrics an ad tech partner chooses to monitor. Consequently, tracking more sensitive metrics means each receives a smaller share of the budget, resulting in increased noise per metric. + +An ad tech can choose to alter how the privacy budget is distributed across different metrics. See [privacy_budget_weight](#privacy_budget_weight) in noise-related-configuration. + +Metrics that aren't privacy-sensitive don't consume any budget, and remain noise-free. Ad tech partners can track these without any alteration. The [table above](#list-of-metrics) details whether noise is added to each metric. + +All metrics are aggregated on the server before being sent to the OTel collector. Noise is applied only to the privacy-sensitive metrics after this aggregation just before being sent to the collector. Non-noised metrics are pushed out aggregated, but without any noise. + +Privacy-sensitive metrics are aggregated within the TEE over a period called the `ExportInterval`. Noise is added just before these aggregated metrics are released externally, to maintain differential privacy. The amount of noise added is independent of the `ExportInterval`. A longer ExportInterval includes more data, which reduces the relative impact of the added noise. In other words, a longer ExportInterval improves the signal-to-noise ratio, resulting in more accurate data. Choosing a longer ExportInterval offers less frequent but less noisy data, whereas a shorter `ExportInterval` means more frequent data releases but with a higher relative noise level. + + +## Server Configuration + +[TELEMETRY_CONFIG](https://github.com/privacysandbox/bidding-auction-servers/blob/b2cfda5f00bcfa5afe92ef178a367797d0707e80/production/deploy/gcp/terraform/environment/demo/seller/seller.tf#L87) is a flag in terraform configuration, used for all metric configuration. The content is expected to be a TextProto conforming to [TelemetryConfig](https://github.com/privacysandbox/bidding-auction-servers/blob/release-1.2/services/common/telemetry/config.proto). + + +### Configuring the metric collection mode + +Metrics can be collected in different modes, based on the kind of [build](https://github.com/privacysandbox/bidding-auction-servers/blob/release-1.2/BUILD#L110) that is running. The build could either be `prod` or `non_prod`. The metrics collected for a `prod` build essentially default to the `PROD` metric collected behavior, i.e sensitive metrics will have noise added before exiting the TEE. + +For a `non_prod` build, the `mode` parameter defines how you want to collect the metrics. The following table defines this behavior in detail. + +| Mode | Build | Behavior | +|:-----------------------:|:-------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------:| +| OFF | non_prod and prod | No metrics are collected | +| PROD | non_prod and prod | Sensitive metrics are noised | +| EXPERIMENT | Only works on non_prod builds. Behavior is the same as PROD on a prod build. | Sensitive metrics are not noised | +| COMPARE | Only works on non_prod builds. Behavior is the same as PROD on a prod build. | Sensitive metrics are duplicated, with one set being noised, and the other being non-noised. Both are released for comparison purposes. | + + +#### Examples + +With the following config, privacy sensitive metrics will have noise added. + + +``` +mode: PROD +``` + + +With the following config, no noise will be added to any metrics on `non_prod` builds. On `prod` build, the mode will behave like `PROD` mode, and privacy sensitive metrics will have noise added. + + +``` +mode: EXPERIMENT +``` + + +See [TelemetryConfig proto definition](https://github.com/privacysandbox/bidding-auction-servers/blob/release-1.2/services/common/telemetry/config.proto) for details. + + +### configuring-collected-metrics + +To configure the set of metrics to monitor, use the [TelemetryConfig::metric](https://github.com/privacysandbox/bidding-auction-servers/blob/release-1.2/services/common/telemetry/config.proto#L44) field. + + +#### Example + + +``` +mode: PROD +metric { name: "m_0" } +metric { name: "m_1" } +``` + + +In the above example, only metrics names `m_0` and `m_1` will be exported. + +The configured metrics can include metrics where noise will be added, or non-sensitive metrics where noise will not be added. The privacy budget will be split evenly across only the metrics where noise will be added. + +If the configured metrics are empty, i.e. no metrics are configured, then all metrics, including all privacy sensitive and non-sensitive metrics are collected. + + +### configuring-export-interval + +Export intervals for metrics with and without noise can be configured separately. The metrics without noise are aggregated purely for performance reasons, whereas the ones with noise can achieve a better signal to noise ratio with aggregation. Thus, we have separate configurations for each. + +`metric_export_interval_ms` configures the export interval of non-noised metrics. The default value is 60,000ms (1 min). + +`dp_export_interval_ms` configures the export interval of noised metric. The default value is 300,000ms (5 min). + +> **Note:** After adjusting "export_interval_ms", dashboard granularity (interval) should be adjusted >= this to avoid showing saw teeth pattern. + +#### Example + + +``` +mode: PROD +metric_export_interval_ms: 100000 +dp_export_interval_ms: 100000 +``` + + +For further information, see the [opentelemetry.io page](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#metric-points). + + +### noise-related-configuration + +We expose several knobs to tune the noise configurations so an ad tech can most optimally use the overall privacy budget. Below are the noising parameters that ad tech can configure for each Privacy Impacting metric. + + +#### max_partitions_contributed + +Default value: 1 + +The maximum number of partitions to be reported per request, higher value results in more noise added. + +Example: + +This metric has buyer as partition, by configuring `max_partitions_contributed` to 2, the metric will record at most 2 buyer's data for each sfe request. +``` +metric { name: \"sfe.initiated_request.to_bfe.count\" max_partitions_contributed: 2 } +``` + +[code link](https://github.com/privacysandbox/data-plane-shared-libraries/blob/main/src/telemetry/flag/config.proto#L63) + +#### lower_bound, upper_bound + +Value is set based on ad tech’s observation on the metric, which range should cover most raw data, larger range results in more noise added. + +Example: + + +``` +metric { name: \"m_0\" lower_bound: 1 upper_bound: 2 } +``` + +[code link](https://github.com/privacysandbox/data-plane-shared-libraries/blob/main/src/telemetry/flag/config.proto#L66) + +#### privacy_budget_weight + +Default value: 1 + +All Privacy Impacting metrics split total privacy budget based on their weight. + +i.e. +`privacy_budget = total_budget * privacy_budget_weight / total_weight` + +Example: + + +``` +metric { name: \"m_0\" privacy_budget_weight: 2 } +``` + +[code link](https://github.com/privacysandbox/data-plane-shared-libraries/blob/main/src/telemetry/flag/config.proto#L72) + +#### drop_noisy_values_probability + +Default value: 0.0 + +The probability that noised metric value will be turned to 0. Setting this to a higher value ensures that the more noise will be turned to 0, but also means that more actual data may also be turned to 0. Setting this to 1 means all values will be turned to 0 (eliminating all noise, but also eliminating all actual data). This value should be set within an open range of (0.0,1.0). + +Example: + + +``` +metric { name: \"m_0\" drop_noisy_values_probability: 0.99 } +``` + + +The rationale behind the approach is that the added noise has a high probability of having a small value and a low probability of having a large value, while real data with large value should be always exported. In this example, the drop_noisy_values_probability parameter 0.99 will remove any data with value no larger than 99% of noise, which could include both noise and actual data. + +[code link](https://github.com/privacysandbox/data-plane-shared-libraries/blob/main/src/telemetry/flag/config.proto#L75) + +## Understanding metric noise + + +### Privacy non-sensitive metrics + +Metrics such as request count (UpDownCounter), request duration(Histogram), CPU usage(Gauge) which are non-sensitive are always exported without noise. Metrics that do not have noise added are exported with the suffix Raw, as highlighted in the figure below. + +![Figure 8. raw metric](images/monitoring_protected_audience_api_services_fig_8.png "image_tooltip") + +**Figure 8.** raw metric + + +### Privacy sensitive metrics + + +#### Noise + +As described earlier, a total privacy budget is split evenly among the privacy sensitive metrics. These metrics define a lower and upper bound of their value. The measured value is first clamped within these bounds. Laplace noise is then applied based on the bounds and available privacy budget. + +The privacy budget split weight and the bounds for metrics will be configurable in a future release. + + +#### Example In Prod + +In `prod` server, noise will be added to Privacy sensitive metrics and exported every `dp_export_interval_ms` (see [Export interval](#configuring-export-interval)). The metrics with noise added have the suffix “Noised” added, as shown in the figure below. + +![Figure 9. noised metric](images/monitoring_protected_audience_api_services_fig_9.png "image_tooltip") + +**Figure 9**. noised metric + + +#### Compare mode (only works in non_prod builds) + +To understand the impact of noise, servers running a `non_prod` build can be deployed with `TELEMETRY_CONFIG`: + + +``` +mode: COMPARE +``` + + +This will output metrics with and without noise as shown in the figure below. This enables their comparison side by side. + + +![Figure 10. compare noised and raw metric](images/monitoring_protected_audience_api_services_fig_10.png "image_tooltip") + +**Figure 10.** compare noised and raw metric + + +## Improving signal-to-noise ratio + +There are tunable parameters and techniques that can be used to increase the signal to noise ratio when exporting metrics with noise. + + +### Higher QPS + +A higher level of queries-per-second (QPS) enables more events to be aggregated per unit time. Since the noise added depends only on the metric bounds and privacy budget, it is independent of how many events are aggregated into the same batch. Increasing the number of events per batch will thus increase the signal to noise ratio and provide a more accurate measurement. + + +### Adjust export intervals + +Similar to a higher QPS, a larger `dp_export_interval_ms` will cause more events in each export batch, thus increasing the signal to noise ratio. This improved accuracy comes at the cost of having less frequent updates. + +[configuring-export-interval](#configuring-export-interval) + + +### Monitoring Subsets of Metrics + +The privacy budget is shared among all privacy-sensitive metrics. Thus, monitoring fewer sensitive metrics means each of them gets a larger portion of the total privacy budget. This results in lesser noise and more accurate measurements. + +[configuring-collected-metrics](#configuring-collected-metrics) + + +### Configure noise parameters + +The noise parameters give a lot of flexibility to fine-tune noising to the specific ad tech requirement. Ad tech can collect data about their actual monitoring ranges and use these parameters to optimize the noise. + +[noise-related-configuration](#noise-related-configuration) diff --git a/production_operation.md b/production_operation.md new file mode 100644 index 0000000..b74735e --- /dev/null +++ b/production_operation.md @@ -0,0 +1,71 @@ +# Production operation + +In this explainer, we’ll review the features available in the Protected Audience services and define the key components for production readiness, so that these services are operable at the desired reliability rate. + +Our goal is to build a set of mitigations and plans so that when outages happen, they can be gracefully and quickly addressed, rather than trying to prevent outages or build “perfect” servers outright. + + +## Design principles + +These pillars are presented in alphabetical order, rather than order of importance. + + +### Change management + +_We’ll offer management tools to address service outages, which typically occur when there’s a system change._ + + + +* Permissions and roles: to control who can make changes and when. +* Server release process: this covers how a new release candidate is created and how the candidate is tested to certify that it’s ready and has no regressions. Also, the ability to run "canary" jobs to test out a new release at a small scale before it goes to full production and understand how rollbacks will work. +* Lifecycle management: releases will not be supported forever, and so we need to work out what commitments there will be and how old releases will be deprecated. This includes the handling of breaking versus non-breaking changes. + + +### Developer experience + +_We’ll create a positive experience for developers by responding to open issues and offering a well-built service with the ability to contribute code._ + + + +* Repository structure: the open source repositories for the servers should be consistent and integrated with tooling. For example, for code indexing and search, continuous integration, continuous build, etc. +* Community contribution policies: these should be consistent between servers, especially where they share shared repositories for common components. +* Debugging tools: these are especially tricky to build for Protected Audience services because of the deliberate limitations of what the servers expose about their running state. We plan to publish separate explainers (e.g. [Debugging protected audience API services][1]) on this topic. + + +### Incident management + +_We’ll offer playbooks for service operators and define emergency procedures_ + + + +* Support structures: when there is an outage, how can server operators report it for triage and remediation? +* Playbooks: for server operators there should be a well defined set of common alerts and how/when to mitigate them. +* Pre-planned mitigations: a set of responses that are expected to be useful. For example, to roll back to a previous server version or to restore corrupt data. + + +### Reliability + +_We’ll create a system that meets reliability requirements._ + + + +* Reliability ranges: the server operators are responsible for deciding the reliability targets that they’re aiming for, e.g. uptime. The servers should be able to meet a range of different reliabilities; this is important because often extremely high reliability comes with significant costs. +* Monitoring and alerting: the ability to detect when servers fail, or stray from reliability targets, is necessary. Without this server operators can’t tell if the servers are performing correctly. See [Monitoring protected audience API services][2] for details. +* Risk management: past outages will be tracked and documented together with changes that come out of them. This both to track system health over time and to make sure that known bugs are corrected. +* Graceful degradation: when servers are overwhelmed they should be able to shed traffic in a predictable way (while also alerting operators that they’re in this state, as above). +* Scale testing: it’s useful to know where the limits of the servers are. For example, how much traffic can one single instance handle before it’s overwhelmed? + + +### Resources + +_We’ll provide information as to what resources the server will use and how best to set it up to use them efficiently._ + + + +* Sensible default deployment configurations: to make sure that common mistakes are avoided. +* Ongoing benchmarking: to prevent performance regressions. +* Cost prediction: to estimate the Cloud costs required to run certain workloads. + + +[1]: https://github.com/privacysandbox/fledge-docs/blob/main/debugging_protected_audience_api_services.md +[2]: https://github.com/privacysandbox/fledge-docs/blob/main/monitoring_protected_audience_api_services.md diff --git a/protected_app_signals_cost.md b/protected_app_signals_cost.md new file mode 100644 index 0000000..79a02d4 --- /dev/null +++ b/protected_app_signals_cost.md @@ -0,0 +1,168 @@ +PAS Cost Estimate Explainer + +**Authors:** +
[Stefano Galarraga][0], Google Privacy Sandbox + + +Overview and Goals +================== + +[Protected App Signals][1] is a new design aimed at the app install ads market. The system is based on server-side execution via the [B&A services][2] and [TEE K/V services][16]. Shifting the auction execution on the server side has benefits in terms of latency and computing power but introduces cost to adtechs to run the cloud instances where B&A will reside. + +This document is based on the existing documentation about cost analysis and reduction ([B&A][3], [K/V][13]) and aims at providing an analysis of how to apply those concepts to the PAS architecture. + +Assumptions +=========== + +The following assumptions are made in computing the cost: + +* About the DSP K/V server used by the SFE we will consider two cases + * TEE Ad Retrieval + * Contextual path ad retrieval +* SFE is using a BYOS K/V server +* Buyside and Sellside B&A services can be running in different networks + * This means that there could be network cost for communications among the trusted components + +Cost Analysis +============= + +The following diagram shows the cost items for the case of BYOS K/V Server + +![Diagram for PAS Servers Cost Analysis](images/pas-cost-elements-diagram.png "Diagram for PAS Server Cost Analysis") + +With reference to the diagram above there are 9 factors contributing to the cost of running PAS: + +* _A,B_: Cost related to increase in payloads received from and sent back to the device +* _C_: Incremental cost of running the _Seller Ad Server_: this is assumed to be negligible +* _D, E_: Incremental cost caused of extra traffic from/to Buyer RTB Server +* _F_: Extra cost of running the Buyer RTB Server: this is assumed to be negligible +* _G_: Cost associated to the traffic from the _Seller Ad Server_ to the SFE + * For this one we will provide two estimates for the cases of TEE ad retrieval and contextual path ad retrieval +* _H_: Cost of running the SFE +* _I_: Cost incurred by the buyer for the SFE to BFE communication. This cost will vary depending on the seller and buyer hosting its services in the same or a different cloud providers +* _S_: Cost incurred by the SFE for the SFE to BFE communication. This cost will vary depending on the seller and buyer hosting its services in the same or a different cloud providers +* _Z_: Cost of running the auction server +* _L_: Cost of running the BFE +* _N_: Cost of running the Bidding Server +* _P_: Cost of running the Retrieval Server. This should include the cost of updating the storage +* _Q,R_: Cost of updating the ads and metadata in the Ad Retrieval Server +* _T_: Cost associated to the traffic from SFE to the K/V Server +* _U_: Cost of running the KV server +* _M,V_: Cost for intra B&A communication + +Seller Cost +----------- + +The seller cost for B&A is analyzed [here][4]. This section will describe the cost differences introduced by PAS. + +### A,B: Cost related to increase in payload sent to and received from the device + +There is no cost expected for the incoming traffic A. + +The exchange will incur additional cost in case of extra data included for PAS in the encrypted payload with the response from SFE to be sent to the device. As of today PAS is requiring no extra data to be sent so the cost is expected to be the same as estimated for [C9 in the B&A cost explainer][4]. + +### D,E: Cost associated to the extra traffic from the Seller Ad Server to the RTBs + +The extra traffic is caused by the data that the RTB will decide to send to their BFE via the Ad Server. The data will likely include, contextual embeddings to use in generateBid and, depending on the solution adopted for ad retrieval, the amount of data sent is expected to be higher since it will contain an additional list of ad identifiers and possibly some per-ad embeddings to be used during TopK selection (in case of hybrid ad retrieval) more details about network cost are discussed in the [B&A cost explainer][5]. + +### G: Cost associated to the traffic from the Seller Ad Server to the SFE + +There are three elements contributing to this cost: + +* The cost for the exchange of sending the [SelectAdRequest][6] including the PAS signals from the device for all the sellers to SFE (refer to C1 in the [B&A cost explainer][7]) +* The cost for the SFE of receiving the SelectAd request from Ad Server: (refer to [C2 in the B&A cost explainer][8]) +* The cost for the SFE of sending the SelectAdResponse response to the Ad Server (see [C8 in the B&A cost explainer][9]) + +In the case of PAS the costs C1 and C2 of the communication from AdServer to SFE will depend on the solution adopted for ad retrieval. In case of TEE ad retrieval the RTB server will send to the Ad Server information to help during retrieval and bidding such as contextual embeddings while in case of contextual the amount of data sent is expected to be higher since it will contain an additional list of ad identifiers and possibly some per-ad embeddings to be used during TopK selection (in case of hybrid ad retrieval) or bidding. + +### H, V: Cost of running the SFE and Auction Server + +The CPU cost of running both the SFE and Auction Server are expected to be similar to that of [serving the PA traffic][10]. + +### T: Cost associated to the traffic from SFE to the K/V Server + +This cost too is expected to be the [same as per the PA traffic][5]. + +Buyer Cost +---------- + +We will refer to [this breakdown][11] from the B&A cost explainer. + +### D,E: Cost associated to the extra traffic from the Seller Ad Server to the RTBs + +This is the cost sustained by the RTB for the contextual path traffic. Please refer to the seller side discussion for details. + +### F: Extra cost of running the Buyer RTB Server + +There will be changes both in the RTB processing of the PAS request and in the size of the data to be sent to the RTB server. We assume this difference in cost to be negligible. + +### L: Cost of running the BFE + +This is the [VM Cost][12] marked C2 and C5 in the B&A explainer. With PAS, the BFE is running an additional flow to PA, thus incurring extra cost. The cost of the PAS flow could be considered equivalent to the cost of processing an extra CustomAudience (or Interest Group in the Chrome parlance). + +### N: Cost of running the Bidding Server + +This is where we expect noticeable differences, mostly because of the computation cost. Initial estimates suggest the AppInstall bidding server may require ~70% additional computational power for multiple javascript executions. We should account the extra workload required to run more complex inference + +The distribution of the cost for running the Bidding Server based on internal estimates is shown below: + +The three factors are: + +* [Inference][15]. This is expected to constitute the vast majority of the cost. Details on how to compute the inference costs will be added later. +* Cost of preparing the data for ad retrieval: essentially the cost for running the `prepareDataForAdRetrievalUDF`. This is a minor contributor. +* Base cost for running the bidding process, this is the cost for coordinating the flow, processing the data from the ads retrieval server and sending it to generateBid, handling the response and returning the results to the BFE. + +There are a lot of changes in bidding server + +1. prepareDataForAdRetrieval - This is a new function that runs before generateBid when using TEE based retrieval and it is skipped in case of contextual based retrieval. The function is intended to be used to decode the signals received from the device and make them available to the retrieval server UDF and to the generateBid. We are expecting that this function might run some inference too. The main factor to consider when estimating this cost are: + 1. Complexity of the model used for feature extraction + 2. Amount of signals received (will likely always be the currently max allowed size of 1.5KB but we are working on enabling the seller to control the payload size alloted for each buyer so this might change). + 3. Logic and language used to implement the UDF (using JS or WASM) + 4. Cost of communicating with the inference service (sidecar process at the moment) +2. Sending request to retrieval server (marked as O in the diagram above) - cost is zero +3. Getting info back from retrieval and sending it to generateBid. This may involve data copying into v8. We need to figure out how much data this is, and what the copying costs are (response size from retrieval, size of embeddings, number of embeddings, number of ads). +4. generateBid: + 1. Cost of the running the script - this is likely similar to the remarketing case. + 2. Cost of sending inference requests to inference sidecar + cost per inference. To compute this you need to know how many ads will be batched in a single inference, how many models will be used during bidding and preparation of data for retrieval, the size of each model and how expensive each inference call is. + 3. Compared to PA, where each Custom Audience will run generateBid with its subset of ads, there is only one generateBid invocation for all the PAS ads +5. General inference cost - download models, general bookkeeping (memory cost of storing models in sidecars). Model storage cost on cloud storage. + +### P: Cost of running the Retrieval Server + Q,R: Cost of updating the ads and metadata in the Ad Retrieval Server + +For details about the K/V server cost you can refer to [this doc][13]. + +#### TEE Ad Retrieval + +The cost contributions are: + +* _Compute cost_: this is the cost for running the ad retrieval UDF, we expect this to be the majority of the cost. This is the C2 cost contributor in the high level cost breakdown [here][14]. +* _Bulk storage cost_: the cost for the data to be stored in the Ad Retrieval server. This is expected to be minimal. This is the C4 cost contributor in the high level cost breakdown [here][14]. +* _Cost to transfer data to bulk storage_: This is the cost for periodically updating the data in the server and depends on the amount of data and frequency of updates. This is covering both C3 and C5 cost contributors in the high level cost breakdown [here][14]. +* _Server Monitoring Cost:_ This is the part of the cost for using the monitoring services and tracked as C7 in the high level cost breakdown [here][14]. + +#### Contextual Ad Retrieval + +The cost contributions are: + +* _Compute cost_: In this case the cost is expected to be much lower than in the case of TEE AdRetrieval because lookup is supported by a simple system-provided UDF and soon no UDF at all. +* _Bulk storage cost_: See the comments for the TEE Ad Retrieval cost +* _Cost to transfer data to bulk storage_: See the comments for the TEE Ad Retrieval cost +* _Server Monitoring Cost:_ This will be the same as for TEE Ad Retrieval + +[0]: https://github.com/galarragas +[1]: https://developers.google.com/privacy-sandbox/relevance/protected-audience/android/protected-app-signals +[2]: https://developers.google.com/privacy-sandbox/relevance/protected-audience/android/bidding-and-auction-services +[3]: ./bidding_auction_cost.md +[4]: ./bidding_auction_cost.md#high-level-cost-breakdown +[5]: ./bidding_auction_cost.md#network-costs +[6]: https://github.com/privacysandbox/bidding-auction-servers/blob/main/api/bidding_auction_servers.proto#L404 +[7]: ./bidding_auction_cost.md#seller-untrusted-server +[8]: ./bidding_auction_cost.md#load-balancer-and-network-address-translation-nat +[9]: ./bidding_auction_cost.md#seller-untrusted-server +[10]: ./bidding_auction_cost.md#vm-costs +[11]: ./bidding_auction_cost.md#high-level-cost-breakdown-1 +[12]: ./bidding_auction_cost.md#vm-costs-1 +[13]: ./key_value_service_cost.md +[14]: ./key_value_service_cost.md#high-level-cost-breakdown +[15]: ./inference_overview.md +[16]: https://github.com/privacysandbox/protected-auction-key-value-service \ No newline at end of file diff --git a/protected_audience_auctions_mixed_mode.md b/protected_audience_auctions_mixed_mode.md new file mode 100644 index 0000000..5c67dde --- /dev/null +++ b/protected_audience_auctions_mixed_mode.md @@ -0,0 +1,396 @@ +**Authors:**
+[Priyanka Chatterjee][1], Google Privacy Sandbox
+[Paul Jensen][2], Google Chrome, Google Privacy Sandbox
+Itay Sharfi, Google Privacy Sandbox + +# Protected audience auctions mixed mode + +Protected Audience auctions **mixed mode** would allow sellers (SSPs) and buyers (DSPs) +to participate in auctions on their preferred platform, [on-device][3] or in [Bidding & Auction services (B&A)][4]. +We call this mixing of on-device and off-device auctions together as **mixed mode**. + +If sellers run Protected Audience auctions either on-device only or in B&A only, that +might limit partner buyers they can work with, given some buyers may prefer to generate +bids either on-device or B&A. Hence, sellers can choose to support mixed-mode auctions +in order to facilitate DSP partners who may prefer on-device or B&A or both types of +platform for bidding. The auctions in mixed mode would allow sellers to run both on-device +and B&A auctions together in one overall Protected Audience auction. + +In any particular seller's mixed component auction, each buyer can choose to bid either +on-device or in B&A, though not both at the same time. + +For a [multi-seller auction][5], each seller can run an auction entirely on-device, +entirely in B&A or both (a seller can further split and run a component auction on-device +for buyers preferring on-device and another component auction in B&A for buyers preferring B&A), +the choice for that is left to the seller. Refer to the [section](#multi--seller-auctions-mixed-mode) +for more details. + +In this way, each Protected Audience auction can facilitate mixed mode whether it is a +[single seller auction](#single--seller-auctions-mixed-mode-architecture) or a +[multi-seller auction](#multi--seller-auctions-mixed-mode). + + +## Seller role in mixed Protected Audience auctions + +With mixed Protected Audience auctions, sellers can support single-seller auctions and +multi-seller auctions on-device,in B&A, or both the platforms. + +* If sellers run auctions on-device, partner buyers generate bids on-device. +* If sellers want to run auctions either in B&A or on-device, mixed mode is an option. + +In case of mixed mode, sellers run two separate auction flows to support the following: + +* Buyers generating bids on-device. +* Buyers generating bids in B&A. + +_Note: Sellers require some additional work to support mixed mode. Refer to the [section](#guide-for-sellers-to-support-mixed-mode) for guidance._ + + +## Buyer role in mixed Protected Audience auctions + +Buyers can choose to generate bids either on-device, in B&A, or support either of +the platforms on a per auction basis. + +Within a Protected Audience mixed auction, buyers can generate bids either on-device or +in B&A. Note that this is true for single seller auctions and [multi-seller auctions for the web][5]. + + +## Support for mixed mode by types of Protected Auctions + +We propose Protected Audience auctions mixed mode for web browsers on desktop and Android. +Protected Auctions mixed mode is not available for Android apps. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Auction type + Auction strategy + Platforms + Is mixed mode recommended? +
Single-seller auctions + + Web browser on desktop and Android + Yes +
+ Android app + No +

+Reason: higher latency for auctions on Android apps +

Multi-seller auctions + Device-orchestrated component auctions + Web browser on desktop and Android + +
Server-orchestrated component auctions + Android app +

+Web browser on desktop and Android +

No +

+Reason: On-device auctions are not feasible +

Waterfall and hybrid mediation + Android app + No +
+ + +## Single-seller auctions mixed mode architecture + +### [Recommended] Parallelization of on-device and server-side auctions + +Seller's JavaScript on publisher website on browser invokes `getInterestGroupAdAuctionData()` +to get the encrypted B&A payload, and then sends ad request to seller's ad service, +where the request includes contextual payload and encrypted B&A payload. Sellers +would stream contextual signals back to the browser as soon as they are available, +along with a [nonce][6] to kick off Protected Audience auctions on-device. In parallel, +sellers would call B&A with the same nonce to kick off Protected Audience auctions +in B&A. After receiving a response from B&A, sellers would stream the B&A winner back +to the browser. + +_Note: This system architecture would facilitate on-device and B&A auctions in +parallel in one overall protected audience auction, such that no incremental +latency is added._ + +![Architecture diagram.](images/protected-audience-mixed-mode-single-seller-parallelized.png) + +_In the above diagram:_ +* _Buyers who prefer generating bids on-device: buyer a, buyer b_ +* _Buyers who prefer generating bids in B&A: buyer x, buyer y_ + + +#### Flow + +_Note: The parts of the system requiring updates and additional changes required to support mixed mode, +are annotated with **[Change required]**._ + +##### Seller's JS on browser + +* **[Change required in seller JS]** Seller's tag on the browser decides the mode to run in the + request path: + * On-device only mode + * Mixed mode + +* **[Change required for payload optimization]** In order to fetch the encrypted B&A payload, + sellers may specify only the partner buyers who prefer B&A as a platform in [`perBuyerConfig`][7] + for payload optimization. Excluding the interest groups of buyers who prefer generating bids + on-device, can optimize the B&A payload size. Refer to this [documentation][8] for more details. + +* The seller's JS on the publisher website on the browser sends one ad request (with contextual + payload and B&A ciphertext) to support both on-device auctions and B&A auctions. The ad request + is a JavaScript Fetch initiated from the browser with `AdAuctionHeaders: true`. + * If on-device Protected Audience parallelization is enabled, the seller's JS invokes `runAdAuction()` + first before sending out the contextual request, so that buyer key/value services are looked up + in parallel to fetch `trustedBiddingSignals`. + + +##### Server side auctions + +* **[Change required in seller's ad service]** Seller's ad service must work with partner buyers + offline to record their choice of platform (on-device or B&A). It is also possible a buyer + supports both the platforms but indicates the choice of platform through RTB response on + a per-auction basis. + * Seller's ad service must maintain a service configuration with choice of platform + (on-device or B&A) of buyers. This would facilitate sending the `perBuyerSignals` + (as part of contextual signals) back to the browser only for on-device buyers. Similarly, + the seller's ad service can send the `perBuyerSignals` to B&A only for the buyers + who prefer bid generation in B&A. + +* Upon receiving the contextual ad request from the browser: + * Seller's ad service sends contextual / real-time-bidding (RTB) requests to buyers. + * _Note: Due to privacy protection, the seller's ad service cannot learn which_ + _buyers have interest groups from the B&A ciphertext._ + + * Sellers can kick off Protected Audience auctions only after the contextual signals (`perBuyerSignals`, `sellerSignals` and `auctionSignals`) + are available. The `perBuyerSignals` are received in RTB response from on-device buyers and B&A buyers. + * On receiving contextual / RTB responses from all buyers, sellers may conduct the contextual ad auction + in the seller's ad service. + + * **[Change required in seller's ad service, B&A]** Sellers should use [HTTP streaming][9] + to stream contextual signals back to the browser as soon as they are available. This + would help kick off on-device bid generation and scoring without being blocked on B&A auctions. + The structure of the server response requires some changes to maintain good parallelization behavior. + + Once contextual signals are available: + + * Seller's ad service generates a nonce based on [version 4 UUID][10] (128 bits). + + * Instead of returning the [`Ad-Auction-Result`][11] HTTP header, return an + `Ad-Auction-Result-Nonce` header passing the nonce. + + * Seller's ad service streams contextual signals (and contextual winner if available) + back to the browser in the HTTP response body back to the JavaScript Fetch initiated + by the browser with `AdAuctionHeaders: true`. + + * Seller's ad service calls B&A ([SellerFrontEnd service][12]) in parallel by sending + [SelectAdRequest][13] and includes a nonce in [SelectAdRequest.AuctionConfig][14]. + + * After Protected Audience auctions conclude in B&A, [SellerFrontEnd service][12] includes + the nonce in the _encrypted [AuctionResult][15] in [SelectAdResponse][16] (B&A response payload) + and returns to the seller's ad service. + + * Once an encrypted B&A response is returned, the seller's ad service can stream that + back to the browser in the HTTP response body. + + +##### On browser + +* **[Change required in browser]** Browser accepts `Ad-Auction-Result-Nonce` header. + +* **[Change required in browser]** Browser verifies it witnessed a Fetch request to the + seller’s origin with `AdAuctionHeaders: true` that included an `Ad-Auction-Result-Nonce` + response header with hash of encrypted B&A response. This would be done similar to how + [Ad-Auction-Result is verified][17]. + +* **[Change required in browser]** Once the encrypted B&A response is received by the browser, + the browser verifies the nonce included in the encrypted B&A response with what included in + the `Ad-Auction-Result-Nonce` header. + +* After Protected Audience auctions conclude on-device, the winner of on-device auction and the + winner of B&A auction would be compared to decide the final winner. + There are two options for the comparison: + * **[Supported]** On-device and B&A winners are fed into seller's scoreAd(). + * **[Not supported]** Alternately, browsers can do the comparison, i.e. compare score between + B&A winner and on-device winner; instead of feeding these into seller's scoreAd(). + +* _Note: No changes required to rendering or reporting flow for on-device auctions._ + + +#### Setting nonce in HTTP header + +_Refer to [Chrome and B&A integration explainer](https://github.com/WICG/turtledove/pull/1233/files) for + additional details around setting the nonce in HTTP header._ + +Some background around why HTTP streaming and nonce is required. + * Web browsers leverage [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) to ensure that + the encrypted B&A response payload is in fact coming from the seller, because TLS authenticates the + seller’s origin. Given that, B&A response must be sent back to the device, with [SHA-256](https://en.wikipedia.org/wiki/SHA-2) + hash of B&A response included in an HTTP response header (Ad-Auction-Result header). This is documented + in [Chrome and B&A integration explainer][11]. + + * However, the problem is that HTTP headers are available to the browser before the response body. This + implies that B&A response and header need to be available before contextual signals can be sent back to + the browser and that leads to higher latency since on-device auctions can not kick off until B&A auctions + conclude. + + This poses a problem in parallelizing B&A and on-device auctions to achieve overall lower end-to-end auction + latency. To solve this problem, [parallelization of on-device and server side auctions][26] is proposed. + + +#### B&A API updates for mixed mode nonce + +The following fields will be added to [SelectAdRequest.AuctionConfig][14] and [AuctionResult][15]. + +``` + +message AuctionConfig { + // Existing fields ..... + + string mixed_mode_nonce = 13; +} + +message AuctionResult { + // Existing fields ..... + + string mixed_mode_nonce = 24; +} + +``` + +### Sequential B&A auction and on-device auction + +This is an alternate proposal to [parallelization of on-device and server side auctions][26]. + +With mixed mode, if sellers do not want to depend on [HTTP streaming][24], they can call +B&A first and then send back contextual signals and B&A response together back in the browser. +In this case, on-device auctions would need to wait for B&A auctions to conclude. Therefore, +this would add an incremental auction latency. + +The flow for this proposal is the same as described for [parallelization of on-device and server side auctions][26]. +The difference being the following: +* Seller's ad server doesn't need to depend on HTTP streaming. +* Seller's ad server doesn't need to create a nonce. +* Seller's ad server calls B&A services after contextual signals are available. The request + payload doesn't include a nonce. +* Seller's ad server continues to depend on the existing [Ad-Auction-Result HTTP header][11] + supported by Chrome browser to include a [SHA-256](https://en.wikipedia.org/wiki/SHA-2) + hash of encrypted B&A response. +* Seller's ad server should send back contextual signals and the encrypted B&A response + together back to the browser. +* On-device auctions kick off only after contextual signals are sent back to the browser. + Therefore, on-device auctions wait until B&A auctions conclude. + + +## Multi-seller auctions mixed mode + +![Architecture diagram.](images/protected-audience-multiseller-mixed-mode.png) + +In case of a [device orchestrated component auction][5], each component level auction: +* can happen on-device only +* can happen in B&A only +* can happen in mixed mode : This is similar to the component seller conducting a + [single-seller auction ][18]in mixed mode. + There are two sub component auctions, one to support on-device buyers and the other to support B&A buyers. + +The choice for the above is up to the component seller conducting the component-level auction. + + +### Top-level scoring + +The two winning bids from on-device auction and B&A auction can directly feed into top-level scoreAd() +that happens on the browser. + +_Note: There is no additional work for sellers to support multi-seller auctions in mixed mode. +If a seller supports mixed mode for single-seller auction, that should be sufficient to support +a component-level auction in mixed mode._ + + +## Guide for sellers to support mixed mode + +* Work with partner buyers to understand the preference of on-device auctions and B&A auctions. + +* Support [single-seller mixed mode][18]. + * The recommended option for overall lower auction latency is [parallelization of on-device and server side auctions][26]. + Refer to the [timeline][19] of support. + * Sellers can start integrating with [sequential B&A auction and on-device auction][20]. Then incorporate + [HTTP streaming and with nonce support][24] as a latency optimization. + +* Support [multi-seller auctions mixed mode](#multi--seller-auctions-mixed-mode). + * There is no additional work required for the sellers to support component level auction in mixed mode. + + +## Timeline for support + +* [Sequential B&A auction and on-device auction][20]: Available + +* Web browsers and B&A services will support handling nonce: Targeting [March 2025][25] + * At this point, sellers will be able to support [parallelization of on-device and server side auctions][26]. + + +## Related Documents + +* [Protected Audience API][3] +* Bidding and Auction services [architecture][22] +* [Browser and B&A integration][21] + + +## Github issues + +Adtechs can file issues related to this publication on [Github][23]. + +[1]: https://github.com/chatterjee-priyanka +[2]: https://github.com/JensenPaul +[3]: https://github.com/WICG/turtledove/blob/main/FLEDGE.md +[4]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md +[5]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_multi_seller_auctions.md#device-orchestrated-component-auctions +[6]: https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-1-get-auction-blob-from-browser +[7]: https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-1-get-auction-blob-from-browser +[8]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding-auction-services-payload-optimization.md#payload-optimization-guide-for-sellers--ssps +[9]: https://en.wikipedia.org/wiki/HTTP_Live_Streaming +[10]: https://www.ietf.org/rfc/rfc4122.html +[11]: https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-3-get-response-blobs-to-browser +[12]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#sellerfrontend-service +[13]: https://github.com/privacysandbox/bidding-auction-servers/blob/6830804052778404bdf6758fbd165dd644e91539/api/bidding_auction_servers.proto#L567 +[14]: https://github.com/privacysandbox/bidding-auction-servers/blob/6830804052778404bdf6758fbd165dd644e91539/api/bidding_auction_servers.proto#L568 +[15]: https://github.com/privacysandbox/bidding-auction-servers/blob/6830804052778404bdf6758fbd165dd644e91539/api/bidding_auction_servers.proto#L259 +[16]: https://github.com/privacysandbox/bidding-auction-servers/blob/6830804052778404bdf6758fbd165dd644e91539/api/bidding_auction_servers.proto#L771 +[17]: https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md#step-4-complete-auction-in-browser +[18]: #single-seller-auctions-mixed-mode-architecture +[19]: #timeline-for-support +[20]: #sequential-BA-auction-and-on--device-auction +[21]: https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md +[22]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#high-level-design +[23]: https://github.com/WICG/protected-auction-services-discussion +[24]: #server-side-auctions +[25]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md#timelines +[26]: #recommended-parallelization-of-on-device-and-server-side-auctions diff --git a/public_cloud_tees.md b/public_cloud_tees.md new file mode 100644 index 0000000..c809525 --- /dev/null +++ b/public_cloud_tees.md @@ -0,0 +1,112 @@ + +_Note: the explainer is posted in the Protected Auction repository, but applies to all Privacy Sandbox services._ + +# Public Cloud TEE Requirements + +Privacy Sandbox has established several APIs that allow off-device processing of user data in order to provide important functionality for API users. For example, the Attribution Reporting API uses [Aggregation Service](https://github.com/WICG/attribution-reporting-api/blob/main/AGGREGATION_SERVICE_TEE.md) to aggregate data across multiple users. The Protected Audience API will use [Bidding & Auction Services](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md) to expand real-time bidding and ad auctions from the user’s device to a server, and access [Key/Value Services](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/key_value_service_trust_model.md) to provide real-time signals for ad auctions. + +Off-device processing must be done in a way that protects confidentiality of user data, and meets the privacy and security goals of Privacy Sandbox and the specific APIs. To meet such goals, Privacy Sandbox requires that processing be done in an attested and isolated environment (also referred to as Trusted Execution Environment, or TEE), using software provided and approved for this purpose by Privacy Sandbox. Other mechanisms to protect user data include adding noise to outputs. + +Privacy Sandbox started by supporting TEEs on Amazon Web Services (AWS) and Google Cloud Platform (GCP). These platforms were chosen for their security standards, availability of solutions, and because of feedback from ad techs. + +In this explainer, we focus on expansion to additional trusted compute environments offered by public Cloud Service Providers (CSPs). We propose criteria these environments must satisfy in order to be eligible for processing user data generated by Privacy Sandbox APIs. The technical and security criteria outlined in this document are aimed at protecting the user data handled by Privacy Sandbox APIs. Each adopter of a Privacy Sandbox API should continue evaluating which cloud solution best fits their needs. + +This explainer focuses on TEEs whose isolation and integrity guarantees are provided by a CSP. TEEs whose isolation and integrity properties come from other sources (such as private clouds or ad tech data centers) pose additional technical and security challenges, and are out of scope for this explainer and will be considered separately. + + +## **Overview of Cloud-based TEEs** + +Data is typically processed in TEEs as follows: + + + +1. A CSP offers a secure computing environment that satisfies Privacy Sandbox physical and logical security, isolation, confidentiality and attestation criteria. +2. An Operator that wishes to use Privacy Sandbox APIs (such as an ad tech company) creates instances of this secure compute environment, configured to run a specific workload binary (service) approved by Privacy Sandbox. +3. The workload requests Coordinators to provide credentials (such as cryptographic keys) that would allow it to process encrypted user data. The Coordinators decide whether to release such credentials after inspecting and verifying: claims about the security, integrity and confidentiality properties of the execution environment, the identity of the workload, and the identity of the Operator. + +## Criteria + +### Cloud-provided Trusted Execution Environment (Cloud TEE) + +A Cloud TEE must satisfy the following criteria to be eligible for running trusted Privacy Sandbox workloads[^1]: + + + +* The environment must be secure, private, and isolated. The CSP must implement controls to prevent all parties (including the workload Operator, and agents or employees of the CSP) from: + * Viewing, adding, removing or altering data while it is in use within the TEE (referred to as Data Confidentiality and Data Integrity) + * Adding, removing, or altering code executing in the TEE (referred to as Code Integrity) + * Otherwise tampering or interacting with the workload (except via APIs explicitly exposed by the workload) +* The environment must be remotely attestable. This means that the CSP must provide an attestation report[^2] with claims about the state of the TEE, including: + + + * Whether the workload meets the above requirements that the environment is secure, private, and isolated + * Cryptographic measurements of the workload and any security-relevant software components running in the TEE + * Identity of the operator + + The above claims must be exposed via a secure API, which is accessible to an external Relying Party (such as a Coordinator) when evaluating whether the instance can be trusted with credentials such as decryption keys. Additional information about the role of Coordinators is available in our service-specific explainers: [Protected Audience services](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/trusted_services_overview.md) and [Aggregation Service](https://github.com/WICG/attribution-reporting-api/blob/main/AGGREGATION_SERVICE_TEE.md). + +* The TEE must be commercially available to a wide range of customers, including academic and security researchers. +* The TEE must be able to run a Linux-based containerized workload with access to functionality such as networking, load balancing, and object storage. + +### Security and trust of the CSP + +CSPs need to employ a combination of technological (hardware and software) and operational measures to provide Cloud TEEs that meet the criteria. For example: + + + +* CSPs play a crucial role in procuring, hosting, maintaining and physically securing server hardware. +* CSPs are responsible for maintaining integrity and security of their internal systems, enforcing proper access controls (such as prevent unilateral access from CSP insiders and human operators), and maintaining isolation of workloads from each other, and from the underlying compute infrastructure +* CSPs ensure isolation of the confidential workload from the workload's operator, and provide secure infrastructure that enables remote attestation by third parties. + +Preferably, CSPs should employ technological measures that are externally verifiable (in concert with internal security measures and operational best practices) to provide protection of their TEEs. For example, CPU technologies such as AMD SEV-SNP and Intel TDX provide a layer of protection and assurance with a root of trust that is external to the CSP. + +We recognize that there are significant hurdles in having all measures externally verifiable (such as limited attestation support by processor-based TEEs, and confidentiality of code and supply chain). We will therefore validate that the CSP and the remote attestations are trustworthy using the following criteria. + + + +* Compliance with recognised standards on cloud security. In particular, [ISO 27001](https://www.iso.org/standard/27001), [ISO 27017](https://www.iso.org/standard/43757.html) and [ISO 27018](https://www.iso.org/standard/76559.html), outlining information security controls for cloud services. +* Certification from cloud security industry bodies. We plan to accept [Level 2 in Cloud Security Alliance’s STAR program](https://cloudsecurityalliance.org/), which requires a third-party audit of the CSP’s adherence to security frameworks. We are open to feedback on other established certification programs which provide comparable guarantees. +* The CSP makes a public commitment to uphold the security and privacy guarantees of their Cloud TEE solutions, including that the Cloud TEE meets the requirements described earlier and the CSP cannot or will not undermine the security of the data processed in environments under their control. In addition, an independent security assessment conducted by a reputable entity is preferred. +* Inclusion in an established global industry research report on public cloud offerings. We plan to accept the [Gartner Magic Quadrant for Strategic Cloud Platform Services report](https://www.gartner.com/en/documents/4970831?ref=null). Gartner established itself as a [trusted consulting firm](https://www.gartner.com/en/why-gartner) for technology, and conducts independent research into the offering by CSPs, including feedback from customers and size of the CSP business. Inclusion in their report indicates that the CSP has gained users’ trust, and has a major reputational stake in ensuring secure provision of cloud infrastructure. We are open to feedback on other established independent research reports which provide comparable guarantees. +* The CSP (including parent company) is headquartered in a supported country or region (currently including the EEA, UK, and US). + +### Updating requirements + +In a competitive marketplace, we expect CSPs to be proactively working on improving the security posture and transparency of their confidential computing offerings over time. As industry standards evolve, we may update the requirements. We may similarly update requirements as we gain more experience in reviewing Cloud TEE solutions. + +We'll provide advance notice if and when requirements are updated. + +## Review Process + +To initiate a review of a Cloud Service Provider, please file a GitHub issue in [this repository](https://github.com/WICG/protected-auction-services-discussion/issues/) using the template provided in this explainer, and provide all required evidence. We may also independently initiate review processes using the same process, and will document our previous decisions to approve AWS and GCP. + +We aim for the review process and discussion to be transparent. However, some discussions may be private to protect confidential information by CSPs and to avoid disclosing internal security practices. We welcome feedback from the ecosystem on any review request. + +If a CSP is approved, we will prioritize starting the work required to enable the CSP based on the benefits to the ecosystem, including demand from users of the Privacy Sandbox TEE services, the effort required to enable the CSP, and other priorities. Initial approval of a CSP is subject to review throughout the development process, as technical challenges may arise that require us to delay or block enabling the cloud. We will provide appropriate notice when we decide which CSPs to support, and the expected timeline (including any changes to the timeline as appropriate). + + +### Sunsetting support + +We will deprecate support for specific Cloud TEE solutions that no longer meet the security or privacy bar. Such deprecation may be a result of updated requirements or of actions (including lack of action) by the CSP which undermine the security of the data processed in environments under their control. + +We'll provide advance notice and support for migration if and when solutions are deprecated. + +### Template for CSP approval GitHub Issue + +Name and web address for the Cloud Service Provider: <> + +Short description of the Cloud TEE solution, including security properties, remote attestation and workload capabilities. Please include links to supporting documentation \*\*. <> + +Short description of security and trust of the CSP, including compliance with ISO standards, Certification from cloud security industry bodies (such as STAR Level 2), and inclusion in a research report on public cloud offerings (such as Gartner’s public cloud report) . Please include links to supporting documentation \*\*. + +(optional) Relevant factors for prioritization of review of the Cloud TEE solution and, if approved, enabling support. For example, demand for solutions \*\*: + +\*\* We understand some information may not be public, and we can accommodate alternative ways to provide supporting evidence. + + +[^1]: + The TEE criteria are aligned with industry definitions for TEE, such as those by the [Confidential Computing Consortium](https://confidentialcomputing.io/) + +[^2]: + + ‘Attestation report’ is the technical term used to describe the report provided by the CSP on the state of the workload and TEE. It is not related to [Privacy Sandbox enrollment attestation](https://github.com/privacysandbox/attestation). diff --git a/release_compatibility_bidding_auction_key_value_services.md b/release_compatibility_bidding_auction_key_value_services.md new file mode 100644 index 0000000..a771358 --- /dev/null +++ b/release_compatibility_bidding_auction_key_value_services.md @@ -0,0 +1,30 @@ +The table below outlines which releases of [Bidding & Auction services](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/bidding_auction_services_api.md) and [Key Value service](https://github.com/privacysandbox/protected-auction-key-value-service/tree/main) are compatible with each other. Please note that it may also include releases that are outside of the general support window (see [KV support windows](https://github.com/WICG/protected-auction-services-discussion/issues/66)). + + + + + + + + + + + + + + + + + + +
Bidding & Auction version + Key Value version +
not supported prior to 4.0 + not supported prior to 0.17.1 +
[4.0-4.1] + 0.17.1 +
>=4.2 + >=1.0 +
+ +Last updated: 2025-01-15 diff --git a/roma_bring_your_own_binary.md b/roma_bring_your_own_binary.md new file mode 100644 index 0000000..fe12f79 --- /dev/null +++ b/roma_bring_your_own_binary.md @@ -0,0 +1,164 @@ +> FLEDGE has been renamed to Protected Audience API. To learn more about the name change, see the +> [blog post](https://privacysandbox.com/intl/en_us/news/protected-audience-api-our-new-name-for-fledge) + +**Authors:**\ +[Shruti Agarwal](https://github.com/a-shruti), Google Privacy Sandbox\ +[Peter Meric](https://github.com/pmeric), Google Privacy Sandbox\ +[Brian Schneider](https://github.com/bjschnei), Google Privacy Sandbox\ +[Edward Gathuru](https://github.com/eggathuru), Google Privacy Sandbox + +# Roma Bring-Your-Own-Binary + +Roma is a C++ library used for executing untrusted code in a secure, isolated environment. + +Currently, Roma supports ad tech-defined functions implemented in JavaScript and Web Assembly. In +this document, we propose an extension to Roma to allow binary executables to be provided as UDFs. + +## Overview + +In Privacy Sandbox, Roma is a C++ library used by Protected Audience's trusted servers to securely +execute ad tech-developed functions, referred to as user-defined functions (UDFs), within a secure, +isolated environment. Central to this sandboxing is the requirement that the UDF execution handles +requests and associated data without any discernible side effects. + +Roma's current design uses [Sandbox2](https://developers.google.com/code-sandboxing/sandbox2) and V8 +as the execution engine, imposing certain limitations on ad techs that restrict them to the use of +JavaScript and WebAssembly (WASM). + +This explainer presents an expansion in Roma's functionality to execute self-contained binaries. + +## Why Bring-Your-Own-Binary? + +Bring-Your-Own-Binary can reduce costs to ad tech by doing the following: + +- **Reducing the barrier-to-entry** - less code to change and time to migrate. +- **Improving latency and reducing cost** - our internal benchmarks have shown compute intensive + UDFs implemented in C++ and Go have lower latency than equivalent ones written in WASM or JS. + +Roma Bring-Your-Own-Binary uses a single instance of a double-sandboxed Virtual Machine Monitor +(VMM) called [gVisor](https://gvisor.dev/). Inside this sandbox, privacy requirements are met +through per-process isolation. This involves calling `clone` with appropriate flags, creating +`pivot_root` for file system isolation, executing the supplied UDF, waiting for execution to +complete and then cleaning up the UDF process and `pivot_root`. + +## Usage + +This section describes how Roma Bring-Your-Own-Binary might be used. + +BYOB integration will be available through Trusted Execution Environment (TEE) based servers : +[Key/Value service](https://github.com/privacysandbox/protected-auction-services-docs/blob/69cef5a4ebb0b4f3077d93e94a9cd9bb56686c54/key_value_service_user_defined_functions.md), +[Bidding & Auction services](https://github.com/privacysandbox/protected-auction-services-docs/blob/ef04c4c6f8d788534130938750ae5573691a66dc/bidding_auction_services_api.md). +Browsers will continue to support JavaScript or WASM-based UDF execution based on +[V8](https://v8.dev/) engine, while TEE based servers depending on Roma for adtech's UDF execution +will continue to +[support v8](https://github.com/privacysandbox/data-plane-shared-libraries/tree/619fc5d4b6383422e54a3624d49a574e56313bc8/src/roma/sandbox/js_engine/v8_engine). + +To simplify UDF debugging and testing, the team plans to release an SDK tool to help the UDF +developer determine if their binary meets the UDF requirements regardless of their development +environment. + +A REPL CLI tool will allow the developer to run their UDF binary in a sandboxed setting in +production or debug modes, making it easier to test or debug. A microbenchmark CLI tool will also be +offered for benchmarking UDF binaries. + +The BYOB SDK for a given UDF contains the Protocol Buffer (protobuf) spec defining the messages +pertinent to the UDF input and output. This protobuf spec defines the expected input and output for +the binary. + +Protobuf messages are used for requests and responses. Communications with the UDF are facilitated +by protobuf messages transmitted using a file descriptor. + +The specification of the UDF communication protocol can be found in +[doc](https://github.com/privacysandbox/data-plane-shared-libraries/blob/619fc5d4b6383422e54a3624d49a574e56313bc8/docs/roma/byob/sdk/docs/udf/Communication%20Interface.md). +The following section offers an illustrative example. + +### Example + +The example below demonstrates how BYOB can be used. Assume you have been supplied with the +following proto as the specification for your UDF. + +```proto +// Echo UDF: An API which reads the request and echoes the response. +// The Echo UDF request. +// Request: EchoRequest +// Response: EchoResponse + +// Request message for the Echo function. +message EchoRequest { + // Message to be echoed. + bytes message = 1; +} + +// The Echo UDF response. +// Response message for the Echo function. +message EchoResponse { + // Response which would be the a copy of the message sent in EchoRequest. + bytes message = 1; +} +``` + +A UDF written in C++ might look like the following. + +```c +#include + +#include "google/protobuf/util/delimited_message_util.h" +#include "src/roma/byob/example/example.pb.h" + +using ::privacy_sandbox::server_common::byob::example::EchoRequest; +using ::privacy_sandbox::server_common::byob::example::EchoResponse; + +EchoRequest ReadRequestFromFd(int fd) { + EchoRequest req; + google::protobuf::io::FileInputStream stream(fd); + google::protobuf::util::ParseDelimitedFromZeroCopyStream(&req, &stream, + nullptr); + return req; +} + +void WriteResponseToFd(int fd, EchoResponse resp) { + google::protobuf::util::SerializeDelimitedToFileDescriptor(resp, fd); +} + +int main(int argc, char* argv[]) { + if (argc != 2) { + std::cerr << "Expecting exactly one argument"; + return -1; + } + int fd = std::stoi(argv[1]); + // Any initialization work can be done before this point. + // The following line will result in a blocking read being performed by the + // binary i.e. waiting for input before execution. + // The EchoRequest proto is defined by the Trusted Server team. The UDF reads + // request from the provided file descriptor. + EchoRequest request = ReadRequestFromFd(fd); + + EchoResponse response; + response.set_message(request.message()); + + // Once the UDF is done executing, it should write the response (EchoResponse + // in this case) to the provided file descriptor. + WriteResponseToFd(fd, std::move(response)); + return 0; +} +``` + +This C++ code should be compiled to a binary and provided to the server. + +For additional examples, refer to +[our code repository](https://github.com/privacysandbox/data-plane-shared-libraries/tree/619fc5d4b6383422e54a3624d49a574e56313bc8/src/roma/byob/example). + +Note that these examples are non-exhaustive. Self-contained executables are generally supported. For details see [doc](https://github.com/privacysandbox/data-plane-shared-libraries/blob/619fc5d4b6383422e54a3624d49a574e56313bc8/docs/roma/byob/sdk/docs/udf/Execution%20Environment%20and%20Interface.md). + +## BYOB availability + +Roma BYOB is open-sourced and the code can be found on +[GitHub](https://github.com/privacysandbox/data-plane-shared-libraries/tree/619fc5d4b6383422e54a3624d49a574e56313bc8/src/roma/byob). + +For details about the execution environment and communication protocol, check out +[documentation](https://github.com/privacysandbox/data-plane-shared-libraries/tree/619fc5d4b6383422e54a3624d49a574e56313bc8/docs/roma/byob/sdk/docs/udf). + +## Github issues + +For questions and bug reports, file an +[issue](https://github.com/WICG/protected-auction-services-discussion) on GitHub. diff --git a/trusted_services_overview.md b/trusted_services_overview.md index 580bf61..3862435 100644 --- a/trusted_services_overview.md +++ b/trusted_services_overview.md @@ -1,3 +1,13 @@ +> FLEDGE has been renamed to Protected Audience API. To learn more about the name change, see the [blog post](https://privacysandbox.com/intl/en_us/news/protected-audience-api-our-new-name-for-fledge) + +**Authors:**
+[Priyanka Chatterjee][1], Google Privacy Sandbox
+[Alexandra White][2], Google Chrome
+[Joe Lewis][3], Google Android
+Chanda Patel, Google
+[Peiwen Hu][4], Google Privacy Sandbox + + # Overview of FLEDGE Services The [Privacy Sandbox](https://privacysandbox.com) aims to develop technologies @@ -76,9 +86,11 @@ Some of the key properties of a TEE include: ### Cloud platform Services running in TEE should be deployed on a cloud platform that supports -necessary security features. **Details specific to cloud platform support +necessary security features. See the [Public Cloud TEE requirements explainer][9] for more details. + +**Details specific to cloud platform support will be published at a later date. We expect to support Amazon Web Services -(AWS), Google Cloud Platform (GCP), and other cloud providers in the future.** +(AWS), Google Cloud Platform (GCP), Azure, and other cloud providers in the future.** ## Privacy considerations @@ -145,13 +157,9 @@ The following entities are inherently without trust: * Service operators. * A client, such as an app or browser. - * For Android, the device must be attested before the client can make an - outgoing request to a FLEDGE service. The details for Android device - attestation will be published in a later document. - * Browsers are not required to attest when querying service APIs. As - such, there is a risk of the information served being visible to - entities beyond the browser. Service operators should be aware of - this when designing their application. + * _Note: We no longer expect to need any device attestation as protection for [Bidding & Auction Service][6]. + As the design has evolved, requests to the B&A Service now pass through the existing untrusted seller's ad service, which can perform + its own protections from invalid traffic. Also, Web Environment Integrity is an entirely unrelated effort._ ## System overview @@ -213,7 +221,7 @@ available to the service after it has been attested. #### Request decryption Upon receiving a request, the FLEDGE service checks the version of the public -key and either lookup corresponding private keys from its in-memory cache. +key and looks up corresponding private keys from its in-memory cache. The FLEDGE service decrypts the request using split private keys, processes the request and then returns an encrypted response back to the client. @@ -237,7 +245,7 @@ The FLEDGE service sends requests to the key management system to fetch private keys and public keys at service bootstrap. Before such keys are granted to the service, the binary hash of the FLEDGE service and guest operating system running on the virtual machine is validated against a hash -of the open source image; this validation process is termed as attestation. +of the open source image; this validation process is termed an attestation. * The FLEDGE service sends requests to private key hosting services to pre-fetch private keys. Private keys are granted to a FLEDGE service only @@ -271,8 +279,8 @@ A _key management system_ includes multiple services that are tasked with: * Provisioning keys to end-user devices. * Provisioning key pairs to real-time services. -In this proposal, two separate trusted parties operate key management -systems. +In this proposal, two separate trusted parties called **Coordinators** operate key +management systems. #### Key Management System A @@ -322,15 +330,20 @@ correspond to the version of the public key that is used for encryption. Public keys have a client side time-to-live (TTL) of N days. Corresponding private keys should have a TTL of at least N+1 days. +#### Adtech authentication by Coordinator + +When an adtech onboards to a [cloud platform](#cloud-platform) to [deploy](#deployment-by-adtechs) +and operate FLEDGE services, they need to be enrolled with Coordinator. Refer [here][7] for more details. + ## Initial plans for release and deployment ### Release by Google -* Developers author open source service code and Google releases source code - to an open source repository (GitHub). -* Google may also publish build artifacts to an open source repo in - [github.com/privacysandbox](https://github.com/privacysandbox) org for - binary validation. +* Developers author open source [Key Management Systems](#key-management-systems) + code and Google releases source code to an open source repository (GitHub). +* Google Privacy Sandbox is publishing [FLEDGE services](#fledge-services) source code and build + artifacts to an open source repo in [github.com/privacysandbox](https://github.com/privacysandbox) + org. Refer [here][8] for the Github code repo of different Privacy Sandbox services. ### Deployment by adtechs @@ -339,8 +352,14 @@ helper guides provided by Google. This includes running binaries of the service in the TEE setup specific to a cloud platform. Adtechs are responsible for the productionization of FLEDGE services operated by them. -Details regarding Key Management Systems deployment will be published at a -later date. +Adtechs will define and be responsible for the Sevice Level Objective (SLO) and Service +Level Agreement (SLA) of the system deployed by them. + +### Deployment by Coordinators + +Two different *Coordinators* will deploy [Key Management System A](#key-management-system-a) and +[Key Management System B](#key-management-system-b) on every [Cloud Platform](#cloud-platform) that +is supported. ## FLEDGE services @@ -363,7 +382,16 @@ infeasible to execute on user's devices. This could be due to system health considerations and ad latency constraints. The FLEDGE Bidding and Auction service executes ad bidding and auctions remotely in the TEE. -Refer to the -[Bidding and auction service API explainer](https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md) -for more information. There will be follow up documents describing the design of -ad bidding and auction services. +Refer to the [Bidding and auction services][6] explainer for more information. + +Refer to all Bidding and Auction explainers [here][5]. + +[1]: https://github.com/chatterjee-priyanka +[2]: https://github.com/heyawhite +[3]: https://github.com/sanbeiji +[4]: https://github.com/peiwenhu +[5]: https://github.com/privacysandbox/fledge-docs#bidding-and-auction-services +[6]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md +[7]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_api.md#enroll-with-coordinators +[8]: https://github.com/privacysandbox +[9]: https://github.com/privacysandbox/protected-auction-services-docs/blob/main/public_cloud_tees.md