Adding Stored Request Support (prebid#207)

* Added openrtb contract classes. * Normalized the documentation. * Ran gofmt. * fixed some more comments. * Added CacheURL to the openrtb types. * Bidder interface. * First pass at appnexus adapter implementation. * Limited the Appnexus adapter to only handle banner and video requests. * Made a first pass at Bidder interfaces. * Removed an unused class. * First pass at openrtb BidRequest cleaning * Added a return so that the code builds. * Renamed adapter.go to bidder.go * Changed appnexus adapter to implement the SingleHttpBidder. * Changed the Bidder type signature slightly. * Pass 2 of BidRequest cleaner * Updated API contract classes, and refactored the singleBidderAdapter so that it compiles again. * Wrote an http bidder which should work for everyone. * Replaced the old base bidder with one which works for both bidding strategies. * Updated the appnexus adapter so that it still implements the interface. * Updated documentation. * Added tests to http_bidder.go, and fixed some bugs. * Exchange probably compiles, but is not done * Swapped the order of the HoldAuction arguments. * Added an openrtb endpoint. Made an interface for Exchange. * End of the week commit. Should be in a relatively sane state * Fixed some build errors. * Fixed a bug in NewExchange. * Fixed some more bugs in the exchange. * Added an openrtb2/auction endpoint for end-to-end tests. * First exchange that can (theoretically) produce a bid response * Fixes for <adapter>.Bid() returning a null pointer * Fixes for <adapter>.Bid() null fix: keep existing debug info * Added input validation for incoming openrtb requests. * Fixes for <adapter>.Bid() : actually save debugging info now * Some input validation cleanup. * Added a test for a successful bid. * Fixed some more bugs. The auction with the appnexus adapter now works from end-to-end. * Made a BidderName enum type, and ported some method signatures to use it. * Removed silly newlines. * Removed some more dead code. * Starting to add exchange tests * Added lots of unit tests for input validation. * Renamed ServerCalls to HttpCalls everywhere. * pre-merge commit * Tests and fixes * Connected bidder param request validation to the json schemas * Fixed some bugs. * Undid some changes to the facebook json schema. * Updated glide.lock. Removed more unexpected changes. * Cleaned up the endpoint_test file. * Added more tests. Improved the endpoint class role to avoid glog.Fatals. * Added some tests for the bidder json-schema validation. * Improved handling of GetAllBids go routine and more tests * Added a benchmark for the openrtb endpoint. * Renamed the openrtb_auction package to openrtb2. * Added a markdown file describing the openrtb2 endpoint. * More tests in, including a null response from an adapter as their bid response. * Add proper error handling of the json Marshal errors to the exchange code. * Implemented the timeout. * Return a 500 if the exchange fails with an error. * Added lots of tests. More to come soon. * More appnexus tests. * Removed some of the TODOs. * Added tests for appnexus param validation * Added another test where unit coverage was spotty. * Deleted some unused code, including the Cache stuff. * Removed some dead comments. * Removed some more TODOs * Added a doc.go to the openrtb_ext package. * Refactoring and cleanup based on suggestions from Nicole. * Moved the appnexus imp.ext into its own file. * Renamed exchange.bidder to exchange.adaptedBidder. * Fixed more docs, and made more things private. * Replaced the response.seatbid.ext map with the contract class. * Added support for Targeting. Adding tests to CleanOpenRTBRequests, work needed here still. * Revert "Added support for Targeting. Adding tests to CleanOpenRTBRequests, work needed here still." This reverts commit f2efcca. * Adding tests to insure proper extensions get passed to adapters during the request cleaning stage. * Fixed docs based on the documentation writer's feedback * Fixed some spaces => tabs formatting. * Fixed go vet errors. * Added docs for adding a new bidder. * Linked the README to the adding a new bidder doc. * Relocated utils.go into exchange, and made the functions private. * Privitized internal exchange functions. * Code review improvements. * A few small fixes from PR comments * Optimizations in cleanOpenRTBRequests * Fixed a bug which resulted in empty seatbid.ext objects. * Starting a branch for processing stored configs. * Fixed some bugs in the contract classes. * Added some more docs, and updated the method signature to include errors. * Sync for auction config cache code * Made a ConfigFetcher which pulls from the filesystem. * Added an empty fetcher which doesn't support configs. Updated the auction_tests to compile again * Return an address. * Updated the glide.lock with the new json-patch dependency. * Made the openrtb-config backend configurable. * Added the openrtb2_config flies to the Dockerfile. * Moved and renamed some stuff to keep it separate from legacy stuff. * Added a postgres fetcher, and made the config fetcher configurable. * Basic config support and tests (working) * Moved some docs around. * Improved two of the unit tests. * Added a context to the config fetchers. * Fixed some minor bugs. Added glogs for config loading info. * Improved the error message for postgres config logging. * Made the APIs consistent with the spec. * Added a few tests, and logged unexpected DB connection errors. Also some doc/style updates. * Added docs for configuration and Stored Requests. * Fixed a naming issue. * Change request processing to hang on to the raw JSON request, so we can merge against the raw request rather than after it gets mangled by the json (Un)Marshal cycle. * Merged upstream changes and added default for max_request_size * Finish reading the request body & return an error message if it exceeded the limit. Made some small performance improvements. * Renamed all "config"-stuff to "stored requests". * Removed some unused contract classes in preference of jsonparser. * Missed a few renames. Optimized request ID finding. * More missed renames. * Removed all traces of account-scoped fetchers, which we dont need to support yet. * Replaced magic maxSize number with a constant. * Reformatted ugly json. Checked if cfg was nil. Renamed NewEagerFetcher to NewFileFetcher. * Lower-cased some HTTP request in the docs, and linked the first mention of Imp to the OpenRTB spec to help clarify things. * Fixed a bug which caused the server to return a 400 when the incoming request was the same as the max configured size.
openx · Dec 6, 2017 · f2e994c · f2e994c
1 parent 7e089b6
commit f2e994c
Show file tree

Hide file tree

Showing 27 changed files with 1,353 additions and 47 deletions.
diff --git a/Dockerfile b/Dockerfile
@@ -2,6 +2,7 @@ FROM alpine
 MAINTAINER Brian O'Kelley <[email protected]>
 ADD prebid-server prebid-server
 COPY static static/
+COPY stored_requests/data stored_requests/data
 EXPOSE 8000
 ENTRYPOINT ["/prebid-server"]
 CMD ["-v", "1", "-logtostderr"]
diff --git a/cache/cache.go → cache/legacy.go b/cache/cache.go → cache/legacy.go
@@ -1,6 +1,8 @@
 package cache
 
-import "github.com/prebid/prebid-server/pbs/buckets"
+import (
+	"github.com/prebid/prebid-server/pbs/buckets"
+)
 
 type Domain struct {
 	Domain string `json:"domain"`

diff --git a/config/config.go b/config/config.go
@@ -1,8 +1,11 @@
 package config
 
 import (
+	"bytes"
+	"errors"
 	"fmt"
 	"github.com/spf13/viper"
+	"strconv"
 	"strings"
 )
 
@@ -18,7 +21,13 @@ type Configuration struct {
 	HostCookie      HostCookie         `mapstructure:"host_cookie"`
 	Metrics         Metrics            `mapstructure:"metrics"`
 	DataCache       DataCache          `mapstructure:"datacache"`
+	StoredRequests  StoredRequests     `mapstructure:"stored_requests"`
 	Adapters        map[string]Adapter `mapstructure:"adapters"`
+	MaxRequestSize  int64              `mapstructure:"max_request_size"`
+}
+
+func (cfg *Configuration) validate() error {
+	return cfg.StoredRequests.validate()
 }
 
 type HostCookie struct {
@@ -59,6 +68,61 @@ type DataCache struct {
 	TTLSeconds int    `mapstructure:"ttl_seconds"`
 }
 
+// StoredRequests configures the backend used to store requests on the server.
+type StoredRequests struct {
+	// Files should be true if Stored Requests should be loaded from the filesystem.
+	Files bool `mapstructure:"filesystem"`
+	// Postgres should be non-nil if Stored Requests should be loaded from a Postgres database.
+	Postgres *PostgresConfig `mapstructure:"postgres"`
+}
+
+func (cfg *StoredRequests) validate() error {
+	if cfg.Files && cfg.Postgres != nil {
+		return errors.New("Only one backend from {filesystem, postgres} can be used at the same time.")
+	}
+
+	return nil
+}
+
+// PostgresConfig configures the Postgres connection for Stored Requests
+type PostgresConfig struct {
+	Database string `mapstructure:"dbname"`
+	Host     string `mapstructure:"host"`
+	Port     int    `mapstructure:"port"`
+	Username string `mapstructure:"user"`
+	Password string `mapstructure:"password"`
+
+	// QueryTemplate is the Postgres Query which can be used to fetch configs from the database.
+	// It is a Template, rather than a full Query, because a single HTTP request may reference multiple Stored Requests.
+	//
+	// In the simplest case, this could be something like:
+	//   SELECT id, requestData FROM stored_requests WHERE id in %ID_LIST%
+	//
+	// The MakeQuery function will transform this query into:
+	//   SELECT id, requestData FROM stored_requests WHERE id in ($1, $2, $3, ...)
+	//
+	// ... where the number of "$x" args depends on how many IDs are nested within the HTTP request.
+	QueryTemplate string `mapstructure:"query"`
+}
+
+// MakeQuery gets a stored-request-fetching query which can be used to fetch numRequests requests at once.
+func (cfg *PostgresConfig) MakeQuery(numRequests int) (string, error) {
+	if numRequests < 1 {
+		return "", fmt.Errorf("can't generate query to fetch %d stored requests", numRequests)
+	}
+	final := bytes.NewBuffer(make([]byte, 0, 2+4*numRequests))
+	final.WriteString("(")
+	for i := 1; i < numRequests; i++ {
+		final.WriteString("$")
+		final.WriteString(strconv.Itoa(i))
+		final.WriteString(", ")
+	}
+	final.WriteString("$")
+	final.WriteString(strconv.Itoa(numRequests))
+	final.WriteString(")")
+	return strings.Replace(cfg.QueryTemplate, "%ID_LIST%", final.String(), 1), nil
+}
+
 type Cache struct {
 	Scheme string `mapstructure:"scheme"`
 	Host   string `mapstructure:"host"`
@@ -76,7 +140,7 @@ func New() (*Configuration, error) {
 	if err := viper.Unmarshal(&c); err != nil {
 		return nil, err
 	}
-	return &c, nil
+	return &c, c.validate()
 }
 
 //Allows for protocol relative URL if scheme is empty

diff --git a/config/config_test.go b/config/config_test.go
@@ -1,10 +1,9 @@
-package config_test
+package config
 
 import (
 	"bytes"
 	"testing"
 
-	"github.com/prebid/prebid-server/config"
 	"github.com/spf13/viper"
 )
 
@@ -27,7 +26,7 @@ func init() {
 
 func TestDefaults(t *testing.T) {
 
-	cfg, err := config.New()
+	cfg, err := New()
 	if err != nil {
 		t.Error(err.Error())
 	}
@@ -115,7 +114,7 @@ func cmpInts(t *testing.T, key string, a int, b int) {
 func TestFullConfig(t *testing.T) {
 	viper.SetConfigType("yaml")
 	viper.ReadConfig(bytes.NewBuffer(fullConfig))
-	cfg, err := config.New()
+	cfg, err := New()
 	if err != nil {
 		t.Fatal(err.Error())
 	}
@@ -158,3 +157,58 @@ func TestFullConfig(t *testing.T) {
 	cmpStrings(t, "adapters.facebook.usersync_url", cfg.Adapters["facebook"].UserSyncURL, "http://facebook.com/ortb/prebid-s2s")
 	cmpStrings(t, "adapters.facebook.platform_id", cfg.Adapters["facebook"].PlatformID, "abcdefgh1234")
 }
+
+func TestValidConfig(t *testing.T) {
+	cfg := Configuration{
+		StoredRequests: StoredRequests{
+			Files: true,
+		},
+	}
+
+	if err := cfg.validate(); err != nil {
+		t.Errorf("OpenRTB filesystem config should work. %v", err)
+	}
+}
+
+func TestInvalidStoredRequestsConfig(t *testing.T) {
+	cfg := Configuration{
+		StoredRequests: StoredRequests{
+			Files:    true,
+			Postgres: &PostgresConfig{},
+		},
+	}
+
+	if err := cfg.validate(); err == nil {
+		t.Error("OpenRTB Configs should not be allowed from both files and postgres.")
+	}
+}
+
+func TestQueryMaker(t *testing.T) {
+	cfg := PostgresConfig{
+		QueryTemplate: "SELECT id, config FROM table WHERE id in %ID_LIST%",
+	}
+	madeQuery, err := cfg.MakeQuery(3)
+	if err != nil {
+		t.Errorf("Unexpected error making query: %v", err)
+	}
+	if madeQuery != "SELECT id, config FROM table WHERE id in ($1, $2, $3)" {
+		t.Errorf(`Final query was not as expeted. Got "%s"`, madeQuery)
+	}
+
+	madeQuery, err = cfg.MakeQuery(11)
+	if err != nil {
+		t.Errorf("Unexpected error making query: %v", err)
+	}
+	if madeQuery != "SELECT id, config FROM table WHERE id in ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)" {
+		t.Errorf(`Final query was not as expeted. Got "%s"`, madeQuery)
+	}
+}
+
+func TestQueryMakerInvalid(t *testing.T) {
+	cfg := PostgresConfig{
+		QueryTemplate: "SELECT id, config FROM table WHERE id in %ID_LIST%",
+	}
+	if _, err := cfg.MakeQuery(0); err == nil {
+		t.Errorf("MakeQuery function should return an error if given no IDs.")
+	}
+}
diff --git a/docs/developers/configuration.md b/docs/developers/configuration.md
@@ -0,0 +1,11 @@
+# Configuration
+
+Configuration is handled by [Viper](https://github.com/spf13/viper), which supports [many ways](https://github.com/spf13/viper#why-viper) of setting config values.
+
+As a general rule, Prebid Server will log its resolved config values on startup and exit immediately if they're not valid.
+
+For development, it's easiest to define your config inside a `pbs.yaml` file in the project root.
+
+## Available options
+
+For now, see [the contract classes](../../config/config.go) in the code.
diff --git a/docs/developers/stored-requests.md b/docs/developers/stored-requests.md
@@ -0,0 +1,138 @@
+# Stored Requests
+
+This document gives a technical overview of the Stored Requests feature.
+
+Docs outlining the motivation and uses will be added sometime in the future.
+
+## Quickstart
+
+Configure your server to read stored requests from the filesystem:
+
+```yaml
+stored_requests:
+  filesystem: true
+```
+
+Choose an ID to reference your stored request data. Throughout this doc, replace {id} with the ID you've chosen.
+
+Add the file `stored_requests/data/by_id/{id}.json` and populate it with some [Imp](https://www.iab.com/wp-content/uploads/2016/03/OpenRTB-API-Specification-Version-2-5-FINAL.pdf#page=17) data.
+
+```json
+{
+  "id": "test-imp-id",
+  "banner": {
+    "format": [
+      {
+        "w": 300,
+        "h": 250
+      },
+      {
+        "w": 300,
+        "h": 600
+      }
+    ]
+  },
+  "ext": {
+    "appnexus": {
+      "placementId": 10433394
+    }
+  }
+}
+```
+
+Start your server.
+
+```bash
+go build .
+./prebid-server
+```
+
+And then `POST` to [`/openrtb2/auction`](../endpoints/openrtb2/auction.md) with your chosen ID.
+
+```json
+{
+  "id": "test-request-id",
+  "imp": [
+    {
+      "ext": {
+        "prebid": {
+          "storedrequest": {
+            "id": "{id}"
+          }
+        }
+      }
+    }
+  ]
+}
+```
+
+The auction will occur as if the HTTP request had included the content from `stored_requests/data/by_id/{id}.json` instead.
+
+## Partially Stored Requests
+
+You can also store _part_ of the Imp on the server. For example:
+
+```json
+{
+  "banner": {
+    "format": [
+      {
+        "w": 300,
+        "h": 250
+      },
+      {
+        "w": 300,
+        "h": 600
+      }
+    ]
+  },
+  "ext": {
+    "appnexus": {
+      "placementId": 10433394
+    }
+  }
+}
+```
+
+This is not _fully_ legal OpenRTB `imp` data, since it lacks an `id`.
+
+However, incoming HTTP requests can fill in the missing data to complete the OpenRTB request:
+
+```json
+{
+    "id": "test-request-id",
+    "imp": [
+      {
+        "id": "test-imp-id",
+        "ext": {
+          "prebid": {
+            "storedrequest": {
+              "id": "{id}"
+            }
+          }
+        }
+      }
+    ]
+  }
+```
+
+If the Stored Request and the HTTP request have conflicting properties,
+they will be resolved with a [JSON Merge Patch](https://tools.ietf.org/html/rfc7386).
+HTTP request properties will overwrite the Stored Request ones.
+
+## Alternate backends
+
+Stored Requests do not need to be saved to files. [Other backends](../../openrtb2_config/) are supported
+with different [configuration options](configuration.yaml). For example:
+
+```yaml
+stored_requests:
+  postgres:
+    host: localhost
+    port: 5432
+    user: db-username
+    dbname: database-name
+    query: SELECT id, requestData FROM stored_requests WHERE id IN %ID_LIST%;
+```
+
+If you need support for a backend that you don't see, please [contribute it](contributing.md).
diff --git a/docs/endpoints/openrtb2/auction.md b/docs/endpoints/openrtb2/auction.md
@@ -191,6 +191,20 @@ PBS requests new syncs by returning the `response.ext.usersync.{bidderName}.sync
 This contains info about every request and response sent by the bidder to its server.
 It is only returned on `test` bids for performance reasons, but may be useful during debugging.
 
+#### Stored Requests
+
+`request.imp[i].ext.prebid.storedrequest` incorporates a [Stored Request](../../developers/stored-requests.md) from the server.
+
+A typical `storedrequest` value looks like this:
+
+```
+{
+  "id": "some-id"
+}
+```
+
+For more information, see the docs for [Stored Requests](../../developers/stored-requests.md).
+
 ### OpenRTB Differences
 
 This section describes the ways in which Prebid Server **breaks** the OpenRTB spec.