From 06df14c481b45e269ab5dba0a8bad0fe051807c1 Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Tue, 12 Sep 2023 15:12:23 +0100 Subject: [PATCH 01/20] docs(broker): Describe what Broker does and how to interact with it --- README.md | 7 +- .../openactive-broker-microservice/README.md | 273 +++++++++++++++++- .../src/util/opportunity-id-cache.js | 15 +- .../src/util/pause-resume.js | 17 +- .../openactive-integration-tests/README.md | 5 +- 5 files changed, 298 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 3c3fe060a1..2db23553cb 100644 --- a/README.md +++ b/README.md @@ -31,13 +31,16 @@ Note that the above command only runs the "core" tests within the test suite, wh The hosted OpenActive Reference Implementation is running on a basic developer tier Azure instance with a burst quota, so it will not handle the load of a test suite run for all tests (hence `npm start -- core`); if the hosted application shuts down, simply wait 5 minutes and try again. ## Configuration + In order to run the test suite against your own implementation, configure the test suite by creating a copy of [`config/default.json`](./config/default.json) named `config/{NODE_ENV}.json` (where `{NODE_ENV}` is the value of your `NODE_ENV` environment variable), including the following properties: - [`broker` microservice configuration](./packages/openactive-broker-microservice/#configuration-for-broker-within-confignode_envjson) - [`integrationTests` and `sellers` configuration](./packages/openactive-integration-tests/#configuration-for-integrationtests-within-confignode_envjson) -The test suite uses the file `config/{NODE_ENV}.json` to override the settings in `default.json`. It is recommended that for development and deployment a such a new file is created instead of making changes to the `default.json` file, so that any new required settings that are added in future versions can be automatically updated in `default.json`. +The test suite uses the file `config/{NODE_ENV}.json` to override the settings in `default.json`. For development and deployment create a new file instead of making changes to the `default.json` file, so that any new required settings that are added in future versions can be automatically updated in `default.json`. + +For more information about this use of `NODE_ENV` see this [documentation](https://github.com/lorenwest/node-config/wiki/Environment-Variables#node_env). -For more information see this [documentation](https://github.com/lorenwest/node-config/wiki/Environment-Variables#node_env). +By convention, much of the documentation assumes you to have created a `config/dev.json` file, which Test Suite will use when env var `NODE_ENV=dev`. But you can use any name you like, and have multiple configuration files for different environments. ## Installation diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index 44a33dfb1a..0ffc328fda 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -1,14 +1,20 @@ # openactive-broker-microservice -This Node.js microservice provides a service to harvest data and proxy calls to an Open Booking API +This Node.js microservice provides a service to harvest data and proxy calls to an Open Booking API. + +It needs to be running, against an Open Booking API implementation, in order for the [openactive-integration-tests](../openactive-integration-tests/) to run. + +TODO perhaps a little more summary about what it does? ## Usage in separate terminal windows To run `openactive-broker-microservice` in separate terminal window to `openactive-integration-tests`, from repository root: -1. `npm install` -2. `export NODE_ENV=dev` -3. `npm run start-broker` +1. Install dependencies: `npm install` +2. Specify Configuration file: `export NODE_ENV=dev` + * This is the command to use if using the `dev.json` config file, which is the default behaviour. See [Configuration](../../README.md#configuration) for more details) +3. Start Broker: `npm run start-broker` +4. Run [openactive-integration-tests](../openactive-integration-tests/) in another terminal window ## Configuration for `broker` within `./config/{NODE_ENV}.json` @@ -16,7 +22,7 @@ The `broker` object within `./config/{NODE_ENV}.json` file of the repository con ### `datasetSiteUrl` -The URL of the dataset site of the booking system under test. This dataset site is used to configure the test suite. +The URL of the dataset site of the Booking System under test. This dataset site is used to configure the test suite. ```json "datasetSiteUrl": "https://reference-implementation.openactive.io/openactive", @@ -154,6 +160,14 @@ While debugging authentication it can be useful to log the configuration that th "logAuthConfig": true, ``` +### `headlessAuth` + +For debugging authentication, it can be useful to perform the browser automation with a different setting for [Puppeteer's `headless` option](https://github.com/puppeteer/puppeteer#default-runtime-settings). By default, this is set to `true`. + +```json + "headlessAuth": true, +``` + ### `bookingPartners` Config for [Booking Partners](https://openactive.io/open-booking-api/EditorsDraft/1.0CR3/#dfn-booking-partner) that the Test Suite will use to connect to your Booking System. @@ -294,3 +308,252 @@ Clients credentials can be defined as follows: "secondary": null } ``` + +## What Broker Microservice does + +TODO2: + +* Parses Dataset Site +* RPDE feed harvesting + validation +* Sets up auth + +## Broker Microservice API + +Broker Microservice exposes an API which is used by the [Integration Tests](../openactive-integration-tests/) to TODO2 + +### User-facing endpoints + +#### `GET /` + +An HTML home page which shows links to other user-facing endpoints + +#### `GET /status` + +A JSON object, which shows the status of the Broker Microservice. This is used by a user to check on the progress of Broker and to help diagnose any potential issues. An annotated example response: + +```json +{ + // How long Broker has been running for + "elapsedTime": "2:58", + // Is Broker currently `harvesting` or is it `paused`? + "harvestingStatus": "harvesting", + // Status about each RPDE feed that Broker is harvesting + "feeds": { + "ScheduledSession": { + // The last page that Broker harvested from the ScheduledSession feed + "currentPage": "https://acme-fitness.org/api/feeds/scheduled-sessions?afterChangeNumber=3710544489", + // The total number of pages that Broker has harvested from the ScheduledSession feed so far + "pages": 8, + // The total number of items that Broker has harvested from the ScheduledSession feed so far + "items": 3992, + /* The reponse times, in milliseconds, of the last 5 pages that Broker + harvested from the ScheduledSession feed */ + "responseTimes": [ + 476.496187210083, + 303.6146593093872, + 313.0219888687134, + 294.4364585876465, + 359.7757930755615 + ], + // Number of items which are queued up for Data Model Validation + "totalItemsQueuedForValidation": 8, + // Number of items which have been validated by Data Model Validation + "validatedItems": 1978, + /* This feed is in `sleepMode` if the last page has been reached, at which point the feed + is polled at a slower rate until more data appears in the feed. */ + "sleepMode": true, + // Time taken for Broker to completely harvest this feed + "timeToHarvestCompletion": "0:07" + }, + "SessionSeries": { + "currentPage": "https://acme-fitness.org/api/feeds/session-series?afterChangeNumber=3662021037", + // ... + }, + "FacilityUse": { /* ... */ }, + "Slot": { /* ... */ }, + /* The Orders feed for Booking Partner `primary`. This is the Booking Partner configured + in the `bookingPartners` config object with the key `primary`. */ + "OrdersFeed (auth:primary)": { + "currentPage": "https://acme-fitness.org/api/booking/orders-rpde?afterChangeNumber=23504", + // ... + }, + // The OrderProposals feed for the same Booking Partner, `primary`. + "OrderProposalsFeed (auth:primary)": { + "currentPage": "https://acme-fitness.org/api/booking/order-proposals-rpde?afterChangeNumber=23504", + // ... + }, + // The Orders and OrderProposals feeds for Booking Partner `secondary` + "OrdersFeed (auth:secondary)": { /* ... */ }, + "OrderProposalsFeed (auth:secondary)": { /* ... */ }, + }, + "orphans": { + /* Tracks the number of items which have been harvested from the ScheduledSession + or Slot feeds, where their corresponding parent SessionSeries or FacilityUse + has not (yet) been harvested. + If all feeds have been harvested and there are orphans, this indicates that + there may be a problem with the Booking System's data. */ + "children": "0 of 13042 (0.00%)" + }, + "totalOpportunitiesHarvested": 13042, + // A summarised view of the numbers of Opportunities in the Buckets + "buckets": { + "TestOpportunityBookable": { + "OpenBookingSimpleFlow": { + "ScheduledSession": { + /* There are 129 ScheduledSessions with this Seller ID that support + Simple Booking Flow and satisfy the criteria `TestOpportunityBookable` */ + "https://acme-fitness.org/api/sellers/1": 129, + "https://acme-fitness.org/api/sellers/2": 27 + }, + "FacilityUseSlot": { + // ... + }, + "IndividualFacilityUseSlot": { + // ... + } + }, + "OpenBookingApprovalFlow": { + // ... + } + }, + "TestOpportunityBookableNonFreeTaxNet": { + "OpenBookingSimpleFlow": { + "ScheduledSession": { + /* Unlike with TestOpporunityBookable, there were no ScheduledSessions + that matched this criteria, `TestOpportunityBookableNonFreeTaxNet`, so + instead of showing Opportunity numbers for each Seller, the status + endpoint shows a summary of why no Opportunities were found. */ + "criteriaErrors": { + /* There are 892 ScheduledSessions which support Simple Booking Flow + and that do not satisfy the `TestOpportunityBookableNonFreeTaxNet` + criteria because they fail this contraint. That is to say that they + do not have seller tax mode TaxNet. */ + "Seller Tax Mode Net": 892, + "Must not require additional details": 88, + "startDate must be 2hrs in advance for random tests to use": 236, + "Only non-free bookable Offers": 496, + // ... + } + }, + "FacilityUseSlot": { + // ... + }, + "IndividualFacilityUseSlot": { + // ... + } + }, + "OpenBookingApprovalFlow": { + // ... + } + }, + // ... so on for every other criteria ... + } +} +``` + +#### `GET /validation-errors` + +An HTML web page which shows any validation errors that have been found for Opportunities that have been harvested from the Booking System. + +### Internal endpoints + +Endpoints used by Tests TODO2 + +#### `GET /health-check` + +Returns a response when the [Initial Harvest](#initial-harvest) is complete. This is used by the [Integration Tests](../openactive-integration-tests/) to check that Broker Microservice is up to date with the Booking System's data. + +This endpoint starts by setting Broker Microservice's [Harvesting Status](#harvesting-status) to `resumed`. So, if it was `paused` before, it will now resume harvesting. + +#### `POST /pause` + +TODO2 use format for POST data interface + +Set Broker Microservice's [Harvesting Status](#harvesting-status) to `paused`. + +This is called by the [Integration Tests](../openactive-integration-tests/) when they have finished running. This reduces the load on the [Booking System](#booking-system-under-test) and on the user's machine in between runs of Test Suite. + +#### `GET /config` + +A JSON object, which [Integration Tests](../openactive-integration-tests/) uses to configure itself. It contains: + +* Broker Microservice config +* Derived Broker Microservice config e.g. `bookingApiBaseUrl` is parsed from the [Dataset Site JSON](https://openactive.io/dataset-api-discovery/EditorsDraft/#embedded-json) that was loaded from the Dataset Site defined by the [`datasetSiteUrl` configuration property](#datasetsiteurl). +* Config from the `OpenActiveTestAuthKeyManager` that Broker Microservice sets up. See [openactive-openid-test-client](../openactive-openid-test-client/) for more info. + +Here is an annotated example: + +```json +{ + // When Broker Microservice started its Initial Harvest + "harvestStartTime": "2023-09-12T10:03:28.452Z", + // The base URL of the Booking System's Open Booking API interface + "bookingApiBaseUrl": "https://acme-fitness.org/api/booking", + // Derived from the `headlessAuth` config property. + "headlessAuth": true, + /* The rest of the config comes straight from calling `OpenActiveTestAuthKeyManager`'s + `config` property */ + "sellersConfig": { + "primary": { + // ... + }, + "secondary": { + // ... + } + }, + "bookingPartnersConfig": { + // ... + }, + "authenticationFailure": false, + "dynamicRegistrationFailure": false, +} +``` + +#### `GET /dataset-site` + +A JSON object, which contains the [Dataset Site JSON](https://openactive.io/dataset-api-discovery/EditorsDraft/#embedded-json) that was loaded from the Dataset Site defined by the [`datasetSiteUrl` configuration property](#datasetsiteurl). + +Here is an example: + +```json +{ + "@context": [ + "https://schema.org/", + "https://openactive.io/" + ], + "@type": "Dataset", + "@id": "https://acme-fitness.org/api/", + "url": "https://acme-fitness.org/api/", + "name": "Acme Fitness", + "accessService": { + "@type": "WebAPI", + "name": "Open Booking API", + // ...etc +``` + +## Concepts + +### Booking System under Test + +TODO2 + +### Buckets + +A **Bucket** is a cache of Opportunity IDs that match a given Opportunity Criteria (TODO link to Opportunity Criteria description in packages/test-interface-criteria/README.md). When Integration Tests sends a request to get a random Opportunity matching a given criteria, the Opportunity is fetched from these buckets. + +### Initial Harvest + +Broker Microservice starts by performing an **Initial Harvest** of the [Booking System](#booking-system-under-test)'s RPDE feeds. It rapidly pages through all the feeds, caching the data that it finds into [Buckets](#buckets) until it reaches the last page of each feed. + +When this Initial Harvest is complete, Broker Microservice's caches are up to date with all of the Booking System's data, and so it can be considered ready for various uses, including running [Integration Tests](../openactive-integration-tests/). A user can manually check whether or not the Initial Harvest is complete by calling the [Health Check endpoint](#get-health-check). + +Broker Microservice will continue to poll the feeds after the Initial Harvest and harvest any new data that it finds. This is essential, as many [Integration Tests](../openactive-integration-tests/) require data to be created in the Booking System after the tests have started. + +### Harvesting Status + +Broker Microservice can have one of two **Harvesting Statuses**: + +* `harvesting`: Broker Microservice is harvesting or polling the [Booking System](#booking-system-under-test)'s RPDE feeds, which means that it is either in its [Initial Harvest](#initial-harvest) phase or it is polling the feeds to keep up to date. +* `paused`: Broker Microservice is **not** harvesting or polling the [Booking System](#booking-system-under-test)'s RPDE feeds. This means that its data is not up to date with the Booking System's data, and so it is not available for use by the [Integration Tests](../openactive-integration-tests/). + +TODO2 ensure that all reasons for becoming paused are documented somewhere. diff --git a/packages/openactive-broker-microservice/src/util/opportunity-id-cache.js b/packages/openactive-broker-microservice/src/util/opportunity-id-cache.js index c348897eaa..6ce4eb9755 100644 --- a/packages/openactive-broker-microservice/src/util/opportunity-id-cache.js +++ b/packages/openactive-broker-microservice/src/util/opportunity-id-cache.js @@ -8,6 +8,7 @@ const { criteria } = require('@openactive/test-interface-criteria'); * @typedef {Map} OpportunityIdCacheType */ +// TODO2 clearly link this to "Buckets"? /** * Cache of Opportunity IDs. They are stored here according to criteria that they match (e.g. Opportunity Criteria, * Opportunity Type, etc). @@ -15,11 +16,15 @@ const { criteria } = require('@openactive/test-interface-criteria'); * * Schema: * - * -> Criteria Name - * -> Booking Flow - * -> Opportunity Type - * -> Seller ID - * -> Set + * -> {Criteria Name} + * -> {Booking Flow} + * -> {Opportunity Type} + * -> contents + * -> {Seller ID} + * -> {Set} + * -> criteriaErrors + * -> {constraint name (name of the constraint that failed)} + * -> {number of items which failed to meet the criteria due to this constraint} */ const OpportunityIdCache = { /** diff --git a/packages/openactive-broker-microservice/src/util/pause-resume.js b/packages/openactive-broker-microservice/src/util/pause-resume.js index 7f3de5add5..cc83719389 100644 --- a/packages/openactive-broker-microservice/src/util/pause-resume.js +++ b/packages/openactive-broker-microservice/src/util/pause-resume.js @@ -3,8 +3,15 @@ const { Mutex } = require('await-semaphore'); class PauseResume { constructor() { this.pauseHarvesting = false; + /** + * Lock that is held while harvesting is paused. It is released when + * harvesting is resumed. This is used to enable `waitIfPaused()` + */ this.pauseHarvestingMutex = new Mutex(); - this.releasePauseSemaphore = null; + /** + * @type {() => void | null} Function that releases `this.pauseHarvestingMutex` + */ + this.releasePauseHarvestingMutex = null; process.on('message', async (msg) => { if (msg === 'pause') { await this.pause(); @@ -17,8 +24,8 @@ class PauseResume { async pause() { if (!this.pauseHarvesting) { this.pauseHarvesting = true; - if (this.releasePauseSemaphore) this.releasePauseSemaphore(); - this.releasePauseSemaphore = await this.pauseHarvestingMutex.acquire(); + if (this.releasePauseHarvestingMutex) this.releasePauseHarvestingMutex(); + this.releasePauseHarvestingMutex = await this.pauseHarvestingMutex.acquire(); } } @@ -29,8 +36,8 @@ class PauseResume { resume() { const wasPaused = this.pauseHarvesting; this.pauseHarvesting = false; - if (this.releasePauseSemaphore) this.releasePauseSemaphore(); - this.releasePauseSemaphore = null; + if (this.releasePauseHarvestingMutex) this.releasePauseHarvestingMutex(); + this.releasePauseHarvestingMutex = null; return wasPaused; } diff --git a/packages/openactive-integration-tests/README.md b/packages/openactive-integration-tests/README.md index fd0955d86a..26383f5d2f 100644 --- a/packages/openactive-integration-tests/README.md +++ b/packages/openactive-integration-tests/README.md @@ -16,8 +16,9 @@ The results of this test suite when run against the reference implementation can To run `openactive-integration-tests` in separate terminal window to `openactive-broker-microservice`, from repository root: 1. Ensure the [openactive-broker-microservice](../openactive-broker-microservice/) is running in another terminal window -2. `export NODE_ENV=dev` -3. `npm run start-tests` +2. Specify Configuration file: `export NODE_ENV=dev` + * This is the command to use if using the `dev.json` config file, which is the default behaviour. See [Configuration](../../README.md#configuration) for more details) +3. Run Tests: `npm run start-tests` ### Running specific tests From 9d2608be31e36a51128d454b6c2068a5c14c5477 Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Tue, 12 Sep 2023 18:54:42 +0100 Subject: [PATCH 02/20] morrrre --- .../openactive-broker-microservice/README.md | 147 ++++++++++++++++-- 1 file changed, 136 insertions(+), 11 deletions(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index 0ffc328fda..f2da6b6014 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -325,11 +325,15 @@ Broker Microservice exposes an API which is used by the [Integration Tests](../o #### `GET /` -An HTML home page which shows links to other user-facing endpoints +**Response Type**: 🌐 HTML + +A home page which shows links to other user-facing endpoints #### `GET /status` -A JSON object, which shows the status of the Broker Microservice. This is used by a user to check on the progress of Broker and to help diagnose any potential issues. An annotated example response: +**Response Type**: 🤖 JSON + +Shows the status of the Broker Microservice. This is used by a user to check on the progress of Broker and to help diagnose any potential issues. An annotated example response: ```json { @@ -387,11 +391,7 @@ A JSON object, which shows the status of the Broker Microservice. This is used b "OrderProposalsFeed (auth:secondary)": { /* ... */ }, }, "orphans": { - /* Tracks the number of items which have been harvested from the ScheduledSession - or Slot feeds, where their corresponding parent SessionSeries or FacilityUse - has not (yet) been harvested. - If all feeds have been harvested and there are orphans, this indicates that - there may be a problem with the Booking System's data. */ + // Tracks the number of Orphans, according to the data that Broker currently has. "children": "0 of 13042 (0.00%)" }, "totalOpportunitiesHarvested": 13042, @@ -453,7 +453,65 @@ A JSON object, which shows the status of the Broker Microservice. This is used b #### `GET /validation-errors` -An HTML web page which shows any validation errors that have been found for Opportunities that have been harvested from the Booking System. +**Response Type**: 🌐 HTML + +Shows any validation errors that have been found for Opportunities that have been harvested from the Booking System. + +#### `GET /orphans` + +**Response Type**: 🤖 JSON + +Data about any [Orphans](#orphans) that Broker Microservice has found. Useful for debugging issues with the [Booking System](#booking-system-under-test)'s data. + +Example response: + +```json +{ + "children": { + // Number of child Opportunities whose parent has been found i.e. not orphaned + "matched": 11946, + // Number of orphaned child Opportunities + "orphaned": 2, + // Total number of child Opportunities + "total": 11948, + // List of the orphaned child Opportunities + "orphanedList": [ + { + "jsonLdType": "ScheduledSession", + // The RPDE item ID of the Orphan + "id": "https://acme-fitness.org/api/scheduled-sessions/1", + "modified": 1234, + // The JSON-LD data for the Orphan + "jsonLd": { + "@context": "https://openactive.io/", + "@type": "ScheduledSession", + "@id": "https://acme-fitness.org/api/scheduled-sessions/1", + "name": "Yoga", + "superEvent": "https://acme-fitness.org/api/session-series/1", + // ...etc + }, + // The JSON-LD ID of the Orphan + "jsonLdId": "https://acme-fitness.org/api/scheduled-sessions/1", + // The anticipated JSON-LD ID of the Orphan's parent + "jsonLdParentId": "https://acme-fitness.org/api/session-series/1", + }, + // ... More orphans + ] + } +} +``` + +#### `GET /opportunity-cache/:id` + +**Response Type**: 🤖 JSON + +**Request params**: + +* `id`: The ID of the [Child Opportunity](#orphans). + +Get an expanded* [Child Opportunity](#orphans) (e.g. a ScheduledSession or IndividualFacilityUseSlot) from Broker Microservice's cache, by its ID. Useful for debugging issues with the [Booking System](#booking-system-under-test)'s data. + +- *expanded: The Opportunity will have been expanded to include its parent Opportunity, if it has one. ### Internal endpoints @@ -461,13 +519,15 @@ Endpoints used by Tests TODO2 #### `GET /health-check` +**Response Type**: 📄 Plain Text + Returns a response when the [Initial Harvest](#initial-harvest) is complete. This is used by the [Integration Tests](../openactive-integration-tests/) to check that Broker Microservice is up to date with the Booking System's data. This endpoint starts by setting Broker Microservice's [Harvesting Status](#harvesting-status) to `resumed`. So, if it was `paused` before, it will now resume harvesting. #### `POST /pause` -TODO2 use format for POST data interface +**Response Type**: 🕳️ Empty Set Broker Microservice's [Harvesting Status](#harvesting-status) to `paused`. @@ -475,7 +535,9 @@ This is called by the [Integration Tests](../openactive-integration-tests/) when #### `GET /config` -A JSON object, which [Integration Tests](../openactive-integration-tests/) uses to configure itself. It contains: +**Response Type**: 🤖 JSON + +Config, which [Integration Tests](../openactive-integration-tests/) uses to configure itself. It contains: * Broker Microservice config * Derived Broker Microservice config e.g. `bookingApiBaseUrl` is parsed from the [Dataset Site JSON](https://openactive.io/dataset-api-discovery/EditorsDraft/#embedded-json) that was loaded from the Dataset Site defined by the [`datasetSiteUrl` configuration property](#datasetsiteurl). @@ -511,7 +573,11 @@ Here is an annotated example: #### `GET /dataset-site` -A JSON object, which contains the [Dataset Site JSON](https://openactive.io/dataset-api-discovery/EditorsDraft/#embedded-json) that was loaded from the Dataset Site defined by the [`datasetSiteUrl` configuration property](#datasetsiteurl). +**Response Type**: 🤖 JSON + +The [Dataset Site JSON](https://openactive.io/dataset-api-discovery/EditorsDraft/#embedded-json) that was loaded from the Dataset Site defined by the [`datasetSiteUrl` configuration property](#datasetsiteurl). + +Some of the tests in the [Integration Tests](../openactive-integration-tests/) use this in order to run checks against the Dataset Site JSON. Here is an example: @@ -531,6 +597,39 @@ Here is an example: // ...etc ``` +#### `DELETE /opportunity-cache` + +**Response Type**: 🕳️ Empty + +Deletes all of Broker Microservice's cached data about harvested [Opportunities](https://openactive.io/open-booking-api/EditorsDraft/#dfn-opportunity). This is used by the [Integration Tests](../openactive-integration-tests/) to reset Broker Microservice's data in between runs of Test Suite. (TODO2 I'm confused by this) + +#### `POST /opportunity-listeners/:id` + +**Response Type**: 🕳️ Empty + +**Request params**: + +* `id`: The ID of the [Child Opportunity](#orphans) to listen for. + +Create an [Opportunity Listener](#opportunity-listeners) for the specified Child Opportunity. This will start the **Listen** phase. + +#### `GET /opportunity-listeners/:id` + +**Response Type**: 🤖 JSON + +**Request params**: + +* `id`: The ID of the [Child Opportunity](#orphans) that is already being listened for. + +Must be called after [`POST /opportunity-listeners/:id`](#post-opportunity-listenersid). + +If and when the specified Child Opportunity is updated, this will return the updated Opportunity. This invokes the **Get** phase of the [Opportunity Listener](#opportunity-listeners). + +Either this endpoint will: + +* (if an update was found) return the updated Opportunity. +* (if no update is found) eventually timeout. + ## Concepts ### Booking System under Test @@ -539,6 +638,8 @@ TODO2 ### Buckets +TODO2 change this to a part of Opportunity ID Cache + A **Bucket** is a cache of Opportunity IDs that match a given Opportunity Criteria (TODO link to Opportunity Criteria description in packages/test-interface-criteria/README.md). When Integration Tests sends a request to get a random Opportunity matching a given criteria, the Opportunity is fetched from these buckets. ### Initial Harvest @@ -557,3 +658,27 @@ Broker Microservice can have one of two **Harvesting Statuses**: * `paused`: Broker Microservice is **not** harvesting or polling the [Booking System](#booking-system-under-test)'s RPDE feeds. This means that its data is not up to date with the Booking System's data, and so it is not available for use by the [Integration Tests](../openactive-integration-tests/). TODO2 ensure that all reasons for becoming paused are documented somewhere. + +### Opportunity Listeners + +An **Opportunity Listener** can be created in Broker Microservice to listen for updates to a given [Child Opportunity](#orphans). This is used by the [Integration Tests](../openactive-integration-tests/) to ensure that certain actions lead to updates to opportunity data. + +An Opportunity Listener is used in two phases: + +1. **Listen**: Start listening for the Opportunity. Any updates to this Opportunity prior to this point will be ignored. +2. **Get**: Get the update if there is one. If there isn't one, wait for one to arrive. + +### Orphans + +There is a parent-child relationship between some types of [Opportunity](https://openactive.io/open-booking-api/EditorsDraft/#dfn-opportunity) Feeds. Some examples: + +* Parent: **FacilityUse** feed + * Child: **Slot** feed or **IndividualFacilityUseSlot** feed +* Parent: **SessionSeries** feed + * Child: **ScheduledSession** feed + +An **Orphan** is an Opportunity from a child feed (e.g. a ScheduledSession) whose corresponding parent Opportunity (e.g. a SessionSeries) does not exist. + +Broker Microservice keeps track of the number of Orphans that it has found. Before its [Initial Harvest](#initial-harvest), it may erroneously identify Orphans because it has not yet seen all the data from the parent feeds. However, after all feeds have been harvested, if there are still Orphans, this indicates that there may be a problem with the Booking System's data. + +For more info about the different configurations of Opportunity feeds available, see [Types of RPDE feed](https://developer.openactive.io/publishing-data/data-feeds/types-of-feed). From 88f6dd694967e56a6f087d5a66b7406c6ebf6629 Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Wed, 13 Sep 2023 15:27:10 +0100 Subject: [PATCH 03/20] finished all API endpoints ---.--- --- .../CONTRIBUTING.md | 6 + .../openactive-broker-microservice/README.md | 176 ++++++++++++++++-- 2 files changed, 167 insertions(+), 15 deletions(-) create mode 100644 packages/openactive-broker-microservice/CONTRIBUTING.md diff --git a/packages/openactive-broker-microservice/CONTRIBUTING.md b/packages/openactive-broker-microservice/CONTRIBUTING.md new file mode 100644 index 0000000000..08675ff841 --- /dev/null +++ b/packages/openactive-broker-microservice/CONTRIBUTING.md @@ -0,0 +1,6 @@ +# Contributing + +## Keeping Documentation up-to-date + +* **If changing Broker Microservice's API** e.g. adding new endpoints or query params: + * Please update the [Broker Microservice API](./README.md#broker-microservice-api) section of the README. diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index f2da6b6014..e60f379750 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -503,16 +503,26 @@ Example response: #### `GET /opportunity-cache/:id` -**Response Type**: 🤖 JSON - **Request params**: * `id`: The ID of the [Child Opportunity](#orphans). +**Response Type**: 🤖 JSON + Get an expanded* [Child Opportunity](#orphans) (e.g. a ScheduledSession or IndividualFacilityUseSlot) from Broker Microservice's cache, by its ID. Useful for debugging issues with the [Booking System](#booking-system-under-test)'s data. - *expanded: The Opportunity will have been expanded to include its parent Opportunity, if it has one. +#### `GET /sample-opportunities` + +**Request body**: The shape of [Child Opportunity](#orphans) to fetch. This data has the same specification as the request body to [Test Interface](https://openactive.io/test-interface/)'s [`POST /test-interface/datasets/:testDatasetIdentifier/opportunities`](https://openactive.io/test-interface/#post-test-interfacedatasetstestdatasetidentifieropportunities) endpoint. + +**Response Type**: 🤖 JSON + +Return a random sample of Opportunities that match the Opportunity Criteria, Seller and Opportunity Type from the request body. This endpoint is very similar to [`POST /test-interface/datasets/:testDatasetIdentifier/opportunities`](#post-test-interfacedatasetstestdatasetidentifieropportunities), but instead is intended for use outside of the [Integration Tests](../openactive-integration-tests/). + +The primary use of this endpoint is to use with something like the [Postman API Platform](https://www.postman.com/) to empower a human user to run manual tests against the [Booking System](#booking-system-under-test). + ### Internal endpoints Endpoints used by Tests TODO2 @@ -605,30 +615,150 @@ Deletes all of Broker Microservice's cached data about harvested [Opportunities] #### `POST /opportunity-listeners/:id` -**Response Type**: 🕳️ Empty - **Request params**: * `id`: The ID of the [Child Opportunity](#orphans) to listen for. -Create an [Opportunity Listener](#opportunity-listeners) for the specified Child Opportunity. This will start the **Listen** phase. +**Response Type**: 🕳️ Empty -#### `GET /opportunity-listeners/:id` +Create an [Opportunity Listener](#two-phase-listeners) to listen for updates to the specified Child Opportunity. -**Response Type**: 🤖 JSON +#### `GET /opportunity-listeners/:id` **Request params**: * `id`: The ID of the [Child Opportunity](#orphans) that is already being listened for. +**Response Type**: 🤖 JSON + Must be called after [`POST /opportunity-listeners/:id`](#post-opportunity-listenersid). -If and when the specified Child Opportunity is updated, this will return the updated Opportunity. This invokes the **Get** phase of the [Opportunity Listener](#opportunity-listeners). +If and when the specified Child Opportunity is updated, this will return the updated Opportunity. This invokes the **Get** phase of the [Opportunity Listener](#two-phase-listeners). Either this endpoint will: -* (if an update was found) return the updated Opportunity. -* (if no update is found) eventually timeout. +* If an update was found: + * return the updated Opportunity. +* If no update is found: + * eventually timeout. + +### `POST /order-listeners/:type/:bookingPartnerIdentifier/:uuid` + +**Request params**: + +* `type`: `orders` or `order-proposals`. +* `bookingPartnerIdentifier`: Identifier of the Booking Partner whose [Orders Feed](https://openactive.io/open-booking-api/EditorsDraft/#dfn-orders-feed) or OrderProposals Feed to listen to. This is the key of the Booking Partner in the [`bookingPartners` configuration property](#bookingpartners) e.g. `primary`. +* `uuid`: UUID of the Order or OrderProposal to listen for. + +**Response Type**: 🤖 JSON + +Create an [Order Listener](#two-phase-listeners) to listen for updates to the specified Order or Order Proposal. + +The response contains info that is useful to a human user for debugging, such as the `startingFeedPage`, which is the page of the feed that Broker Microservice will start listening from. + +### `GET /order-listeners/:type/:bookingPartnerIdentifier/:uuid` + +**Request params**: + +* `type`: `orders` or `order-proposals`. +* `bookingPartnerIdentifier`: Identifier of the Booking Partner whose [Orders Feed](https://openactive.io/open-booking-api/EditorsDraft/#dfn-orders-feed) or OrderProposals Feed is being listened for. This is the key of the Booking Partner in the [`bookingPartners` configuration property](#bookingpartners) e.g. `primary`. +* `uuid`: UUID of the Order or OrderProposal that is being listened for. + +**Response Type**: 🤖 JSON + +Must be called after [`POST /order-listeners/:type/:bookingPartnerIdentifier/:uuid`](#post-order-listenerstypebookingpartneridentifieruuid). + +If and when the specified Order or OrderProposal is updated, this will return its updated data. This invokes the **Get** phase of the [Order Listener](#two-phase-listeners). + +Either this endpoint will: + +* If an update was found: + * return the updated Order or OrderProposal. +* If no update is found: + * eventually timeout. + +### `GET /is-order-uuid-present/:type/:bookingPartnerIdentifier/:uuid` + +**Request params**: + +* `type`: `orders` or `order-proposals`. +* `bookingPartnerIdentifier`: Identifier of the Booking Partner whose [Orders Feed](https://openactive.io/open-booking-api/EditorsDraft/#dfn-orders-feed) or OrderProposals Feed is being checked. This is the key of the Booking Partner in the [`bookingPartners` configuration property](#bookingpartners) e.g. `primary`. +* `uuid`: UUID of the Order or OrderProposal that is being checked. + +**Response Type**: 🤖 JSON + +Check whether or not the specified Order or OrderProposal is present in the [Booking System](#booking-system-under-test)'s feeds. + +This endpoint will do one of the following: + +1. If the Order or OrderProposal has been seen in the feeds already: + * return `true` +2. If the Order or OrderProposal has not been seen in the feeds yet and the feeds have been fully harvested: + * return `false` +3. If the Order or OrderProposal has not been seen in the feeds yet but the feeds have not been fully harvested yet: + * wait until either of conditions #1 or #2 are met, then return the appropriate result. + +### `GET /opportunity/:id` + +**Request params**: + +* `id`: The ID of the [Child Opportunity](#orphans). + +**Request query params**: + +* `useCacheIfAvailable` (OPTIONAL): If `true`, the Opportunity will be retrieved from Broker Microservice's caches if available. If `false`, Broker Microservice will not look at its existing caches and will await an update to the Opportunity in the [Booking System](#booking-system-under-test)'s feeds. Defaults to `false`. +* `expectedCapacity` (OPTIONAL): If included, the Opportunity will only be returned if its capacity is equal to the specified value. Any updates which have a different value for capacity will be ignored. This is useful for [Integration Tests](../openactive-integration-tests/) tests which need to check that an Opportunity's capacity has been updated to a certain value. + +**Response Type**: 🤖 JSON + +Get the specified [Child Opportunity](#orphans) from either Broker Microservice's caches or from updates to the [Booking System](#booking-system-under-test)'s feeds that happen after this call is made. + +This endpoint will do one of the following: + +1. If the Opportunity is found: + * return the Opportunity. +2. If the Opportunity is never found: + * eventually timeout + +### `POST /test-interface/datasets/:testDatasetIdentifier/opportunities` + +**Request params**: + +* `testDatasetIdentifier`: The identifier of the [Test Dataset](https://openactive.io/test-interface/#datasets-endpoints) + +**Request body**: The shape of [Child Opportunity](#orphans) to fetch. This data has the same specification as the request body to [Test Interface](https://openactive.io/test-interface/)'s [`POST /test-interface/datasets/:testDatasetIdentifier/opportunities`](https://openactive.io/test-interface/#post-test-interfacedatasetstestdatasetidentifieropportunities) endpoint. + +**Response type**: 🤖 JSON + +This endpoint mirrors the [Test Interface](https://openactive.io/test-interface/)'s [`POST /test-interface/datasets/:testDatasetIdentifier/opportunities`](https://openactive.io/test-interface/#post-test-interfacedatasetstestdatasetidentifieropportunities) endpoint exactly. But, instead of creating a new Opportunity when called, it fetches an existing Opportunity from Broker Microservice's caches. + +In this way, it can be used by [Integration Tests](../openactive-integration-tests/) when it is running in [Random mode](../openactive-integration-tests/README.md#userandomopportunities). + +If an Opportunity matching the Opportunity Criteria, Seller and Opportunity Type from the request body is found, it will be returned. Otherwise, the response will be a 404. + +Any Opportunity returned from this endpoint will be [locked](#opportunity-locks) and so will not be available for subsequent calls until locks are released. + +### `DELETE /test-interface/datasets/:testDatasetIdentifier` + +**Request params**: + +* `testDatasetIdentifier`: The identifier of the [Test Dataset](https://openactive.io/test-interface/#datasets-endpoints) + +**Response type**: 🕳️ Empty + +This endpoint mirrors the [Test Interface](https://openactive.io/test-interface/)'s [`DELETE /test-interface/datasets/:testDatasetIdentifier`](https://openactive.io/test-interface/#delete-test-interfacedatasetstestdatasetidentifier) endpoint exactly. But, instead of deleting Opportunities, it releases all [Opportunity Locks](#opportunity-locks) created for the specified `testDatasetIdentifier`. + +In this way, it can be used by [Integration Tests](../openactive-integration-tests/) when it is running in [Random mode](../openactive-integration-tests/README.md#userandomopportunities) to get ready for a new test run. + +### `POST /assert-unmatched-criteria` + +**Request body**: The shape of [Child Opportunity](#orphans) to check. This data has the same specification as the request body to [Test Interface](https://openactive.io/test-interface/)'s [`POST /test-interface/datasets/:testDatasetIdentifier/opportunities`](https://openactive.io/test-interface/#post-test-interfacedatasetstestdatasetidentifieropportunities) endpoint. + +**Response type**: 🤖 JSON + +Assert that an Opportunity matching the Opportunity Criteria, Seller and Opportunity Type from the request body is **not** found in the [Booking System](#booking-system-under-test)'s data. If such an Opportunity is found, the response will be a `404`. Otherwise, the response will be a `204`. + +This is used by some [Non-Implemented tests](../openactive-integration-tests/README.md#structure) to ensure that a test feature MUST be implemented if a certain criteria of Opportunity is found in the Booking System's data. e.g. the [`cancellation-window` feature](../openactive-integration-tests/test/features/cancellation/cancellation-window) has a non-implemented test that asserts that there are no Opportunities with cancellation windows defined. ## Concepts @@ -659,14 +789,15 @@ Broker Microservice can have one of two **Harvesting Statuses**: TODO2 ensure that all reasons for becoming paused are documented somewhere. -### Opportunity Listeners +### Opportunity Locks -An **Opportunity Listener** can be created in Broker Microservice to listen for updates to a given [Child Opportunity](#orphans). This is used by the [Integration Tests](../openactive-integration-tests/) to ensure that certain actions lead to updates to opportunity data. +When running tests in [Random mode](../openactive-integration-tests/README.md#userandomopportunities), it is important that any Opportunity is only used in one test. This increases test coverage by ensuring that tests use a variety of different Opportunities and, crucially, it prevents unexpected behaviours when tests are run in parallel. -An Opportunity Listener is used in two phases: +Broker Microservice uses **Opportunity Locks** to ensure that an Opportunity is only used in one test. When an Opportunity is retrieved in Random mode, using [`POST /test-interface/datasets/:testDatasetIdentifier/opportunities`](#post-test-interfacedatasetstestdatasetidentifieropportunities), it is **locked**. Subsequent calls to get a random Opportunity with the same critera will not return the same Opportunity, as it is locked. -1. **Listen**: Start listening for the Opportunity. Any updates to this Opportunity prior to this point will be ignored. -2. **Get**: Get the update if there is one. If there isn't one, wait for one to arrive. +In between runs of the [Integration Tests](../openactive-integration-tests/) tests, these locks need to be **released**, to ensure that all Opportunities are available for subsequent test runs. + +These locks are contained within each [Test Dataset](https://openactive.io/test-interface/#datasets-endpoints), so that an Opportunity that is locked in one Test Dataset will be avaiable to another. ### Orphans @@ -682,3 +813,18 @@ An **Orphan** is an Opportunity from a child feed (e.g. a ScheduledSession) whos Broker Microservice keeps track of the number of Orphans that it has found. Before its [Initial Harvest](#initial-harvest), it may erroneously identify Orphans because it has not yet seen all the data from the parent feeds. However, after all feeds have been harvested, if there are still Orphans, this indicates that there may be a problem with the Booking System's data. For more info about the different configurations of Opportunity feeds available, see [Types of RPDE feed](https://developer.openactive.io/publishing-data/data-feeds/types-of-feed). + + +### Two-Phase Listeners + +A **Two-Phase Listener** can be created in Broker Miroservice to listen to updates to some object (e.g. [Opportunity](https://openactive.io/open-booking-api/EditorsDraft/#dfn-opportunity) or Order) in the [Booking System](#booking-system-under-test). These are used by the [Integration Tests](../openactive-integration-tests/) to ensure that certain actions lead to updates to data e.g. cancellation should lead to the Order's cancelled OrderItems updating their status. + +The two phases for these listeners are: + +1. **Listen**: Start listening for the object. Any updates to the object prior to this point will be ignored. +2. **Get**: Get the update if there is one. If there isn't one, wait for one to arrive. + +Types of Two-Phase Listeners in Broker Microservice: + +* **Opportunity Listeners**: Listen for updates to a given [Child Opportunity](#orphans). +* **Order Listeners**: Listen for updates to a given Order. From d90f622afc228134f6d89369c75bf818c42244d8 Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Wed, 13 Sep 2023 18:09:33 +0100 Subject: [PATCH 04/20] completed concepts and summary, linking with API --- .../openactive-broker-microservice/README.md | 61 +++++++++++++------ 1 file changed, 41 insertions(+), 20 deletions(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index e60f379750..e7cc4f5fe8 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -1,10 +1,12 @@ # openactive-broker-microservice -This Node.js microservice provides a service to harvest data and proxy calls to an Open Booking API. +Broker Microservice sits in front of a [Booking System](#booking-system-under-test), which is an implementation of the [Open Booking API](https://openactive.io/open-booking-api/EditorsDraft/), and provides the following services, to enable Test Suite: -It needs to be running, against an Open Booking API implementation, in order for the [openactive-integration-tests](../openactive-integration-tests/) to run. +1. [Harvests](#initial-harvest) data from the Booking System's Opportunity and Order RPDE feeds. This data is then [validated](#data-model-validation) and [cached](#opportunity-id-cache) and any [listeners](#two-phase-listeners) are notified. +2. Fetch and parse data from the Booking System's [Dataset Site](https://openactive.io/dataset-api-discovery/EditorsDraft/). +3. Sets up and maintains [auth credentials](#auth) for access to the Booking System. -TODO perhaps a little more summary about what it does? +It needs to be running, against a Booking System, in order for the [openactive-integration-tests](../openactive-integration-tests/) to run. ## Usage in separate terminal windows @@ -309,20 +311,14 @@ Clients credentials can be defined as follows: } ``` -## What Broker Microservice does - -TODO2: - -* Parses Dataset Site -* RPDE feed harvesting + validation -* Sets up auth - ## Broker Microservice API -Broker Microservice exposes an API which is used by the [Integration Tests](../openactive-integration-tests/) to TODO2 +Broker Microservice exposes an API which is used by [Integration Tests](../openactive-integration-tests/) and can be used by users to debug potential issues with the [Booking System](#booking-system-under-test). ### User-facing endpoints +Endpoints which caan be used by users to debug potential issues with the [Booking System](#booking-system-under-test). + #### `GET /` **Response Type**: 🌐 HTML @@ -455,7 +451,7 @@ Shows the status of the Broker Microservice. This is used by a user to check on **Response Type**: 🌐 HTML -Shows any validation errors that have been found for Opportunities that have been harvested from the Booking System. +Shows any [validation](#data-model-validation) errors that have been found for Opportunities that have been harvested from the Booking System. #### `GET /orphans` @@ -525,7 +521,7 @@ The primary use of this endpoint is to use with something like the [Postman API ### Internal endpoints -Endpoints used by Tests TODO2 +Endpoints used by [Integration Tests](../openactive-integration-tests/). #### `GET /health-check` @@ -611,7 +607,7 @@ Here is an example: **Response Type**: 🕳️ Empty -Deletes all of Broker Microservice's cached data about harvested [Opportunities](https://openactive.io/open-booking-api/EditorsDraft/#dfn-opportunity). This is used by the [Integration Tests](../openactive-integration-tests/) to reset Broker Microservice's data in between runs of Test Suite. (TODO2 I'm confused by this) +Deletes all of Broker Microservice's cached data about harvested [Opportunities](https://openactive.io/open-booking-api/EditorsDraft/#dfn-opportunity). This is used by the [Integration Tests](../openactive-integration-tests/) to reset Broker Microservice's data in between runs of Test Suite. (TODO I'm confused by this) #### `POST /opportunity-listeners/:id` @@ -760,17 +756,34 @@ Assert that an Opportunity matching the Opportunity Criteria, Seller and Opportu This is used by some [Non-Implemented tests](../openactive-integration-tests/README.md#structure) to ensure that a test feature MUST be implemented if a certain criteria of Opportunity is found in the Booking System's data. e.g. the [`cancellation-window` feature](../openactive-integration-tests/test/features/cancellation/cancellation-window) has a non-implemented test that asserts that there are no Opportunities with cancellation windows defined. +## Browser Automation for Auth Endpoints + +Broker Microservice also exposes endpoints created from the `setupBrowserAutomationRoutes(..)` function from [openactive-openid-test-client](../openactive-openid-test-client/). +These endpoints are used by [Integration Tests](../openactive-integration-tests/) for tests which check the [OpenID Connect](https://openid.net/developers/how-connect-works/) authentication flow. + ## Concepts +### Auth + +Broker Microservice sets up and maintains auth credentials for access to the [Booking System](#booking-system-under-test). + +[Integration Tests](../openactive-integration-tests/) can also obtain these credentials from [`GET /config`](#get-config) and use them to access the Booking System. + +These credentials are set up according to the auth strategy defined in the [`bookingPartners` configuration property](#bookingpartners). + +To do this, Broker Microservice uses the [openactive-openid-test-client](../openactive-openid-test-client/) library. + ### Booking System under Test -TODO2 +An implementation of the [Open Booking API specification](https://openactive.io/open-booking-api/EditorsDraft/), which is being tested by the [Integration Tests](../openactive-integration-tests/). + +Broker Microservice connects to this Booking System in order to fetch metadata, harvest its Opportunity and Order RPDE feeds and acquire auth credentials. All of these then empower [Integration Tests](../openactive-integration-tests/) to run a suite of tests against this same Booking System. -### Buckets +### Data Model Validation -TODO2 change this to a part of Opportunity ID Cache +For a [Booking System](#booking-system-under-test) to be considered valid, all of the data in its Opportunity and Order RPDE feeds must conform to the specifications for [Opportunity Data](https://openactive.io/modelling-opportunity-data/) and the [Open Booking API](https://openactive.io/open-booking-api/EditorsDraft/). -A **Bucket** is a cache of Opportunity IDs that match a given Opportunity Criteria (TODO link to Opportunity Criteria description in packages/test-interface-criteria/README.md). When Integration Tests sends a request to get a random Opportunity matching a given criteria, the Opportunity is fetched from these buckets. +Broker Microservice automatically runs validation on all of the data that it harvests from the Booking System's feeds, using the [Data Model Validator library](https://github.com/openactive/data-model-validator). ### Initial Harvest @@ -787,7 +800,15 @@ Broker Microservice can have one of two **Harvesting Statuses**: * `harvesting`: Broker Microservice is harvesting or polling the [Booking System](#booking-system-under-test)'s RPDE feeds, which means that it is either in its [Initial Harvest](#initial-harvest) phase or it is polling the feeds to keep up to date. * `paused`: Broker Microservice is **not** harvesting or polling the [Booking System](#booking-system-under-test)'s RPDE feeds. This means that its data is not up to date with the Booking System's data, and so it is not available for use by the [Integration Tests](../openactive-integration-tests/). -TODO2 ensure that all reasons for becoming paused are documented somewhere. +### Opportunity ID Cache + +Broker Microservice keeps a cache of all the [Opportunities](https://openactive.io/open-booking-api/EditorsDraft/#dfn-opportunity) that it has harvested from the [Booking System](#booking-system-under-test)'s feeds. + +This cache is stored in various structures. For the purposes of interfacing with Broker Microservice, the most important one to know about is **Buckets**: + +#### Buckets + +A **Bucket** is a cache of Opportunity IDs that match a given [Opportunity Criteria](../test-interface-criteria/README.md). When Integration Tests sends a request to get a random Opportunity matching a given criteria, the Opportunity is fetched from these buckets. ### Opportunity Locks From 038ea9fedffabddb4b47bf35c58dba1907fa3fe5 Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Wed, 13 Sep 2023 18:40:37 +0100 Subject: [PATCH 05/20] . --- .../src/util/opportunity-id-cache.js | 1 - 1 file changed, 1 deletion(-) diff --git a/packages/openactive-broker-microservice/src/util/opportunity-id-cache.js b/packages/openactive-broker-microservice/src/util/opportunity-id-cache.js index 6ce4eb9755..6a639c7a90 100644 --- a/packages/openactive-broker-microservice/src/util/opportunity-id-cache.js +++ b/packages/openactive-broker-microservice/src/util/opportunity-id-cache.js @@ -8,7 +8,6 @@ const { criteria } = require('@openactive/test-interface-criteria'); * @typedef {Map} OpportunityIdCacheType */ -// TODO2 clearly link this to "Buckets"? /** * Cache of Opportunity IDs. They are stored here according to criteria that they match (e.g. Opportunity Criteria, * Opportunity Type, etc). From 3665c043ee146ebaeb4944d8598dae13a5d818c8 Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Wed, 13 Sep 2023 18:51:45 +0100 Subject: [PATCH 06/20] . --- .../openactive-broker-microservice/README.md | 35 ++++++++++--------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index e7cc4f5fe8..7a3f87d196 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -1,10 +1,11 @@ # openactive-broker-microservice -Broker Microservice sits in front of a [Booking System](#booking-system-under-test), which is an implementation of the [Open Booking API](https://openactive.io/open-booking-api/EditorsDraft/), and provides the following services, to enable Test Suite: +Broker Microservice sits in front of a [Booking System](#booking-system-under-test), which is an implementation of the [Open Booking API specification](https://openactive.io/open-booking-api/EditorsDraft/), and provides the following services, to enable Test Suite: -1. [Harvests](#initial-harvest) data from the Booking System's Opportunity and Order RPDE feeds. This data is then [validated](#data-model-validation) and [cached](#opportunity-id-cache) and any [listeners](#two-phase-listeners) are notified. +1. [Harvests](#initial-harvest) data from the Booking System's Opportunity and Order RPDE feeds. + * This data is then [validated](#data-model-validation) and [cached](#opportunity-id-cache) and any [listeners](#two-phase-listeners) are notified. 2. Fetch and parse data from the Booking System's [Dataset Site](https://openactive.io/dataset-api-discovery/EditorsDraft/). -3. Sets up and maintains [auth credentials](#auth) for access to the Booking System. +3. Set up and maintains [auth credentials](#auth) for access to the Booking System. It needs to be running, against a Booking System, in order for the [openactive-integration-tests](../openactive-integration-tests/) to run. @@ -14,7 +15,7 @@ To run `openactive-broker-microservice` in separate terminal window to `openacti 1. Install dependencies: `npm install` 2. Specify Configuration file: `export NODE_ENV=dev` - * This is the command to use if using the `dev.json` config file, which is the default behaviour. See [Configuration](../../README.md#configuration) for more details) + * This is the command to use if using the `dev.json` config file, which is the default behaviour. See [Configuration](../../README.md#configuration) for more details 3. Start Broker: `npm run start-broker` 4. Run [openactive-integration-tests](../openactive-integration-tests/) in another terminal window @@ -317,7 +318,7 @@ Broker Microservice exposes an API which is used by [Integration Tests](../opena ### User-facing endpoints -Endpoints which caan be used by users to debug potential issues with the [Booking System](#booking-system-under-test). +Endpoints which can be used by users to debug potential issues with the [Booking System](#booking-system-under-test). #### `GET /` @@ -329,7 +330,7 @@ A home page which shows links to other user-facing endpoints **Response Type**: 🤖 JSON -Shows the status of the Broker Microservice. This is used by a user to check on the progress of Broker and to help diagnose any potential issues. An annotated example response: +Shows the status of the Broker Microservice. Use this to check on the progress of Broker and to help diagnose any potential issues. An annotated example response: ```json { @@ -451,7 +452,7 @@ Shows the status of the Broker Microservice. This is used by a user to check on **Response Type**: 🌐 HTML -Shows any [validation](#data-model-validation) errors that have been found for Opportunities that have been harvested from the Booking System. +Shows any [validation](#data-model-validation) errors that have been found for Opportunities that have been harvested from the [Booking System](#booking-system-under-test). #### `GET /orphans` @@ -515,9 +516,9 @@ Get an expanded* [Child Opportunity](#orphans) (e.g. a ScheduledSession or Indiv **Response Type**: 🤖 JSON -Return a random sample of Opportunities that match the Opportunity Criteria, Seller and Opportunity Type from the request body. This endpoint is very similar to [`POST /test-interface/datasets/:testDatasetIdentifier/opportunities`](#post-test-interfacedatasetstestdatasetidentifieropportunities), but instead is intended for use outside of the [Integration Tests](../openactive-integration-tests/). +Return a random sample of Opportunities that match the Opportunity Criteria, Seller and Opportunity Type from the request body. This endpoint is very similar to [`POST /test-interface/datasets/:testDatasetIdentifier/opportunities`](#post-test-interfacedatasetstestdatasetidentifieropportunities), but instead is intended for use outside of [Integration Tests](../openactive-integration-tests/). -The primary use of this endpoint is to use with something like the [Postman API Platform](https://www.postman.com/) to empower a human user to run manual tests against the [Booking System](#booking-system-under-test). +The primary use of this endpoint is to use with something like the [Postman API Platform](https://www.postman.com/) to empower a user to run manual tests against the [Booking System](#booking-system-under-test). ### Internal endpoints @@ -527,7 +528,7 @@ Endpoints used by [Integration Tests](../openactive-integration-tests/). **Response Type**: 📄 Plain Text -Returns a response when the [Initial Harvest](#initial-harvest) is complete. This is used by the [Integration Tests](../openactive-integration-tests/) to check that Broker Microservice is up to date with the Booking System's data. +Returns a response when the [Initial Harvest](#initial-harvest) is complete. This is used by [Integration Tests](../openactive-integration-tests/) to check that Broker Microservice is up to date with the Booking System's data. This endpoint starts by setting Broker Microservice's [Harvesting Status](#harvesting-status) to `resumed`. So, if it was `paused` before, it will now resume harvesting. @@ -537,7 +538,7 @@ This endpoint starts by setting Broker Microservice's [Harvesting Status](#harve Set Broker Microservice's [Harvesting Status](#harvesting-status) to `paused`. -This is called by the [Integration Tests](../openactive-integration-tests/) when they have finished running. This reduces the load on the [Booking System](#booking-system-under-test) and on the user's machine in between runs of Test Suite. +This is called by [Integration Tests](../openactive-integration-tests/) when they have finished running. This reduces the load on the [Booking System](#booking-system-under-test) and on the user's machine in between runs of Test Suite. #### `GET /config` @@ -583,7 +584,7 @@ Here is an annotated example: The [Dataset Site JSON](https://openactive.io/dataset-api-discovery/EditorsDraft/#embedded-json) that was loaded from the Dataset Site defined by the [`datasetSiteUrl` configuration property](#datasetsiteurl). -Some of the tests in the [Integration Tests](../openactive-integration-tests/) use this in order to run checks against the Dataset Site JSON. +Some of the tests in [Integration Tests](../openactive-integration-tests/) use this in order to run checks against the Dataset Site JSON. Here is an example: @@ -607,7 +608,7 @@ Here is an example: **Response Type**: 🕳️ Empty -Deletes all of Broker Microservice's cached data about harvested [Opportunities](https://openactive.io/open-booking-api/EditorsDraft/#dfn-opportunity). This is used by the [Integration Tests](../openactive-integration-tests/) to reset Broker Microservice's data in between runs of Test Suite. (TODO I'm confused by this) +Deletes all of Broker Microservice's cached data about harvested [Opportunities](https://openactive.io/open-booking-api/EditorsDraft/#dfn-opportunity). This is used by [Integration Tests](../openactive-integration-tests/) to reset Broker Microservice's data in between runs of Test Suite. (TODO I'm confused by this) #### `POST /opportunity-listeners/:id` @@ -775,7 +776,7 @@ To do this, Broker Microservice uses the [openactive-openid-test-client](../open ### Booking System under Test -An implementation of the [Open Booking API specification](https://openactive.io/open-booking-api/EditorsDraft/), which is being tested by the [Integration Tests](../openactive-integration-tests/). +An implementation of the [Open Booking API specification](https://openactive.io/open-booking-api/EditorsDraft/), which is being tested by [Integration Tests](../openactive-integration-tests/). Broker Microservice connects to this Booking System in order to fetch metadata, harvest its Opportunity and Order RPDE feeds and acquire auth credentials. All of these then empower [Integration Tests](../openactive-integration-tests/) to run a suite of tests against this same Booking System. @@ -798,7 +799,7 @@ Broker Microservice will continue to poll the feeds after the Initial Harvest an Broker Microservice can have one of two **Harvesting Statuses**: * `harvesting`: Broker Microservice is harvesting or polling the [Booking System](#booking-system-under-test)'s RPDE feeds, which means that it is either in its [Initial Harvest](#initial-harvest) phase or it is polling the feeds to keep up to date. -* `paused`: Broker Microservice is **not** harvesting or polling the [Booking System](#booking-system-under-test)'s RPDE feeds. This means that its data is not up to date with the Booking System's data, and so it is not available for use by the [Integration Tests](../openactive-integration-tests/). +* `paused`: Broker Microservice is **not** harvesting or polling the [Booking System](#booking-system-under-test)'s RPDE feeds. This means that its data is not up to date with the Booking System's data, and so it is not available for use by [Integration Tests](../openactive-integration-tests/). ### Opportunity ID Cache @@ -816,7 +817,7 @@ When running tests in [Random mode](../openactive-integration-tests/README.md#us Broker Microservice uses **Opportunity Locks** to ensure that an Opportunity is only used in one test. When an Opportunity is retrieved in Random mode, using [`POST /test-interface/datasets/:testDatasetIdentifier/opportunities`](#post-test-interfacedatasetstestdatasetidentifieropportunities), it is **locked**. Subsequent calls to get a random Opportunity with the same critera will not return the same Opportunity, as it is locked. -In between runs of the [Integration Tests](../openactive-integration-tests/) tests, these locks need to be **released**, to ensure that all Opportunities are available for subsequent test runs. +In between runs of the tests in [Integration Tests](../openactive-integration-tests/), these locks need to be **released**, to ensure that all Opportunities are available for subsequent test runs. These locks are contained within each [Test Dataset](https://openactive.io/test-interface/#datasets-endpoints), so that an Opportunity that is locked in one Test Dataset will be avaiable to another. @@ -838,7 +839,7 @@ For more info about the different configurations of Opportunity feeds available, ### Two-Phase Listeners -A **Two-Phase Listener** can be created in Broker Miroservice to listen to updates to some object (e.g. [Opportunity](https://openactive.io/open-booking-api/EditorsDraft/#dfn-opportunity) or Order) in the [Booking System](#booking-system-under-test). These are used by the [Integration Tests](../openactive-integration-tests/) to ensure that certain actions lead to updates to data e.g. cancellation should lead to the Order's cancelled OrderItems updating their status. +A **Two-Phase Listener** can be created in Broker Miroservice to listen to updates to some object (e.g. [Opportunity](https://openactive.io/open-booking-api/EditorsDraft/#dfn-opportunity) or Order) in the [Booking System](#booking-system-under-test). These are used by [Integration Tests](../openactive-integration-tests/) to ensure that certain actions lead to updates to data e.g. cancellation should lead to the Order's cancelled OrderItems updating their status. The two phases for these listeners are: From d7ea91aa451c6a7a4f21117de3199d0a888c1576 Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Wed, 13 Sep 2023 18:59:58 +0100 Subject: [PATCH 07/20] . --- packages/openactive-broker-microservice/README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index 7a3f87d196..94a56b2668 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -528,7 +528,7 @@ Endpoints used by [Integration Tests](../openactive-integration-tests/). **Response Type**: 📄 Plain Text -Returns a response when the [Initial Harvest](#initial-harvest) is complete. This is used by [Integration Tests](../openactive-integration-tests/) to check that Broker Microservice is up to date with the Booking System's data. +Returns a response when the [Initial Harvest](#initial-harvest) is complete. This is used by [Integration Tests](../openactive-integration-tests/) to check that Broker Microservice is up to date with the [Booking System](#booking-system-under-test)'s data. This endpoint starts by setting Broker Microservice's [Harvesting Status](#harvesting-status) to `resumed`. So, if it was `paused` before, it will now resume harvesting. @@ -608,7 +608,10 @@ Here is an example: **Response Type**: 🕳️ Empty -Deletes all of Broker Microservice's cached data about harvested [Opportunities](https://openactive.io/open-booking-api/EditorsDraft/#dfn-opportunity). This is used by [Integration Tests](../openactive-integration-tests/) to reset Broker Microservice's data in between runs of Test Suite. (TODO I'm confused by this) +Deletes all of Broker Microservice's cached data about harvested [Opportunities](https://openactive.io/open-booking-api/EditorsDraft/#dfn-opportunity). + +This is used by [Integration Tests](../openactive-integration-tests/) to reset Broker Microservice's cache in between runs of Test Suite in [Controlled mode](../openactive-integration-tests/README.md#userandomopportunities). +This is necessary as the [Booking System](#booking-system-under-test) deletes the entire Test Dataset in between runs of Test Suite in Controlled mode ([Test Interface docs](https://openactive.io/test-interface/#delete-test-interfacedatasetstestdatasetidentifier)). #### `POST /opportunity-listeners/:id` From 149ca30307cb5c78b7b9ad2b37e81a6a44fbb439 Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Wed, 13 Sep 2023 19:00:52 +0100 Subject: [PATCH 08/20] . --- packages/openactive-broker-microservice/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index 94a56b2668..e58ff85754 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -601,7 +601,10 @@ Here is an example: "accessService": { "@type": "WebAPI", "name": "Open Booking API", - // ...etc + // ...etc + }, + // ... +} ``` #### `DELETE /opportunity-cache` From d24a601faf95d39342622c62ded474263cdcb64e Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Wed, 13 Sep 2023 19:11:26 +0100 Subject: [PATCH 09/20] . --- packages/openactive-broker-microservice/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index e58ff85754..d79991bfe9 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -709,8 +709,8 @@ This endpoint will do one of the following: **Request query params**: -* `useCacheIfAvailable` (OPTIONAL): If `true`, the Opportunity will be retrieved from Broker Microservice's caches if available. If `false`, Broker Microservice will not look at its existing caches and will await an update to the Opportunity in the [Booking System](#booking-system-under-test)'s feeds. Defaults to `false`. -* `expectedCapacity` (OPTIONAL): If included, the Opportunity will only be returned if its capacity is equal to the specified value. Any updates which have a different value for capacity will be ignored. This is useful for [Integration Tests](../openactive-integration-tests/) tests which need to check that an Opportunity's capacity has been updated to a certain value. +* `useCacheIfAvailable` (OPTIONAL): If `true`, the Opportunity will be retrieved from Broker Microservice's [caches](#opportunity-id-cache) if available. If `false`, Broker Microservice will not look at its existing caches and will await an update to the Opportunity in the [Booking System](#booking-system-under-test)'s feeds. Defaults to `false`. +* `expectedCapacity` (OPTIONAL): If included, the Opportunity will only be returned if its capacity is equal to the specified value. Any updates which have a different value for capacity will be ignored. This is useful for [Integration Tests](../openactive-integration-tests/) tests which need to check that an Opportunity's capacity has been updated to a certain value e.g. it should go down after a successful booking. **Response Type**: 🤖 JSON From 0f4622902a9c88849c1b6987dc3c368567a67e04 Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Tue, 19 Sep 2023 15:13:37 +0100 Subject: [PATCH 10/20] some review comments --- packages/openactive-broker-microservice/README.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index d79991bfe9..e46a0a1f0d 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -1,11 +1,15 @@ # openactive-broker-microservice -Broker Microservice sits in front of a [Booking System](#booking-system-under-test), which is an implementation of the [Open Booking API specification](https://openactive.io/open-booking-api/EditorsDraft/), and provides the following services, to enable Test Suite: +This "Broker Microservice" project is a component of the OpenActive Test Suite. Although it can be run stand-alone for some advanced use-cases, it is run automatically as part of the OpenActive Test Suite `npm start` command (or when the OpenActive Test Suite Docker container is started). For general information about running the OpenActive Test Suite, please see the [relevant documentation](https://developer.openactive.io/open-booking-api/test-suite). -1. [Harvests](#initial-harvest) data from the Booking System's Opportunity and Order RPDE feeds. +## How it works + +Broker Microservice sits in front of a [Booking System](#booking-system-under-test), which is an implementation of the [Open Booking API specification](https://openactive.io/open-booking-api/EditorsDraft/), and provides the following services, to support OpenActive Test Suite: + +1. Fetch, parse and validate data from the Booking System's [Dataset Site](https://openactive.io/dataset-api-discovery/EditorsDraft/). +2. Set up and maintain [auth credentials](#auth) for access to the Booking System. +3. [Harvests](#initial-harvest) data from the Booking System's Opportunity and Order RPDE feeds. * This data is then [validated](#data-model-validation) and [cached](#opportunity-id-cache) and any [listeners](#two-phase-listeners) are notified. -2. Fetch and parse data from the Booking System's [Dataset Site](https://openactive.io/dataset-api-discovery/EditorsDraft/). -3. Set up and maintains [auth credentials](#auth) for access to the Booking System. It needs to be running, against a Booking System, in order for the [openactive-integration-tests](../openactive-integration-tests/) to run. From d1ac9a56db8b6320abbe984a1a468948d39d129f Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Tue, 19 Sep 2023 15:15:37 +0100 Subject: [PATCH 11/20] headlessAuth puppeteer setting elaboration Co-authored-by: Nick Evans <2616208+nickevansuk@users.noreply.github.com> --- packages/openactive-broker-microservice/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index e46a0a1f0d..8b4255641b 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -169,7 +169,7 @@ While debugging authentication it can be useful to log the configuration that th ### `headlessAuth` -For debugging authentication, it can be useful to perform the browser automation with a different setting for [Puppeteer's `headless` option](https://github.com/puppeteer/puppeteer#default-runtime-settings). By default, this is set to `true`. +When debugging authentication, it can be useful to see the browser window in which the OpenID Connect authentication tests take place. This setting configures [Puppeteer's `headless` option](https://github.com/puppeteer/puppeteer#default-runtime-settings), which can be set to `false` to show the browser window. By default, this is set to `true`, so that the browser window is hidden. ```json "headlessAuth": true, From 6cdaa4421d625df08ef3f87d5d2fc33f317ea808 Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Tue, 19 Sep 2023 15:17:09 +0100 Subject: [PATCH 12/20] include tip with description of status report for non-found criteria Co-authored-by: Nick Evans <2616208+nickevansuk@users.noreply.github.com> --- packages/openactive-broker-microservice/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index 8b4255641b..45560f5189 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -423,7 +423,9 @@ Shows the status of the Broker Microservice. Use this to check on the progress o /* Unlike with TestOpporunityBookable, there were no ScheduledSessions that matched this criteria, `TestOpportunityBookableNonFreeTaxNet`, so instead of showing Opportunity numbers for each Seller, the status - endpoint shows a summary of why no Opportunities were found. */ + endpoint shows a summary of why no Opportunities were found. + The key with the highest value is likely to be the reason that there are + no items in this bucket. */ "criteriaErrors": { /* There are 892 ScheduledSessions which support Simple Booking Flow and that do not satisfy the `TestOpportunityBookableNonFreeTaxNet` From b4dc1739913aea464aff9f611c041500eb262d3f Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Tue, 19 Sep 2023 15:19:39 +0100 Subject: [PATCH 13/20] consistency of reference Co-authored-by: Nick Evans <2616208+nickevansuk@users.noreply.github.com> --- packages/openactive-broker-microservice/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index 45560f5189..5592abbd8a 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -430,7 +430,7 @@ Shows the status of the Broker Microservice. Use this to check on the progress o /* There are 892 ScheduledSessions which support Simple Booking Flow and that do not satisfy the `TestOpportunityBookableNonFreeTaxNet` criteria because they fail this contraint. That is to say that they - do not have seller tax mode TaxNet. */ + do not have seller `taxMode` of `TaxNet`. */ "Seller Tax Mode Net": 892, "Must not require additional details": 88, "startDate must be 2hrs in advance for random tests to use": 236, From 75cd21be9f98724de46ac7641f6c221ba414b6be Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Tue, 19 Sep 2023 15:23:38 +0100 Subject: [PATCH 14/20] improve sellerTaxModeNet criteria msg --- packages/openactive-broker-microservice/README.md | 2 +- .../src/criteria/TestOpportunityBookableNonFreeTaxNet.js | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index e46a0a1f0d..89df243da5 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -429,7 +429,7 @@ Shows the status of the Broker Microservice. Use this to check on the progress o and that do not satisfy the `TestOpportunityBookableNonFreeTaxNet` criteria because they fail this contraint. That is to say that they do not have seller tax mode TaxNet. */ - "Seller Tax Mode Net": 892, + "Seller must have taxMode of TaxNet": 892, "Must not require additional details": 88, "startDate must be 2hrs in advance for random tests to use": 236, "Only non-free bookable Offers": 496, diff --git a/packages/test-interface-criteria/src/criteria/TestOpportunityBookableNonFreeTaxNet.js b/packages/test-interface-criteria/src/criteria/TestOpportunityBookableNonFreeTaxNet.js index 10d969d5cc..535bb418c7 100644 --- a/packages/test-interface-criteria/src/criteria/TestOpportunityBookableNonFreeTaxNet.js +++ b/packages/test-interface-criteria/src/criteria/TestOpportunityBookableNonFreeTaxNet.js @@ -9,7 +9,7 @@ const { TestOpportunityBookableNonFree } = require('./TestOpportunityBookableNon /** * @type {OpportunityConstraint} */ -function sellerTaxModeNet(opportunity) { +function sellerMustHaveTaxModeTaxNet(opportunity) { const organization = getOrganizerOrProvider(opportunity); return organization && organization.taxMode === 'https://openactive.io/TaxNet'; } @@ -21,8 +21,8 @@ const TestOpportunityBookableNonFreeTaxNet = createCriteria({ name: 'TestOpportunityBookableNonFreeTaxNet', opportunityConstraints: [ [ - 'Seller Tax Mode Net', - sellerTaxModeNet, + 'Seller must have taxMode of TaxNet', + sellerMustHaveTaxModeTaxNet, ], ], offerConstraints: [], From 0ccfb9f15d634df33f32bfac6981ec3e449ac029 Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Tue, 19 Sep 2023 15:27:08 +0100 Subject: [PATCH 15/20] if it references one that exists --- packages/openactive-broker-microservice/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index c1a8ef5a9b..a94d98afd8 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -514,7 +514,7 @@ Example response: Get an expanded* [Child Opportunity](#orphans) (e.g. a ScheduledSession or IndividualFacilityUseSlot) from Broker Microservice's cache, by its ID. Useful for debugging issues with the [Booking System](#booking-system-under-test)'s data. -- *expanded: The Opportunity will have been expanded to include its parent Opportunity, if it has one. +- *expanded: The Opportunity will have been expanded to include its parent Opportunity, if it references one that exists. #### `GET /sample-opportunities` From 68365d354757d057691c71ad754e08a8c5d56cf9 Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Tue, 19 Sep 2023 15:31:39 +0100 Subject: [PATCH 16/20] create order listener response example --- packages/openactive-broker-microservice/README.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index a94d98afd8..e1f981cb6f 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -663,7 +663,14 @@ Either this endpoint will: Create an [Order Listener](#two-phase-listeners) to listen for updates to the specified Order or Order Proposal. -The response contains info that is useful to a human user for debugging, such as the `startingFeedPage`, which is the page of the feed that Broker Microservice will start listening from. +The response contains info that may be useful to a user for debugging. Example: + +```json +{ + // The page of the feed that Broker Microservice will start listening from. + "startingFeedPage": "https://acme-fitness.org/api/booking/orders-rpde?afterChangeNumber=23504", +} +``` ### `GET /order-listeners/:type/:bookingPartnerIdentifier/:uuid` From 6329db98aa9d7f33bba6fafa5165c8533bbb408e Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Tue, 19 Sep 2023 15:41:38 +0100 Subject: [PATCH 17/20] need to be -> are Co-authored-by: Nick Evans <2616208+nickevansuk@users.noreply.github.com> --- packages/openactive-broker-microservice/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index e1f981cb6f..36c567c1ce 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -836,7 +836,7 @@ When running tests in [Random mode](../openactive-integration-tests/README.md#us Broker Microservice uses **Opportunity Locks** to ensure that an Opportunity is only used in one test. When an Opportunity is retrieved in Random mode, using [`POST /test-interface/datasets/:testDatasetIdentifier/opportunities`](#post-test-interfacedatasetstestdatasetidentifieropportunities), it is **locked**. Subsequent calls to get a random Opportunity with the same critera will not return the same Opportunity, as it is locked. -In between runs of the tests in [Integration Tests](../openactive-integration-tests/), these locks need to be **released**, to ensure that all Opportunities are available for subsequent test runs. +In between runs of the tests in [Integration Tests](../openactive-integration-tests/), these locks are **released**, to ensure that all Opportunities are available for subsequent test runs. These locks are contained within each [Test Dataset](https://openactive.io/test-interface/#datasets-endpoints), so that an Opportunity that is locked in one Test Dataset will be avaiable to another. From 27712a4af833a3a39fc1db396ded1ecbbdc35196 Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Tue, 19 Sep 2023 15:45:17 +0100 Subject: [PATCH 18/20] how opps get into buckets --- packages/openactive-broker-microservice/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/openactive-broker-microservice/README.md b/packages/openactive-broker-microservice/README.md index e1f981cb6f..a2fd85e69d 100644 --- a/packages/openactive-broker-microservice/README.md +++ b/packages/openactive-broker-microservice/README.md @@ -828,7 +828,7 @@ This cache is stored in various structures. For the purposes of interfacing with #### Buckets -A **Bucket** is a cache of Opportunity IDs that match a given [Opportunity Criteria](../test-interface-criteria/README.md). When Integration Tests sends a request to get a random Opportunity matching a given criteria, the Opportunity is fetched from these buckets. +A **Bucket** is a cache of Opportunity IDs that match a given [Opportunity Criteria](../test-interface-criteria/README.md). During harvesting, each Opportunity found in the feeds is added to the buckets whose criteria it matches. When Integration Tests sends a request to get a random Opportunity matching a given criteria, the Opportunity is fetched from these buckets. ### Opportunity Locks From 831050b1fa44215969244eb5ee0daf72cdf6e8dc Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Tue, 19 Sep 2023 15:46:33 +0100 Subject: [PATCH 19/20] failed -> was not met Co-authored-by: Nick Evans <2616208+nickevansuk@users.noreply.github.com> --- .../src/util/opportunity-id-cache.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/openactive-broker-microservice/src/util/opportunity-id-cache.js b/packages/openactive-broker-microservice/src/util/opportunity-id-cache.js index 6a639c7a90..e71fe8c16d 100644 --- a/packages/openactive-broker-microservice/src/util/opportunity-id-cache.js +++ b/packages/openactive-broker-microservice/src/util/opportunity-id-cache.js @@ -22,7 +22,7 @@ const { criteria } = require('@openactive/test-interface-criteria'); * -> {Seller ID} * -> {Set} * -> criteriaErrors - * -> {constraint name (name of the constraint that failed)} + * -> {constraint name (name of the constraint that was not met)} * -> {number of items which failed to meet the criteria due to this constraint} */ const OpportunityIdCache = { From 22162267f0d30ba67cd8d8eb927206fba6f752ef Mon Sep 17 00:00:00 2001 From: Luke Winship Date: Tue, 19 Sep 2023 15:47:23 +0100 Subject: [PATCH 20/20] no "default behaviour" Co-authored-by: Nick Evans <2616208+nickevansuk@users.noreply.github.com> --- packages/openactive-integration-tests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/openactive-integration-tests/README.md b/packages/openactive-integration-tests/README.md index 26383f5d2f..eed53e419c 100644 --- a/packages/openactive-integration-tests/README.md +++ b/packages/openactive-integration-tests/README.md @@ -17,7 +17,7 @@ To run `openactive-integration-tests` in separate terminal window to `openactive 1. Ensure the [openactive-broker-microservice](../openactive-broker-microservice/) is running in another terminal window 2. Specify Configuration file: `export NODE_ENV=dev` - * This is the command to use if using the `dev.json` config file, which is the default behaviour. See [Configuration](../../README.md#configuration) for more details) + * This is the command to use if using the `dev.json` config file. See [Configuration](../../README.md#configuration) for more details) 3. Run Tests: `npm run start-tests` ### Running specific tests