From e4705775ea517245cc5f23629636950d6bffc631 Mon Sep 17 00:00:00 2001 From: TomJKing Date: Fri, 27 Oct 2023 09:24:07 +0100 Subject: [PATCH 1/5] Update README --- CHANGELOG.md | 18 ----------------- README.md | 36 ++++++++++++++++++++++++++++++++++ terraform/da-terraform-modules | 2 +- 3 files changed, 37 insertions(+), 19 deletions(-) delete mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md deleted file mode 100644 index d0a0cd4..0000000 --- a/CHANGELOG.md +++ /dev/null @@ -1,18 +0,0 @@ -# CHANGELOG - -All notable changes to this project will be documented in this file. - -The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), -and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - -It is part of the [TRE template repository](https://github.com/nationalarchives/da-tre-template) - -## [0.0.1] - 2023-01-31 - -### Added - -- The keep a change log CHANGELOG - -### Fixed - -- Minor typos diff --git a/README.md b/README.md index 63d2008..48984e5 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,10 @@ Repository containing code for a self-contained service to generate unique "references" for records transferred to The National Archives (TNA). +## Reference Schema + +TBC + ![](https://raw.githubusercontent.com/nationalarchives/tdr-dev-documentation/master/beta-architecture/diagrams/reference-generator.svg) The service consists of three main components: @@ -24,6 +28,14 @@ A 500 response body will be returned if any issues occur when calling the Lambda * numberofrefs parameter isn't an int * any dynamodb Exceptions (key not found, unable to update, etc) +The number of references that can be returned in a single call is limited, because: +* prevention of single call using up all possible reference; +* limits to the permitted size of the response. + +Calling clients will need to handle this limit by making more than a single call to retrieve the required number of references if it is greater than the limit. + +The limit is store here for use by calling clients: [reference_generator_limit](https://github.com/nationalarchives/da-terraform-configurations/blob/main/tdr/main.tf) + ## DynamoDb Table The DynamoDb stores the current counter used for generating unique references. The DynamoDb is encrypted so that it cannot be directly modified via the AWS console or AWS CLI. @@ -33,6 +45,13 @@ Below is an example of what the table looks like: |-------------|------------------| | fileCounter | 6 | +### Security + +The DynamoDb is monitored by a specific set of Cloud Custodian rules: +* `Reference-counter-table-kms-key-check`: Checks if reference counter table encrypted with specific KMS key. +* `Reference-counter-table-pitr-check`: Checks that point in time recovery (PITR) is enabled for reference counter table + +Full details and code are here: [Reference counter custodian rules](https://github.com/nationalarchives/tna-custodian/tree/master/custodian/policies/dynamodb/reference-counter) ## API Gateway @@ -43,6 +62,8 @@ It can be called directly by providing the parameter `numberofrefs={value}` by m The api gateway has a resource policy which restricts which services can call it. +Any new calling clients will need to provide an AWS IAM role which can call the API Gateway and this needs to be added to the API Gateway resource policy. + ## Deployment Deployment process of the service will depend on the hosting environment. @@ -100,3 +121,18 @@ It relies on the `da-terraform-configurations` and `da-terraform-modules` projec Commit and push all the changes made in the terraform directory to its GitHub repo, then (in the GitHub repo): Go the Actions tab -> Click ["Apply Terraform and deploy lambda"] -> Click "Run workflow" -> select the branch with the workflow file you want to use -> type the version to deploy -> Click the green "Run worklfow" button + +## Moving to new hosting project + +Should the reference generator service need to be moved to a different hosting project then the following steps will need to be taken: + +***NOTE*** Before the move the current counter value will need to be noted down to ensure the new DynamoDb table is seeded with the correct counter to prevent duplicate references + +* Add relevant Github Actions workflows for the new project to allow testing and deployment +* Update the [da-terraform-configurations](https://github.com/nationalarchives/da-terraform-configurations) repo with the new values for the reference generator service in the relevant project file: + * `reference_generator_limit` + * `reference_generator_intg_url` + * `reference_generator_staging_url` + * `reference_generator_prod_url` +* Ensure the Cloud Custodian rules are implemented for on the new hosting project: [DynamoDb > Security](#security) +* Set up relevant GitHub actions in the new hosting project: [Deployment](#deployment) diff --git a/terraform/da-terraform-modules b/terraform/da-terraform-modules index c65c877..e8322dd 160000 --- a/terraform/da-terraform-modules +++ b/terraform/da-terraform-modules @@ -1 +1 @@ -Subproject commit c65c877a4adf208923c16a0d097848de3b3dbe0b +Subproject commit e8322dd83cd921f609ba366896d110fcae2f83ee From 5a203b652e578fb7a3d0bb0139d94d30c4418ab9 Mon Sep 17 00:00:00 2001 From: TomJKing Date: Fri, 27 Oct 2023 09:28:02 +0100 Subject: [PATCH 2/5] Fix editorial issues --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 48984e5..35a1e9f 100644 --- a/README.md +++ b/README.md @@ -2,10 +2,6 @@ Repository containing code for a self-contained service to generate unique "references" for records transferred to The National Archives (TNA). -## Reference Schema - -TBC - ![](https://raw.githubusercontent.com/nationalarchives/tdr-dev-documentation/master/beta-architecture/diagrams/reference-generator.svg) The service consists of three main components: @@ -29,10 +25,10 @@ A 500 response body will be returned if any issues occur when calling the Lambda * any dynamodb Exceptions (key not found, unable to update, etc) The number of references that can be returned in a single call is limited, because: -* prevention of single call using up all possible reference; +* prevent a single call using up all possible reference; * limits to the permitted size of the response. -Calling clients will need to handle this limit by making more than a single call to retrieve the required number of references if it is greater than the limit. +Calling clients will need to handle this limit by making multiple calls to retrieve the required number of references if it is greater than the limit. The limit is store here for use by calling clients: [reference_generator_limit](https://github.com/nationalarchives/da-terraform-configurations/blob/main/tdr/main.tf) @@ -64,6 +60,10 @@ The api gateway has a resource policy which restricts which services can call it Any new calling clients will need to provide an AWS IAM role which can call the API Gateway and this needs to be added to the API Gateway resource policy. +## Reference Schema + +TBC + ## Deployment Deployment process of the service will depend on the hosting environment. From 98e220423be374c5d6f0ca6c0c321a5444040f55 Mon Sep 17 00:00:00 2001 From: TomJKing Date: Wed, 1 Nov 2023 06:08:46 +0000 Subject: [PATCH 3/5] Pull request feedback --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 35a1e9f..d5cdd80 100644 --- a/README.md +++ b/README.md @@ -25,12 +25,12 @@ A 500 response body will be returned if any issues occur when calling the Lambda * any dynamodb Exceptions (key not found, unable to update, etc) The number of references that can be returned in a single call is limited, because: -* prevent a single call using up all possible reference; +* prevent a single call using up all possible references; * limits to the permitted size of the response. Calling clients will need to handle this limit by making multiple calls to retrieve the required number of references if it is greater than the limit. -The limit is store here for use by calling clients: [reference_generator_limit](https://github.com/nationalarchives/da-terraform-configurations/blob/main/tdr/main.tf) +The limit is stored here for use by calling clients: [reference_generator_limit](https://github.com/nationalarchives/da-terraform-configurations/blob/main/tdr/main.tf) ## DynamoDb Table From 9d4c53969d815f80a0e8e232256e7f9121f0249e Mon Sep 17 00:00:00 2001 From: tna-digital-archiving-jenkins Date: Wed, 1 Nov 2023 08:21:41 +0000 Subject: [PATCH 4/5] Update aws-java-sdk-dynamodb to 1.12.578 --- project/Dependencies.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/project/Dependencies.scala b/project/Dependencies.scala index d3f202d..bc92f5a 100644 --- a/project/Dependencies.scala +++ b/project/Dependencies.scala @@ -7,7 +7,7 @@ object Dependencies { lazy val scalaTest = "org.scalatest" %% "scalatest" % "3.2.17" lazy val awsSdkDynamoDbV2 = "software.amazon.awssdk" % "dynamodb" % "2.21.7" - lazy val awsSdkDynamoDbV1 = "com.amazonaws" % "aws-java-sdk-dynamodb" % "1.12.573" + lazy val awsSdkDynamoDbV1 = "com.amazonaws" % "aws-java-sdk-dynamodb" % "1.12.578" lazy val lambdaJavaCore = "com.amazonaws" % "aws-lambda-java-core" % "1.2.3" lazy val lambdaJavaEvents = "com.amazonaws" % "aws-lambda-java-events" % "3.11.3" lazy val ocitools = "uk.gov.nationalarchives.oci" % "oci-tools-scala_2.13" % "0.3.0" From 88072bc7243fcdd10e27a2733a8b6c21aa735a50 Mon Sep 17 00:00:00 2001 From: tna-digital-archiving-jenkins Date: Wed, 1 Nov 2023 08:21:44 +0000 Subject: [PATCH 5/5] Update dynamodb to 2.21.12 --- project/Dependencies.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/project/Dependencies.scala b/project/Dependencies.scala index d3f202d..a70e92a 100644 --- a/project/Dependencies.scala +++ b/project/Dependencies.scala @@ -6,7 +6,7 @@ object Dependencies { private val testContainersVersion = "0.41.0" lazy val scalaTest = "org.scalatest" %% "scalatest" % "3.2.17" - lazy val awsSdkDynamoDbV2 = "software.amazon.awssdk" % "dynamodb" % "2.21.7" + lazy val awsSdkDynamoDbV2 = "software.amazon.awssdk" % "dynamodb" % "2.21.12" lazy val awsSdkDynamoDbV1 = "com.amazonaws" % "aws-java-sdk-dynamodb" % "1.12.573" lazy val lambdaJavaCore = "com.amazonaws" % "aws-lambda-java-core" % "1.2.3" lazy val lambdaJavaEvents = "com.amazonaws" % "aws-lambda-java-events" % "3.11.3"