Skip to content

Latest commit

 

History

History
95 lines (77 loc) · 3.75 KB

README.md

File metadata and controls

95 lines (77 loc) · 3.75 KB

WIBL AWS processing setup instructions and usage

The following instructions describe how to setup WIBL processing lambdas and WIBL manager web service in AWS.

Setup

1. Edit configuration

Edit configuration-parameters.sh as needed. The main thing you will probably need to do change is the DCDB_PROVIER_ID variable. Once you do this, make sure your DCDB authorization key is placed in a file named ingest-external-${DCDB_PROVIDER_ID}.txt in the same directory as configuration-parameters.sh.

2. Create S3 buckets

Run configure-buckets.sh to create S3 buckets for incoming WIBL files and staging/outgoing GeoJSON files:

$ ./configure-buckets.sh

3. Create SNS topics

Run configure-sns.sh to create SNS topics that will be used to orchestrate WIBL file processing lambdas:

$ ./configure-sns.sh

4. Configure WIBL manager ECS service

Run configure-manager-ecs.sh to create WIBL manager ECS service. This will also create all VPC resources needed to run WIBL processing lambdas:

$ ./configure-manager-ecs.sh

5. Configure WIBL processing lambdas

Run configure-lambdas.sh to create WIBL processing lambdas and SNS subscriptions needed to trigger them:

$ ./configure-lambdas.sh

Update WIBL processing lambda code

Run update-lambda-code.sh to update code for the WIBL processing lambdas after they have been created:

$ ./update-lambda-code.sh

6. Configure test EC2 instances (Optional)

Run configure-ec2-test-instance.sh to create EC2 instances on the private and public subnet of the WIBL VPC. These EC2 instances can use useful for testing and debugging WIBL manager and processing service lambdas:

$ ./configure-ec2-test-instance.sh

Usage

  1. First upload a WIBL file into the WIBL incoming S3 bucket, for example a WIBL file named 6FF5B14E-65F3-4BFD-85D1-50F8CA02BC0B.wibl.

  2. Then trigger WIBL file conversion by issuing an HTTPS request to the conversion start lambda:

curl -v \
'https://fdjofdjofewjofjo5546jogfljfdlmfl3368.lambda-url.us-east-2.on.aws/' \
-H 'Content-Type: application/json' \
-d '{"object": "6FF5B14E-65F3-4BFD-85D1-50F8CA02BC0B.wibl"}'

Note the URL of the conversion start lambda can be obtained from the $CONVERSION_START_URL environment variable within configure-lambdas.sh, or by running: echo "$(cat ${WIBL_BUILD_LOCATION}/create_url_lambda_conversion_start.json | jq -r '.FunctionUrl')".

This will result in the length of the file being returned via the HTTP response:

content-length: 63641

Further, if successful, the conversion start lambda will publish a message to the conversion SNS topic, which will cause the conversion lambda to be invoked.

Running vizualization lambda

The vizualization lambda allows multiple GeoJSON files stored in $STAGING_BUCKET to be combined and vizualized over GEBCO data using Generic Mapping Tools and written to $VIZ_BUCKET. The script vizlambdainvoke.py shows how to invoke this lambda (make sure to set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables before running).

When invoking the visualisation lambda, you may get the following error:

botocore.errorfactory.ResourceNotReadyException: An error occurred (ResourceNotReadyException) when calling the Invoke operation (reached max retries: 4): Resources for the function are being restored.

This can happen if the lambda hasn't been run in a while. Try again after a few minutes.

Note: Any service wishing to invoke this lambda will need to setup a resources-based policy.