Skip to content

Commit

Permalink
Update to version v1.1.0
Browse files Browse the repository at this point in the history
  • Loading branch information
pwrmiller committed Nov 22, 2021
1 parent a21c75a commit 70270dc
Show file tree
Hide file tree
Showing 123 changed files with 4,400 additions and 506 deletions.
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Steps to reproduce the behavior.
A clear and concise description of what you expected to happen.

**Please complete the following information about the solution:**
- [ ] Version: [e.g. v1.0.0]
- [ ] Version: [e.g. v0.0.1]

To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "(SO0170) Maintaining Personalized Experiences with Machine Learning [...]".

Expand Down
10 changes: 6 additions & 4 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,9 @@ __pycache__/
# Generated test assets
source/infrastructure/tests/assets/*
!source/infrastructure/tests/assets/.keep
source/aws_lambda/get_next_scheduled_event/.gradle
source/aws_lambda/get_next_scheduled_event/build
source/aws_lambda/get_next_scheduled_event/.idea
source/scheduler/cdk/aws_solutions/scheduler/cdk/aws_lambda/get_next_scheduled_event/build
source/scheduler/cdk/aws_solutions/scheduler/cdk/aws_lambda/get_next_scheduled_event/.gradle
source/scheduler/cdk/aws_solutions/scheduler/cdk/aws_lambda/get_next_scheduled_event/.idea

# gradle build files
**/.gradle/*
Expand All @@ -60,4 +60,6 @@ source/aws_lambda/get_next_scheduled_event/.idea

# python build files
source/cdk_solution_helper_py/helpers_cdk/build/*
source/cdk_solution_helper_py/helpers_common/build/*
source/cdk_solution_helper_py/helpers_common/build/*
source/scheduler/common/build/*
source/scheduler/cdk/build/*
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,13 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.1.0] - 2021-11-22
### Added
- The solution now creates an Amazon EventBridge event bus, and puts messages to the bus when resources have been
created by the workflow. This can be useful when integrating with external systems.
- The solution now contains a command line interface (CLI) that allows schedule creation for existing resources in
Amazon Personalize.

## [1.0.1] - 2021-10-01
### Added
- The solution now exports the Amazon SNS Topic ARN as `SNSTopicArn`.
Expand Down
1 change: 1 addition & 0 deletions NOTICE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ pytest-env under the Massachusetts Institute of Technology (MIT) license
PyYAML under the Massachusetts Institute of Technology (MIT) license
requests under the Apache License Version 2.0
requests-mock under the Apache License Version 2.0
rich under the Massachusetts Institute of Technology (MIT) license
tenacity under the Apache License Version 2.0
quartz-scheduler under the Apache License Version 2.0

Expand Down
10 changes: 7 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,10 @@ The template includes the following components:
2. Perform solution FULL retraining on schedule (and update associated campaigns)
3. Perform solution UPDATE retraining on schedule (and update associated campaigns)
4. Create batch inference jobs
9. An Amazon EventBridge event bus, where resource status notification updates are posted throughout the AWS Step
functions workflow
10. A command line interface (CLI) lets existing resources be imported and allows schedules to be established for
resources that already exist in Amazon Personalize


**Note**: From v1.0.0, AWS CloudFormation template resources are created by the [AWS CDK](https://aws.amazon.com/cdk/)
Expand Down Expand Up @@ -287,9 +291,9 @@ To customize the solution, follow the steps below:
The following procedures assumes that all the OS-level configuration has been completed. They are:

* [AWS Command Line Interface](https://aws.amazon.com/cli/)
* [Python](https://www.python.org/) 3.7 or newer
* [Python](https://www.python.org/) 3.9 or newer
* [Node.js](https://nodejs.org/en/) 16.x or newer
* [AWS CDK](https://aws.amazon.com/cdk/) 1.95.2 or newer
* [AWS CDK](https://aws.amazon.com/cdk/) 1.126.0 or newer
* [Amazon Corretto OpenJDK](https://docs.aws.amazon.com/corretto/) 11

> **Please ensure you test the templates before updating any production deployments.**
Expand Down Expand Up @@ -360,7 +364,7 @@ build-s3-cdk-dist \
S3 bucket where the name is `<DIST_BUCKET_PREFIX>-<REGION_NAME>`. The solution's CloudFormation template will expect the
source code to be located in the bucket matching that name.
- `$SOLUTION_NAME` - The name of This solution (example: personalize-solution-customization)
- `$VERSION` - The version number to use (example: v1.0.1)
- `$VERSION` - The version number to use (example: v0.0.1)
- `$REGION_NAME` - The region name to use (example: us-east-1)

This will result in all global assets being pushed to the `DIST_BUCKET_PREFIX`, and all regional assets being pushed to
Expand Down
1 change: 1 addition & 0 deletions source/.coveragerc
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ source =
infrastructure
aws_lambda
cdk_solution_helper_py
scheduler

[report]
fail_under = 80.0
6 changes: 6 additions & 0 deletions source/aws_lambda/create_batch_inference_job/handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,12 @@
"default": "omit",
"as": "seconds",
},
"timeStarted": {
"source": "event",
"path": "workflowConfig.timeStarted",
"default": "omit",
"as": "iso8601",
},
},
)
def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:
Expand Down
8 changes: 7 additions & 1 deletion source/aws_lambda/create_campaign/handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,14 @@
"default": "omit",
"as": "seconds",
},
"timeStarted": {
"source": "event",
"path": "workflowConfig.timeStarted",
"default": "omit",
"as": "iso8601",
},
},
status="campaign.status",
status="campaign.latestCampaignUpdate.status || campaign.status",
)
def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:
"""Create a campaign in Amazon Personalize based on the configuration in `event`
Expand Down
File renamed without changes.
42 changes: 42 additions & 0 deletions source/aws_lambda/create_config/handler.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# ######################################################################################################################
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. #
# #
# Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance #
# with the License. You may obtain a copy of the License at #
# #
# http://www.apache.org/licenses/LICENSE-2.0 #
# #
# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed #
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for #
# the specific language governing permissions and limitations under the License. #
# ######################################################################################################################
import json
import os

from aws_lambda_powertools import Logger, Tracer, Metrics
from aws_lambda_powertools.metrics import MetricUnit
from aws_lambda_powertools.utilities.data_classes import S3Event

from shared.personalize.service_model import ServiceModel
from shared.personalize_service import Personalize


logger = Logger()
tracer = Tracer()
metrics = Metrics()


@metrics.log_metrics
@tracer.capture_lambda_handler
def lambda_handler(event, context):
"""Generate and return a solution configuration file derived from the properties of a dataset group
:param dict event: AWS Lambda Event (in this case, the dataset group and schedules)
:param context: AWS Lambda Context
:return: Dict
"""
dataset_group_name = event["datasetGroupName"]
schedules = event.get("schedules")

cli = Personalize()
model = ServiceModel(cli, dataset_group_name=dataset_group_name)
return model.get_config(dataset_group_name=dataset_group_name, schedules=schedules)
14 changes: 10 additions & 4 deletions source/aws_lambda/create_dataset/handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,17 +30,23 @@
config={
"name": {
"source": "event",
"path": "name",
"path": "serviceConfig.name",
},
"datasetType": {
"source": "event",
"path": "datasetType",
"path": "serviceConfig.datasetType",
},
"datasetGroupArn": {
"source": "event",
"path": "datasetGroupArn",
"path": "serviceConfig.datasetGroupArn",
},
"schemaArn": {"source": "event", "path": "serviceConfig.schemaArn"},
"timeStarted": {
"source": "event",
"path": "workflowConfig.timeStarted",
"default": "omit",
"as": "iso8601",
},
"schemaArn": {"source": "event", "path": "schemaArn"},
},
)
def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:
Expand Down
6 changes: 6 additions & 0 deletions source/aws_lambda/create_dataset_group/handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,12 @@
"path": "KMS_KEY_ARN",
"default": "omit",
},
"timeStarted": {
"source": "event",
"path": "workflowConfig.timeStarted",
"default": "omit",
"as": "iso8601",
},
},
)
def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:
Expand Down
6 changes: 6 additions & 0 deletions source/aws_lambda/create_dataset_import_job/handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,12 @@
"default": "omit",
"as": "seconds",
},
"timeStarted": {
"source": "event",
"path": "workflowConfig.timeStarted",
"default": "omit",
"as": "iso8601",
},
},
)
def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:
Expand Down
10 changes: 8 additions & 2 deletions source/aws_lambda/create_event_tracker/handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,17 @@
config={
"name": {
"source": "event",
"path": "name",
"path": "serviceConfig.name",
},
"datasetGroupArn": {
"source": "event",
"path": "datasetGroupArn",
"path": "serviceConfig.datasetGroupArn",
},
"timeStarted": {
"source": "event",
"path": "workflowConfig.timeStarted",
"default": "omit",
"as": "iso8601",
},
},
)
Expand Down
6 changes: 6 additions & 0 deletions source/aws_lambda/create_filter/handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,12 @@
"source": "event",
"path": "serviceConfig.filterExpression",
},
"timeStarted": {
"source": "event",
"path": "workflowConfig.timeStarted",
"default": "omit",
"as": "iso8601",
},
},
)
def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:
Expand Down
6 changes: 6 additions & 0 deletions source/aws_lambda/create_solution/handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,12 @@
"path": "serviceConfig.solutionConfig",
"default": "omit",
},
"timeStarted": {
"source": "event",
"path": "workflowConfig.timeStarted",
"default": "omit",
"as": "iso8601",
},
},
)
def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:
Expand Down
6 changes: 6 additions & 0 deletions source/aws_lambda/create_solution_version/handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,12 @@
"path": "workflowConfig.solutionVersionArn",
"default": "omit",
},
"timeStarted": {
"source": "event",
"path": "workflowConfig.timeStarted",
"default": "omit",
"as": "iso8601",
},
},
)
def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:
Expand Down
File renamed without changes.
33 changes: 33 additions & 0 deletions source/aws_lambda/prepare_input/handler.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# ######################################################################################################################
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. #
# #
# Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance #
# with the License. You may obtain a copy of the License at #
# #
# http://www.apache.org/licenses/LICENSE-2.0 #
# #
# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed #
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for #
# the specific language governing permissions and limitations under the License. #
# ######################################################################################################################

from typing import Dict, Any

from aws_lambda_powertools import Logger, Tracer, Metrics
from aws_lambda_powertools.utilities.typing import LambdaContext

from shared.sfn_middleware import set_workflow_config

logger = Logger()
tracer = Tracer()
metrics = Metrics()


def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:
"""Add timeStarted to the workflowConfig of all items
:param event: AWS Lambda Event
:param context: AWS Lambda Context
:return: the modified input
"""
config = set_workflow_config(event)
return config
78 changes: 78 additions & 0 deletions source/aws_lambda/shared/events.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# ######################################################################################################################
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. #
# #
# Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance #
# with the License. You may obtain a copy of the License at #
# #
# http://www.apache.org/licenses/LICENSE-2.0 #
# #
# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed #
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for #
# the specific language governing permissions and limitations under the License. #
# ######################################################################################################################
from datetime import datetime
from typing import Dict, Optional

from aws_lambda_powertools import Logger

from shared.exceptions import (
NotificationError,
SolutionVersionPending,
)
from shared.notifiers import NotifyEventBridge
from shared.resource import Resource

logger = Logger()


NOTIFY_LIST = [NotifyEventBridge()]


class Notifies:
"""Decorates a resource creation or describe call to provide event notifications"""

def __init__(self, status: str):
self.status = status

def __call__(self, function):
def wrapper(caller, resource: Resource, **kwargs):
try:
result = function(caller, resource, **kwargs)
except SolutionVersionPending as exc:
# because of how solution versions are handled, we must manually notify and re-raise
self.notify(
resource=resource,
result={
"solutionVersionArn": str(exc),
"status": "CREATE IN_PROGRESS",
},
cutoff=None,
)
raise exc

# run the notifier
cutoff = kwargs.get("timeStarted")
self.notify(resource, result, cutoff)

return result

return wrapper

def notify(
self, resource: Resource, result: Dict, cutoff: Optional[datetime]
) -> None:
"""
Notify each target in the NOTIFY_LIST
:param resource: the subject of the notification
:param result: the description of the subject of the notification
:param cutoff: the cutoff datetime for notifications (UTC required, timezone aware)
:return: None
"""
for notifier in NOTIFY_LIST:
notifier.set_cutoff(cutoff)
try:
notifier.notify(self.status, resource, result)
except NotificationError as exc:
logger.error(
f"notifier {notifier.name} failed: {str(exc)}"
) # log and continue through notifiers
Loading

0 comments on commit 70270dc

Please sign in to comment.