Merge pull request #9 from aws-solutions/feature/v1.1.0

Release candidate/v1.1.0

.github/ISSUE_TEMPLATE/bug_report.md

@@ -17,7 +17,7 @@ Steps to reproduce the behavior.
 A clear and concise description of what you expected to happen.
 
 **Please complete the following information about the solution:**
-- [ ] Version: [e.g. v1.0.0]
+- [ ] Version: [e.g. v0.0.1]
 
 To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "(SO0170) Maintaining Personalized Experiences with Machine Learning [...]".
 

.gitignore

@@ -48,9 +48,9 @@ __pycache__/
 # Generated test assets
 source/infrastructure/tests/assets/*
 !source/infrastructure/tests/assets/.keep
-source/aws_lambda/get_next_scheduled_event/.gradle
-source/aws_lambda/get_next_scheduled_event/build
-source/aws_lambda/get_next_scheduled_event/.idea
+source/scheduler/cdk/aws_solutions/scheduler/cdk/aws_lambda/get_next_scheduled_event/build
+source/scheduler/cdk/aws_solutions/scheduler/cdk/aws_lambda/get_next_scheduled_event/.gradle
+source/scheduler/cdk/aws_solutions/scheduler/cdk/aws_lambda/get_next_scheduled_event/.idea
 
 # gradle build files
 **/.gradle/*
@@ -60,4 +60,6 @@ source/aws_lambda/get_next_scheduled_event/.idea
 
 # python build files
 source/cdk_solution_helper_py/helpers_cdk/build/*
-source/cdk_solution_helper_py/helpers_common/build/*
+source/cdk_solution_helper_py/helpers_common/build/*
+source/scheduler/common/build/*
+source/scheduler/cdk/build/*

CHANGELOG.md

@@ -4,6 +4,13 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.1.0] - 2021-11-22
+### Added
+- The solution now creates an Amazon EventBridge event bus, and puts messages to the bus when resources have been
+created by the workflow. This can be useful when integrating with external systems. 
+- The solution now contains a command line interface (CLI) that allows schedule creation for existing resources in
+Amazon Personalize. 
+
 ## [1.0.1] - 2021-10-01
 ### Added
 - The solution now exports the Amazon SNS Topic ARN as `SNSTopicArn`.

NOTICE.txt

@@ -34,6 +34,7 @@ pytest-env under the Massachusetts Institute of Technology (MIT) license
 PyYAML under the Massachusetts Institute of Technology (MIT) license
 requests under the Apache License Version 2.0
 requests-mock under the Apache License Version 2.0
+rich under the Massachusetts Institute of Technology (MIT) license
 tenacity under the Apache License Version 2.0
 quartz-scheduler under the Apache License Version 2.0
 

README.md

@@ -35,6 +35,10 @@ The template includes the following components:
     2. Perform solution FULL retraining on schedule (and update associated campaigns)
     3. Perform solution UPDATE retraining on schedule (and update associated campaigns)
     4. Create batch inference jobs
+9. An Amazon EventBridge event bus, where resource status notification updates are posted throughout the AWS Step
+functions workflow 
+10. A command line interface (CLI) lets existing resources be imported and allows schedules to be established for
+resources that already exist in Amazon Personalize
 
 
 **Note**: From v1.0.0, AWS CloudFormation template resources are created by the [AWS CDK](https://aws.amazon.com/cdk/) 
@@ -287,9 +291,9 @@ To customize the solution, follow the steps below:
 The following procedures assumes that all the OS-level configuration has been completed. They are:
 
 * [AWS Command Line Interface](https://aws.amazon.com/cli/)
-* [Python](https://www.python.org/) 3.7 or newer
+* [Python](https://www.python.org/) 3.9 or newer
 * [Node.js](https://nodejs.org/en/) 16.x or newer 
-* [AWS CDK](https://aws.amazon.com/cdk/) 1.95.2 or newer 
+* [AWS CDK](https://aws.amazon.com/cdk/) 1.126.0 or newer 
 * [Amazon Corretto OpenJDK](https://docs.aws.amazon.com/corretto/) 11 
 
 > **Please ensure you test the templates before updating any production deployments.**
@@ -360,7 +364,7 @@ build-s3-cdk-dist \
   S3 bucket where the name is `<DIST_BUCKET_PREFIX>-<REGION_NAME>`. The solution's CloudFormation template will expect the
   source code to be located in the bucket matching that name.
 - `$SOLUTION_NAME` - The name of This solution (example: personalize-solution-customization)
-- `$VERSION` - The version number to use (example: v1.0.1)
+- `$VERSION` - The version number to use (example: v0.0.1)
 - `$REGION_NAME` - The region name to use (example: us-east-1)
 
 This will result in all global assets being pushed to the `DIST_BUCKET_PREFIX`, and all regional assets being pushed to 

source/.coveragerc

@@ -7,6 +7,7 @@ source =
     infrastructure
     aws_lambda
     cdk_solution_helper_py
+    scheduler
 
 [report]
 fail_under = 80.0

source/aws_lambda/create_batch_inference_job/handler.py

@@ -64,6 +64,12 @@
             "default": "omit",
             "as": "seconds",
         },
+        "timeStarted": {
+            "source": "event",
+            "path": "workflowConfig.timeStarted",
+            "default": "omit",
+            "as": "iso8601",
+        },
     },
 )
 def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:

source/aws_lambda/create_campaign/handler.py

@@ -52,8 +52,14 @@
             "default": "omit",
             "as": "seconds",
         },
+        "timeStarted": {
+            "source": "event",
+            "path": "workflowConfig.timeStarted",
+            "default": "omit",
+            "as": "iso8601",
+        },
     },
-    status="campaign.status",
+    status="campaign.latestCampaignUpdate.status || campaign.status",
 )
 def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:
     """Create a campaign in Amazon Personalize based on the configuration in `event`

source/aws_lambda/create_config/handler.py

@@ -0,0 +1,42 @@
+# ######################################################################################################################
+#  Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.                                                  #
+#                                                                                                                      #
+#  Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance      #
+#  with the License. You may obtain a copy of the License at                                                           #
+#                                                                                                                      #
+#   http://www.apache.org/licenses/LICENSE-2.0                                                                         #
+#                                                                                                                      #
+#  Unless required by applicable law or agreed to in writing, software distributed under the License is distributed    #
+#  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for   #
+#  the specific language governing permissions and limitations under the License.                                      #
+# ######################################################################################################################
+import json
+import os
+
+from aws_lambda_powertools import Logger, Tracer, Metrics
+from aws_lambda_powertools.metrics import MetricUnit
+from aws_lambda_powertools.utilities.data_classes import S3Event
+
+from shared.personalize.service_model import ServiceModel
+from shared.personalize_service import Personalize
+
+
+logger = Logger()
+tracer = Tracer()
+metrics = Metrics()
+
+
+@metrics.log_metrics
+@tracer.capture_lambda_handler
+def lambda_handler(event, context):
+    """Generate and return a solution configuration file derived from the properties of a dataset group
+    :param dict event: AWS Lambda Event (in this case, the dataset group and schedules)
+    :param context: AWS Lambda Context
+    :return: Dict
+    """
+    dataset_group_name = event["datasetGroupName"]
+    schedules = event.get("schedules")
+
+    cli = Personalize()
+    model = ServiceModel(cli, dataset_group_name=dataset_group_name)
+    return model.get_config(dataset_group_name=dataset_group_name, schedules=schedules)

source/aws_lambda/create_dataset/handler.py

@@ -30,17 +30,23 @@
     config={
         "name": {
             "source": "event",
-            "path": "name",
+            "path": "serviceConfig.name",
         },
         "datasetType": {
             "source": "event",
-            "path": "datasetType",
+            "path": "serviceConfig.datasetType",
         },
         "datasetGroupArn": {
             "source": "event",
-            "path": "datasetGroupArn",
+            "path": "serviceConfig.datasetGroupArn",
+        },
+        "schemaArn": {"source": "event", "path": "serviceConfig.schemaArn"},
+        "timeStarted": {
+            "source": "event",
+            "path": "workflowConfig.timeStarted",
+            "default": "omit",
+            "as": "iso8601",
         },
-        "schemaArn": {"source": "event", "path": "schemaArn"},
     },
 )
 def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:

source/aws_lambda/create_dataset_group/handler.py

@@ -43,6 +43,12 @@
             "path": "KMS_KEY_ARN",
             "default": "omit",
         },
+        "timeStarted": {
+            "source": "event",
+            "path": "workflowConfig.timeStarted",
+            "default": "omit",
+            "as": "iso8601",
+        },
     },
 )
 def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:

source/aws_lambda/create_dataset_import_job/handler.py

@@ -48,6 +48,12 @@
             "default": "omit",
             "as": "seconds",
         },
+        "timeStarted": {
+            "source": "event",
+            "path": "workflowConfig.timeStarted",
+            "default": "omit",
+            "as": "iso8601",
+        },
     },
 )
 def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:

source/aws_lambda/create_event_tracker/handler.py

@@ -31,11 +31,17 @@
     config={
         "name": {
             "source": "event",
-            "path": "name",
+            "path": "serviceConfig.name",
         },
         "datasetGroupArn": {
             "source": "event",
-            "path": "datasetGroupArn",
+            "path": "serviceConfig.datasetGroupArn",
+        },
+        "timeStarted": {
+            "source": "event",
+            "path": "workflowConfig.timeStarted",
+            "default": "omit",
+            "as": "iso8601",
         },
     },
 )

source/aws_lambda/create_filter/handler.py

@@ -41,6 +41,12 @@
             "source": "event",
             "path": "serviceConfig.filterExpression",
         },
+        "timeStarted": {
+            "source": "event",
+            "path": "workflowConfig.timeStarted",
+            "default": "omit",
+            "as": "iso8601",
+        },
     },
 )
 def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:

source/aws_lambda/create_solution/handler.py

@@ -62,6 +62,12 @@
             "path": "serviceConfig.solutionConfig",
             "default": "omit",
         },
+        "timeStarted": {
+            "source": "event",
+            "path": "workflowConfig.timeStarted",
+            "default": "omit",
+            "as": "iso8601",
+        },
     },
 )
 def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:

source/aws_lambda/create_solution_version/handler.py

@@ -49,6 +49,12 @@
             "path": "workflowConfig.solutionVersionArn",
             "default": "omit",
         },
+        "timeStarted": {
+            "source": "event",
+            "path": "workflowConfig.timeStarted",
+            "default": "omit",
+            "as": "iso8601",
+        },
     },
 )
 def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:

source/aws_lambda/prepare_input/handler.py

@@ -0,0 +1,33 @@
+# ######################################################################################################################
+#  Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.                                                  #
+#                                                                                                                      #
+#  Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance      #
+#  with the License. You may obtain a copy of the License at                                                           #
+#                                                                                                                      #
+#   http://www.apache.org/licenses/LICENSE-2.0                                                                         #
+#                                                                                                                      #
+#  Unless required by applicable law or agreed to in writing, software distributed under the License is distributed    #
+#  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for   #
+#  the specific language governing permissions and limitations under the License.                                      #
+# ######################################################################################################################
+
+from typing import Dict, Any
+
+from aws_lambda_powertools import Logger, Tracer, Metrics
+from aws_lambda_powertools.utilities.typing import LambdaContext
+
+from shared.sfn_middleware import set_workflow_config
+
+logger = Logger()
+tracer = Tracer()
+metrics = Metrics()
+
+
+def lambda_handler(event: Dict[str, Any], context: LambdaContext) -> Dict:
+    """Add timeStarted to the workflowConfig of all items
+    :param event: AWS Lambda Event
+    :param context: AWS Lambda Context
+    :return: the modified input
+    """
+    config = set_workflow_config(event)
+    return config

source/aws_lambda/shared/events.py

@@ -0,0 +1,78 @@
+# ######################################################################################################################
+#  Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.                                                  #
+#                                                                                                                      #
+#  Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance      #
+#  with the License. You may obtain a copy of the License at                                                           #
+#                                                                                                                      #
+#   http://www.apache.org/licenses/LICENSE-2.0                                                                         #
+#                                                                                                                      #
+#  Unless required by applicable law or agreed to in writing, software distributed under the License is distributed    #
+#  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for   #
+#  the specific language governing permissions and limitations under the License.                                      #
+# ######################################################################################################################
+from datetime import datetime
+from typing import Dict, Optional
+
+from aws_lambda_powertools import Logger
+
+from shared.exceptions import (
+    NotificationError,
+    SolutionVersionPending,
+)
+from shared.notifiers import NotifyEventBridge
+from shared.resource import Resource
+
+logger = Logger()
+
+
+NOTIFY_LIST = [NotifyEventBridge()]
+
+
+class Notifies:
+    """Decorates a resource creation or describe call to provide event notifications"""
+
+    def __init__(self, status: str):
+        self.status = status
+
+    def __call__(self, function):
+        def wrapper(caller, resource: Resource, **kwargs):
+            try:
+                result = function(caller, resource, **kwargs)
+            except SolutionVersionPending as exc:
+                # because of how solution versions are handled, we must manually notify and re-raise
+                self.notify(
+                    resource=resource,
+                    result={
+                        "solutionVersionArn": str(exc),
+                        "status": "CREATE IN_PROGRESS",
+                    },
+                    cutoff=None,
+                )
+                raise exc
+
+            # run the notifier
+            cutoff = kwargs.get("timeStarted")
+            self.notify(resource, result, cutoff)
+
+            return result
+
+        return wrapper
+
+    def notify(
+        self, resource: Resource, result: Dict, cutoff: Optional[datetime]
+    ) -> None:
+        """
+        Notify each target in the NOTIFY_LIST
+        :param resource: the subject of the notification
+        :param result: the description of the subject of the notification
+        :param cutoff: the cutoff datetime for notifications (UTC required, timezone aware)
+        :return: None
+        """
+        for notifier in NOTIFY_LIST:
+            notifier.set_cutoff(cutoff)
+            try:
+                notifier.notify(self.status, resource, result)
+            except NotificationError as exc:
+                logger.error(
+                    f"notifier {notifier.name} failed: {str(exc)}"
+                )  # log and continue through notifiers