diff --git a/.vscode/spellright.dict b/.vscode/spellright.dict index 75658e5d6..16e79590d 100644 --- a/.vscode/spellright.dict +++ b/.vscode/spellright.dict @@ -25,6 +25,8 @@ arn qid Nutritionix 'qna' +Logstash +quniqueterms Node.js ap ctz @@ -33,7 +35,3 @@ gotanswer qnabotcontext nnn-nn-nnnn Conne -ssml -cardtitle -cardimageurl -buttonvalue diff --git a/README.md b/README.md index 38b88763a..7ba84ec5d 100644 --- a/README.md +++ b/README.md @@ -6,8 +6,15 @@ This repository contains code for the QnABot, described in the AWS AI blog post [“Creating a Question and Answer Bot with Amazon Lex and Amazon Alexa”](https://aws.amazon.com/blogs/ai/creating-a-question-and-answer-bot-with-amazon-lex-and-amazon-alexa/). -See the "Getting Started" to launch your own QnABot +See the "Getting Started" to launch your own QnABot. +4.7.0 QnABot now supports LexV2 with voice interaction in multiple languages. + - Two installation/update modes are now available: + - (i) LexV1 + LexV2 (default, recommended for most AWS regions. + - (ii) LexV2-only (currently recommended for AWS regions where LexV1 is not available). + +**See all the new features in 4.7.0** [LexV2 support, Excel Import, New Canada Central region](#new-features) + 4.6.0 provides a number of new features described below. Several to call attention to are the following: - Kendra custom no_hits item required in earlier releases is no longer required to enable Kendra Fallback and should be removed, configurable confidence thresholds now available for filtering Kendra results. @@ -18,22 +25,6 @@ See the "Getting Started" to launch your own QnABot **New features in 4.5.0** [Kendra Web Crawler, Comprehend PII Detection, Translate Custom Terminology, Increased deployment regions](#new-features) -**New features in 4.4.0** [Preview version of VPC Deployment Support, Preview version of BotRouter, Upgrade to ES 7.9, Slack client detection and Markdown Support](#new-features) - -**New features in 4.3.0** [Connect Wizard to assist in Connect / Lex / QnABot use case, Security enhancement in API Gateway, Four node elastic search cluster support](#new-features) - -**New features in 4.2.0** [Beta Kendra FAQ Support, Bug fixes, Multiple document chaining, Repeat question, Elastic Search 7.7 upgrade](#new-features) - -**New features in 4.1.0** [Encryption at rest for S3 and Elastic Search Cluster, option to require Cognito user pool authorization to access embedded web UI, enhanced Kendra integration, enhanced Connect integration, and others](#new-features) - -**New features in 4.0.0** [Update to Elasticsearch 7.4, improved question matching accuracy, fuzzy matching, new multi-language support debug setting, SSML for Amazon Connect, improved Kendra integration, full upgrade support](#new-features) - -**New features in 3.0.3** [New content tuning Readme, Enhanced CFN Lex Resource to work with manually created Bot versions](#new-features) - -**New features in 3.0.2** [New Elicit Response Bots, Lambda Functions backing conditional chaining, Lex Bot versioning use](#new-features) - -**New features in 3.0.0** [ElicitResponse, Conditional Chaining, new Launch regions](#new-features) - ## Upgrade Notes During an upgrade, we recommend that existing QnABot content first be exported and downloaded from the Content Designer prior to @@ -65,6 +56,7 @@ Click a button to launch QnABot CloudFormation stack in the desired region | Frankfurt | | | Singapore | | | Tokyo | | +| Canada Central | | ### Clone the git repo and build a version @@ -107,78 +99,6 @@ If you have an existing stack you can run the following to update your stack: npm run update ``` -## Components - -### CloudFormation Templates - -The CloudFormation test templates are in the templates/test folder. The current templates are: - -1. Master: the template contains all the resources for QnABot. -2. Public: this is a version of the Master template with less parameters, less outputs, and the bootstrap bucket hardcoded to the publicBucket in config.json -3. various templates in /templates/dev: needed for local testing of the lambda functions. - -Run a template test with: - -```shell -npm run stack test/{template-name} -``` - -For example, if you want to test the domain template run: - -```shell -npm run stack test/domain -``` - -To understand the command more run: - -```shell -npm run stack -h -``` - -You also can check a template's syntax with: - -```shell -npm run check {template-name} -``` - -ex. - -```shell -npm run check domain -``` - -To understand the command more run: - -```shell -npm check stack -h -``` - -### Lambda Functions - -Lambda functions are found in the /lambda directory. Refer to the README.md file in each directory for instructions on setting up a dev environment and testing. -[Fulfillment](lambda/fulfillment/README.md) -[CFN](lambda/handler/README.md) -[Lex-Build](lambda/lex-build/README.md) -[Import](lambda/import/README.md) - -### Web Interface - -The Designer UI and client UI code is in the /website directory. - -To Test the web ui, Launch a development master stack: - -```shell -npm run stack dev/master up -``` - -when that stack has finished run: - -```shell -cd ./website ; make dev -``` - -this will launch a running webpack process that will watch for changes to files and upload the changes to your running dev/master stack. - #### Designer UI Compatibility Currently the only browsers supported are: @@ -198,6 +118,26 @@ See the [LICENSE.md](LICENSE.md) file for details ## New features +### Version 4.7.0 + +- QnABot now supports LexV2 with voice interaction in multiple languages. + - Two installation/update modes are now available: + - (i) LexV1 + LexV2 (default, recommended for most AWS regions. + - (ii) LexV2-only (currently recommended for AWS regions where LexV1 is not available). + - LexV2 locales are specified via a new CloudFormation parameter + - The default locales are US English, US Spanish and Canadian French. +- The QnABot web client now uses LexV2 and supports dynamic bot locale selection from a new title bar menu. +- Custom LexV2 Elicit Response bots are now supported. The built-in response bots still use LexV1 and are + available only when QnABot is installed in LexV1+LexV2 mode. +- CloudFormation deployment is now available for Canada/Montreal region (LexV2-only mode). +- Amazon Connect integration in the Canada/Montreal region supports multiple voice languages using LexV2. +- The Content Designer 'Test All' feature now uses LexV2. +- Content Designer's "Rebuild Lex Bot" feature now rebuilds both LexV2 and LexV1 bots +- Non-English LexV2 bot locales are automatically generated with sample utterances translated from English questions using Amazon Translate. +- Content Designer's Import feature now supports Excel spreadsheets as well as the existing JSON format. +- QnABot's Elasticsearch cache is now automatically kept warm to improve query time consistency. +- Negative feedback (thumbs down) messages can now generate notifications (text, email, etc.) using Amazon SNS. + ### Version 4.6.0 - Kendra integration is now fully automated during install or update when the new default Kendra Index Id parameter is provided. @@ -232,347 +172,5 @@ See the [LICENSE.md](LICENSE.md) file for details if desired. - QnABot distribution regions now available for one click deployment have increased to 8 regions. These are Northern Virginia (us-east-1), Oregon (us-west-2), Ireland (eu-west-1), London (eu-west-2), Frankfurt (eu-central-1), Sydney (ap-southeast-2), Singapore (ap-southeast-1), and Tokyo (ap-northeast-1). -### Version 4.4.0 - -- Preview VPC support - [readme](./VPCSupportREADME.md) -- Preview BotRouter support - [readme](./BotRoutingREADME.md) -- Upgrade to Elasticsearch service version 7.9 -- Slack client support via Lex with Slack specific markdown support -- Added support for Alexa re-prompt functionality - -VPC support is enabled in beta mode through a new template available in the distribution repos. Please understand -the content in [readme](./VPCSupportREADME.md) before proceeding with this type of deployment. - -- artifacts/aws-ai-qna-bot/templates/public-vpc-support.json -This beta template exposes two new additional parameters that can be specified when deployed using the CloudFormation console. -These parameters are: -- VPCSubnetIdList -- VPCSecurityGroupIdList -As one might expect a set of SubnetIds and SecurityGroupIds need to be specified. Two private subnets with appropriate NAT -based gateway to public internet should be selected. The security group specified must allow at a minimum inbound -connectivity on port 443. The Elasticsearch cluster and all Lambdas will be attached to these private subnets. The -Designer UI is still available outside of the VPC but requires login via the Cognito user pool. The Elasticsearch -cluster will not be available externally. Users wishing to use the Kibana console will need VPN connectivity to the -VPC and is outside the scope of this document. - -### Version 4.3.0 - -- New Connect Wizard available in the Content Designer UI to assist integration with a Connect Contact Flow. -- New 4-node Elasticsearch domain support for improved fault tolerance in deployment template. -- Elicit Response bot support for confirmation responses using phone keypad 1 = yes 2 = no. -- Security improvements in API Gateway. -- ID token values removed from session event after validation and redacted from logging. -- Setting to limit the number of Kendra fallback search results. -- Setting to enable signed URLs for S3 documents in Kendra search results. - -Provides the ability to deploy QnABot components within VPC infrastructure via a new template named public-vpc-support.json. -This template is made available for use as a separate installation mechanism. It is not the default template utilized in the -public distribution buckets. Please take great care in deploying QnABot in VPC. The Elasticsearch Cluster -becomes bound to the VPC as well as the Lambda's installed. The Elasticsearch cluster is no longer available -outside of the VPC. All Lambdas are bound to the VPC to allow communication with the cluster. - -The following limitations exist: - -- A QnABot deployed within VPC can NOT be modified to operated in non-VPC. -- A properly configured VPC with public/private subnets using proper Internet Gateway and Nat Gateway must be available. -- Two private subnets must be specified as parameters for the CloudFormation template. -- A Security Group allowing inbound port 443 from connections within the VPC CIDR block must be configured and specified -as a parameter for the CloudFormation template. -- The Kibana dashboard will only be available from clients that have access to the VPC. -- The VPC must be configured to allow access to Lex, S3, Lambda, DynamoDB, Systems Manager (SSM Parameter Store), Kendra, and -Comprehend. As more features are added to QnABot the required service access may increase. VPC Endpoints using PrivateLink -is not required but may be used if available. - -### Version 4.2.0 - -- New Kendra FAQ support (Beta version) using the setting KENDRA_FAQ_INDEX. New menu item in Designer UI to export Questions as a Kendra FAQ. See revised Blog Post for details. -- New GetSessionAttribute Handlebars helper to obtain session attribute. Works similar to lodash get(). Will not through exception and will return a default value. -- Enhanced handlebars to support string concatenation including handlebar 'variables' like Session Attributes and UserInfo, etc. Use case, e.g. to build a url containing a users email, eg a google calendar URL. Example of syntax now supported - in this case to dynamically build a personalized URL based on user info. `{{setSessionAttr 'link' 'https://calendar.google.com/calendar/embed?src=' UserInfo.Email '&ctz=America%2FNew_York'}}` -- Moved 'previous' and 'navigation' session attributes under a new 'qnabotcontext' session attribute so that Connect (and other) clients have fewer session attributes to preserve. -- Allows Chaining rule Lambda function to return a modified session object in addition to the string for chaining. -- Allows Chaining of up to 10 documents. Each document's Lambda hooks will also be invoked in sequence if defined. -- Added a new Repeat QID in the QNAUtility example package. Allows QnABot to easily repeat the last answer. -- Allow the chaining rule to specify a specific QID rather than an answer. A QID can be specified in the chaining rule by using string such as QID:: e.g. QID::Admin.001. Note, the new QID:: syntax can also be used from the webUI, say as button values if/when you prefer to target a specific QID (exact query) rather than rely on question matching. -- Fixed a defect to allow conditional chaining to be invoked after an elicit response bot failure. -- Upgrades to and installs ElasticSearch 7.7. - -### Version 4.1.0 - -- Install / Upgrade now supports the option to configure S3 Buckets and Elastic Search cluster using encryption at rest -- Install / Upgrade now supports the option to require Cognito based user authorization to access the built-in full screen web UI (Public/Private parameter in template) - Public is the default -- Added two settings parameters to enforce user identity verification check, so that bot can be secured for use by authenticated users only - - ENFORCE_VERIFIED_IDENTITY. Default is false. Set to true to make QnABot require verified identity from client - - NO_VERIFIED_IDENTITY_QUESTION. The default is "no_verified_identity". If user identity cannot be verified, replace question string with this. If not verified, the system will respond to user's question with the result of searching for NO_VERIFIED_IDENTITY_QUESTION. This allows a customizable message which informs the user that they must log in. A default question with qid "no_verified_identity" is included in QNAUtility example package. -- Enhanced Kendra fallback integration to use a specific answer if there is a best answer available and bold face highlighted words from Kendra response -- Added Comprehend sentiment analysis to all utterances and text captured by the QNAFreeText elicit response bot -- Enhanced Kibana dashboard to identify Lex client channels - Connect, Web, SMS -- Improved internal use of Booleans from settings configuration -- Enhanced Connect integration - - Added session attribute named "qnabot_qid" that holds the matching question id found in elastic search - - Added session attribute "qnabot_gotanswer" that holds boolean true/false if an answer was found - - Encapsulating all Kendra and Elicit Response Bot session attributes into a single "qnabotcontext" attribute making it easier to store and reset in Connect contact flow -- Added new QNAYesNoExit elicit response bot which allows a user to exit the YesNoExit question using "exit", "bye", "quit", "admin", "rep","representative","stop", "help", "bye", "goodbye" which sets the Yes_No_Exit slot value / session attribute to "Exit". -- Update to 0.17.0 of embedded lex-web-ui -- Resolved additional dependabot identified security issues with dependent packages -- Fixed lambda/fulfillment unit tests -- Fixed defect where response bot was not triggered on next question when using lambda function for conditional chaining - -### Version 4.0.0 - -- Update to Elasticsearch 7.4 -- Improved question matching accuracy and tuning -- Tolerance for typos and minor spelling errors with fuzzy matching setting -- Easier troubleshooting when using voice or multi-language support with new debug setting -- SSML support when using Amazon Connect -- Improvements to Amazon Kendra integration -- Full upgrade support without data loss when upgrading from previous versions. - -### Content Tuning and Accuracy Guide - -Content Tuning an and Accuracy Guide now available as a Markdown Readme. [README](tuning_accuracy_guide/AWS_QnABot_tuning_recognition_accuracy_guide.md) - -### Optional Redact feature for log and metric output - -QnABot can be configured to redact information written to CloudWatch logs, S3 metrics, and Kibana metrics logs. -This feature is disabled by default. Use the Designer UI Settings form to enable this feature. One can configure -the RegEx applied to strings as they are logged. If RegEx matches are found, the match is replaced with the string -'XXXXXX'. - -The initial RegEx is - -```regex -\b\d{4}\b(?![-])|\b\d{9}\b|\b\d{3}-\d{2}-\d{4}\b -``` - -This replaces 4 digit numbers not followed by a hyphen, a 9 digit number (SSN without hyphens), and a typical -SSN using nnn-nn-nnnn syntax with hyphens. - -### New Connect Callback Example - -New example demonstrating how QnABot can be asked by a user for a live agent based phone callback. The -implementation provides a new LambdaHook example as well as four sample questions that ask a user for -their name and phone number prior to handing off to an Amazon Connect instance to initiate the callback. - -**Two configuration updates are required to use this example with Amazon Connect.** - -The IAM Role/Policy used by the ConnectCallback Lambda must include a new policy that allows -the action "connect:StartOutboundVoiceContact" to be used with the resource -`"arn:aws:connect:*:*:instance//*"`. The following is an example of this policy - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "VisualEditor0", - "Effect": "Allow", - "Action": "connect:StartOutboundVoiceContact", - "Resource": "arn:aws:connect:*:*:instance//*" - } - ] -} -``` - -1) Find the Lambda ConnectCallback Function in the AWS Lambda Console -2) Open the AWS Console and select the Lambda Service -3) In the Console's filter enter 'ConnectCallback' and press enter -4) The displayed function will start with `-ExamplePYTHONLambdaConne...` If you have multiple QnABot stacks -installed you'll see multiple functions listed -5) Open the Lambda function by clicking on the function -6) Select the Permissions tab -7) Click on the Role name to open this Role in a new tab -8) Click on + Add inline policy -9) Select the JSON tab -10) Copy the sample text above, paste as JSON, and change `` to the Instance ID identified in the Connect Console. -11) Click on Review policy -12) Enter a name for the policy and click Create policy - -You've now enabled Lambda functions using this role to start outbound calls via the connect instance - -Lambda Hook Arguments need to be updated. Before being used, the item with qid CONNECT_TO_AGENT.04 -should have its Arguments field adjusted to reflect identifiers from the Connect instance: - -```bash -"AWS_connect_instance_id": "", -"AWS_connect_contact_flow_id": "", -"AWS_connect_queue_id": "", -``` - -Once these configuration changes are in place, QnABot can be successfully use Amazon Connect to place -outbound calls. - -### Conditional Chaining now supports Lambda Function - -You can specify that a Conditional Chaining rule runs a Lambda Function using the "Lambda::FunctionName" syntax. -The Lambda function name must start with "QNA". The Lambda will receive arguments the same as Lambda Hooks. The -function must return a String that is used to resolve the next item just the same as other conditional chaining rules. - -### Improved Scale - -This release of QnABot utilizes Lex Bot aliases and Lex versioning. The QnA Bot and all ElicitResponse Bots are -now installed using the alias 'live'. 'live' points at the latest numbered Bot version. All Lex resources are now -versioned starting with '1' after initial installation. - -### Additional Elicit Response Bots - -This release adds several elicit response bots for -QNAWage - 7 digit wage value - returns slot named "Wage" -QNASocialSecurity - SSN Numbers in the nnn-nn-nnnn format - returns slot named "SSN" -QNAPin - 4 digit pin value - Returns slot named "Pin" - -### Configuring QnABot to ask the questions - -QnABot was designed to answer questions, but now it can also ask questions and process -the user’s answers. Use this feature for data collection and validation; -or implement surveys, quizzes, personalized recommendations; or to build a triage -chatbot application. See the [blog post](https://aws.amazon.com/blogs/machine-learning/creating-a-question-and-answer-bot-with-amazon-lex-and-amazon-alexa/) for details about this new feature. -This feature is know as ElicitResponse from a configuration perspective. - -### Automatically advancing and branching along a tree of questions - -QnABot can now chain the user from one answer to another. This can be used with ElicitResponse -to ask multiple questions from a user. Conditional chaining allows branching based on -the state of session variables. Please see the [blog post](https://aws.amazon.com/blogs/machine-learning/creating-a-question-and-answer-bot-with-amazon-lex-and-amazon-alexa/) -for a description on how to use this feature. - -### Kendra Fallback Support - -QnABot version 2.6.0 optionally supports integration with Amazon Kendra as a fallback mechanism if a question/answer can not -be found in QnABot. - -**Important note. Use of Kendra as a fallback mechanism will incur additional charges for your AWS Account. Please -review the Kendra pricing structure. The fallback mechanism for QnABot can be useful when deploying Kendra as an -Enterprise search solution.** - -To enable this support for your Kendra indexes, use the Settings UI in the Designer and add your -index to the ALT_SEARCH_KENDRA_INDEXES parameter. This parameter takes an array of strings and uses the form below. - -```json -["a672e3a2-nnnn-nnnn-nnnn-7b3abc81c313"] -``` - -**Don't forget to use your Kendra Index ID rather than the one in the sample** -Next use the QnABot Designer UI to import a Sample/Extension named KendraFallback. - -This loads a new question with a qid of "KendraFallback". Edit this question in the Designer UI and change its question from "no_hits_alternative" to "no_hits" and save the changes. - -If you have previously loaded the QnAUtility.json from Examples/Extensions you need to either remove -the question with the ID "CustomNoMatches" or change the question for this ID from "no_hits" to "no_hits_original" - -Once the new question, "KendraFallback" is configured as the response for "no_hits", the Kendra index will be -searched for an answer whenever a curated answer can not be found. This feature provides a fallback mechanism -prior to telling the user an answer could not be found. - -A [workshop](https://github.com/aws-samples/aws-ai-qna-bot/tree/master/workshops/reinvent2019/readme.md) is available in GitHub. -that will walk you through setting up this feature. - -**Important note. Use of Kendra as a fallback mechanism will incur additional charges for your AWS Account. Please review the Kendra pricing structure. The fallback mechanism for QnABot can be useful when deploying Kendra as an Enterprise search solution.** - -### MultiLanguage Support - -QnABot version 2.6.0 supports use of multiple languages with these limitations: - -- MultiLanguage support for voice is limited to use with Alexa skills only. (Amazon Lex currently supports voice recognition in English only) -- MutiLanguage support for text is available via all text clients (e.g. Lex Web UI, SMS, etc.) - - ####Lex (text mode only): - -If the user enters a question in a language other than english, QnABot will attempt to return an answer in the other language. -It does this by using Amazon Comprehend to identify the language typed. If Comprehend can identify the language based on a configured minimum confidence, -QnABot will serve up content based on that locale. - -Users can also set a preferred language whereby QnABot will always attempt to respond with content in the chosen -locale. If the user sets the preferred language to be Spanish, QnABot will always try and serve up content using -Spanish when possible. - -#### Alexa (voice) - -You will need to add each language you want to use to your QnABot skill using the Alexa Developer console. The intent schema for each -language will be identical except for the skill invocation name. Give the skill a unique invocation name for each language that you add. - -QnABot will use the language setting provided by Alexa, and will attempt to respond in that language. - -#### How it works - -QnABot converts the question posed by the user to English, using Amazon Translate, and performs a lookup of the answer in Elastic Search -just as it normally does, using the English translation of the question. ElasticSearch searches are done in English only since QnABot documents -are indexed using the English text analyzer (stemming, stop words, etc.) - -To ensure good matching of translated questions, you can use the Amazon Translate console to see the English translation of your local language question. Use QnABot content designer -to ensure your QnA item has the right sample questions to ensure a match. - -Once it finds the question, QnABot will serve up the configured answer. - -You can use Handlebar blocks to define explicit answers in each different language you want to support. OR, if you do not -explicitly define an answer in the user's language, QnABot will automatically use Amazon Translate to convert the default English. -answer to the target language. - -#### Configuration - -By default this feature is disabled. Use the following three steps to enable and configure this feature. Step 1 enables the feature. Step 2 loads in two questions from this extension that allow the user to select a preferred language. The defaults supplied in this question are English, Spanish, French, German, and Italian. You can extend this list to -support other languages. - -Step 1) Enable multi language support - -a) QnABot uses a property named ENABLE_MULTI_LANGUAGE_SUPPORT, default value of "false". -You can change this setting using the Content Designer Settings page. Set it to "true" to enable multi language support. - -Step 2) Use the Designer UI to import the Sample/Extension named Language / Multiple Language Support. - -This will add two questions to the system: Language.000 and Language.001. When using Lex text clients, these questions will allow you to set your preferred language. -The preferred language, if set, will take precedence over -the auto detected language. - -_When using Alexa, the language is automatically set by the skill. You will not be able to override the preferred language when using Alexa._ - -Language.000 provides a question that allows the user to set the current sessions preferred output saying a simple word -such as French, German, or Spanish, or Italian. - -Language.001 resets the preferred language. This can be performed by saying or typing 'reset language' or 'detect language'. -You can also input using your language of choice assuming AWS Translate can translate the input back to English. - -Once you've imported this extension question try typing the question 'Spanish'. You should see a Spanish response. - -Next enter 'English' and you will have switched your preference back to English. - -Next enter 'reset language' and your preference will be reset and language auto detection will occur again. - -The answer for Language.000 uses the following handlebar syntax - -```handlebars -{{#setLang 'fr' false}} D'accord. J'ai défini votre langue préférée sur l'anglais. {{/setLang}} -{{#setLang 'es' false}} Okay. He configurado tu idioma preferido al inglés. {{/setLang}} -{{#setLang 'de' false}} In Ordnung. Ich habe Ihre bevorzugte Sprache auf Englisch eingestellt. {{/setLang}} -{{#setLang 'it' false}} Ok. Ho impostato la tua lingua preferita sull'inglese.{{/setLang}} -{{#setLang 'en' true}} Ok. I've set your preferred language to English. {{/setLang}} -``` - -The helper function setLang performs the necessary processing to set the preferred language/locale for the session. To -add support for other languages just extend the answer in Language.000 with additional locales. - -Step 3) In order to serve up content that is locale specific you can - -- allow QnABot to automatically translate your english answers to the session language using Amazon Translate. -- OR provide explicitly curated answers in QnA items, in multiple languages, using handlebars, as shown below. - -Lets modify the question sun.1. The following would be an example where the handlebar function ifLang is used to specify a response for Spanish. - -Use the handlebar template defaultLang to specify the response QnABot should provide when the language is unknown. By -default this is typically in English but could be in any language as needed. - -`{{#defaultLant}}{{/defaultLang}}` must be the last element in the answer block. - -```handlebars -{{#ifLang 'es'}} -Nuestro sol tiene 4.600 millones de años. Se considera una enana amarilla con un diámetro de 1,392,684 kilómetros y una circunferencia de 4,370,005 kilómetros. Tiene una masa que es igual a 333,060 tierras y una temperatura superficial de 5,500 grados centígrados. ¡Muy caliente! -{{/ifLang}} -{{#defaultLang}} -Our sun is 4.6 billion years old. Its considered a yellow dwarf with a diameter of 1,392,684 kilometers and a circumference of 4,370,005 kilometers. It has a mass that is equal to 333,060 earths and a surface temperature of 5,500 degrees celsius. Really Hot! -{{/defaultLang}} -``` - -The handlebar function ifLang takes locale as a quoted parameter. This tells QnABot which locale to associate with the subsequent -text. - A [workshop](https://github.com/aws-samples/aws-ai-qna-bot/tree/master/workshops/reinvent2019/readme.md) is available in GitHub that will walk you through setting up this feature. diff --git a/VPCSupportREADME.md b/VPCSupportREADME.md index 62cfb96e8..9d3a42349 100644 --- a/VPCSupportREADME.md +++ b/VPCSupportREADME.md @@ -1,14 +1,26 @@ # VPC Support - Preview Mode (version 1.0 - December 2020) -QnABot now provides in Preview Mode deployment using a VPC. +This feature allows deployment of QnABot components within VPC infrastructure via a new template named public-vpc-support.json. +This template is made available for use as a separate installation mechanism. It is not the default template utilized in the +public distribution buckets. Please take care in deploying QnABot in VPC. The Elasticsearch Cluster +becomes bound to the VPC as well as the Lambda's installed. The Elasticsearch cluster is no longer available +outside of the VPC. All Lambdas are bound to the VPC to allow communication with the cluster. + +Two additional parameters are required by this template. + +- VPCSubnetIdList (two private subnets spread over two availability zones - see below) +- VPCSecurityGroupIdList (see below) ### Requirements In order for deployment of resources attaching within a VPC two requirements must be met. -1) A fully functioning VPC with a minimum of two private subnets spread over two availability zones is required. In addition +1) A fully functioning VPC with a minimum of two private subnets spread over two availability zones is required. + These private VPC subnets should have access to AWS services. This can be accomplished using NAT Gateway with proper IGW + configuration / routing. Other third party gateway implementations can be used that provide access to AWS services. + 2) A pre-configured VPC security group that 1) allows inbound connections on port 443 from other addresses in the VPC CIDR block. For example, if the VPC's CIDR block is 10.178.0.0/16, inbound connections in the security @@ -55,4 +67,3 @@ VPN or Direct Connect to the VPC. * The API Gateway used by the Designer UI is still available publicly and access is still authorized using Cognito. The Lambda's backing the API will run within the VPC. - diff --git a/docs/connect_callback.md b/docs/connect_callback.md new file mode 100644 index 000000000..fd7bb0b61 --- /dev/null +++ b/docs/connect_callback.md @@ -0,0 +1,53 @@ +# New Connect Callback Example + +New example demonstrating how QnABot can be asked by a user for a live agent based phone callback. The +implementation provides a new LambdaHook example as well as four sample questions that ask a user for +their name and phone number prior to handing off to an Amazon Connect instance to initiate the callback. + +**Two configuration updates are required to use this example with Amazon Connect.** + +The IAM Role/Policy used by the ConnectCallback Lambda must include a new policy that allows +the action "connect:StartOutboundVoiceContact" to be used with the resource +`"arn:aws:connect:*:*:instance//*"`. The following is an example of this policy + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": "connect:StartOutboundVoiceContact", + "Resource": "arn:aws:connect:*:*:instance//*" + } + ] +} +``` + +1) Find the Lambda ConnectCallback Function in the AWS Lambda Console +2) Open the AWS Console and select the Lambda Service +3) In the Console's filter enter 'ConnectCallback' and press enter +4) The displayed function will start with `-ExamplePYTHONLambdaConne...` If you have multiple QnABot stacks +installed you'll see multiple functions listed +5) Open the Lambda function by clicking on the function +6) Select the Permissions tab +7) Click on the Role name to open this Role in a new tab +8) Click on + Add inline policy +9) Select the JSON tab +10) Copy the sample text above, paste as JSON, and change `` to the Instance ID identified in the Connect Console. +11) Click on Review policy +12) Enter a name for the policy and click Create policy + +You've now enabled Lambda functions using this role to start outbound calls via the connect instance + +Lambda Hook Arguments need to be updated. Before being used, the item with qid CONNECT_TO_AGENT.04 +should have its Arguments field adjusted to reflect identifiers from the Connect instance: + +```bash +"AWS_connect_instance_id": "", +"AWS_connect_contact_flow_id": "", +"AWS_connect_queue_id": "", +``` + +Once these configuration changes are in place, QnABot can be successfully use Amazon Connect to place +outbound calls. diff --git a/docs/multilanguage_support.md b/docs/multilanguage_support.md new file mode 100644 index 000000000..2d4fa3693 --- /dev/null +++ b/docs/multilanguage_support.md @@ -0,0 +1,105 @@ +# MultiLanguage Support + +QnABot version 2.6.0 supports use of multiple languages with these limitations: + +- MultiLanguage support for voice is limited to use with Alexa skills only. (Amazon Lex currently supports voice recognition in English only) +- MutiLanguage support for text is available via all text clients (e.g. Lex Web UI, SMS, etc.) + + ####Lex (text mode only): + +If the user enters a question in a language other than english, QnABot will attempt to return an answer in the other language. +It does this by using Amazon Comprehend to identify the language typed. If Comprehend can identify the language based on a configured minimum confidence, +QnABot will serve up content based on that locale. + +Users can also set a preferred language whereby QnABot will always attempt to respond with content in the chosen +locale. If the user sets the preferred language to be Spanish, QnABot will always try and serve up content using +Spanish when possible. + +## Alexa (voice) + +You will need to add each language you want to use to your QnABot skill using the Alexa Developer console. The intent schema for each +language will be identical except for the skill invocation name. Give the skill a unique invocation name for each language that you add. + +QnABot will use the language setting provided by Alexa, and will attempt to respond in that language. + +## How it works + +QnABot converts the question posed by the user to English, using Amazon Translate, and performs a lookup of the answer in Elastic Search +just as it normally does, using the English translation of the question. ElasticSearch searches are done in English only since QnABot documents +are indexed using the English text analyzer (stemming, stop words, etc.) + +To ensure good matching of translated questions, you can use the Amazon Translate console to see the English translation of your local language question. Use QnABot content designer +to ensure your QnA item has the right sample questions to ensure a match. + +Once it finds the question, QnABot will serve up the configured answer. + +You can use Handlebar blocks to define explicit answers in each different language you want to support. OR, if you do not +explicitly define an answer in the user's language, QnABot will automatically use Amazon Translate to convert the default English. +answer to the target language. + +## Configuration + +By default this feature is disabled. Use the following three steps to enable and configure this feature. Step 1 enables the feature. Step 2 loads in two questions from this extension that allow the user to select a preferred language. The defaults supplied in this question are English, Spanish, French, German, and Italian. You can extend this list to +support other languages. + +Step 1) Enable multi language support + +a) QnABot uses a property named ENABLE_MULTI_LANGUAGE_SUPPORT, default value of "false". +You can change this setting using the Content Designer Settings page. Set it to "true" to enable multi language support. + +Step 2) Use the Designer UI to import the Sample/Extension named Language / Multiple Language Support. + +This will add two questions to the system: Language.000 and Language.001. When using Lex text clients, these questions will allow you to set your preferred language. +The preferred language, if set, will take precedence over +the auto detected language. + +_When using Alexa, the language is automatically set by the skill. You will not be able to override the preferred language when using Alexa._ + +Language.000 provides a question that allows the user to set the current sessions preferred output saying a simple word +such as French, German, or Spanish, or Italian. + +Language.001 resets the preferred language. This can be performed by saying or typing 'reset language' or 'detect language'. +You can also input using your language of choice assuming AWS Translate can translate the input back to English. + +Once you've imported this extension question try typing the question 'Spanish'. You should see a Spanish response. + +Next enter 'English' and you will have switched your preference back to English. + +Next enter 'reset language' and your preference will be reset and language auto detection will occur again. + +The answer for Language.000 uses the following handlebar syntax + +```handlebars +{{#setLang 'fr' false}} D'accord. J'ai défini votre langue préférée sur l'anglais. {{/setLang}} +{{#setLang 'es' false}} Okay. He configurado tu idioma preferido al inglés. {{/setLang}} +{{#setLang 'de' false}} In Ordnung. Ich habe Ihre bevorzugte Sprache auf Englisch eingestellt. {{/setLang}} +{{#setLang 'it' false}} Ok. Ho impostato la tua lingua preferita sull'inglese.{{/setLang}} +{{#setLang 'en' true}} Ok. I've set your preferred language to English. {{/setLang}} +``` + +The helper function setLang performs the necessary processing to set the preferred language/locale for the session. To +add support for other languages just extend the answer in Language.000 with additional locales. + +Step 3) In order to serve up content that is locale specific you can + +- allow QnABot to automatically translate your english answers to the session language using Amazon Translate. +- OR provide explicitly curated answers in QnA items, in multiple languages, using handlebars, as shown below. + +Lets modify the question sun.1. The following would be an example where the handlebar function ifLang is used to specify a response for Spanish. + +Use the handlebar template defaultLang to specify the response QnABot should provide when the language is unknown. By +default this is typically in English but could be in any language as needed. + +`{{#defaultLant}}{{/defaultLang}}` must be the last element in the answer block. + +```handlebars +{{#ifLang 'es'}} +Nuestro sol tiene 4.600 millones de años. Se considera una enana amarilla con un diámetro de 1,392,684 kilómetros y una circunferencia de 4,370,005 kilómetros. Tiene una masa que es igual a 333,060 tierras y una temperatura superficial de 5,500 grados centígrados. ¡Muy caliente! +{{/ifLang}} +{{#defaultLang}} +Our sun is 4.6 billion years old. Its considered a yellow dwarf with a diameter of 1,392,684 kilometers and a circumference of 4,370,005 kilometers. It has a mass that is equal to 333,060 earths and a surface temperature of 5,500 degrees celsius. Really Hot! +{{/defaultLang}} +``` + +The handlebar function ifLang takes locale as a quoted parameter. This tells QnABot which locale to associate with the subsequent +text. \ No newline at end of file diff --git a/docs/overview/README.md b/docs/overview/README.md new file mode 100644 index 000000000..a78ffceac --- /dev/null +++ b/docs/overview/README.md @@ -0,0 +1,37 @@ +# Overview + +Chatbots are a great way to make information available for your users. With QnABot you can deploy a chatbot in just a few steps and have a fully functional chat experience setup in under an hour. + + +Once the solution is deployed, you have a QnABot designer console where you can build and manage your question and answer bank. The question and answer bank become your knowledge base and the main source of information for QnABot to interact with to provide users' with relevant answers. + +## Solution architecture and how It works + +Three key AWS services are at the core of the solution: + +- **Amazon Lex** is a service for building conversational interfaces into any application using voice and text. Amazon Lex provides the advanced deep learning functionalities of automatic speech recognition (ASR) for converting speech to text, and natural language understanding (NLU) to recognize the intent of the text to enable you to build applications with highly engaging user experiences and lifelike conversational interactions. +- **Amazon ElasticSearch** is an open-source search and analytics engine for use cases such as log analytics, real-time application monitoring, and clickstream analysis. Amazon ElasticSearch is a managed service that makes it simple to deploy, operate, and scale Elasticsearch clusters in the AWS Cloud. The service offers open-source Amazon ElasticSearch APIs, managed Kibana, and integrations with Logstash and other AWS services, enabling you to securely ingest data from any source and search, analyze, and visualize it in real time. +- **Amazon Kendra** is an intelligent search service powered by machine learning. Kendra reimagines enterprise search for your websites and applications so your employees and customers can easily find the content they are looking for, even when it’s scattered across multiple locations and content repositories within your organization. + +Let's take a closer look at these three services and how they help power +the QnABot solution. + +![Solution architecture and data flow](image2.png) + +When you ask QnABot a question, a few things happen: + +1. The question gets processed and transcribed by Amazon Lex using a Natural Language Understanding (NLU) and Processing (NLP) engine. + - QnABot initially trains the NLP to match a wide variety of possible questions and statements, so that the Amazon Lex bot can accept just about any question a user might ask. The Amazon Lex interaction model is setup with: + - **Intents**: An intent represents an action that fulfills a user's spoken request. Intents can optionally have arguments called **slots**. The QnABot uses **slots** to capture user input and fulfills the Intent via Lambda function. + - **Sample utterances**: A set of likely spoken phrases mapped to the intents. This should include as many representative phrases as possible. The sample utterances specify the words and phrases users can say to invoke your intents. QnABot updates the **Sample utterances** with the various questions to train the chatbot to understand different user input +2. The Bot fulfillment Lambda function generates an ElasticSearch query containing the transcribed question. The query attempts to find the best match from all the questions and answers you’ve previously provided. +3. This request is then sent to Amazon ElasticSearch. QnABot attempts to match a user's question to the list of questions and answers (created in the QnABot content designer) stored in Amazon ElasticSearch. +4. (Optional) If an answer is not found in ElasticSearch and Amazon Kendra is configured, QnABot will search documents or web pages in your Kendra index. + +## Monitoring Usage + +![Kibana Dashboard](image9.png) + +QnABot includes a visualization tool (using Kibana) to analyze QnABot usage. Kibana is an open-source data visualization and exploration tool used for log and time-series analytics, application monitoring, and operational intelligence use cases. It offers powerful and easy-to-use features such as histograms, line graphs, pie charts, heat maps, and built-in geospatial support. Also, it provides tight integration with Elasticsearch, which makes Kibana the default choice for visualizing data stored in Elasticsearch. + +The Kibana dashboard can be used to view usage history, logged utterances, no hits utterances, positive user feedback, and negative user feedback and also provides the ability to create custom reports. diff --git a/docs/overview/image2.png b/docs/overview/image2.png new file mode 100755 index 000000000..dbfa42f8d Binary files /dev/null and b/docs/overview/image2.png differ diff --git a/docs/overview/image9.png b/docs/overview/image9.png new file mode 100755 index 000000000..e8988f714 Binary files /dev/null and b/docs/overview/image9.png differ diff --git a/docs/redact.md b/docs/redact.md new file mode 100644 index 000000000..55ff6a1e3 --- /dev/null +++ b/docs/redact.md @@ -0,0 +1,15 @@ +# Optional Redact feature for log and metric output + +QnABot can be configured to redact information written to CloudWatch logs, S3 metrics, and Kibana metrics logs. +This feature is disabled by default. Use the Designer UI Settings form to enable this feature. One can configure +the RegEx applied to strings as they are logged. If RegEx matches are found, the match is replaced with the string +'XXXXXX'. + +The initial RegEx is + +```regex +\b\d{4}\b(?![-])|\b\d{9}\b|\b\d{3}-\d{2}-\d{4}\b +``` + +This replaces 4 digit numbers not followed by a hyphen, a 9 digit number (SSN without hyphens), and a typical +SSN using nnn-nn-nnnn syntax with hyphens. \ No newline at end of file diff --git a/test-do-not-use-obsolete/.gitignore b/test-do-not-use-obsolete/.gitignore deleted file mode 100644 index 41d9bbb82..000000000 --- a/test-do-not-use-obsolete/.gitignore +++ /dev/null @@ -1,2 +0,0 @@ -*.log -output diff --git a/test-do-not-use-obsolete/Makefile b/test-do-not-use-obsolete/Makefile deleted file mode 100644 index a527bb5fe..000000000 --- a/test-do-not-use-obsolete/Makefile +++ /dev/null @@ -1,7 +0,0 @@ - -up: ../build/templates/test.json - npm run stack -- --input $$(pwd)/cfn/test.js --operation up - -../build/templates/test.json: ./cfn/* - ../bin/build.js --input $$(pwd)/cfn/test.js --output $$(pwd)/../build/templates/test.json --verbose - diff --git a/test-do-not-use-obsolete/README.md b/test-do-not-use-obsolete/README.md deleted file mode 100644 index c7d836502..000000000 --- a/test-do-not-use-obsolete/README.md +++ /dev/null @@ -1,15 +0,0 @@ -# QnABot Testing -testing infastructure. goal is to run all tests within a codebuild project. -```shell -make #builds and launches template -``` -can use individual scripts localing as well -scripts run in the following order - -1) configure.sh -2) setup.sh -3) test.sh -4) teardown.sh - -Cloudformation template in ./cfn is designed to be a single template not dependent on the bootstrap template - diff --git a/test-do-not-use-obsolete/alexa/.gitignore b/test-do-not-use-obsolete/alexa/.gitignore deleted file mode 100644 index 200dcfbde..000000000 --- a/test-do-not-use-obsolete/alexa/.gitignore +++ /dev/null @@ -1 +0,0 @@ -files/* diff --git a/test-do-not-use-obsolete/alexa/README.md b/test-do-not-use-obsolete/alexa/README.md deleted file mode 100644 index aaea2f57a..000000000 --- a/test-do-not-use-obsolete/alexa/README.md +++ /dev/null @@ -1,16 +0,0 @@ -# Alexa tests -scripts to automaticly build alexa skill and test - -steps -1) install and configure alexa ask cli. [instructions](https://developer.amazon.com/docs/smapi/quick-start-alexa-skills-kit-command-line-interface.html) -2) launch dev/master stack -```shell -npm run stack dev/master up -``` -3) run setup and build skill -```shell -./setup.js && ./create.sh -``` -4) then go to [amazon developer console](https://developer.amazon.com/home.html) and enable new skill named "QnABot Test" - - diff --git a/test-do-not-use-obsolete/alexa/assets/en-US.json b/test-do-not-use-obsolete/alexa/assets/en-US.json deleted file mode 100644 index 535f293ee..000000000 --- a/test-do-not-use-obsolete/alexa/assets/en-US.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "interactionModel":{ - "languageModel":{ - "invocationName":"q and a", - "types":[{ - "name":"EXAMPLE_QUESTIONS", - "values":[] - }], - "intents":[] - } - } -} diff --git a/test-do-not-use-obsolete/alexa/assets/skill.json b/test-do-not-use-obsolete/alexa/assets/skill.json deleted file mode 100644 index 78a04ed97..000000000 --- a/test-do-not-use-obsolete/alexa/assets/skill.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "skillManifest": { - "publishingInformation": { - "locales": { - "en-US": { - "summary": "QnABot Alexa Test", - "examplePhrases": [ - "Alexa ask QnA who am i" - ], - "name": "QnABot Test", - "description": "Sample Full Description" - } - }, - "isAvailableWorldwide": true, - "testingInstructions": "Sample Testing Instructions.", - "category": "EDUCATION_AND_REFERENCE", - "distributionCountries": [] - }, - "apis": { - "custom": { - "endpoint": { - "uri":"" - } - } - }, - "manifestVersion": "1.0" - }, -"privacyAndCompliance": { - "allowsPurchases": false, - "usesPersonalInfo": false, - "isChildDirected": false, - "isExportCompliant": true, - "containsAds": false, - "locales": { - "en-US": { - "privacyPolicyUrl": "http://www.myprivacypolicy.sampleskill.com", - "termsOfUseUrl": "http://www.termsofuse.sampleskill.com" - } - } -} -} diff --git a/test-do-not-use-obsolete/alexa/create.sh b/test-do-not-use-obsolete/alexa/create.sh deleted file mode 100755 index ed8bf8989..000000000 --- a/test-do-not-use-obsolete/alexa/create.sh +++ /dev/null @@ -1,12 +0,0 @@ -#! /bin/bash -__dirname="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" - -node $__dirname/setup.js -BIN=$(npm bin) -OUTPUT=$($BIN/ask api create-skill -f $__dirname/files/skill.json) -echo $OUTPUT -ID=$(echo "$OUTPUT" | grep amzn1.ask.skill | cut -d' ' -f3) - -$BIN/ask api update-model --skill-id $ID \ - --file $__dirname/files/model.json \ - --locale en-US --debug diff --git a/test-do-not-use-obsolete/alexa/run.sh b/test-do-not-use-obsolete/alexa/run.sh deleted file mode 100755 index 451ac0f8d..000000000 --- a/test-do-not-use-obsolete/alexa/run.sh +++ /dev/null @@ -1,9 +0,0 @@ -#! /bin/bash - -ask simulate \ - --skill-id amzn1.ask.skill.99499268-0dac-495c-957e-99833ae4b50a \ - --text "hello" \ - --locale en-US \ - --debug - - diff --git a/test-do-not-use-obsolete/alexa/setup.js b/test-do-not-use-obsolete/alexa/setup.js deleted file mode 100755 index 4022ac524..000000000 --- a/test-do-not-use-obsolete/alexa/setup.js +++ /dev/null @@ -1,46 +0,0 @@ -#! /usr/bin/env node -var util=require('./util') -var Promise=require('bluebird') -var skill=require('./assets/skill') -var model=require('./assets/en-US') -var fs=Promise.promisifyAll(require('fs')) - -Promise.join( - util.api({ - path:"/bot/alexa", - method:"GET" - }) - .then(x=>x.schema.intents), - util.api({ - path:"/bot", - method:"GET" - }) - .then(x=>x.lambdaArn), - util.api({ - path:"/bot/utterances", - method:"GET" - }) -) -.spread(function(intents,lambda,utterances){ - skill.skillManifest.apis.custom.endpoint.uri=lambda - - intents.forEach(function(x){ - x.name=x.intent - delete x.intent - }) - model.interactionModel.languageModel.intents=intents - - model.interactionModel.languageModel.types[0] - .values=utterances.map(x=>{ return { - name:{value:x} - }}) - model.interactionModel.languageModel.intents[0].samples=["{QnA_slot}"] - - - return Promise.join( - fs.writeFileAsync(__dirname+'/files/model.json',JSON.stringify(model,null,2)), - fs.writeFileAsync(__dirname+'/files/skill.json',JSON.stringify(skill,null,2)) - ) -}) - - diff --git a/test-do-not-use-obsolete/alexa/test/.ask/config b/test-do-not-use-obsolete/alexa/test/.ask/config deleted file mode 100644 index e527ea5a2..000000000 --- a/test-do-not-use-obsolete/alexa/test/.ask/config +++ /dev/null @@ -1,19 +0,0 @@ -{ - "deploy_settings": { - "default": { - "skill_id": "amzn1.ask.skill.97607164-a819-4cc2-9b66-89bb00b11146", - "was_cloned": false, - "merge": { - "skillManifest": { - "apis": { - "custom": { - "endpoint": { - "uri": "ask-custom-test-default" - } - } - } - } - } - } - } -} diff --git a/test-do-not-use-obsolete/alexa/test/lambda/custom/index.js b/test-do-not-use-obsolete/alexa/test/lambda/custom/index.js deleted file mode 100644 index 0f43149c1..000000000 --- a/test-do-not-use-obsolete/alexa/test/lambda/custom/index.js +++ /dev/null @@ -1,55 +0,0 @@ -'use strict'; -var Alexa = require("alexa-sdk"); - -// For detailed tutorial on how to making a Alexa skill, -// please visit us at http://alexa.design/build - - -exports.handler = function(event, context) { - var alexa = Alexa.handler(event, context); - alexa.registerHandlers(handlers); - alexa.execute(); -}; - -var handlers = { - 'LaunchRequest': function () { - this.emit('SayHello'); - }, - 'HelloWorldIntent': function () { - this.emit('SayHello'); - }, - 'MyNameIsIntent': function () { - this.emit('SayHelloName'); - }, - 'SayHello': function () { - this.response.speak('Hello World!') - .cardRenderer('hello world', 'hello world'); - this.emit(':responseReady'); - }, - 'SayHelloName': function () { - var name = this.event.request.intent.slots.name.value; - this.response.speak('Hello ' + name) - .cardRenderer('hello world', 'hello ' + name); - this.emit(':responseReady'); - }, - 'SessionEndedRequest' : function() { - console.log('Session ended with reason: ' + this.event.request.reason); - }, - 'AMAZON.StopIntent' : function() { - this.response.speak('Bye'); - this.emit(':responseReady'); - }, - 'AMAZON.HelpIntent' : function() { - this.response.speak("You can try: 'alexa, hello world' or 'alexa, ask hello world my" + - " name is awesome Aaron'"); - this.emit(':responseReady'); - }, - 'AMAZON.CancelIntent' : function() { - this.response.speak('Bye'); - this.emit(':responseReady'); - }, - 'Unhandled' : function() { - this.response.speak("Sorry, I didn't get that. You can try: 'alexa, hello world'" + - " or 'alexa, ask hello world my name is awesome Aaron'"); - } -}; diff --git a/test-do-not-use-obsolete/alexa/test/lambda/custom/package.json b/test-do-not-use-obsolete/alexa/test/lambda/custom/package.json deleted file mode 100644 index 8df570155..000000000 --- a/test-do-not-use-obsolete/alexa/test/lambda/custom/package.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "name": "skill-sample-nodejs-hello-world", - "version": "1.0.0", - "description": "Hello world sample skill", - "scripts": { - "test": "echo \"Error: no test specified\" && exit 1" - }, - "dependencies": { - "alexa-sdk": "^1.0.0" - } -} diff --git a/test-do-not-use-obsolete/alexa/test/models/en-US.json b/test-do-not-use-obsolete/alexa/test/models/en-US.json deleted file mode 100644 index e1ebf23a7..000000000 --- a/test-do-not-use-obsolete/alexa/test/models/en-US.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "interactionModel":{ - "languageModel":{ - "invocationName":"hello world", - "types":[], - "intents":[ - { - "name": "AMAZON.CancelIntent", - "samples": [] - }, - { - "name": "AMAZON.HelpIntent", - "samples": [] - }, - { - "name": "AMAZON.StopIntent", - "samples": [] - }, - { - "name":"HelloWorldIntent", - "slots":[ - - ], - "samples":[ - "hello", - "say hello", - "say hello world" - ] - }, - { - "name":"MyNameIsIntent", - "slots":[ - { - "name":"name", - "type":"AMAZON.US_FIRST_NAME" - } - ], - "samples":[ - "my name is {name}", - "i am {name}", - "you can call me {name}" - ] - } - ] - } - } -} diff --git a/test-do-not-use-obsolete/alexa/test/skill.json b/test-do-not-use-obsolete/alexa/test/skill.json deleted file mode 100644 index 3e4523c36..000000000 --- a/test-do-not-use-obsolete/alexa/test/skill.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "skillManifest": { - "publishingInformation": { - "locales": { - "en-US": { - "summary": "Sample Short Description", - "examplePhrases": [ - "Alexa open hello world", - "Alexa tell hello world I am Jeff", - "Alexa tell hello world my name is Peter" - ], - "name": "test", - "description": "Sample Full Description" - } - }, - "isAvailableWorldwide": true, - "testingInstructions": "Sample Testing Instructions.", - "category": "EDUCATION_AND_REFERENCE", - "distributionCountries": [] - }, - "apis": { - "custom": { - "endpoint": { - "sourceDir": "lambda/custom" - } - } - }, - "manifestVersion": "1.0" - } -} diff --git a/test-do-not-use-obsolete/alexa/util.js b/test-do-not-use-obsolete/alexa/util.js deleted file mode 100644 index 96c58723c..000000000 --- a/test-do-not-use-obsolete/alexa/util.js +++ /dev/null @@ -1,43 +0,0 @@ -var config=require('../../config') -process.env.AWS_PROFILE=config.profile -process.env.AWS_DEFAULT_REGION=config.region -var query=require('query-string').stringify -var _=require('lodash') -var Promise=require('bluebird') -var axios=require('axios') -var Url=require('url') -var sign=require('aws4').sign -var fs=require('fs') -var aws=require('aws-sdk') -aws.config.setPromisesDependency(Promise) -aws.config.region=config.region -var outputs=require('../../bin/exports') - - -exports.api=api -function api(opts){ - return outputs('dev/master',{wait:true}).then(function(output){ - var href=opts.path ? output.ApiEndpoint+'/'+opts.path : opts.href - var url=Url.parse(href) - var request={ - host:url.hostname, - method:opts.method.toUpperCase(), - url:url.href, - path:url.path, - headers:opts.headers || {} - } - if(opts.body){ - request.body=JSON.stringify(opts.body), - request.data=opts.body, - request.headers['content-type']='application/json' - } - - var credentials=aws.config.credentials - var signed=sign(request,credentials) - delete request.headers["Host"] - delete request.headers["Content-Length"] - - return Promise.resolve(axios(signed)) - .get('data') - }) -} diff --git a/test-do-not-use-obsolete/cfn/Dockerfile b/test-do-not-use-obsolete/cfn/Dockerfile deleted file mode 100644 index dc3879472..000000000 --- a/test-do-not-use-obsolete/cfn/Dockerfile +++ /dev/null @@ -1,39 +0,0 @@ -FROM selenium/standalone-chrome -LABEL authors=JohnCalhoun -USER root -RUN apt-get update && \ - apt-get upgrade -y - -# Set the timezone -RUN echo "Europe/Berlin" | tee /etc/timezone && \ - ln -fs /usr/share/zoneinfo/Europe/Berlin /etc/localtime && \ - dpkg-reconfigure -f noninteractive tzdata - -# Set the locale for UTF-8 support -RUN echo en_US.UTF-8 UTF-8 >> /etc/locale.gen && \ - locale-gen && \ - update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 -ENV LANG en_US.UTF-8 -ENV LANGUAGE en_US:en -ENV LC_ALL en_US.UTF-8 - -# AWS CLI needs the PYTHONIOENCODING environment varialbe to handle UTF-8 correctly: -ENV PYTHONIOENCODING=UTF-8 - -# man and less are needed to view 'aws help' -# ssh allows us to log in to new instances -# vim is useful to write shell scripts -# python* is needed to install aws cli using pip install - -RUN apt-get install -y \ - zip \ - git \ - curl \ - python \ - python-pip \ - python-virtualenv - -RUN pip install awscli -RUN curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash - && \ - apt-get install -y nodejs - diff --git a/test-do-not-use-obsolete/cfn/README.md b/test-do-not-use-obsolete/cfn/README.md deleted file mode 100644 index 176364348..000000000 --- a/test-do-not-use-obsolete/cfn/README.md +++ /dev/null @@ -1,6 +0,0 @@ -# QnABot Test Cloudformation -Cloudformation assets to run tests in codebuild - -steps -- make test image that contains aws-cli,nodejs,selenium,and chrome -- use container to run all test diff --git a/test-do-not-use-obsolete/cfn/buildspec.yml b/test-do-not-use-obsolete/cfn/buildspec.yml deleted file mode 100644 index 597e64b26..000000000 --- a/test-do-not-use-obsolete/cfn/buildspec.yml +++ /dev/null @@ -1,18 +0,0 @@ -version: 0.2 - -phases: - pre_build: - commands: - - echo Logging in to Amazon ECR... - - $(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION) - build: - commands: - - echo Build started on `date` - - echo Building the Docker image... - - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG . - - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG - post_build: - commands: - - echo Build completed on `date` - - echo Pushing the Docker image... - - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG diff --git a/test-do-not-use-obsolete/cfn/config/buildspec.yml b/test-do-not-use-obsolete/cfn/config/buildspec.yml deleted file mode 100644 index 761462f17..000000000 --- a/test-do-not-use-obsolete/cfn/config/buildspec.yml +++ /dev/null @@ -1,19 +0,0 @@ -version: 0.2 - -phases: - install: - commands: - - npm install - pre_build: - commands: - - ./test/configure.sh - - ./test/setup.sh - build: - commands: - - ./test/test.sh - post_build: - commands: - - ./test/teardown.js -artifacts: - files: - - ./test/output/**/* diff --git a/test-do-not-use-obsolete/cfn/lambda/README.md b/test-do-not-use-obsolete/cfn/lambda/README.md deleted file mode 100644 index 751046146..000000000 --- a/test-do-not-use-obsolete/cfn/lambda/README.md +++ /dev/null @@ -1 +0,0 @@ -lambda functions for use in test script. they are written in to cloudformation template directly diff --git a/test-do-not-use-obsolete/cfn/lambda/build.js b/test-do-not-use-obsolete/cfn/lambda/build.js deleted file mode 100644 index f0bc7d173..000000000 --- a/test-do-not-use-obsolete/cfn/lambda/build.js +++ /dev/null @@ -1,64 +0,0 @@ -var response = require('cfn-response') -var aws=require('aws-sdk') -aws.config.region=process.env.AWS_REGION -var cb=new aws.CodeBuild() -var s3=new aws.S3() -var ecr=new aws.ECR() -var lambda=new aws.Lambda() - -exports.build = function(event, context) { - console.log(JSON.stringify(event,null,2)) - if(event.StackId){ - if(event.RequestType==="Create"){ - var params=Object.assign({},event.ResourceProperties) - delete params.ServiceToken - - cb.startBuild(params).promise() - .then(x=>new Promise((res,rej)=>setTimeout(()=>res(x),10*1000))) - .then(x=>{ - return lambda.invoke({ - FunctionName:process.env.AWS_LAMBDA_FUNCTION_NAME, - InvocationType:"Event", - Payload:JSON.stringify({ - event,context,id:x.build.id - }) - }).promise() - }) - .catch(x=>{ - console.log(x) - response.send(event, context, response.FAILED) - }) - }else{ - response.send(event, context, response.SUCCESS) - } - }else{ - event.context.done=context.done - cb.batchGetBuilds({ - ids:[event.id] - }).promise() - .then(x=>{ - console.log(x) - var status=x.builds[0].buildStatus - if(status==="SUCCEEDED"){ - response.send(event.event, event.context, response.SUCCESS) - }else if(status==="IN_PROGRESS"){ - setTimeout(()=>lambda.invoke({ - FunctionName:process.env.AWS_LAMBDA_FUNCTION_NAME, - InvocationType:"Event", - Payload:JSON.stringify({ - event:event.event, - context:event.context, - id:x.builds[0].id - }) - }).promise(),10*1000) - }else{ - response.send(event.event, event.context, response.FAILED) - } - }) - .catch(x=>{ - console.log(x) - response.send(event.event, event.context, response.FAILED) - }) - } -} - diff --git a/test-do-not-use-obsolete/cfn/lambda/clear.js b/test-do-not-use-obsolete/cfn/lambda/clear.js deleted file mode 100644 index cd34161a9..000000000 --- a/test-do-not-use-obsolete/cfn/lambda/clear.js +++ /dev/null @@ -1,55 +0,0 @@ -var response = require('cfn-response') -var aws=require('aws-sdk') -aws.config.region=process.env.AWS_REGION -var cb=new aws.CodeBuild() -var s3=new aws.S3() -var ecr=new aws.ECR() -var lambda=new aws.Lambda() - -exports.clear = function(event, context) { - console.log(JSON.stringify(event,null,2)) - - if(event.RequestType==="Delete"){ - Delete(event.ResourceProperties) - .then(()=>response.send(event, context, response.SUCCESS)) - .catch(x=>{ - console.log(x) - response.send(event, context, response.FAILED) - }) - }else{ - response.send(event, context, response.SUCCESS) - } -} - - -function Delete(params){ - return new Promise(function(res,rej){ - function next(){ - s3.listObjectsV2({ - Bucket:params.Bucket - }).promise() - .then(x=>x.Contents) - .then(function(files){ - return files.map(file=>{return { - Key:file.Key - } }) - }) - .then(function(keys){ - console.log("going to delete",keys) - if(keys.length>0){ - return s3.deleteObjects({ - Bucket:params.Bucket, - Delete:{ - Objects:keys - } - }).promise() - .then(()=>next()) - .catch(rej) - }else{ - res() - } - }) - } - next() - }) -} diff --git a/test-do-not-use-obsolete/cfn/lambda/clearImage.js b/test-do-not-use-obsolete/cfn/lambda/clearImage.js deleted file mode 100644 index e4ca7e394..000000000 --- a/test-do-not-use-obsolete/cfn/lambda/clearImage.js +++ /dev/null @@ -1,26 +0,0 @@ -var response = require('cfn-response') -var aws=require('aws-sdk') -aws.config.region=process.env.AWS_REGION -var cb=new aws.CodeBuild() -var s3=new aws.S3() -var ecr=new aws.ECR() -var lambda=new aws.Lambda() - -exports.clearImage = function(event, context) { - console.log(JSON.stringify(event,null,2)) - - if(event.RequestType==="Delete"){ - ecr.batchDeleteImage({ - imageIds:[{imageTag:event.ResourceProperties.tag}], - repositoryName:event.ResourceProperties.repo - }).promise() - .then(()=>response.send(event, context, response.SUCCESS)) - .catch(x=>{ - console.log(x) - response.send(event, context, response.SUCCESS) - }) - }else{ - response.send(event, context, response.SUCCESS) - } -} - diff --git a/test-do-not-use-obsolete/cfn/lambda/zip.js b/test-do-not-use-obsolete/cfn/lambda/zip.js deleted file mode 100644 index 139e1a742..000000000 --- a/test-do-not-use-obsolete/cfn/lambda/zip.js +++ /dev/null @@ -1,27 +0,0 @@ -var response = require('cfn-response') -var aws=require('aws-sdk') -aws.config.region=process.env.AWS_REGION -var cb=new aws.CodeBuild() -var s3=new aws.S3() -var ecr=new aws.ECR() -var lambda=new aws.Lambda() - -exports.zip = function(event, context) { - console.log(JSON.stringify(event,null,2)) - - if(event.RequestType==="Create"){ - s3.putObject({ - Bucket:event.ResourceProperties.bucket, - Key:event.ResourceProperties.key, - Body:new Buffer(event.ResourceProperties.body,"base64") - }).promise() - .then(x=>response.send(event,context,response.SUCCESS)) - .catch(x=>{ - console.log(x) - response.send(event, context, response.FAILED) - }) - }else{ - response.send(event, context, response.SUCCESS) - } -} - diff --git a/test-do-not-use-obsolete/cfn/test.js b/test-do-not-use-obsolete/cfn/test.js deleted file mode 100644 index 9d56dda1e..000000000 --- a/test-do-not-use-obsolete/cfn/test.js +++ /dev/null @@ -1,327 +0,0 @@ -var fs=require('fs') -var Promise=require('bluebird') -var child=Promise.promisifyAll(require('child_process')) -var JSZip = require("jszip"); -var zip=new JSZip() -var package=require('../../package.json') -zip.file('buildspec.yml',fs.readFileSync(__dirname+'/buildspec.yml','utf-8')) -zip.file('Dockerfile',fs.readFileSync(__dirname+'/Dockerfile','utf-8')) - -var tag="test" -var source="source.zip" -async function run(){ - var buff=await Promise.resolve(zip.generateAsync({type:'nodebuffer'})) - var info=await child.execAsync('git rev-parse --symbolic-full-name --abbrev-ref @{u}',{ - cwd:__dirname - }) - var info_parse=info.match(/(.*)\/(.*)/) - var remote=info_parse[1] - var branch=info_parse[2] - - var remote_info=await child.execAsync(`git remote get-url ${remote}`,{ - cwd:__dirname - }) - - var path=remote_info.match(/(.*):(.*)/)[2] - var url=`https://github.com/${path}` - - return { - "Description": "This template creates test infastructure for testing QnABot", - "Resources":{ - "Repo":{ - "Type" : "AWS::ECR::Repository", - "Properties":{ - RepositoryPolicyText:{ - "Version": "2012-10-17", - "Statement": [{ - "Effect": "Allow", - "Principal": { - "Service": "codebuild.amazonaws.com" - }, - "Action": [ - "ecr:GetDownloadUrlForLayer", - "ecr:BatchGetImage", - "ecr:BatchCheckLayerAvailability" - ] - }] - } - } - }, - "testBuild":{ - "Type":"AWS::CodeBuild::Project", - "Properties":{ - "Artifacts":{ - Type:"S3", - Location:{"Ref":"Bucket"} - }, - "Environment":{ - ComputeType:"BUILD_GENERAL1_LARGE", - Image:{"Fn::Sub":"${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${Repo}:"+tag}, - EnvironmentVariables:[], - Type:"LINUX_CONTAINER" - }, - "Name":{"Fn::Sub":"${AWS::StackName}-test-build"}, - ServiceRole:{"Ref":"TestServiceRole"}, - Source:{ - Type:"GITHUB", - Location:url, - Auth:{ - Type:"OAUTH" - } - }, - TimeoutInMinutes:4*60 - } - }, - "ImageBuild":{ - "Type":"AWS::CodeBuild::Project", - "Properties":{ - "Artifacts":{ - Type:"NO_ARTIFACTS" - }, - "Environment":{ - ComputeType:"BUILD_GENERAL1_LARGE", - EnvironmentVariables:[{ - Name:"IMAGE_REPO_NAME", - Value:{"Ref":"Repo"} - },{ - Name:"IMAGE_TAG", - Value:tag, - },{ - Name:"AWS_DEFAULT_REGION", - Value:{"Ref":"AWS::Region"} - },{ - Name:"AWS_ACCOUNT_ID", - Value:{"Ref":"AWS::AccountId"} - }], - Image:"aws/codebuild/docker:17.09.0", - PrivilegedMode:true, - Type:"LINUX_CONTAINER" - }, - "Name":{"Fn::Sub":"${AWS::StackName}-test-image-build"}, - ServiceRole:{"Ref":"ServiceRole"}, - Source:{ - Type:"S3", - Location:{"Fn::Sub":"${Bucket}/"+source} - } - } - }, - "Bucket": { - "Type": "AWS::S3::Bucket", - "Properties": {} - }, - "Clear":{ - "Type": "Custom::S3Clear", - "DependsOn":["Bucket"], - "Properties": { - "ServiceToken": { "Fn::GetAtt" : ["ClearLambda", "Arn"] }, - "Bucket":{"Ref":"Bucket"} - } - }, - "ClearImage":{ - "Type": "Custom::ClearImage", - "DependsOn":["Bucket"], - "Properties": { - "ServiceToken": { "Fn::GetAtt" : ["ClearImageLambda", "Arn"] }, - "repo":{"Ref":"Repo"}, - "tag":tag - } - }, - "upload":{ - "Type": "Custom::S3Upload", - "Properties": { - "ServiceToken": { "Fn::GetAtt" : ["UploadLambda", "Arn"] }, - "bucket":{"Ref":"Bucket"}, - "key":source, - "body":buff.toString('base64') - } - }, - "Build":{ - "Type": "Custom::CodeBuildStart", - "DependsOn":["upload"], - "Properties": { - "ServiceToken": { "Fn::GetAtt" : ["BuildLambda", "Arn"] }, - "projectName":{"Ref":"ImageBuild"} - } - }, - "test":{ - "Type": "Custom::CodeBuildStart", - "DependsOn":["Build"], - "Properties": { - "ServiceToken": { "Fn::GetAtt" : ["BuildLambda", "Arn"] }, - "projectName":{"Ref":"testBuild"}, - "sourceVersion":branch, - "buildspecOverride":fs.readFileSync( - __dirname+'/config/buildspec.yml','utf-8') - } - }, - "ClearImageLambda":lambda("clearImage"), - "BuildLambda": lambda("build"), - "UploadLambda": lambda("zip"), - "ClearLambda": lambda("clear"), - "TestServiceRole": { - "Type": "AWS::IAM::Role", - "Properties": { - "AssumeRolePolicyDocument": { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Service": "codebuild.amazonaws.com" - }, - "Action": "sts:AssumeRole" - } - ] - }, - "Path": "/", - "ManagedPolicyArns": [ - "arn:aws:iam::aws:policy/AdministratorAccess" - ], - } - }, - "ServiceRole": { - "Type": "AWS::IAM::Role", - "Properties": { - "AssumeRolePolicyDocument": { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Service": "codebuild.amazonaws.com" - }, - "Action": "sts:AssumeRole" - } - ] - }, - "Path": "/", - "ManagedPolicyArns": [ - "arn:aws:iam::aws:policy/AmazonECS_FullAccess", - "arn:aws:iam::aws:policy/AmazonS3FullAccess" - ], - "Policies":[{ - "PolicyName":"codebuild", - "PolicyDocument": { - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "CloudWatchLogsPolicy", - "Effect": "Allow", - "Action": [ - "logs:CreateLogGroup", - "logs:CreateLogStream", - "logs:PutLogEvents" - ], - "Resource": [ - "*" - ] - }, - { - "Sid": "CodeCommitPolicy", - "Effect": "Allow", - "Action": [ - "codecommit:GitPull", - "ecr:*" - ], - "Resource": [ - "*" - ] - }, - { - "Sid": "S3GetObjectPolicy", - "Effect": "Allow", - "Action": [ - "s3:GetObject", - "s3:GetObjectVersion" - ], - "Resource": [ - "*" - ] - }, - { - "Sid": "S3PutObjectPolicy", - "Effect": "Allow", - "Action": [ - "s3:PutObject" - ], - "Resource": [ - "*" - ] - } - ] - } - }] - } - }, - "LambdaRole": { - "Type": "AWS::IAM::Role", - "Properties": { - "AssumeRolePolicyDocument": { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Service": "lambda.amazonaws.com" - }, - "Action": "sts:AssumeRole" - } - ] - }, - "Path": "/", - "ManagedPolicyArns": [ - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole", - "arn:aws:iam::aws:policy/AmazonECS_FullAccess", - "arn:aws:iam::aws:policy/AmazonS3FullAccess", - "arn:aws:iam::aws:policy/AWSCodeBuildAdminAccess" - ], - "Policies":[{ - "PolicyName":"codebuild", - "PolicyDocument": { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "lambda:InvokeFunction", - "ecr:*" - ], - "Resource": [ - "*" - ] - } - ] - } - }] - } - } - }, - "Outputs": { - "BuildConsole":{ - "Value":{"Fn::Sub":"https://console.aws.amazon.com/codebuild/home?region=${AWS::Region}#/projects/${testBuild}/view"} - }, - "ImageConsole":{ - "Value":{"Fn::Sub":"https://console.aws.amazon.com/codebuild/home?region=${AWS::Region}#/projects/${ImageBuild}/view"} - } - } - } -} -run() -module.exports=run() - - -function lambda(name){ - return { - "Type": "AWS::Lambda::Function", - "Properties": { - "Code": { - "ZipFile":fs.readFileSync(__dirname+`/lambda/${name}.js`,'utf-8') - }, - "Handler": "index."+name, - "MemorySize": "128", - "Role": {"Fn::GetAtt": ["LambdaRole","Arn"]}, - "Runtime": "nodejs12.x", - "Timeout": 300 - } - } -} diff --git a/test-do-not-use-obsolete/configure.sh b/test-do-not-use-obsolete/configure.sh deleted file mode 100755 index a39841f26..000000000 --- a/test-do-not-use-obsolete/configure.sh +++ /dev/null @@ -1,38 +0,0 @@ -#! /bin/bash -__dirname="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" -BASE=$__dirname/.. -BIN=$__dirname/../bin -cd $BASE - -PROFILE="default" -NAMESPACE="test" - -if aws s3 ls --profile $PROFILE >> /dev/null; then - echo "aws cli configured" -else - echo "configuring aws cli" - region=$AWS_REGION - creds=$(curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI) - - aws configure set aws_access_key_id $( echo $creds | $BIN/json.js AccessKeyId) - aws configure set aws_secret_access_key $( echo $creds | $BIN/json.js SecretAccessKey) - aws configure set aws_session_token $( echo $creds | $BIN/json.js Token ) - aws configure set $PROFILE.region $region -fi -aws configure list --profile $PROFILE - -if [ ! -f ./config.json ]; then - CLI_REGION=$(aws configure get region --profile $PROFILE) - echo "creating config" - - node $BIN/config.js john@example.com $CLI_REGION \ - | $BIN/json.js -e "this.profile='$PROFILE'" \ - | $BIN/json.js -e "this.namespace='$NAMESPACE'" \ - > $BASE/config.json -fi - -if [ -n "$ASK_CREDENTIALS" ]; then - echo "$ASK_CREDENTIALS" >> ~/.ask/cli_config -fi - - diff --git a/test-do-not-use-obsolete/kendra_tests/context_FAQ_md.json b/test-do-not-use-obsolete/kendra_tests/context_FAQ_md.json deleted file mode 100644 index 9fa21fa26..000000000 --- a/test-do-not-use-obsolete/kendra_tests/context_FAQ_md.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "callbackWaitsForEmptyEventLoop": true, - "functionVersion": "$LATEST", - "functionName": "QNA-dev-dev-dev-master-2-Example-EXTKendraFallback-1NKAGNPBF6VVY", - "memoryLimitInMB": "2048", - "logGroupName": "/aws/lambda/QNA-dev-dev-dev-master-2-Example-EXTKendraFallback-1NKAGNPBF6VVY", - "logStreamName": "2020/07/02/[$LATEST]e5eaf1b6bade474a82aa21a378b9a71f", - "invokedFunctionArn": "arn:aws:lambda:us-east-1:425742325899:function:QNA-dev-dev-dev-master-2-Example-EXTKendraFallback-1NKAGNPBF6VVY", - "awsRequestId": "a0665572-c2b8-424f-a22e-502f6bb317cb" -} diff --git a/test-do-not-use-obsolete/kendra_tests/context_doc_query.json b/test-do-not-use-obsolete/kendra_tests/context_doc_query.json deleted file mode 100644 index 5e1065214..000000000 --- a/test-do-not-use-obsolete/kendra_tests/context_doc_query.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "callbackWaitsForEmptyEventLoop": true, - "functionVersion": "$LATEST", - "functionName": "QNA-dev-dev-dev-master-1-Example-EXTKendraFallback-1CKOPNQ8CI049", - "memoryLimitInMB": "2048", - "logGroupName": "/aws/lambda/QNA-dev-dev-dev-master-1-Example-EXTKendraFallback-1CKOPNQ8CI049", - "logStreamName": "2020/06/25/[$LATEST]98a19ebfac874a36b5d450ec4726a6a3", - "invokedFunctionArn": "arn:aws:lambda:us-east-1:425742325899:function:QNA-dev-dev-dev-master-1-Example-EXTKendraFallback-1CKOPNQ8CI049", - "awsRequestId": "c6dd6ab6-4ab0-4839-931b-2418beded229" -} diff --git a/test-do-not-use-obsolete/kendra_tests/context_top_ans.json b/test-do-not-use-obsolete/kendra_tests/context_top_ans.json deleted file mode 100644 index f7b1fede8..000000000 --- a/test-do-not-use-obsolete/kendra_tests/context_top_ans.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "callbackWaitsForEmptyEventLoop": true, - "functionVersion": "$LATEST", - "functionName": "QNAbot-dev-dev-dev-master-4-Exam-EXTKendraFallback-T1KJ602HXNIW", - "memoryLimitInMB": "2048", - "logGroupName": "/aws/lambda/QNAbot-dev-dev-dev-master-4-Exam-EXTKendraFallback-T1KJ602HXNIW", - "logStreamName": "2020/06/24/[$LATEST]a70c615cee674b69a916f679fcdc4459", - "invokedFunctionArn": "arn:aws:lambda:us-east-1:425742325899:function:QNAbot-dev-dev-dev-master-4-Exam-EXTKendraFallback-T1KJ602HXNIW", - "awsRequestId": "55459ee4-3f84-470d-ae35-399ce38a333f" -} - diff --git a/test-do-not-use-obsolete/kendra_tests/event_FAQ_md.json b/test-do-not-use-obsolete/kendra_tests/event_FAQ_md.json deleted file mode 100644 index 3c6c071e4..000000000 --- a/test-do-not-use-obsolete/kendra_tests/event_FAQ_md.json +++ /dev/null @@ -1,135 +0,0 @@ -{ - "req": { - "_event": { - "messageVersion": "1.0", - "invocationSource": "FulfillmentCodeHook", - "userId": "us-east-1:7ace124d-bfd1-4356-ac5a-ed8a98785fd9", - "sessionAttributes": {}, - "requestAttributes": null, - "bot": { - "name": "QNA_dev_dev_dev_master_two_BotUSFgl", - "alias": "live", - "version": "1" - }, - "outputDialogMode": "Text", - "currentIntent": { - "name": "fulfilment_IntentfiuhxvkSil", - "slots": { - "slot": "How many planets orbit the sun" - }, - "slotDetails": { - "slot": { - "resolutions": [], - "originalValue": "How many planets orbit the sun" - } - }, - "confirmationStatus": "None" - }, - "inputTranscript": "How many planets orbit the sun?", - "recentIntentSummaryView": null, - "sentimentResponse": null, - "kendraResponse": null, - "errorFound": false - }, - "_settings": { - "ENABLE_DEBUG_RESPONSES": "false", - "ES_USE_KEYWORD_FILTERS": "true", - "ES_NO_HITS_QUESTION": "no_hits", - "ES_USE_FUZZY_MATCH": "false", - "ES_KEYWORD_SYNTAX_TYPES": "NOUN,PROPN,VERB,INTJ", - "ES_SYNTAX_CONFIDENCE_LIMIT": ".20", - "ES_MINIMUM_SHOULD_MATCH": "2<75%", - "ES_PHRASE_BOOST": "4", - "ES_SCORE_ANSWER_FIELD": "false", - "ERRORMESSAGE": "Unfortunately I encountered an error when searching for your answer. Please ask me again later.", - "EMPTYMESSAGE": "You stumped me! Sadly I don't know how to answer your question.", - "DEFAULT_ALEXA_LAUNCH_MESSAGE": "Hello, Please ask a question", - "DEFAULT_ALEXA_STOP_MESSAGE": "Goodbye", - "SMS_HINT_REMINDER_ENABLE": "true", - "SMS_HINT_REMINDER": " (Feedback? Reply THUMBS UP or THUMBS DOWN. Ask HELP ME at any time)", - "SMS_HINT_REMINDER_INTERVAL_HRS": "24", - "IDENTITY_PROVIDER_JWKS_URLS": [], - "ENABLE_MULTI_LANGUAGE_SUPPORT": "false", - "MINIMUM_CONFIDENCE_SCORE": 0.6, - "ALT_SEARCH_KENDRA_INDEXES": "[\"e1c23860-e5c8-4409-ae26-b05bd6ced00a\"]", - "ELICIT_RESPONSE_MAX_RETRIES": 3, - "ELICIT_RESPONSE_RETRY_MESSAGE": "Please try again?", - "ELICIT_RESPONSE_BOT_FAILURE_MESSAGE": "Your response was not understood. Please start again.", - "ELICIT_RESPONSE_DEFAULT_MSG": "Ok. ", - "ENABLE_REDACTING": "false", - "REDACTING_REGEX": "\\b\\d{4}\\b(?![-])|\\b\\d{9}\\b|\\b\\d{3}-\\d{2}-\\d{4}\\b", - "DEFAULT_USER_POOL_JWKS_URL": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_jt2cXkCEb/.well-known/jwks.json" - }, - "_type": "LEX", - "_userId": "us-east-1:7ace124d-bfd1-4356-ac5a-ed8a98785fd9", - "question": "How many planets orbit the sun?", - "session": {}, - "_preferredResponseType": "PlainText", - "_userInfo": { - "UserId": "us-east-1:7ace124d-bfd1-4356-ac5a-ed8a98785fd9", - "InteractionCount": 0, - "TimeSinceLastInteraction": 1593663418.209 - }, - "_info": { - "es": { - "address": "search-qna-dev-elasti-mtl6ojldf3ja-w5begr25vpfuyidnlgb4izlbtq.us-east-1.es.amazonaws.com", - "index": "qna-dev-dev-dev-master-2", - "type": "qna", - "service": { - "qid": "QNA-dev-dev-dev-master-2-ESQidLambda-1I9JEHU6U4K9I", - "proxy": "QNA-dev-dev-dev-master-2-ESProxyLambda-1NMAY6LNBRJDY" - } - } - } - }, - "res": { - "type": "PlainText", - "message": "The Kendra Fallback search was not able to identify any results", - "session": { - "appContext": { - "altMessages": {} - }, - "previous": { - "qid": "KendraFallback", - "a": "The Kendra Fallback search was not able to identify any results", - "alt": {}, - "q": "How many planets orbit the sun?" - }, - "navigation": { - "next": "", - "previous": [], - "hasParent": true - } - }, - "card": { - "send": false, - "title": "", - "text": "", - "url": "" - }, - "_userInfo": { - "UserId": "us-east-1:7ace124d-bfd1-4356-ac5a-ed8a98785fd9", - "InteractionCount": 1, - "TimeSinceLastInteraction": 1593663418.209, - "FirstSeen": "Thu Jul 02 2020 16:16:58 GMT+0000 (Coordinated Universal Time)", - "LastSeen": "Thu Jul 02 2020 16:16:58 GMT+0000 (Coordinated Universal Time)" - }, - "got_hits": 0, - "result": { - "qid": "KendraFallback", - "quniqueterms": " no_hits ", - "questions": [ - { - "q": "no_hits" - } - ], - "a": "The Kendra Fallback search was not able to identify any results", - "l": "QNA:EXTKendraFallback", - "type": "qna", - "autotranslate": { - "a": true - } - }, - "plainMessage": "The Kendra Fallback search was not able to identify any results" - } -} diff --git a/test-do-not-use-obsolete/kendra_tests/event_doc_query.json b/test-do-not-use-obsolete/kendra_tests/event_doc_query.json deleted file mode 100644 index ed91ded36..000000000 --- a/test-do-not-use-obsolete/kendra_tests/event_doc_query.json +++ /dev/null @@ -1,179 +0,0 @@ -{ - "req": { - "_event": { - "messageVersion": "1.0", - "invocationSource": "FulfillmentCodeHook", - "userId": "us-east-1:36231782-ac61-483c-b4ab-13c20c1efb57", - "sessionAttributes": { - "kendraResponsibleQid": "KendraFallback", - "navigation": "{\"next\":\"\",\"previous\":[],\"hasParent\":true}", - "previous": "{\"qid\":\"KendraFallback\",\"a\":\"The Kendra Fallback search was not able to identify any results\",\"alt\":{},\"q\":\"How many regions does the sun have?\"}", - "qnabotcontext": "{\"kendra\":{\"kendraIndexId\":\"e1c23860-e5c8-4409-ae26-b05bd6ced00a\", \"kendraQueryId\":\"9b04b1d3-2409-43de-9d07-8c0ce3979e37\", \"kendraResultId\":\"9b04b1d3-2409-43de-9d07-8c0ce3979e37-2f0c0b44-bd10-4f23-9969-702e97a2286a\"}}" - }, - "requestAttributes": null, - "bot": { - "name": "QNA_dev_dev_dev_master_one_BotpvdFf", - "alias": "live", - "version": "1" - }, - "outputDialogMode": "Text", - "currentIntent": { - "name": "fulfilment_IntentDOKFOzfCnr", - "slots": { - "slot": "What is the composition of the sun" - }, - "slotDetails": { - "slot": { - "resolutions": [], - "originalValue": "What is the composition of the sun" - } - }, - "confirmationStatus": "None" - }, - "inputTranscript": "What is the composition of the sun?", - "recentIntentSummaryView": [ - { - "intentName": "fulfilment_IntentDOKFOzfCnr", - "checkpointLabel": null, - "slots": { - "slot": "How many regions does the sun have" - }, - "confirmationStatus": "None", - "dialogActionType": "Close", - "fulfillmentState": "Fulfilled", - "slotToElicit": null - } - ], - "sentimentResponse": null, - "kendraResponse": null, - "errorFound": false - }, - "_settings": { - "ENABLE_DEBUG_RESPONSES": "false", - "ES_USE_KEYWORD_FILTERS": "true", - "ES_NO_HITS_QUESTION": "no_hits", - "ES_USE_FUZZY_MATCH": "false", - "ES_KEYWORD_SYNTAX_TYPES": "NOUN,PROPN,VERB,INTJ", - "ES_SYNTAX_CONFIDENCE_LIMIT": ".20", - "ES_MINIMUM_SHOULD_MATCH": "2<75%", - "ES_PHRASE_BOOST": "4", - "ES_SCORE_ANSWER_FIELD": "false", - "ERRORMESSAGE": "Unfortunately I encountered an error when searching for your answer. Please ask me again later.", - "EMPTYMESSAGE": "You stumped me! Sadly I don't know how to answer your question.", - "DEFAULT_ALEXA_LAUNCH_MESSAGE": "Hello, Please ask a question", - "DEFAULT_ALEXA_STOP_MESSAGE": "Goodbye", - "SMS_HINT_REMINDER_ENABLE": "true", - "SMS_HINT_REMINDER": " (Feedback? Reply THUMBS UP or THUMBS DOWN. Ask HELP ME at any time)", - "SMS_HINT_REMINDER_INTERVAL_HRS": "24", - "IDENTITY_PROVIDER_JWKS_URLS": [], - "ENABLE_MULTI_LANGUAGE_SUPPORT": "false", - "MINIMUM_CONFIDENCE_SCORE": 0.6, - "ALT_SEARCH_KENDRA_INDEXES": "[\"e1c23860-e5c8-4409-ae26-b05bd6ced00a\"]", - "ELICIT_RESPONSE_MAX_RETRIES": 3, - "ELICIT_RESPONSE_RETRY_MESSAGE": "Please try again?", - "ELICIT_RESPONSE_BOT_FAILURE_MESSAGE": "Your response was not understood. Please start again.", - "ELICIT_RESPONSE_DEFAULT_MSG": "Ok. ", - "ENABLE_REDACTING": "false", - "REDACTING_REGEX": "\\b\\d{4}\\b(?![-])|\\b\\d{9}\\b|\\b\\d{3}-\\d{2}-\\d{4}\\b", - "DEFAULT_USER_POOL_JWKS_URL": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_asIXCtVxR/.well-known/jwks.json" - }, - "_type": "LEX", - "_userId": "us-east-1:36231782-ac61-483c-b4ab-13c20c1efb57", - "question": "What is the composition of the sun?", - "session": { - "navigation": { - "next": "", - "previous": [], - "hasParent": true - }, - "previous": { - "qid": "KendraFallback", - "a": "The Kendra Fallback search was not able to identify any results", - "alt": {}, - "q": "How many regions does the sun have?" - }, - "qnabotcontext": { - "kendra":{ - "kendraIndexId":"e1c23860-e5c8-4409-ae26-b05bd6ced00a", - "kendraQueryId":"9b04b1d3-2409-43de-9d07-8c0ce3979e37", - "kendraResultId":"9b04b1d3-2409-43de-9d07-8c0ce3979e37-2f0c0b44-bd10-4f23-9969-702e97a2286a", - "kendraResponsibleQid": "KendraFallback" - } - } - }, - "_preferredResponseType": "PlainText", - "_userInfo": { - "InteractionCount": 3, - "UserId": "us-east-1:36231782-ac61-483c-b4ab-13c20c1efb57", - "FirstSeen": "Thu Jun 25 2020 16:59:02 GMT+0000 (Coordinated Universal Time)", - "LastSeen": "Thu Jun 25 2020 16:59:28 GMT+0000 (Coordinated Universal Time)", - "TimeSinceLastInteraction": 11.653 - }, - "_info": { - "es": { - "address": "search-qna-dev-elasti-43wg8xxw528o-hrcx54m7voo7lncykifol6ucuy.us-east-1.es.amazonaws.com", - "index": "qna-dev-dev-dev-master-1", - "type": "qna", - "service": { - "qid": "QNA-dev-dev-dev-master-1-ESQidLambda-1WWM866K5R7G5", - "proxy": "QNA-dev-dev-dev-master-1-ESProxyLambda-F87F2OUBD8KW" - } - } - } - }, - "res": { - "type": "PlainText", - "message": "The Kendra Fallback search was not able to identify any results", - "session": { - "kendraResponsibleQid": "KendraFallback", - "navigation": { - "next": "", - "previous": [], - "hasParent": true - }, - "previous": { - "qid": "KendraFallback", - "a": "The Kendra Fallback search was not able to identify any results", - "alt": {}, - "q": "What is the composition of the sun?" - }, - "kendraIndexId": "e1c23860-e5c8-4409-ae26-b05bd6ced00a", - "kendraQueryId": "9b04b1d3-2409-43de-9d07-8c0ce3979e37", - "kendraResultId": "9b04b1d3-2409-43de-9d07-8c0ce3979e37-2f0c0b44-bd10-4f23-9969-702e97a2286a", - "appContext": { - "altMessages": {} - } - }, - "card": { - "send": false, - "title": "", - "text": "", - "url": "" - }, - "_userInfo": { - "InteractionCount": 4, - "UserId": "us-east-1:36231782-ac61-483c-b4ab-13c20c1efb57", - "FirstSeen": "Thu Jun 25 2020 16:59:02 GMT+0000 (Coordinated Universal Time)", - "LastSeen": "Thu Jun 25 2020 16:59:39 GMT+0000 (Coordinated Universal Time)", - "TimeSinceLastInteraction": 11.653 - }, - "got_hits": 0, - "result": { - "qid": "KendraFallback", - "quniqueterms": " no_hits ", - "questions": [ - { - "q": "no_hits" - } - ], - "a": "The Kendra Fallback search was not able to identify any results", - "l": "QNA:EXTKendraFallback", - "type": "qna", - "autotranslate": { - "a": true - } - }, - "plainMessage": "The Kendra Fallback search was not able to identify any results" - } -} - diff --git a/test-do-not-use-obsolete/kendra_tests/event_top_ans.json b/test-do-not-use-obsolete/kendra_tests/event_top_ans.json deleted file mode 100644 index dc3b499bd..000000000 --- a/test-do-not-use-obsolete/kendra_tests/event_top_ans.json +++ /dev/null @@ -1,178 +0,0 @@ -{ - "req": { - "_event": { - "messageVersion": "1.0", - "invocationSource": "FulfillmentCodeHook", - "userId": "us-east-1:4f20bd96-a761-4541-a1d9-fcbb0e51c236", - "sessionAttributes": { - "navigation": "{\"next\":\"\",\"previous\":[],\"hasParent\":true}", - "userLocale": "en", - "previous": "{\"qid\":\"KendraFallback\",\"a\":\"The Kendra Fallback search was not able to identify any results\",\"alt\":{},\"q\":\"How many layers does the sun have?\"}", - "userDetectedLocaleConfidence": "0.9988220930099487", - "qnabotcontext": { - "kendra":{ - "kendraIndexId":"e1c23860-e5c8-4409-ae26-b05bd6ced00a", - "kendraQueryId":"09c519d4-4b9a-4355-a586-e25e64285c4f", - "kendraResultId":"09c519d4-4b9a-4355-a586-e25e64285c4f-3527f152-b335-4619-a934-b5d7eecd1982", - "kendraResponsibleQid": "KendraFallback" - } - }, - "userDetectedLocale": "en" - }, - "requestAttributes": null, - "bot": { - "name": "QNAbot_dev_dev_dev_master_four_BotoNDIv", - "alias": "live", - "version": "1" - }, - "outputDialogMode": "Text", - "currentIntent": { - "name": "fulfilment_IntentrDEldIYchy", - "slots": { - "slot": "What is the closest star to the earth" - }, - "slotDetails": { - "slot": { - "resolutions": [], - "originalValue": "What is the closest star to the earth" - } - }, - "confirmationStatus": "None" - }, - "inputTranscript": "What is the closest star to the earth?", - "recentIntentSummaryView": null, - "sentimentResponse": null, - "kendraResponse": null, - "origQuestion": "What is the closest star to the earth?", - "errorFound": false - }, - "_settings": { - "ENABLE_DEBUG_RESPONSES": "false", - "ES_USE_KEYWORD_FILTERS": "true", - "ES_NO_HITS_QUESTION": "no_hits", - "ES_USE_FUZZY_MATCH": "false", - "ES_KEYWORD_SYNTAX_TYPES": "NOUN,PROPN,VERB,INTJ", - "ES_SYNTAX_CONFIDENCE_LIMIT": ".20", - "ES_MINIMUM_SHOULD_MATCH": "2<75%", - "ES_PHRASE_BOOST": "4", - "ES_SCORE_ANSWER_FIELD": "false", - "ERRORMESSAGE": "Unfortunately I encountered an error when searching for your answer. Please ask me again later.", - "EMPTYMESSAGE": "You stumped me! Sadly I don't know how to answer your question.", - "DEFAULT_ALEXA_LAUNCH_MESSAGE": "Hello, Please ask a question", - "DEFAULT_ALEXA_STOP_MESSAGE": "Goodbye", - "SMS_HINT_REMINDER_ENABLE": "true", - "SMS_HINT_REMINDER": " (Feedback? Reply THUMBS UP or THUMBS DOWN. Ask HELP ME at any time)", - "SMS_HINT_REMINDER_INTERVAL_HRS": "24", - "IDENTITY_PROVIDER_JWKS_URLS": [], - "ENABLE_MULTI_LANGUAGE_SUPPORT": "true", - "MINIMUM_CONFIDENCE_SCORE": 0.6, - "ALT_SEARCH_KENDRA_INDEXES": "[\"e1c23860-e5c8-4409-ae26-b05bd6ced00a\"]", - "ELICIT_RESPONSE_MAX_RETRIES": 3, - "ELICIT_RESPONSE_RETRY_MESSAGE": "Please try again?", - "ELICIT_RESPONSE_BOT_FAILURE_MESSAGE": "Your response was not understood. Please start again.", - "ELICIT_RESPONSE_DEFAULT_MSG": "Ok. ", - "ENABLE_REDACTING": "false", - "REDACTING_REGEX": "\\b\\d{4}\\b(?![-])|\\b\\d{9}\\b|\\b\\d{3}-\\d{2}-\\d{4}\\b", - "DEFAULT_USER_POOL_JWKS_URL": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_EZg5Ia82Z/.well-known/jwks.json" - }, - "_type": "LEX", - "_userId": "us-east-1:4f20bd96-a761-4541-a1d9-fcbb0e51c236", - "question": "What is the closest star to the earth?", - "session": { - "kendraResponsibleQid": "KendraFallback", - "navigation": { - "next": "", - "previous": [], - "hasParent": true - }, - "userLocale": "en", - "previous": { - "qid": "KendraFallback", - "a": "The Kendra Fallback search was not able to identify any results", - "alt": {}, - "q": "How many layers does the sun have?" - }, - "userDetectedLocaleConfidence": 0.9990218877792358, - "kendraIndexId": "e1c23860-e5c8-4409-ae26-b05bd6ced00a", - "kendraQueryId": "09c519d4-4b9a-4355-a586-e25e64285c4f", - "userDetectedLocale": "en", - "kendraResultId": "09c519d4-4b9a-4355-a586-e25e64285c4f-3527f152-b335-4619-a934-b5d7eecd1982" - }, - "_preferredResponseType": "PlainText", - "_userInfo": { - "InteractionCount": 1, - "UserId": "us-east-1:4f20bd96-a761-4541-a1d9-fcbb0e51c236", - "FirstSeen": "Wed Jun 24 2020 17:18:02 GMT+0000 (Coordinated Universal Time)", - "LastSeen": "Wed Jun 24 2020 17:18:02 GMT+0000 (Coordinated Universal Time)", - "TimeSinceLastInteraction": 746.572 - }, - "_info": { - "es": { - "address": "search-qnabot-elasti-4n2sjlh1tlwf-ziyzyejzynvs6rs2cvx7yaqxma.us-east-1.es.amazonaws.com", - "index": "qnabot-dev-dev-dev-master-4", - "type": "qna", - "service": { - "qid": "QNAbot-dev-dev-dev-master-4-ESQidLambda-636ARD1AGUJH", - "proxy": "QNAbot-dev-dev-dev-master-4-ESProxyLambda-NF5QMNB8RE00" - } - } - } - }, - "res": { - "type": "PlainText", - "message": "The Kendra Fallback search was not able to identify any results", - "session": { - "kendraResponsibleQid": "KendraFallback", - "navigation": { - "next": "", - "previous": [], - "hasParent": true - }, - "userLocale": "en", - "previous": { - "qid": "KendraFallback", - "a": "The Kendra Fallback search was not able to identify any results", - "alt": {}, - "q": "What is the closest star to the earth?" - }, - "userDetectedLocaleConfidence": 0.9990218877792358, - "kendraIndexId": "e1c23860-e5c8-4409-ae26-b05bd6ced00a", - "kendraQueryId": "09c519d4-4b9a-4355-a586-e25e64285c4f", - "userDetectedLocale": "en", - "kendraResultId": "09c519d4-4b9a-4355-a586-e25e64285c4f-3527f152-b335-4619-a934-b5d7eecd1982", - "appContext": { - "altMessages": {} - } - }, - "card": { - "send": false, - "title": "", - "text": "", - "url": "" - }, - "_userInfo": { - "InteractionCount": 2, - "UserId": "us-east-1:4f20bd96-a761-4541-a1d9-fcbb0e51c236", - "FirstSeen": "Wed Jun 24 2020 17:18:02 GMT+0000 (Coordinated Universal Time)", - "LastSeen": "Wed Jun 24 2020 17:30:28 GMT+0000 (Coordinated Universal Time)", - "TimeSinceLastInteraction": 746.572 - }, - "got_hits": 0, - "result": { - "qid": "KendraFallback", - "quniqueterms": " no_hits ", - "questions": [ - { - "q": "no_hits" - } - ], - "a": "The Kendra Fallback search was not able to identify any results", - "l": "QNA:EXTKendraFallback", - "type": "qna", - "autotranslate": { - "a": true - } - }, - "plainMessage": "The Kendra Fallback search was not able to identify any results" - } -} diff --git a/test-do-not-use-obsolete/kendra_tests/kendraData_FAQ_md.json b/test-do-not-use-obsolete/kendra_tests/kendraData_FAQ_md.json deleted file mode 100644 index c2d3098da..000000000 --- a/test-do-not-use-obsolete/kendra_tests/kendraData_FAQ_md.json +++ /dev/null @@ -1,94 +0,0 @@ -{ - "QueryId": "b904542d-af85-447d-808b-d86b5df16c06", - "ResultItems": [ - { - "Id": "b904542d-af85-447d-808b-d86b5df16c06-ee0db51d-15f4-4786-9144-b51d29d37f66", - "Type": "QUESTION_ANSWER", - "AdditionalAttributes": [ - { - "Key": "QuestionText", - "ValueType": "TEXT_WITH_HIGHLIGHTS_VALUE", - "Value": { - "TextWithHighlightsValue": { - "Text": "How many planets orbit the Sun?", - "Highlights": [ - { - "BeginOffset": 4, - "EndOffset": 8, - "TopAnswer": false - }, - { - "BeginOffset": 9, - "EndOffset": 16, - "TopAnswer": false - }, - { - "BeginOffset": 17, - "EndOffset": 22, - "TopAnswer": false - }, - { - "BeginOffset": 27, - "EndOffset": 30, - "TopAnswer": false - } - ] - } - } - }, - { - "Key": "AnswerText", - "ValueType": "TEXT_WITH_HIGHLIGHTS_VALUE", - "Value": { - "TextWithHighlightsValue": { - "Text": "Eight planets orbit the sun. Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune. Sorry I'm required to leave out my favorite, Pluto.", - "Highlights": [ - { - "BeginOffset": 6, - "EndOffset": 13, - "TopAnswer": false - }, - { - "BeginOffset": 14, - "EndOffset": 19, - "TopAnswer": false - }, - { - "BeginOffset": 24, - "EndOffset": 27, - "TopAnswer": false - } - ] - } - } - } - ], - "DocumentId": "a212c9bbe37131a9777efc915d837915ee33e5b327ec7df4645cb96d6d165f4a1593702980960", - "DocumentTitle": { - "Text": "" - }, - "DocumentExcerpt": { - "Text": "Eight planets orbit the sun. Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune. Sorry I'm required to leave out my favorite, Pluto.", - "Highlights": [ - { - "BeginOffset": 0, - "EndOffset": 147, - "TopAnswer": false - } - ] - }, - "DocumentURI": "{\"args\":[\"\"],\"next\":\"\",\"a\":\"Eight planets orbit the sun. Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune. Sorry I'm required to leave out my favorite, Pluto.\",\"r\":{\"buttons\":[{\"text\":\"What is a solar flare?\",\"value\":\"What is a solar flare\"}],\"subTitle\":\"\",\"imageUrl\":\"\",\"title\":\"Suggestions\",\"text\":\"\",\"url\":\"\"},\"t\":\"\",\"alt\":{\"markdown\":\"\",\"ssml\":\"\"},\"l\":\"\",\"qid\":\"sun.10\",\"type\":\"qna\",\"q\":[\"How many planets orbit the Sun?\",\"Which planets orbit the Sun?\"]}", - "DocumentAttributes": [ - { - "Key": "_source_uri", - "Value": { - "StringValue": "{\"args\":[\"\"],\"next\":\"\",\"a\":\"Eight planets orbit the sun. Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune. Sorry I'm required to leave out my favorite, Pluto.\",\"r\":{\"buttons\":[{\"text\":\"What is a solar flare?\",\"value\":\"What is a solar flare\"}],\"subTitle\":\"\",\"imageUrl\":\"\",\"title\":\"Suggestions\",\"text\":\"\",\"url\":\"\"},\"t\":\"\",\"alt\":{\"markdown\":\"\",\"ssml\":\"\"},\"l\":\"\",\"qid\":\"sun.10\",\"type\":\"qna\",\"q\":[\"How many planets orbit the Sun?\",\"Which planets orbit the Sun?\"]}" - } - } - ] - } - ], - "FacetResults": [], - "TotalNumberOfResults": 1, - "originalKendraIndexId": "e1c23860-e5c8-4409-ae26-b05bd6ced00a" -} diff --git a/test-do-not-use-obsolete/kendra_tests/kendraData_doc_query.json b/test-do-not-use-obsolete/kendra_tests/kendraData_doc_query.json deleted file mode 100644 index fb4771dbc..000000000 --- a/test-do-not-use-obsolete/kendra_tests/kendraData_doc_query.json +++ /dev/null @@ -1,193 +0,0 @@ -{ - "QueryId": "7563d4b1-0646-46ae-9013-d04fdb7f0cb4", - "ResultItems": [ - { - "Id": "7563d4b1-0646-46ae-9013-d04fdb7f0cb4-cbf7edb4-e12c-43bc-aff9-c90fc2b0e68d", - "Type": "ANSWER", - "AdditionalAttributes": [ - { - "Key": "AnswerText", - "ValueType": "TEXT_WITH_HIGHLIGHTS_VALUE", - "Value": { - "TextWithHighlightsValue": { - "Text": "Anatomy of the Sun\nMysteries of the Sun\n\n\nThe Sun is an incandescent mass of hydrogen, helium, and other heavier elements. While it appears constant and unchanging from \nour vantage point on Earth, it actually has a dynamic and variable system of twisting magnetic fields that cause solar events of almost \nunimaginable power.\n\n\nThe Convection Zone\nEnergy continues to move toward the surface \nthrough convection currents of heated and \ncooled gas in the convection zone.\n\n\n \n\n\nThe Radiative Zone\nEnergy moves slowly outward—taking \nmore than 170,000 years to radiate through \nthe layer of the Sun known as the radiative \nzone.", - "Highlights": [ - { - "BeginOffset": 42, - "EndOffset": 121, - "TopAnswer": false - }, - { - "BeginOffset": 53, - "EndOffset": 121, - "TopAnswer": false - }, - { - "BeginOffset": 15, - "EndOffset": 18, - "TopAnswer": false - }, - { - "BeginOffset": 36, - "EndOffset": 39, - "TopAnswer": false - }, - { - "BeginOffset": 46, - "EndOffset": 49, - "TopAnswer": false - }, - { - "BeginOffset": 594, - "EndOffset": 597, - "TopAnswer": false - } - ] - } - } - } - ], - "DocumentId": "s3://explore-kendra-solar/637244main_MysteriesOfTheSun_Book.pdf", - "DocumentTitle": { - "Text": "637244main_MysteriesOfTheSun_Book" - }, - "DocumentExcerpt": { - "Text": "Anatomy of the Sun\nMysteries of the Sun\n\n\nThe Sun is an incandescent mass of hydrogen, helium, and other heavier elements. While it appears constant and unchanging from \nour vantage point on Earth, it actually has a dynamic and variable system of twisting magnetic fields that cause solar events of a", - "Highlights": [ - { - "BeginOffset": 0, - "EndOffset": 300, - "TopAnswer": false - } - ] - }, - "DocumentURI": "https://s3.us-east-1.amazonaws.com/explore-kendra-solar/637244main_MysteriesOfTheSun_Book.pdf", - "DocumentAttributes": [] - }, - { - "Id": "7563d4b1-0646-46ae-9013-d04fdb7f0cb4-fdcd7c67-02e9-4a73-8ffa-03fadd6cc880", - "Type": "ANSWER", - "AdditionalAttributes": [ - { - "Key": "AnswerText", - "ValueType": "TEXT_WITH_HIGHLIGHTS_VALUE", - "Value": { - "TextWithHighlightsValue": { - "Text": "a star, composed mostly of hydrogen, at the center of the Solar \n\n\nSystem, and with planets orbiting around it. But ancient people didn’t have access to the same tools we have today. \n\n\nTheir understanding about the Sun was far more concerned with the day-to-day needs of living. As such, their notions \n\n\nhave influenced the way we (still) think of the Sun.", - "Highlights": [ - { - "BeginOffset": 27, - "EndOffset": 35, - "TopAnswer": false - }, - { - "BeginOffset": 216, - "EndOffset": 219, - "TopAnswer": false - }, - { - "BeginOffset": 354, - "EndOffset": 357, - "TopAnswer": false - } - ] - } - } - } - ], - "DocumentId": "s3://explore-kendra-solar/637244main_MysteriesOfTheSun_Book.pdf", - "DocumentTitle": { - "Text": "637244main_MysteriesOfTheSun_Book" - }, - "DocumentExcerpt": { - "Text": "a star, composed mostly of hydrogen, at the center of the Solar \n\n\nSystem, and with planets orbiting around it. But ancient people didn’t have access to the same tools we have today. \n\n\nTheir understanding about the Sun was far more concerned with the day-to-day needs of living. As such, their notio", - "Highlights": [ - { - "BeginOffset": 0, - "EndOffset": 300, - "TopAnswer": false - } - ] - }, - "DocumentURI": "https://s3.us-east-1.amazonaws.com/explore-kendra-solar/637244main_MysteriesOfTheSun_Book.pdf", - "DocumentAttributes": [] - }, - { - "Id": "7563d4b1-0646-46ae-9013-d04fdb7f0cb4-9f4ad51f-5fc0-4d61-bc61-af58f0e4ba23", - "Type": "DOCUMENT", - "AdditionalAttributes": [], - "DocumentId": "s3://explore-kendra-solar/Sun_Lithograph.pdf", - "DocumentTitle": { - "Text": "Sun_Lithograph", - "Highlights": [ - { - "BeginOffset": 0, - "EndOffset": 3, - "TopAnswer": false - } - ] - }, - "DocumentExcerpt": { - "Text": "...Mass 1.989 × 1030 kg \nDensity 1.409 g/cm3 \nComposition 92.1% hydrogen, 7.8% helium, \n 0.1% other elements \nSurface Temperature (Photosphere) 5,500 deg C \n\n\n(10,000 deg F) \nLuminosity* 3.83 × 1033 ergs/sec\n\n\n*The total energy radiated by the Sun (or any star) per second at all wavelengths...", - "Highlights": [ - { - "BeginOffset": 46, - "EndOffset": 57, - "TopAnswer": false - }, - { - "BeginOffset": 248, - "EndOffset": 251, - "TopAnswer": false - } - ] - }, - "DocumentURI": "https://s3.us-east-1.amazonaws.com/explore-kendra-solar/Sun_Lithograph.pdf", - "DocumentAttributes": [ - { - "Key": "_source_uri", - "Value": { - "StringValue": "https://s3.us-east-1.amazonaws.com/explore-kendra-solar/Sun_Lithograph.pdf" - } - } - ] - }, - { - "Id": "7563d4b1-0646-46ae-9013-d04fdb7f0cb4-8ef98066-6bbe-4317-8ea9-346d3b34b7ae", - "Type": "DOCUMENT", - "AdditionalAttributes": [], - "DocumentId": "s3://explore-kendra-solar/637244main_MysteriesOfTheSun_Book.pdf", - "DocumentTitle": { - "Text": "637244main_MysteriesOfTheSun_Book", - "Highlights": [] - }, - "DocumentExcerpt": { - "Text": "...gases may \ncontribute to a change in tempera-\nture or water composition in \nthe atmosphere. \n\n\n \n\n\nStratosphere\n10–31 Miles\n\n\nThe ozone layer lies within the \nstratosphere and absorbs ultraviolet \nradiation from the Sun.\n\n\nTroposphere\n0–10 Miles\n\n\nThe troposphere is the layer of the...", - "Highlights": [ - { - "BeginOffset": 63, - "EndOffset": 74, - "TopAnswer": false - }, - { - "BeginOffset": 221, - "EndOffset": 224, - "TopAnswer": false - } - ] - }, - "DocumentURI": "https://s3.us-east-1.amazonaws.com/explore-kendra-solar/637244main_MysteriesOfTheSun_Book.pdf", - "DocumentAttributes": [ - { - "Key": "_source_uri", - "Value": { - "StringValue": "https://s3.us-east-1.amazonaws.com/explore-kendra-solar/637244main_MysteriesOfTheSun_Book.pdf" - } - } - ] - } - ], - "FacetResults": [], - "TotalNumberOfResults": 4, - "originalKendraIndexId": "e1c23860-e5c8-4409-ae26-b05bd6ced00a" -} diff --git a/test-do-not-use-obsolete/kendra_tests/kendraData_top_ans.json b/test-do-not-use-obsolete/kendra_tests/kendraData_top_ans.json deleted file mode 100644 index 406e760c2..000000000 --- a/test-do-not-use-obsolete/kendra_tests/kendraData_top_ans.json +++ /dev/null @@ -1,164 +0,0 @@ -{ - "QueryId": "9f2d96e0-dbed-4192-a5b8-c8a9a86dc738", - "ResultItems": [ - { - "Id": "9f2d96e0-dbed-4192-a5b8-c8a9a86dc738-f46c83da-5aac-4200-8a6e-440a2867a7e2", - "Type": "ANSWER", - "AdditionalAttributes": [ - { - "Key": "AnswerText", - "ValueType": "TEXT_WITH_HIGHLIGHTS_VALUE", - "Value": { - "TextWithHighlightsValue": { - "Text": "Ancient Greek \nastronomers and philosophers knew this “geocentric” concept \nfrom as early as the 6th century BCE. Now we know, of course, \nthat all the planets orbit our lone star — the Sun.\n\n\nThe Sun is the closest star to Earth, at a mean distance from \nour planet of 149.60 million kilometers (92.96 million miles). This \ndistance is known as an astronomical unit (abbreviated AU), and \nsets the scale for measuring distances all across the solar sys-\ntem. The Sun, a huge sphere of mostly ionized gas, supports life \non Earth. The connection and interactions between the Sun and \nEarth drive the seasons, ocean currents, weather, and climate.", - "Highlights": [ - { - "BeginOffset": 186, - "EndOffset": 189, - "TopAnswer": true - }, - { - "BeginOffset": 182, - "EndOffset": 189, - "TopAnswer": false - }, - { - "BeginOffset": 175, - "EndOffset": 179, - "TopAnswer": false - }, - { - "BeginOffset": 208, - "EndOffset": 215, - "TopAnswer": false - }, - { - "BeginOffset": 216, - "EndOffset": 220, - "TopAnswer": false - }, - { - "BeginOffset": 224, - "EndOffset": 229, - "TopAnswer": false - }, - { - "BeginOffset": 524, - "EndOffset": 529, - "TopAnswer": false - }, - { - "BeginOffset": 584, - "EndOffset": 589, - "TopAnswer": false - } - ] - } - } - } - ], - "DocumentId": "s3://explore-kendra-solar/Sun_Lithograph.pdf", - "DocumentTitle": { - "Text": "Sun_Lithograph" - }, - "DocumentExcerpt": { - "Text": "Ancient Greek \nastronomers and philosophers knew this “geocentric” concept \nfrom as early as the 6th century BCE. Now we know, of course, \nthat all the planets orbit our lone star — the Sun.\n\n\nThe Sun is the closest star to Earth, at a mean distance from \nour planet of 149.60 million kilometers (92.", - "Highlights": [ - { - "BeginOffset": 0, - "EndOffset": 300, - "TopAnswer": false - } - ] - }, - "DocumentURI": "https://s3.us-east-1.amazonaws.com/explore-kendra-solar/Sun_Lithograph.pdf", - "DocumentAttributes": [] - }, - { - "Id": "9f2d96e0-dbed-4192-a5b8-c8a9a86dc738-0b7134ed-257c-4de1-9b5f-01bef33935d6", - "Type": "DOCUMENT", - "AdditionalAttributes": [], - "DocumentId": "s3://explore-kendra-solar/Sun_Lithograph.pdf", - "DocumentTitle": { - "Text": "Sun_Lithograph", - "Highlights": [] - }, - "DocumentExcerpt": { - "Text": "...from as early as the 6th century BCE. Now we know, of course, \nthat all the planets orbit our lone star — the Sun.\n\n\nThe Sun is the closest star to Earth, at a mean distance from \nour planet of 149.60 million kilometers (92.96 million miles...", - "Highlights": [ - { - "BeginOffset": 102, - "EndOffset": 106, - "TopAnswer": false - }, - { - "BeginOffset": 135, - "EndOffset": 142, - "TopAnswer": false - }, - { - "BeginOffset": 143, - "EndOffset": 147, - "TopAnswer": false - }, - { - "BeginOffset": 151, - "EndOffset": 156, - "TopAnswer": false - } - ] - }, - "DocumentURI": "https://s3.us-east-1.amazonaws.com/explore-kendra-solar/Sun_Lithograph.pdf", - "DocumentAttributes": [ - { - "Key": "_source_uri", - "Value": { - "StringValue": "https://s3.us-east-1.amazonaws.com/explore-kendra-solar/Sun_Lithograph.pdf" - } - } - ] - }, - { - "Id": "9f2d96e0-dbed-4192-a5b8-c8a9a86dc738-511c93c5-0285-4014-b7bf-a4d9fec683e2", - "Type": "DOCUMENT", - "AdditionalAttributes": [], - "DocumentId": "s3://explore-kendra-solar/637244main_MysteriesOfTheSun_Book.pdf", - "DocumentTitle": { - "Text": "637244main_MysteriesOfTheSun_Book", - "Highlights": [] - }, - "DocumentExcerpt": { - "Text": "...Solar Cycle\n\n\nSolar Storms\n\n\nEarth’s Magnetosphere\n\n\nEarth’s Upper Atmosphere\n\n\nSpace Weather\n\n\nCredits\n\n\n2\n\n\n4\n\n\n6\n\n\n8\n\n\n10\n\n\n14\n\n\n16\n\n\n18\n\n\n\n\n\n\n\nPrologue and Introduction\n\n\nNow in the early 21st century, we know that the Sun is a star, composed mostly of hydrogen, at the center of the Solar...", - "Highlights": [ - { - "BeginOffset": 32, - "EndOffset": 37, - "TopAnswer": false - }, - { - "BeginOffset": 56, - "EndOffset": 61, - "TopAnswer": false - }, - { - "BeginOffset": 235, - "EndOffset": 239, - "TopAnswer": false - } - ] - }, - "DocumentURI": "https://s3.us-east-1.amazonaws.com/explore-kendra-solar/637244main_MysteriesOfTheSun_Book.pdf", - "DocumentAttributes": [ - { - "Key": "_source_uri", - "Value": { - "StringValue": "https://s3.us-east-1.amazonaws.com/explore-kendra-solar/637244main_MysteriesOfTheSun_Book.pdf" - } - } - ] - } - ], - "FacetResults": [], - "TotalNumberOfResults": 3, - "originalKendraIndexId": "e1c23860-e5c8-4409-ae26-b05bd6ced00a" -} diff --git a/test-do-not-use-obsolete/kendra_tests/mockClient.js b/test-do-not-use-obsolete/kendra_tests/mockClient.js deleted file mode 100644 index 6c4381f0f..000000000 --- a/test-do-not-use-obsolete/kendra_tests/mockClient.js +++ /dev/null @@ -1,7 +0,0 @@ -// mockClient.js - -// query calls func(err, data) to return the Data from Kendra JSON from the CloudWatch log -exports.query = function(params, func) { - const data = require('./kendraData_FAQ_md.json'); - func(undefined, data); -}; diff --git a/test-do-not-use-obsolete/kendra_tests/mockClient1.js b/test-do-not-use-obsolete/kendra_tests/mockClient1.js deleted file mode 100644 index 6c4381f0f..000000000 --- a/test-do-not-use-obsolete/kendra_tests/mockClient1.js +++ /dev/null @@ -1,7 +0,0 @@ -// mockClient.js - -// query calls func(err, data) to return the Data from Kendra JSON from the CloudWatch log -exports.query = function(params, func) { - const data = require('./kendraData_FAQ_md.json'); - func(undefined, data); -}; diff --git a/test-do-not-use-obsolete/kendra_tests/mockClient2.js b/test-do-not-use-obsolete/kendra_tests/mockClient2.js deleted file mode 100644 index 81f4040ec..000000000 --- a/test-do-not-use-obsolete/kendra_tests/mockClient2.js +++ /dev/null @@ -1,7 +0,0 @@ -// mockClient2.js for testing top_ans feature - -// query calls func(err, data) to return the Data from Kendra JSON from the CloudWatch log -exports.query = function(params, func) { - const data = require('./kendraData_top_ans.json'); - func(undefined, data); -}; diff --git a/test-do-not-use-obsolete/kendra_tests/mockClient3.js b/test-do-not-use-obsolete/kendra_tests/mockClient3.js deleted file mode 100644 index 302ed51e4..000000000 --- a/test-do-not-use-obsolete/kendra_tests/mockClient3.js +++ /dev/null @@ -1,7 +0,0 @@ -// mockClient3.js - -// query calls func(err, data) to return the Data from Kendra JSON from the CloudWatch log -exports.query = function(params, func) { - const data = require('./kendraData_doc_query.json'); - func(undefined, data); -}; diff --git a/test-do-not-use-obsolete/kendra_tests/run_test.js b/test-do-not-use-obsolete/kendra_tests/run_test.js deleted file mode 100644 index d59d9c278..000000000 --- a/test-do-not-use-obsolete/kendra_tests/run_test.js +++ /dev/null @@ -1,63 +0,0 @@ -var assert = require('assert'); - -// kendra fallback test -const kendraFallback = require('../KendraFallback.js'); - -async function test_markdown() { - const event = require('./event_FAQ_md.json'); - const context = require('./context_FAQ_md.json'); - event.test = 1; - const actual_resp = await kendraFallback.handler(event, context); - return actual_resp; -} - -async function test_top_ans() { - const event = require('./event_top_ans.json'); - const context = require('./context_top_ans.json'); - event.test = 2; - const actual_resp = await kendraFallback.handler(event, context); - return actual_resp; -} - -async function test_doc_query() { - const event = require('./event_doc_query.json'); - const context = require('./context_doc_query.json'); - event.test = 3; - const actual_resp = await kendraFallback.handler(event, context); - return actual_resp; -} - - -describe('#test_kendra_highlights()', () => { - it('test_markdown', async function() { - let resp = await test_markdown(); - - // tests that in markdown format, highlights are boldened - assert.equal(resp.res.session.appContext.altMessages.markdown, - "*Answer from Amazon Kendra FAQ.* \n \n\nEight **planets** **orbit** the **sun**. Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune. Sorry I'm required to leave out my favorite, Pluto."); - }); - it('test_top_answer', async function() { - let resp = await test_top_ans(); - - // tests that in markdown format, only the top answer phrase is returned with the link from where it is extracted - assert.equal(resp.res.session.appContext.altMessages.markdown, - "*Amazon Kendra suggested answer.* \n \n\n**Sun** \n\n Source Link: [https://s3.us-east-1.amazonaws.com/explore-kendra-solar/Sun_Lithograph.pdf](https://s3.us-east-1.amazonaws.com/explore-kendra-solar/Sun_Lithograph.pdf)"); - - // tests that in SSML format, only the top answer phrase is returned - assert.equal(resp.res.session.appContext.altMessages.ssml, - " Amazon Kendra suggested answer. Sun "); - }); - it('test_doc_query', async function() { - let resp = await test_doc_query(); - - // tests that when querying an unstructured document, highlights are boldened in the excerpts when no top answer is found - assert.equal(resp.res.session.appContext.altMessages.markdown, - "*While I did not find an exact answer, these search results from Amazon Kendra might be helpful.* \n \n\nAnatomy of the Sun Mysteries of the Sun **The Sun is an incandescent mass of hydrogen, helium, and other heavier elements**. While it appears constant and unchanging from our vantage point on Earth, it actually has a dynamic and variable system of twisting magnetic fields that cause solar events of almost unimaginable power. The Convection Zone Energy continues to move toward the surface through convection currents of heated and cooled gas in the convection zone. The Radiative Zone Energy moves slowly outward—taking more than 170,000 years to radiate through the layer of the **Sun** known as the radiative zone.\n\n Source Link: [https://s3.us-east-1.amazonaws.com/explore-kendra-solar/637244main_MysteriesOfTheSun_Book.pdf](https://s3.us-east-1.amazonaws.com/explore-kendra-solar/637244main_MysteriesOfTheSun_Book.pdf)\n\n***\n\n
\n\n ...Mass 1.989 × 1030 kg Density 1.409 g/cm3 **Composition** 92.1% hydrogen, 7.8% helium, 0.1% other elements Surface Temperature (Photosphere) 5,500 deg C (10,000 deg F) Luminosity* 3.83 × 1033 ergs/sec *The total energy radiated by the **Sun** (or any star) per second at all wavelengths...\n\n Source Link: [https://s3.us-east-1.amazonaws.com/explore-kendra-solar/Sun_Lithograph.pdf](https://s3.us-east-1.amazonaws.com/explore-kendra-solar/Sun_Lithograph.pdf)\n\n***\n\n
\n\n ...gases may contribute to a change in tempera- ture or water **composition** in the atmosphere. Stratosphere 10–31 Miles The ozone layer lies within the stratosphere and absorbs ultraviolet radiation from the **Sun**. Troposphere 0–10 Miles The troposphere is the layer of the...\n\n Source Link: [https://s3.us-east-1.amazonaws.com/explore-kendra-solar/637244main_MysteriesOfTheSun_Book.pdf](https://s3.us-east-1.amazonaws.com/explore-kendra-solar/637244main_MysteriesOfTheSun_Book.pdf)"); - - // tests that when querying an unstructured document, the SSML response returns the longest highlight from the response when no top answer is found - assert.equal(resp.res.session.appContext.altMessages.ssml, - " Anatomy of the Sun Mysteries of the Sun The Sun is an incandescent mass of hydrogen, helium, and other heavier elements. While it appears constant and unchanging from our vantage point on Earth, it actually has a dynamic and variable system of twisting magnetic fields that cause solar events of almost unimaginable power "); - }); -}); - - diff --git a/test-do-not-use-obsolete/kendra_tests/sample_return.json b/test-do-not-use-obsolete/kendra_tests/sample_return.json deleted file mode 100644 index 87858a060..000000000 --- a/test-do-not-use-obsolete/kendra_tests/sample_return.json +++ /dev/null @@ -1,146 +0,0 @@ -{ - "req": { - "_event": { - "messageVersion": "1.0", - "invocationSource": "FulfillmentCodeHook", - "userId": "us-east-1:7ace124d-bfd1-4356-ac5a-ed8a98785fd9", - "sessionAttributes": {}, - "requestAttributes": null, - "bot": { - "name": "QNA_dev_dev_dev_master_two_BotUSFgl", - "alias": "live", - "version": "1" - }, - "outputDialogMode": "Text", - "currentIntent": { - "name": "fulfilment_IntentfiuhxvkSil", - "slots": { - "slot": "How many planets orbit the sun" - }, - "slotDetails": { - "slot": { - "resolutions": [], - "originalValue": "How many planets orbit the sun" - } - }, - "confirmationStatus": "None" - }, - "inputTranscript": "How many planets orbit the sun?", - "recentIntentSummaryView": null, - "sentimentResponse": null, - "kendraResponse": null, - "errorFound": false - }, - "_settings": { - "ENABLE_DEBUG_RESPONSES": "false", - "ES_USE_KEYWORD_FILTERS": "true", - "ES_NO_HITS_QUESTION": "no_hits", - "ES_USE_FUZZY_MATCH": "false", - "ES_KEYWORD_SYNTAX_TYPES": "NOUN,PROPN,VERB,INTJ", - "ES_SYNTAX_CONFIDENCE_LIMIT": ".20", - "ES_MINIMUM_SHOULD_MATCH": "2<75%", - "ES_PHRASE_BOOST": "4", - "ES_SCORE_ANSWER_FIELD": "false", - "ERRORMESSAGE": "Unfortunately I encountered an error when searching for your answer. Please ask me again later.", - "EMPTYMESSAGE": "You stumped me! Sadly I don't know how to answer your question.", - "DEFAULT_ALEXA_LAUNCH_MESSAGE": "Hello, Please ask a question", - "DEFAULT_ALEXA_STOP_MESSAGE": "Goodbye", - "SMS_HINT_REMINDER_ENABLE": "true", - "SMS_HINT_REMINDER": " (Feedback? Reply THUMBS UP or THUMBS DOWN. Ask HELP ME at any time)", - "SMS_HINT_REMINDER_INTERVAL_HRS": "24", - "IDENTITY_PROVIDER_JWKS_URLS": [], - "ENABLE_MULTI_LANGUAGE_SUPPORT": "false", - "MINIMUM_CONFIDENCE_SCORE": 0.6, - "ALT_SEARCH_KENDRA_INDEXES": "[\"e1c23860-e5c8-4409-ae26-b05bd6ced00a\"]", - "ELICIT_RESPONSE_MAX_RETRIES": 3, - "ELICIT_RESPONSE_RETRY_MESSAGE": "Please try again?", - "ELICIT_RESPONSE_BOT_FAILURE_MESSAGE": "Your response was not understood. Please start again.", - "ELICIT_RESPONSE_DEFAULT_MSG": "Ok. ", - "ENABLE_REDACTING": "false", - "REDACTING_REGEX": "\\b\\d{4}\\b(?![-])|\\b\\d{9}\\b|\\b\\d{3}-\\d{2}-\\d{4}\\b", - "DEFAULT_USER_POOL_JWKS_URL": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_jt2cXkCEb/.well-known/jwks.json" - }, - "_type": "LEX", - "_userId": "us-east-1:7ace124d-bfd1-4356-ac5a-ed8a98785fd9", - "question": "How many planets orbit the sun?", - "session": {}, - "_preferredResponseType": "PlainText", - "_userInfo": { - "UserId": "us-east-1:7ace124d-bfd1-4356-ac5a-ed8a98785fd9", - "InteractionCount": 0, - "TimeSinceLastInteraction": 1593663418.209 - }, - "_info": { - "es": { - "address": "search-qna-dev-elasti-mtl6ojldf3ja-w5begr25vpfuyidnlgb4izlbtq.us-east-1.es.amazonaws.com", - "index": "qna-dev-dev-dev-master-2", - "type": "qna", - "service": { - "qid": "QNA-dev-dev-dev-master-2-ESQidLambda-1I9JEHU6U4K9I", - "proxy": "QNA-dev-dev-dev-master-2-ESProxyLambda-1NMAY6LNBRJDY" - } - } - } - }, - "res": { - "type": "PlainText", - "message": "Answer from Amazon Kendra FAQ.\n\n Eight planets orbit the sun. Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune. Sorry I'm required to leave out my favorite, Pluto.", - "session": { - "appContext": { - "altMessages": { - "markdown": "*Answer from Amazon Kendra FAQ.* \n \n\nEight **planets** **orbit** the **sun**. Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune. Sorry I'm required to leave out my favorite, Pluto.", - "ssml": " Answer from Amazon Kendra FAQ. Eight planets orbit the sun. Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune. Sorry I'm required to leave out my favorite, Pluto " - } - }, - "previous": { - "qid": "KendraFallback", - "a": "The Kendra Fallback search was not able to identify any results", - "alt": {}, - "q": "How many planets orbit the sun?" - }, - "navigation": { - "next": "", - "previous": [], - "hasParent": true - }, - "qnabotcontext": { - "kendra":{ - "kendraIndexId":"e1c23860-e5c8-4409-ae26-b05bd6ced00a", - "kendraQueryId":"ec35a915-8990-4ef7-a5da-f490911cad42", - "kendraResultId":"ec35a915-8990-4ef7-a5da-f490911cad42-2db7aa5b-1fcb-4910-a2fc-b3922a6069e3", - "kendraResponsibleQid": "KendraFallback" - } - } - }, - "card": { - "send": false, - "title": "", - "text": "", - "url": "" - }, - "_userInfo": { - "UserId": "us-east-1:7ace124d-bfd1-4356-ac5a-ed8a98785fd9", - "InteractionCount": 1, - "TimeSinceLastInteraction": 1593663418.209, - "FirstSeen": "Thu Jul 02 2020 16:16:58 GMT+0000 (Coordinated Universal Time)", - "LastSeen": "Thu Jul 02 2020 16:16:58 GMT+0000 (Coordinated Universal Time)" - }, - "got_hits": 0, - "result": { - "qid": "KendraFallback", - "quniqueterms": " no_hits ", - "questions": [ - { - "q": "no_hits" - } - ], - "a": "The Kendra Fallback search was not able to identify any results", - "l": "QNA:EXTKendraFallback", - "type": "qna", - "autotranslate": { - "a": true - } - }, - "plainMessage": "The Kendra Fallback search was not able to identify any results" - } -} diff --git a/test-do-not-use-obsolete/kendra_tests/test_json.json b/test-do-not-use-obsolete/kendra_tests/test_json.json deleted file mode 100644 index 489f89fa7..000000000 --- a/test-do-not-use-obsolete/kendra_tests/test_json.json +++ /dev/null @@ -1,42 +0,0 @@ - -{ - "Key": "AnswerText", - "ValueType": "TEXT_WITH_HIGHLIGHTS_VALUE", - "Value": { - "TextWithHighlightsValue": { - "Text": "Anatomy of the Sun\nMysteries of the Sun\n\n\nThe Sun is an incandescent mass of hydrogen, helium, and other heavier elements. While it appears constant and unchanging from \nour vantage point on Earth, it actually has a dynamic and variable system of twisting magnetic fields that cause solar events of almost \nunimaginable power.\n\n\nThe Convection Zone\nEnergy continues to move toward the surface \nthrough convection currents of heated and \ncooled gas in the convection zone.\n\n\n \n\n\nThe Radiative Zone\nEnergy moves slowly outward—taking \nmore than 170,000 years to radiate through \nthe layer of the Sun known as the radiative \nzone.", - "Highlights": [ - { - "BeginOffset": 42, - "EndOffset": 121, - "TopAnswer": false - }, - { - "BeginOffset": 53, - "EndOffset": 121, - "TopAnswer": false - }, - { - "BeginOffset": 15, - "EndOffset": 18, - "TopAnswer": false - }, - { - "BeginOffset": 36, - "EndOffset": 39, - "TopAnswer": false - }, - { - "BeginOffset": 46, - "EndOffset": 49, - "TopAnswer": false - }, - { - "BeginOffset": 594, - "EndOffset": 597, - "TopAnswer": false - } - ] - } - } -} diff --git a/test-do-not-use-obsolete/reporter.js b/test-do-not-use-obsolete/reporter.js deleted file mode 100644 index 8fb2c291c..000000000 --- a/test-do-not-use-obsolete/reporter.js +++ /dev/null @@ -1,131 +0,0 @@ -/*! - * Nodeunit - * Copyright (c) 2010 Caolan McMahon - * MIT Licensed - */ - -/** - * Module dependencies - */ -var base="../node_modules/nodeunit/lib/reporters/" -var chalk=require('chalk') -const ora = require('ora'); -var _=require('lodash') -var Promise=require('bluebird') -var nodeunit = require(base+'../nodeunit'), - utils = require(base+'../utils'), - fs = require('fs'), - track = require(base+'../track'), - path = require('path'), - AssertionError = require(base+'../assert').AssertionError; - -/** - * Reporter info string - */ - -exports.info = "Default tests reporter"; - - -/** - * Run all tests within each module, reporting the results to the command-line. - * - * @param {Array} files - * @api public - */ - -exports.run = function (files, options={}, callback,_log) { -return new Promise(function(res,rej){ - var options=options || {} - var start = new Date().getTime(); - var report=[] - var tracker = track.createTracker(function (tracker) { - if (tracker.unfinished()) { - _log(''); - _log('FAILURES: Undone tests (or their setups/teardowns): '); - var names = tracker.names(); - for (var i = 0; i < names.length; i += 1) { - _log('/' + names[i]); - } - _log(''); - _log('To fix this, make sure all tests call test.done()'); - rej(tracker.unfinished()) - } - }); - var timer - var spinner - var opts = { - testspec: options.testspec, - testFullSpec: options.testFullSpec, - recursive: options.recursive, - moduleStart: function (name) { - _log('\n' + name); - }, - testDone: function (name, assertions) { - tracker.remove(name); - var duration=(new Date())-timer - if (!assertions.failures()) { - spinner.succeed(' ('+assertions.length+') '+ name +" "+duration/1000+'s' ); - report.push({name:name.toString(),status:"success",duration}) - } - else { - spinner.fail(' ' + name + '\n'); - var tmp={name:name.toString(),status:"fail",duration,messages:[]} - assertions.forEach(function (a) { - if (a.failed()) { - a = utils.betterErrors(a); - if (a.error instanceof AssertionError && a.message) { - tmp.message.push(a.message) - _log( - 'Assertion Message: ' + - assertion_message(a.message) - ); - } - _log(a.error.stack + '\n'); - } - }); - report.push(tmp) - } - }, - done: function (assertions, end) { - var end = end || new Date().getTime(); - var duration = end - start; - if (assertions.failures()) { - _log( - '\n' + 'FAILURES: ' + assertions.failures() + - '/' + assertions.length + ' assertions failed (' + - assertions.duration/1000 + 's)' - ); - } - else { - _log( - '\n' + 'OK: ' + assertions.length + - ' assertions (' + assertions.duration/1000 + 's)' - ); - } - res(report) - if (callback) callback(assertions.failures() ? new Error('We have got test failures.') : undefined); - }, - testStart: function(name) { - name.toString=()=>{ - return name.filter(x=>_.isString(x)).join('/') - } - spinner = ora({ - text:name, - spinner:"simpleDots" - }).start(); - spinner.color = 'yellow'; - - timer=new Date() - tracker.put(name); - } - }; - if (files && files.length) { - var paths = files.map(function (p) { - return path.resolve(p); - }); - nodeunit.runFiles(paths, opts); - } else { - nodeunit.runModules(files,opts); - } -}) -}; diff --git a/test-do-not-use-obsolete/run.js b/test-do-not-use-obsolete/run.js deleted file mode 100755 index 5318d79ae..000000000 --- a/test-do-not-use-obsolete/run.js +++ /dev/null @@ -1,29 +0,0 @@ -#! /usr/bin/env node -var fs = require('fs'); -var util = require('util'); -var label=process.argv[3] -var log_file = fs.createWriteStream(__dirname + `/output/log-${label}.log`, {flags : 'a'}); -var output_file = fs.createWriteStream(__dirname + `/output/output-${label}.log`, {flags : 'a'}); -var _log=console.log -const stripAnsi = require('strip-ansi'); - -var log=function(x){ - log_file.write(stripAnsi(util.format(x)) + '\n'); - output_file.write(stripAnsi(util.format(x)) + '\n'); - _log(x) -} - -console.log=function(x){ - log_file.write(stripAnsi(util.format(x)) + '\n'); -}.bind(console) - -var reporter = require('./reporter'); - -reporter - .run([process.argv[2]],null,null,log) - .then(x=>fs.writeFileSync(__dirname+`/output/output-${label}.json`,JSON.stringify(x))) - - - - - diff --git a/test-do-not-use-obsolete/setup.sh b/test-do-not-use-obsolete/setup.sh deleted file mode 100755 index a62b6c207..000000000 --- a/test-do-not-use-obsolete/setup.sh +++ /dev/null @@ -1,21 +0,0 @@ -#! /bin/bash -__dirname="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" -BASE=$__dirname/.. -SELENIUM_START=/opt/bin/entry_point.sh -cd $BASE - -if [ -e $SELENIUM_START ]; then - $SELENIUM_START & -fi - -npm run --silent bootstrap - -if [ $? -ne 0 ]; then - echo "failed to create bootstrap bucket" - exit 1 -fi - -mkdir -p $__dirname/output -rm $__dirname/output/*; - -echo "ready" diff --git a/test-do-not-use-obsolete/teardown.js b/test-do-not-use-obsolete/teardown.js deleted file mode 100755 index 18599a24f..000000000 --- a/test-do-not-use-obsolete/teardown.js +++ /dev/null @@ -1,45 +0,0 @@ -#! /usr/bin/env node -var name=require('../bin/name') -var config=require('../config') -process.env.AWS_PROFILE=config.profile -process.env.AWS_DEFAULT_REGION=config.profile -var aws=require('aws-sdk') -var Promise=require('bluebird') -aws.config.setPromisesDependency(Promise) -aws.config.region=require('../config').region -var cf=new aws.CloudFormation() -var _=require('lodash') - -var prefix=name('a',{prefix:true}) -cf.listStacks({ - StackStatusFilter:[ -"CREATE_IN_PROGRESS","CREATE_FAILED","CREATE_COMPLETE","ROLLBACK_FAILED","ROLLBACK_COMPLETE","DELETE_FAILED","UPDATE_COMPLETE","UPDATE_ROLLBACK_FAILED","UPDATE_ROLLBACK_COMPLETE" - ] -}).promise() -.then(x=>x.StackSummaries.filter(x=>x.StackName.match(prefix)) -.map(x=>x.StackName)) -.map(StackName=>{ - console.log(`deleting: ${StackName}`) - return cf.deleteStack({StackName}).promise() -}) -.then(()=>{ - return new Promise(function(res,rej){ - next() - function next(){ - cf.listStacks({ - StackStatusFilter:["DELETE_IN_PROGRESS"] - }).promise() - .then(x=>x.StackSummaries.filter(x=>x.StackName.match(prefix)).length) - .then(count=>{ - console.log('stacks up:'+count) - if(count<1){ - res() - }else{ - setTimeout(()=>next(),10*1000) - } - }) - .catch(rej) - } - }) -}) - diff --git a/test-do-not-use-obsolete/test.json b/test-do-not-use-obsolete/test.json deleted file mode 100644 index 291e2b5cb..000000000 --- a/test-do-not-use-obsolete/test.json +++ /dev/null @@ -1,161 +0,0 @@ -{ - "Description": "This template creates test infastructure for testing QnABot", - "Resources": { - "Repo": {"Type": "AWS::ECR::Repository", "Properties": {}}, - "testBuilb": { - "Type": "AWS::CodeBuild::Project", - "Properties": { - "Artifacts": {"Type": "S3", "Location": {"Ref": "Bucket"}}, - "Environment": { - "ComputeType": "BUILD_GENERAL1_LARGE", - "Image": { - "Fn::Sub": "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${Repo}:test" - }, - "EnvironmentVariables": [ - { - "Name": "CODEBUILD_SOURCE_VERSION", - "Value": "feature-remove-exports" - } - ], - "Type": "LINUX_CONTAINER" - }, - "Name": {"Fn::Sub": "${AWS::StackName}-test-build"}, - "ServiceRole": {"Ref": "ServiceRole"}, - "Source": { - "Type": "GITHUB", - "Location": "https://github.com/rstrahan/aws-ai-qna-bot-2.git", - "auth": {"type": "OATH"} - } - } - }, - "ImageBuild": { - "Type": "AWS::CodeBuild::Project", - "Properties": { - "Artifacts": {"Type": "NO_ARTIFACTS"}, - "Environment": { - "ComputeType": "BUILD_GENERAL1_LARGE", - "EnvironmentVariables": [ - {"Name": "IMAGE_REPO_NAME", "Value": {"Ref": "Repo"}}, - {"Name": "IMAGE_TAG", "Value": "test"}, - {"Name": "AWS_DEFAULT_REGION", "Value": {"Ref": "AWS::Region"}}, - {"Name": "AWS_ACCOUNT_ID", "Value": {"Ref": "AWS::AccountId"}} - ], - "Image": "ubuntu/docker/17.09.0", - "PrivilegedMode": true, - "Type": "LINUX_CONTAINER" - }, - "Name": {"Fn::Sub": "${AWS::StackName}-test-image-build"}, - "ServiceRole": {"Ref": "ServiceRole"}, - "Source": { - "Type": "S3", - "Location": {"Fn::Sub": "${Bucket}/source.zip"} - } - } - }, - "Bucket": {"Type": "AWS::S3::Bucket", "Properties": {}}, - "Clear": { - "Type": "Custom::S3Clear", - "Properties": { - "ServiceToken": {"Fn::GetAtt": ["ClearLambda", "Arn"]}, - "Bucket": {"Ref": "Bucket"} - } - }, - "upload": { - "Type": "Custom::S3Upload", - "Properties": { - "ServiceToken": {"Fn::GetAtt": ["UploadLambda", "Arn"]}, - "bucket": {"Ref": "Bucket"}, - "key": "source.zip", - "body": "UEsDBAoAAAAAAAGbK0yFEUoNCwAAAAsAAAANAAAAYnVpbGRzcGVjLnltbGhlbGxvIHdvcmxkUEsDBAoAAAAAAAGbK0yFEUoNCwAAAAsAAAAKAAAARG9ja2VyZmlsZWhlbGxvIHdvcmxkUEsBAhQACgAAAAAAAZsrTIURSg0LAAAACwAAAA0AAAAAAAAAAAAAAAAAAAAAAGJ1aWxkc3BlYy55bWxQSwECFAAKAAAAAAABmytMhRFKDQsAAAALAAAACgAAAAAAAAAAAAAAAAA2AAAARG9ja2VyZmlsZVBLBQYAAAAAAgACAHMAAABpAAAAAAA=" - } - }, - "Build": { - "Type": "Custom::CodeBuildStart", - "Properties": { - "ServiceToken": {"Fn::GetAtt": ["BuildLambda", "Arn"]}, - "name": {"Ref": "ImageBuild"} - } - }, - "BuildLambda": { - "Type": "AWS::Lambda::Function", - "Properties": { - "Code": { - "ZipFile": "var response = require('cfn-response')\nvar aws=require('aws-sdk')\naws.config.region=process.env.AWS_REGION\nvar cb=new aws.CodeBuild()\nvar s3=new aws.S3()\n\nexports.build = function(event, context) {\n console.log(JSON.stringify(event,null,2))\n\n if(event.RequestType===\"Create\"){\n cb.startBuild({\n projectName:event.ResourceProperties.name\n }).promise()\n .then(x=>response.send(event,context,response.SUCCESS))\n .catch(x=>response.send(event,context,response.FAIL))\n }else{\n response.send(event, context, response.SUCCESS)\n }\n}\n\nexports.zip = function(event, context) {\n console.log(JSON.stringify(event,null,2))\n\n if(event.RequestType===\"Create\"){\n s3.putObject({\n Bucket:event.ResourceProperties.bucket,\n Key:event.ResourceProperties.key,\n Body:new Buffer(event.ResourceProperties.body,\"base64\")\n })\n .then(x=>response.send(event,context,response.SUCCESS))\n .catch(x=>response.send(event,context,response.FAIL))\n }else{\n response.send(event, context, response.SUCCESS)\n }\n}\n\nexports.clear = function(event, context) {\n console.log(JSON.stringify(event,null,2))\n\n if(event.RequestType===\"Delete\"){\n Delete(event.ResourceProperties)\n .then(()=>response.send(event, context, response.SUCCESS))\n .catch(x=>{\n console.log(x)\n response.send(event, context, response.FAILED)\n })\n }else{\n response.send(event, context, response.SUCCESS)\n }\n}\n\nfunction Delete(params){\n return new Promise(function(res,rej){\n function next(){\n s3.listObjectVersions({\n Bucket:params.Bucket\n }).promise()\n .then(x=>x.Versions.concat(x.DeleteMarkers))\n .then(function(files){\n return files.map(file=>{return {\n Key:file.Key,\n VersionId:file.VersionId\n } })\n })\n .then(function(keys){\n console.log(\"going to delete\",keys)\n if(keys.length>0){ \n return s3.deleteObjects({\n Bucket:params.Bucket,\n Delete:{\n Objects:keys\n }\n }).promise()\n .then(()=>next())\n .catch(rej)\n }else{\n res()\n }\n })\n }\n next()\n })\n}\n" - }, - "Handler": "index.build", - "MemorySize": "128", - "Role": {"Fn::GetAtt": ["LambdaRole", "Arn"]}, - "Runtime": "nodejs12.x", - "Timeout": 300 - } - }, - "UploadLambda": { - "Type": "AWS::Lambda::Function", - "Properties": { - "Code": { - "ZipFile": "var response = require('cfn-response')\nvar aws=require('aws-sdk')\naws.config.region=process.env.AWS_REGION\nvar cb=new aws.CodeBuild()\nvar s3=new aws.S3()\n\nexports.build = function(event, context) {\n console.log(JSON.stringify(event,null,2))\n\n if(event.RequestType===\"Create\"){\n cb.startBuild({\n projectName:event.ResourceProperties.name\n }).promise()\n .then(x=>response.send(event,context,response.SUCCESS))\n .catch(x=>response.send(event,context,response.FAIL))\n }else{\n response.send(event, context, response.SUCCESS)\n }\n}\n\nexports.zip = function(event, context) {\n console.log(JSON.stringify(event,null,2))\n\n if(event.RequestType===\"Create\"){\n s3.putObject({\n Bucket:event.ResourceProperties.bucket,\n Key:event.ResourceProperties.key,\n Body:new Buffer(event.ResourceProperties.body,\"base64\")\n })\n .then(x=>response.send(event,context,response.SUCCESS))\n .catch(x=>response.send(event,context,response.FAIL))\n }else{\n response.send(event, context, response.SUCCESS)\n }\n}\n\nexports.clear = function(event, context) {\n console.log(JSON.stringify(event,null,2))\n\n if(event.RequestType===\"Delete\"){\n Delete(event.ResourceProperties)\n .then(()=>response.send(event, context, response.SUCCESS))\n .catch(x=>{\n console.log(x)\n response.send(event, context, response.FAILED)\n })\n }else{\n response.send(event, context, response.SUCCESS)\n }\n}\n\nfunction Delete(params){\n return new Promise(function(res,rej){\n function next(){\n s3.listObjectVersions({\n Bucket:params.Bucket\n }).promise()\n .then(x=>x.Versions.concat(x.DeleteMarkers))\n .then(function(files){\n return files.map(file=>{return {\n Key:file.Key,\n VersionId:file.VersionId\n } })\n })\n .then(function(keys){\n console.log(\"going to delete\",keys)\n if(keys.length>0){ \n return s3.deleteObjects({\n Bucket:params.Bucket,\n Delete:{\n Objects:keys\n }\n }).promise()\n .then(()=>next())\n .catch(rej)\n }else{\n res()\n }\n })\n }\n next()\n })\n}\n" - }, - "Handler": "index.zip", - "MemorySize": "128", - "Role": {"Fn::GetAtt": ["LambdaRole", "Arn"]}, - "Runtime": "nodejs12.x", - "Timeout": 300 - } - }, - "ClearLambda": { - "Type": "AWS::Lambda::Function", - "Properties": { - "Code": { - "ZipFile": "var response = require('cfn-response')\nvar aws=require('aws-sdk')\naws.config.region=process.env.AWS_REGION\nvar cb=new aws.CodeBuild()\nvar s3=new aws.S3()\n\nexports.build = function(event, context) {\n console.log(JSON.stringify(event,null,2))\n\n if(event.RequestType===\"Create\"){\n cb.startBuild({\n projectName:event.ResourceProperties.name\n }).promise()\n .then(x=>response.send(event,context,response.SUCCESS))\n .catch(x=>response.send(event,context,response.FAIL))\n }else{\n response.send(event, context, response.SUCCESS)\n }\n}\n\nexports.zip = function(event, context) {\n console.log(JSON.stringify(event,null,2))\n\n if(event.RequestType===\"Create\"){\n s3.putObject({\n Bucket:event.ResourceProperties.bucket,\n Key:event.ResourceProperties.key,\n Body:new Buffer(event.ResourceProperties.body,\"base64\")\n })\n .then(x=>response.send(event,context,response.SUCCESS))\n .catch(x=>response.send(event,context,response.FAIL))\n }else{\n response.send(event, context, response.SUCCESS)\n }\n}\n\nexports.clear = function(event, context) {\n console.log(JSON.stringify(event,null,2))\n\n if(event.RequestType===\"Delete\"){\n Delete(event.ResourceProperties)\n .then(()=>response.send(event, context, response.SUCCESS))\n .catch(x=>{\n console.log(x)\n response.send(event, context, response.FAILED)\n })\n }else{\n response.send(event, context, response.SUCCESS)\n }\n}\n\nfunction Delete(params){\n return new Promise(function(res,rej){\n function next(){\n s3.listObjectVersions({\n Bucket:params.Bucket\n }).promise()\n .then(x=>x.Versions.concat(x.DeleteMarkers))\n .then(function(files){\n return files.map(file=>{return {\n Key:file.Key,\n VersionId:file.VersionId\n } })\n })\n .then(function(keys){\n console.log(\"going to delete\",keys)\n if(keys.length>0){ \n return s3.deleteObjects({\n Bucket:params.Bucket,\n Delete:{\n Objects:keys\n }\n }).promise()\n .then(()=>next())\n .catch(rej)\n }else{\n res()\n }\n })\n }\n next()\n })\n}\n" - }, - "Handler": "index.clear", - "MemorySize": "128", - "Role": {"Fn::GetAtt": ["LambdaRole", "Arn"]}, - "Runtime": "nodejs12.x", - "Timeout": 300 - } - }, - "ServiceRole": { - "Type": "AWS::IAM::Role", - "Properties": { - "AssumeRolePolicyDocument": { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": {"Service": "codebuild.amazonaws.com"}, - "Action": "sts:AssumeRole" - } - ] - }, - "Path": "/", - "ManagedPolicyArns": [ - "arn:aws:iam::aws:policy/AmazonECS_FullAccess", - "arn:aws:iam::aws:policy/AmazonS3FullAccess" - ] - } - }, - "LambdaRole": { - "Type": "AWS::IAM::Role", - "Properties": { - "AssumeRolePolicyDocument": { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": {"Service": "lambda.amazonaws.com"}, - "Action": "sts:AssumeRole" - } - ] - }, - "Path": "/", - "ManagedPolicyArns": [ - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole", - "arn:aws:iam::aws:policy/AmazonECS_FullAccess", - "arn:aws:iam::aws:policy/AmazonS3FullAccess" - ] - } - } - }, - "Outputs": {} -} \ No newline at end of file diff --git a/test-do-not-use-obsolete/test.sh b/test-do-not-use-obsolete/test.sh deleted file mode 100755 index f0955cb4a..000000000 --- a/test-do-not-use-obsolete/test.sh +++ /dev/null @@ -1,15 +0,0 @@ -#! /bin/bash -__dirname="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" -BASE=$__dirname/.. -cd $BASE - -#npm run up - -echo "Testing Api/Lex" -if $__dirname/run.js $BASE/templates/master/test/index.js master; then - echo "Finished Testing API/Lex" -else - exit 1 -fi - - diff --git a/tuning_accuracy_guide/AWS_QnABot_tuning_recognition_accuracy_guide.md b/tuning_accuracy_guide/AWS_QnABot_tuning_recognition_accuracy_guide.md index 58eac67ad..750e54853 100644 --- a/tuning_accuracy_guide/AWS_QnABot_tuning_recognition_accuracy_guide.md +++ b/tuning_accuracy_guide/AWS_QnABot_tuning_recognition_accuracy_guide.md @@ -1,8 +1,6 @@ -**Tuning Recognition Accuracy** -=============================== +# Tuning Recognition Accuracy -Overview -======== +## Overview Chatbots are a great way to make information available for your users. With the Amazon Web Services (AWS) QnABot solution you can deploy a @@ -15,8 +13,7 @@ provides a common language to describe and provision all the infrastructure resources in your AWS cloud environment. This AWS QnABot CloudFormation template will setup resources such as Amazon Lex, Amazon Elasticsearch Service, Amazon API Gateway, and AWS Lambda. Additionally, -this solution is also available in GitHub -(). +this solution is also available in [GitHub](https://github.com/aws-samples/aws-ai-qna-bot). Once the solution is deployed, you have a QnABot designer console where you can build and manage your question and answer bank. The question and @@ -32,8 +29,7 @@ works in identifying user questions and providing the relevant answers. Learn how you can improve the recognition accuracy and further augment the chatbot to get better over time. -Solution architecture and how It works -====================================== +### Solution architecture and how It works When the QnABot solution is deployed, resources such as Amazon Lex, Amazon Elasticsearch Service (Amazon ES), Amazon API Gateway, and AWS @@ -41,7 +37,7 @@ Lambda are provisioned and setup in your AWS account. Two key AWS services are at the core of the solution: -- **Amazon Lex** is a service for building conversational interfaces +- **Amazon Lex** is a service for building conversational interfaces into any application using voice and text. Amazon Lex provides the advanced deep learning functionalities of automatic speech recognition (ASR) for converting speech to text, and natural @@ -49,7 +45,7 @@ Two key AWS services are at the core of the solution: enable you to build applications with highly engaging user experiences and lifelike conversational interactions. -- **Amazon ES** is an open-source search and analytics engine for use +- **Amazon ES** is an open-source search and analytics engine for use cases such as log analytics, real-time application monitoring, and clickstream analysis. Amazon ES is a managed service that makes it simple to deploy, operate, and scale Elasticsearch clusters in the @@ -61,7 +57,7 @@ Two key AWS services are at the core of the solution: Let's take a closer look at these two services and how they help power the AWS QnABot solution. -![](image2.png) +![Solution architecture and data flow](image2.png) *Figure 1.0 -- Solution architecture and data flow* When you ask QnABot a question, a few things happen: @@ -69,85 +65,85 @@ When you ask QnABot a question, a few things happen: 1. The question gets processed and transcribed by Amazon Lex using a Natural Language Understanding (NLU) and Processing (NLP) engine. - - QnABot initially trains the NLP to match a wide variety of possible + - QnABot initially trains the NLP to match a wide variety of possible questions and statements, so that the Amazon Lex bot can accept just about any question a user might ask. The Amazon Lex interaction model is setup with: - - **Intents**: An intent represents an action that fulfills a + - **Intents**: An intent represents an action that fulfills a user\'s spoken request. Intents can optionally have arguments called **slots**. The QnABot uses **slots** to capture user - input and fulfils the Intent via Lambda function. + input and fulfills the Intent via Lambda function. - - **Sample utterances**: A set of likely spoken phrases mapped to + - **Sample utterances**: A set of likely spoken phrases mapped to the intents. This should include as many representative phrases as possible. The sample utterances specify the words and phrases users can say to invoke your intents. QnABot updates the **Sample utterances** with the various questions to train the chatbot to understand different user input -2. This request is then sent to Amazon ES. QnABot attempts to match a +2. This request is then sent to Amazon ES. QnABot attempts to match a user's question to the list of questions and answers (created in the QnABot content designer) stored in Amazon ES. - - The QnABot uses full text search to find the most relevant ranked + - The QnABot uses full text search to find the most relevant ranked document from the searchable index. Relevancy ranking is based on a few properties: - - Count: How many search terms appear in a document + - Count: How many search terms appear in a document - - Frequency: How often the specified keywords occur in a given + - Frequency: How often the specified keywords occur in a given document - - Importance: How rare or new the specified keywords are and how + - Importance: How rare or new the specified keywords are and how closely the keywords occur together in a phrase - - The closer the alignment between a question associated with an item + - The closer the alignment between a question associated with an item and a question asked by the user, the greater the probability that the QnABot will choose that item as the most relevant answer. Noise words such as article, preposition in sentence constructs have lower weighting than unique keywords. - - - The keyword filter feature helps QnABot to be more accurate when - answering questions, and to admit more readily when it doesn’t know the - answer. The keyword filter feature works by using Amazon Comprehend to - determine the ‘part of speech’ that applies to each word you say to QnABot. - By default, nouns (including proper nouns), verbs, and interjections are - used as ‘keywords’. Any answer returned by QnABot must have questions + + - The keyword filter feature helps QnABot to be more accurate when + answering questions, and to admit more readily when it doesn’t know the + answer. The keyword filter feature works by using Amazon Comprehend to + determine the ‘part of speech’ that applies to each word you say to QnABot. + By default, nouns (including proper nouns), verbs, and interjections are + used as ‘keywords’. Any answer returned by QnABot must have questions that match these keywords, using the following (default) rule: - + - if there are 1 or 2 keywords, then all keywords must match. - if there are 3 or more keywords, then 75% of the keywords must match. - - If QnABot can’t find any answers that match these keyword filter rules, - then it will admit that it doesn’t know the answer rather than guessing - an answer that doesn’t match the keywords. QnABot logs every question - that it can’t answer so you can see them in the Kibana Dashboard which - we’ll show you a little later. The keyword filters feature is enabled by - default, but if it causes problems for you, you can customize its + - If QnABot can’t find any answers that match these keyword filter rules, + then it will admit that it doesn’t know the answer rather than guessing + an answer that doesn’t match the keywords. QnABot logs every question + that it can’t answer so you can see them in the Kibana Dashboard which + we’ll show you a little later. The keyword filters feature is enabled by + default, but if it causes problems for you, you can customize its settings, or disable it altogether - see [Modifying configuration settings](https://www.amazon.com/qnabot/#config-settings) - - - The Bot fulfillment Lambda function generates an Amazon ES query - containing the transcribed question. The query attempts to find the best - match from all the questions and answers you’ve previously provided, - filtering items to apply the keyword filters and using Amazon ES - relevance scoring to rank the results. Scoring is based on (i) matching - the words in the user’s question against the unique set of words used in - the stored questions (“quniqueterms”), (ii) matching the phrasing of the + + - The Bot fulfillment Lambda function generates an Amazon ES query + containing the transcribed question. The query attempts to find the best + match from all the questions and answers you’ve previously provided, + filtering items to apply the keyword filters and using Amazon ES + relevance scoring to rank the results. Scoring is based on (i) matching + the words in the user’s question against the unique set of words used in + the stored questions (“quniqueterms”), (ii) matching the phrasing of the user’s question to the text of stored questions (nested field: questions.q), - and (iii) matching the topic value assigned to the previous answer (if any) - to increase the overall relevance score when topic value (field t) matches. + and (iii) matching the topic value assigned to the previous answer (if any) + to increase the overall relevance score when topic value (field t) matches. The following example shows the query: - - ![](image3.png) - + + ![ElasticSearch Query Example](image3.png) + *Figure 2.0 -- Example Elasticsearch query* - - If an item has a large number of questions representing many unique - words, you might find that a short question (one or two words) might not - generate a high enough score to match the correct answer, even if it - exactly matches one of the stored questions in the item. You can + - If an item has a large number of questions representing many unique + words, you might find that a short question (one or two words) might not + generate a high enough score to match the correct answer, even if it + exactly matches one of the stored questions in the item. You can experiment by increasing the weight (boost) given to exact 'phrase' matches: - + a. Log in to the content designer, choose the tools menu ( ☰ ) and choose Settings. @@ -155,11 +151,11 @@ When you ask QnABot a question, a few things happen: or '4' to a larger number, say '8'. c. Scroll to the bottom and choose Save. - - - QnABot now supports Elasticsearch fuzzy matching which makes it more - tolerant of misspellings and typing errors in the user input. Enable + + - QnABot now supports Elasticsearch fuzzy matching which makes it more + tolerant of misspellings and typing errors in the user input. Enable this feature in the Settings page: - + a. Log in to the content designer, choose the tools menu ( ☰ ) and choose Settings. @@ -167,14 +163,14 @@ When you ask QnABot a question, a few things happen: to true. c. Scroll to the bottom and choose Save. - - - In previous releases, QnABot also included matches found in the - answer field to give greater relevance to items where the answer - contains multiple references to the terms used in the user’s question. - This is not always desirable, however, since term matches contained in - the answer field can make it harder to predict and control the items - you want to score highest for any question. QnABot v4.0.0 and later - disables this feature by default. You can, of course, turn it on + + - In previous releases, QnABot also included matches found in the + answer field to give greater relevance to items where the answer + contains multiple references to the terms used in the user’s question. + This is not always desirable, however, since term matches contained in + the answer field can make it harder to predict and control the items + you want to score highest for any question. QnABot v4.0.0 and later + disables this feature by default. You can, of course, turn it on again using the Settings page:: a. Log in to the content designer, choose the tools menu ( ☰ ) and @@ -185,13 +181,12 @@ When you ask QnABot a question, a few things happen: c. Scroll to the bottom and choose Save. - - The QnABot fulfillment Lambda function uses the answer field {a} (or + - The QnABot fulfillment Lambda function uses the answer field {a} (or the alternate Markdown or SSML fields if they have been populated) from the item with the highest score to create a response to send back to Amazon Lex. -Tuning recognition accuracy and monitoring -========================================== +## Tuning recognition accuracy and monitoring Now, we have a background on how the QnABot solution works to provide answers to users' questions. We will now take a look at how you can @@ -206,33 +201,33 @@ conflict with another question with similar keywords. You can solve this in a couple of ways: -1. The QnABot solution has a built-in tool that can help you quickly +1. The QnABot solution has a built-in tool that can help you quickly test all your questions and see how well they perform. Based on the results, you can then review any incorrect matches and update the needed questions. The following steps below show how you can enable testing: - - Log in to the QnABot designer and choose TEST ALL. + - Log in to the QnABot designer and choose TEST ALL. - - Use the default Filename, or enter your own. + - Use the default Filename, or enter your own. - - If you want to test only a subset of questions, you can optionally + - If you want to test only a subset of questions, you can optionally filter by {qid} prefix. Leave this field blank to test all the questions. - - Choose the TEST ALL button and wait for the tests to complete. + - Choose the TEST ALL button and wait for the tests to complete. - ![](image5.png) + ![icon](image5.png) - - Choose the view results icon - (![](image6.png)) on the far right to view the test + - Choose the view results icon + (![Test all](image6.png)) on the far right to view the test results. - ![](image7.png) + ![Test all](image7.png) Behind the scenes, a test function sends every question for every item to the QnABot via Amazon Lex and checks that the QnABot matched the question to the expected item. Any incorrect matches are highlighted in red. Test results can be viewed in the browser or downloaded as a CSV file. -2. The QnABot solution also includes a visualization tool (using +2. The QnABot solution also includes a visualization tool (using Kibana) to analyze chatbot usage. Kibana is an open-source data visualization and exploration tool used for log and time-series analytics, application monitoring, and operational intelligence use @@ -244,7 +239,7 @@ You can solve this in a couple of ways: This visualization tool is available from the QnABot designer console. - ![](image8.png) + ![menu](image8.png) The Kibana dashboard can be used to view usage history, logged utterances, no hits utterances, positive user feedback, and negative @@ -258,7 +253,7 @@ no question and answer exists in the knowledge base. This can be another way to see how well the chatbot is performing. You can then augment the knowledge base where needed. -![](image9.png) +![Kibana Dashboard](image9.png) *Figure 4.0 -- Kibana dashboard showing* No Hits *report* @@ -275,39 +270,37 @@ question. In the below example, you can see there are many variations of the same question that can be included to further train the bot to understand user input. -![](image10.png) +![no hits](image10.png) -*Figure 5.0 -- Example question variations* +_Figure 5.0 -- Example question variations_ In previous releases (before v4.0.0) repeated terms like "*how do I use*" could skew scoring toward items with higher numbers of repeating words. -In QnABot 4.0.0 and onward, this problem has been eliminated by changing the +In QnABot 4.0.0 and onward, this problem has been eliminated by changing the elasticsearch index and query to score now only on unique (not repeating) word matches. Previously we recommended to avoid repeating phrases in multiple -questions by putting all the alternatives into a single question like +questions by putting all the alternatives into a single question like `"How do I use Q&A / q and a chatbot?"`. This practice is no longer recommended. -Instead, just add new questions as needed to represent various synonyms and -phrases that a user might say to mean the same thing. +Instead, just add new questions as needed to represent various synonyms and +phrases that a user might say to mean the same thing. The example questions you provide are used to help improve the accuracy of the -answers from QnABot. +answers from QnABot. The example questions are also used to train the Lex Bot (or Alexa skill) to provide more accurate voice recognition, especially when questions are expected to contain uncommon words like acronyms, proper names, or specialized vocabulary. -When using voice, for example when using QnABot in an Amazon Connect contact -center, you will find that transcribed questions are more accurate when you -provide many examples, and use the 'LEX REBUILD' feature as described in +When using voice, for example when using QnABot in an Amazon Connect contact +center, you will find that transcribed questions are more accurate when you +provide many examples, and use the 'LEX REBUILD' feature as described in [Tuning the Bot’s Natural Language Processing](https://www.amazon.com/qnabot/#troubleshoot-tune). - These options can further help tune the chatbot to better understand user input and be able to support different ways of asking the same question. -Conclusion -========== +## Conclusion There are of course many ways we can improve the search experience, such as adding support for synonyms, lexicons, taxonomies, and [Amazon Kendra integration](https://www.amazon.com/qnabot/#kendra)