Skip to content
This repository has been archived by the owner on Oct 28, 2022. It is now read-only.

Latest commit

 

History

History
88 lines (66 loc) · 3.81 KB

amazon-sqs.md

File metadata and controls

88 lines (66 loc) · 3.81 KB
Raw

Amazon SQS

Sync overview

This source will sync messages from an SQS Queue.

Output schema

This source will output one stream for the configured SQS Queue.
The stream record data will have three fields:

  • id (a UUIDv4 as a STRING)
  • body (message body as a STRING)
  • attributes (attributes of the messages as an OBJECT or NULL)

Features

Feature Supported?(Yes/No) Notes
Full Refresh Sync yes
Incremental Sync no
Namespaces no

Performance considerations

Getting started

Requirements

  • AWS IAM Access Key
  • AWS IAM Secret Key
  • AWS SQS Queue

Properties

Required properties are 'Queue URL', 'AWS Region' and 'Delete Messages After Read' as noted in bold below.

  • Queue URL (STRING)
    • The full AWS endpoint URL of the queue e.g. https://sqs.eu-west-1.amazonaws.com/1234567890/example-queue-url
  • AWS Region (STRING)
    • The region code for the SQS Queue e.g. eu-west-1
  • Delete Messages After Read (BOOLEAN)
    • WARNING: Setting this option to TRUE can result in data loss, do not enable this option unless you understand the risk. See the Data loss warning section below.
    • Should the message be deleted from the SQS Queue after being read? This prevents the message being read more than once
    • By default messages are NOT deleted, thus can be re-read after the Message Visibility Timeout
    • Default: False
  • Max Batch Size (INTEGER)
    • The max amount of messages to consume in a single poll e.g. 5
    • Minimum of 1, maximum of 10
    • Default: 10
  • Max Wait Time (INTEGER)
    • The max amount of time (in seconds) to poll for messages before commiting a batch (or timing out) unless we fill a batch (as per Max Batch Size)
    • Minimum of 1, maximum of 20
    • Default: 20
  • Message Attributes To Return (STRING)
    • A comma separated list of Attributes to return for each message
    • Default: All
  • Message Visibility Timeout (INTEGER)
    • After a message is read, how much time (in seconds) should the message be hidden from other consumers
    • After this timeout, the message is not deleted and can be re-read
    • Default: 30
  • AWS IAM Access Key ID (STRING)
    • The Access Key for the IAM User with permissions on this Queue
    • If Delete Messages After Read is false then only sqs:ReceiveMessage
    • If Delete Messages After Read is true then sqs:DeleteMessage is also needed
  • AWS IAM Secret Key (STRING)
    • The Secret Key for the IAM User with permissions on this Queue

Data loss warning

When enabling Delete Messages After Read, the Source will delete messages from the SQS Queue after reading them. The message is deleted after the configured Destination takes the message from this Source, but makes no guarentee that the downstream destination has commited/persisted the message. This means that it is possible for the Airbyte Destination to read the message from the Source, the Source deletes the message, then the downstream application fails - resulting in the message being lost permanently.

Extra care should be taken to understand this risk before enabling this option.

Setup guide

NOTE:

  • If Delete Messages After Read is false then the IAM User needs only sqs:ReceiveMessage in the AWS IAM Policy
  • If Delete Messages After Read is true then both sqs:ReceiveMessage and sqs:DeleteMessage are needed in the AWS IAM Policy

CHANGELOG

Version Date Pull Request Subject
0.1.0 2021-10-10 #0000 Initial version