Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the InfluxDB Timestream connector #201

Draft
wants to merge 9 commits into
base: mainline
Choose a base branch
from

Commits on Sep 25, 2024

  1. Initial commit for the InfluxDB Timestream Connector.

    Signed-off-by: forestmvey <[email protected]>
    forestmvey committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    cfce8f0 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Add pre-commit hook for secrets scanning

    *Issue #, if available:*
    
    N/A.
    
    *Description of changes:*
    
    - A pre-commit hook has been added that uses `aws-secrets` in order to prevent secrets from being committed.
    
    
    By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
    trevorbonas authored Oct 1, 2024
    Configuration menu
    Copy the full SHA
    eea8a7c View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. Add SAM template

    *Issue #, if available:*
    
    N/A
    
    *Description of changes:*
    
    - SAM template `template.yml` added.
        - Asynchronous and synchronous invocation supported.
    - Documentation for how the connector can be deployed using the SAM template added.
    - `lambda_runtime` crate added.
    - `LambdaEvent<serde_json::Value>` used instead of `lambda_http::Request`, in order to support more types of requests, instead of just AWS service integrations.
    - Tests added for handling different kinds of `queryParameters` keys in requests.
    - DLQ implemented for asynchronous invocation.
    - Integration tests changed to check the newly returned `serde_json::Value` struct.
    
    - [x] Unit tests passed.
    - [x] Integration tests passed.
    
    
    By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
    trevorbonas authored Oct 7, 2024
    Configuration menu
    Copy the full SHA
    da989ef View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Add support for customer-defined partition keys

    - Users can now define partition keys for their newly-created tables. Partition key configuration is controlled using three environment variables, `custom_partition_key_type`, `custom_partition_key_dimension`, and `enforce_custom_partition_key`.
    - Environment variables added for CDPK support.
    - SAM template parameters added for CDPK support.
    - Documentation added for CDPK support.
    - Integration tests added for CDPK support.
    - Integration test asserts moved to end of tests, in order to ensure resources are cleaned up.
    
    - [x] Unit tests passed.
    - [x] Integration tests passed.
    trevorbonas authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    8c8bc2f View commit details
    Browse the repository at this point in the history
  2. Improve IAM permissions

    *Issue #, if available:*
    
    N/A.
    
    *Description of changes:*
    
    - Deployment permissions have been updated according to the required permissions as discovered by testing deploying the connector using its SAM template.
    - `samconfig.toml` added with default stack deployment options.
    - "Troubleshooting" section added to the README with two known errors users may encounter.
        - Issue with cross-platform Rust compilation.
        - ConflictExceptions occurring with concurrent instances of the connector.
    
    
    By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
    trevorbonas authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    3070ee6 View commit details
    Browse the repository at this point in the history
  3. Return 2.0 formatted Lambda responses for local invocation

    - The environment variable `local_invocation` has been added and is set to `true` by default when the Lambda function is run with `cargo lambda watch`. This environment enables responses to be returned in a format `cargo lambda` expects, in the 2.0 formatting. Otherwise, the 1.0 format will be returned, which the synchronous API Gateway expects.
    - `cargo fmt` run.
    
    - [x] Tested locally.
    - [x] Tested with synchronous invocation.
    - [x] Tested with asynchronous invocation.
    trevorbonas authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    996d3f7 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Optimize connector

    *Issue #, if available:*
    
    N/A.
    
    *Description of changes:*
    - Ingestion to multiple tables done in parallel.
    - Ingestion of 100 records to a single table done in parallel.
    - Chunking of records into batches of 100 done in parallel.
    - Limit on maximum possible number of threads added.
    - Print statement for each 100 records removed.
    - Logging option added to SAM template.
    - All `println!` calls changed to `info!`.
    - Default logging level for the connector changed to `INFO`.
    - Trace statements that measure the execution time of each function added.
    - Instructions added to README for how to configure logging levels.
    - Database and tables are no longer checked if their corresponding environment variables for creation are not set.
    - The checking of whether tables exist has been moved within the asynchronous code block used to ingest records to a table. This checking is now done in parallel and the hashmap is looped through once, instead of twice.
    - Instructions added to README for how to reduce stack costs.
    
    - [x] Integration tests passed.
    - [x] Unit tests passed.
    
    By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
    trevorbonas authored Oct 22, 2024
    Configuration menu
    Copy the full SHA
    0e6fdf7 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2024

  1. Add Single Table Mapping for InfluxDB Timestream Connector (#23)

    * Initial implementation for adding single-table mapping.
    
    Signed-off-by: forestmvey <[email protected]>
    
    * Adding environment variables and updating documentation.
    
    Signed-off-by: forestmvey <[email protected]>
    
    * Adding tests and revising single-table mapping.
    
    Signed-off-by: forestmvey <[email protected]>
    
    * Adding single table example to README.
    
    Signed-off-by: forestmvey <[email protected]>
    
    * Fixing measure-name using line protocol metric name for single table mapping.
    
    Signed-off-by: forestmvey <[email protected]>
    
    * Fix typo in README.
    
    Signed-off-by: forestmvey <[email protected]>
    
    * Fixing documentation typos and test setting wrong environment variable.
    
    Signed-off-by: forestmvey <[email protected]>
    
    * Fixing table for single table multi measure records example in README.
    
    Signed-off-by: forestmvey <[email protected]>
    
    * Fix invalid line protocol examples with additional comma separating the timestamp.
    
    Signed-off-by: forestmvey <[email protected]>
    
    ---------
    
    Signed-off-by: forestmvey <[email protected]>
    forestmvey authored Oct 31, 2024
    Configuration menu
    Copy the full SHA
    4c51820 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2024

  1. Multi-table Mapping Default for InfluxDB Timestream Connector (#25)

    * Set default table mapping to multi-table for the InfluxDB Timestream Connector.
    
    Signed-off-by: forestmvey <[email protected]>
    
    * Fixing table formatting in README for InfluxDB Timestream Connector.
    
    Signed-off-by: forestmvey <[email protected]>
    
    * Updating README to reference multi-table for the default table mapping.
    
    Signed-off-by: forestmvey <[email protected]>
    
    ---------
    
    Signed-off-by: forestmvey <[email protected]>
    forestmvey authored Nov 15, 2024
    Configuration menu
    Copy the full SHA
    4487277 View commit details
    Browse the repository at this point in the history