Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds S3 GUCs #81

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Adds S3 GUCs #81

wants to merge 2 commits into from

Commits on Nov 27, 2024

  1. Match fields by name via option 

    We add an option for `COPY FROM` called `match_by_name` which matches Parquet file fields to PostgreSQL table columns
`by their names` rather than `by their order` in the schema. By default, the option is `false`. The option is useful
when field order differs between the Parquet file and the table, but their names match.

**!!IMPORTANT!!**: This is a breaking change. Before the PR, we match always by name. This is a bit strict and not common
way to match schemas. (e.g. COPY FROM csv at postgres or COPY FROM of duckdb match by field position by default)
This is why we match by position by default and have a COPY FROM option `match_by_name` that can be set to true
for the old behaviour.

Closes #39.
    @aykut-bozkurt
    aykut-bozkurt committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    4241cae View commit details
    Browse the repository at this point in the history

  2. Adds S3 GUCs 

    To make it more Postgres way, we define 3 s3 related GUCs.

- `pg_parquet.aws_config_file`: an absolute path to the configuration file used by s3 client. Note that when set,
   the GUC overrides `AWS_CONFIG_FILE` environment variable. By default the GUC is unset,
- `pg_parquet.aws_shared_credentials_file`: an absolute path to the shared credentials file used by s3 client.
   Note that when set, the GUC overrides `AWS_SHARED_CREDENTIALS_FILE` environment variable. By default the GUC is unset,
- `pg_parquet.aws_profile`: the profile name used by s3 client. Note that when set, the GUC overrides `AWS_PROFILE`
   environment variable. By default the GUC is unset.

These GUCs can only be set by a superuser. You can easily set these GUCs to use different configurations per session.
You do not need to restart your session to change the config file, shared credentials file or profile name.

Closes #70.
    @aykut-bozkurt
    aykut-bozkurt committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    2acb492 View commit details
    Browse the repository at this point in the history