Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

As a User, I can provide all arguments as click options for the GCS Reader, S3 Reader, GCS Writer and BQ Writer #93

Open
gabrielleberanger opened this issue Feb 8, 2021 · 0 comments · May be fixed by #146
Assignees
Labels
enhancement Refactoring, factorizing, adding tests, etc. P2 2nd priority

Comments

@gabrielleberanger
Copy link
Contributor

WHY
The following readers/writers are using implicit config.<VAR> variables:

  • gcs_reader, via the object_storage_reader
    • config.PROJECT_ID
  • s3_reader, via the object_storage_reader
    • config.REGION_NAME, config.AWS_ACCESS_KEY_ID, config.AWS_SECRET_ACCESS_KEY
  • gcs_writer
    • config.PROJECT_ID
  • bq_writer
    • config.PROJECT_ID

These variables are not provided by the user as click options, but as environment variables. These environment variables are retrieved from the following snippet that appears in the nck/config.py module:

for key, var in os.environ.items():
    locals()[key] = var

This can be quite confusing for the user, and does not match the common NCK logic.

HOW
Explicitly implement these variables as click options

@gabrielleberanger gabrielleberanger added enhancement Refactoring, factorizing, adding tests, etc. P2 2nd priority labels Feb 8, 2021
@gabrielleberanger gabrielleberanger changed the title As a User, I can provide all arguments of as click options for the GCS Reader, S3 Reader, GCS Writer and BQ Writer As a User, I can provide all arguments as click options for the GCS Reader, S3 Reader, GCS Writer and BQ Writer Mar 18, 2021
@louisRDSC louisRDSC mentioned this issue Apr 22, 2021
5 tasks
@pol-defont-reaulx pol-defont-reaulx self-assigned this Apr 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Refactoring, factorizing, adding tests, etc. P2 2nd priority
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants