Use JSONL to upload to BigQuery

[judahrand]

Problem

Describe the problem your PR is trying to solve

Proposed changes

• Use JSONL rather than Avro to reduce computation and simplify code.
• Dump the JSONL to GCS and let BigQuery work out how to efficiently load the data.
• Only breaking change that I'm aware of is that you will now need to define a gcs_bucket in the config.

#85

Types of changes

What types of changes does your code introduce to target-bigquery?
Put an x in the boxes that apply

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation Update (if none of the other choices apply)

Checklist

• Description above provides context of the change
• I have added tests that prove my fix is effective or that my feature works
• Unit tests for changes (not needed for documentation changes)
• CI checks pass with my changes
• Bumping version in setup.py is an individual PR and not mixed with feature or bugfix PRs
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions

[judahrand]

@jmriego This passes the integration tests on our fork so I assume it will here too.

[judahrand]

The integration tests also seem to run faster using this implementation. Obviously, there's some noise to them but I've seen around 24 minutes for the Avro implementation and 19 minutes for this JSONL + GCS implementation.

[jmriego]

that's really good! do you mind giving me some time to test this in my environment as we have quite a few weird schemas and data we can use for testing?

[jmriego]

can we make this optional instead? I'd prefer if we don't force people to use GCS as I know not everyone is using it and it's a breaking change.
Should be OK to upload then read from GCS if this parameter is provided or use load_table_from_file if not

[judahrand]

IMO there's no real reason not to ingest via GCS. It's certainly not a priority for us and I don't think I'd have the time to make the change + add additional testing in the near future. Though you're more than welcome to make the necessary changes. It will uglify the code a bit too.

[jmriego]

the thing is that we have been using it without uploading to GCS for years now and this would make it more difficult for us to have PII compliance unless we deleted the files later. I'm going to merge this to a new jsonl branch and I'll do that change and some testing on it

[judahrand]

This implementation does delete the files:

pipelinewise-target-bigquery/target_bigquery/db_sync.py

Lines 443 to 446 in c1272d3

	try:
	job.result()
	finally:
	blob.delete()

[jmriego]

sounds good! just for the sake of conversation. Let's say we make GCS mandatory. What would be the benefits of doing that?
I can see that it would be more similar to the Snowflake target maintained by PipelineWise that forces you to use a stage or S3. But I don't think you'd win any speed, debug capabilities (if we delete the file in a finally) or anything else.
What do you think?

[judahrand]

from https://www.oreilly.com/library/view/google-bigquery-the/9781492044451/ch04.html

^ This suggests that when your network is fast loading into GCS is better. At least the way we're running this (and I'd guess is common) both our DB and (obviously) BQ are in GCP (in the same region in fact). So the network connection is very fast (10Gbps+) This makes GCS the obvious choice (and probably uncompressed better than compressed).

Do you disagree?

[judahrand]

Also given that we're already not using compression that also suggests that outside GCP loading via GCS is still quicker.

[judahrand]

@jmriego We have now deployed these changes internally and have seen a dramatic increase in throughput both in BQ ingestion times and in individual record processing.