Skip to content

Commit

Permalink
fix: fix pipeline failure due to bigquery Provided Schema does not ma…
Browse files Browse the repository at this point in the history
…tch Table. (#6344)

fix kokoro job failing with error message:
```
+ bq load --autodetect --project_id=cloud-java-metrics --source_format=CSV client_library_versions.cloud_java_client_library_release_dates cloud_java_client_library_release_dates.csv
Upload complete.Waiting on bqjob_r28424c53fa9642fd_0000018b47445529_1 ... (0s) Current status: RUNNING                                                                                      Waiting on bqjob_r28424c53fa9642fd_0000018b47445529_1 ... (0s) Current status: DONE  

BigQuery error in load operation: Error processing job
'cloud-java-metrics:bqjob_r28424c53fa9642fd_0000018b47445529_1': Provided Schema
does not match Table cloud-java-metrics:client_library_versions.cloud_java_clien
t_library_release_dates. Cannot add fields (field: string_field_0)
Failure details:
- It looks like you are appending to an existing table with
autodetect enabled. Disabling autodetect may resolve this.
```
Relevant gcloud documentation: [bq load](https://cloud.google.com/bigquery/docs/reference/bq-cli-reference#bq_load)
- remove `--autodetect` and use default false so destination table schema is used.
- `--skip_leading_rows=1` since no need to infer schema as above.
- add --null_marker="-" to deal with empty release dates from source, e.g. for `google-cloud-accessapproval:1.0.10`

![image](https://github.com/googleapis/java-cloud-bom/assets/3340152/7edc8ecf-7f0e-4b8a-969a-b7d5f475cfa0)


Manual script run and successfully run script to upload data.
  • Loading branch information
zhumin8 authored Dec 26, 2023
1 parent 3fe0c17 commit 17c18f1
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion .kokoro/nightly/create-versions-csv.sh
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ echo "Inserting client_library_versions.cloud_java_client_library_release_dates.
head cloud_java_client_library_release_dates.csv
echo "===================="

bq load --autodetect --project_id=cloud-java-metrics --source_format=CSV \
bq load --skip_leading_rows=1 --project_id=cloud-java-metrics --source_format=CSV --null_marker="-" \
client_library_versions.cloud_java_client_library_release_dates \
cloud_java_client_library_release_dates.csv

Expand Down

0 comments on commit 17c18f1

Please sign in to comment.