Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PG to BQ, CDC messes up format of JSONB columns, whereas initial load doesn't #828

Closed
saisrirampur opened this issue Dec 15, 2023 · 2 comments · Fixed by #948
Closed

PG to BQ, CDC messes up format of JSONB columns, whereas initial load doesn't #828

saisrirampur opened this issue Dec 15, 2023 · 2 comments · Fixed by #948
Assignees

Comments

@saisrirampur
Copy link
Contributor

saisrirampur commented Dec 15, 2023

Setup steps in Postgres for JSONB

CREATE TABLE test_jsonb(id serial primary key, j jsonb);
INSERT INTO test_jsonb(j) VALUES ('{"isDemo":false}');
--Kick off mirror now and after initial load insert another row on Postgres
INSERT INTO test_jsonb(j) VALUES ('{"isDemo":false}');

Data in BQ for JSONB

Screen Shot 2023-12-14 at 7 19 48 PM

Setup steps in Postgres for ARRAY

CREATE TABLE test_array(id serial primary key, t text[], i int[]);
INSERT INTO  test_array(t,i) VALUES ('{sai,krishna}'::text[],'{1,2}'::int[]);
--Kick off mirror now and after initial load insert another row on Postgres
INSERT INTO  test_array(t,i) VALUES ('{sai,krishna}'::text[],'{1,2}'::int[]);

Data in BQ for JSONB

Screen Shot 2023-12-14 at 8 25 09 PM
@saisrirampur saisrirampur changed the title PG to BQ, CDC messes up format of JSONB columns PG to BQ, CDC messes up format of JSONB columns, whereas initial load doesn't Dec 15, 2023
@Amogh-Bharadwaj
Copy link
Contributor

Fixed by #830

@iskakaushik
Copy link
Contributor

re-opening this as @heavycrystal mentioned that this is still an issue for SF.

@iskakaushik iskakaushik reopened this Dec 15, 2023
iskakaushik pushed a commit that referenced this issue Jan 1, 2024
Similar to how #830 fixes JSON structure for BigQuery, this one does it
for Snowflake by performing a [PARSE_JSON
transformation](https://docs.snowflake.com/en/sql-reference/functions/parse_json)
in Copy (for initial load) and Merge (CDC) steps

<img width="1111" alt="Screenshot 2024-01-01 at 5 23 52 PM"
src="https://github.com/PeerDB-io/peerdb/assets/65964360/03549553-56c5-475b-b668-0e83b68c8eb0">

You can now query synced Postgres JSONs by PeerDB to Snowflake Variants
using:
```sql
SELECT j:tags FROM mysftable
``` 
Adds a test for this. Also tested manually

Fixes #828
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants