Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: add test for multiple identical schema messages #220

Merged
merged 2 commits into from
Nov 27, 2023
Merged

fix: add test for multiple identical schema messages #220

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e1fd37d
SDK bump
sebastianswms Oct 25, 2023
bbe5364
Remove SDK bump (handled by dependabot)
sebastianswms Oct 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ packages = [
[tool.poetry.dependencies]
python = "<3.12,>=3.7.1"
requests = "^2.25.1"
singer-sdk = ">=0.28,<0.34"
singer-sdk = ">=0.31,<0.34"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is handled automatically by Depandabot PRs here we should probably just leave this be as I don't see a big value in bumping up the lower end of the singer-sdk range right now (Generally Poetry should just pull the latest SingerSDK version so maybe there's an argument for not having a range but that's pretty seperate

#199

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My logic was that 0.31 is the version that merged meltano/sdk#1864 which was mentioned as a fix for #124. But if Dependabot is handling it, that makes sense.

psycopg2-binary = "2.9.9"
sqlalchemy = ">=2.0,<3.0"
sshtunnel = "0.4.0"
Expand Down
12 changes: 12 additions & 0 deletions target_postgres/tests/data_files/multiple_schema_messages.singer
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{"type": "SCHEMA", "stream": "test_multiple_schema_messages", "key_properties": ["id"], "schema": {"required": ["id"], "type": "object", "properties": {"id": {"type": "integer"}, "metric": {"type": "integer"}}}}
{"type": "SCHEMA", "stream": "test_multiple_schema_messages", "key_properties": ["id"], "schema": {"required": ["id"], "type": "object", "properties": {"id": {"type": "integer"}, "metric": {"type": "integer"}}}}
{"type": "SCHEMA", "stream": "test_multiple_schema_messages", "key_properties": ["id"], "schema": {"required": ["id"], "type": "object", "properties": {"id": {"type": "integer"}, "metric": {"type": "integer"}}}}
{"type": "SCHEMA", "stream": "test_multiple_schema_messages", "key_properties": ["id"], "schema": {"required": ["id"], "type": "object", "properties": {"id": {"type": "integer"}, "metric": {"type": "integer"}}}}
{"type": "RECORD", "stream": "test_multiple_schema_messages", "record": {"id": 1, "metric": 100}}
{"type": "RECORD", "stream": "test_multiple_schema_messages", "record": {"id": 2, "metric": 200}}
{"type": "RECORD", "stream": "test_multiple_schema_messages", "record": {"id": 3, "metric": 300}}
{"type": "SCHEMA", "stream": "test_multiple_schema_messages", "key_properties": ["id"], "schema": {"required": ["id"], "type": "object", "properties": {"id": {"type": "integer"}, "metric": {"type": "integer"}}}}
{"type": "SCHEMA", "stream": "test_multiple_schema_messages", "key_properties": ["id"], "schema": {"required": ["id"], "type": "object", "properties": {"id": {"type": "integer"}, "metric": {"type": "integer"}}}}
{"type": "RECORD", "stream": "test_multiple_schema_messages", "record": {"id": 4, "metric": 400}}
{"type": "RECORD", "stream": "test_multiple_schema_messages", "record": {"id": 5, "metric": 500}}
{"type": "RECORD", "stream": "test_multiple_schema_messages", "record": {"id": 6, "metric": 600}}
14 changes: 14 additions & 0 deletions target_postgres/tests/test_standard_target.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,20 @@ def test_multiple_state_messages(postgres_target):
singer_file_to_target(file_name, postgres_target)


# TODO test that data is correct
def test_multiple_schema_messages(postgres_target, caplog):
"""Test multiple identical schema messages.

Multiple schema messages with the same schema should not cause 'schema has changed'
logging statements. See: https://github.com/MeltanoLabs/target-postgres/issues/124

Caplog docs: https://docs.pytest.org/en/latest/how-to/logging.html#caplog-fixture
"""
file_name = "multiple_schema_messages.singer"
singer_file_to_target(file_name, postgres_target)
assert "Schema has changed for stream" not in caplog.text


def test_relational_data(postgres_target):
engine = create_engine(postgres_target)
file_name = "user_location_data.singer"
Expand Down
Loading