Skip to content

Commit 60db822

Browse files
mjsqumjsqu
authored
Merge pull request wintersrd#75 from s7clarke10/feature/patched_singer_python_framework
Feature/patched singer python framework
2 parents e4f90bd + bca10ba commit 60db822

File tree

5 files changed

+769
-530
lines changed

5 files changed

+769
-530
lines changed

‎.bumpversion.cfg

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[bumpversion]
2-
current_version = 2.3.1
2+
current_version = 2.4.0
33
parse = (?P<major>\d+)
44
\.(?P<minor>\d+)
55
\.(?P<patch>\d+)

‎CHANGELOG.md

+88-77
Original file line numberDiff line numberDiff line change
@@ -1,77 +1,88 @@
1-
# tap-mssql 2.3.1 2024-07-22
2-
3-
* Bug Fix. Issue #62 - change to column selection using lists instead of sets to preserve column ordering
4-
5-
# tap-mssql 2.3.0 2024-04-18
6-
7-
* Bug Fix. Change pendulum DateTime type to datetime.datetime as pymssql 2.3.0 is no longer compatible with query parameters as pendulum DateTime (https://github.com/pymssql/pymssql/issues/889)
8-
9-
# tap-mssql 2.2.3 2024-02-26
10-
11-
* Bug Fix. Enhancing singer-decimal output for numeric data to correctly output the correct datatype - string and precision.
12-
13-
# tap-mssql 2.2.1 2023-10-30
14-
15-
* Bug Fix. Removing test code which slipped into the release, and adjusting offending code with correct dynamic column name.
16-
17-
# tap-mssql 2.2.0 2023-08-23
18-
19-
This feature increases support for SQL Datatypes.
20-
21-
* Adds support for datetimeoffset - https://github.com/wintersrd/pipelinewise-tap-mssql/issues/50
22-
* Resolves issue with datetime2 rounding with high precision timestamps https://github.com/wintersrd/pipelinewise-tap-mssql/issues/51
23-
* Resolves the max and min range for tinyints. The current min and max are correct for MariaDB and MySQL only. MSSQL Server only supports
24-
positive integers (unsigned tinyint). https://github.com/wintersrd/pipelinewise-tap-mssql/issues/2
25-
26-
# tap-mssql 2.1.0 2023-08-01
27-
28-
This is a number of new enhancements to extend capability and resolving a few bugs.
29-
30-
## Features
31-
* Adds `use_singer_decimal` setting, capability i.e. the ability to output decimal, float, and integer data as strings to avoid loss of precision. A target like this custom [target-snowflake](https://github.com/mjsqu/pipelinewise-target-snowflake) can decode this back into numeric data.
32-
* Ordering columns by their ordinal position so the schema matches the database definition
33-
* Support for new `cursor_array_size` parameter. This allows fetching records in batches
34-
speeding up network traffic. Recommend a setting of `10000` records.
35-
* Fixing bug. We have encountered a scenario where a stream starts. It failed and there
36-
was no bookmark written.
37-
When the job is re-run the state file had currently_syncing for the given stream but with
38-
no bookmark written it didn't appear in the streams_with_state list. This small change will
39-
ensure that the currently_syncing_list will be populated whether there is a bookmark or not
40-
by filtering against the ordered_streams list which includes both streams_without_state and
41-
streams_with_state.
42-
* Removed test for "INCREMENTAL" and not primary_keys.
43-
INCREMENTAL loads can be performed without primary keys as long as there
44-
is a replication key.
45-
* Updating the CDC documentation
46-
* Fixing bug. Where encountered a scenario where state lsn == max lsn.
47-
If the lsn coming from state is equal to the maximum lsn, then we should not be incrementing
48-
this lsn as it causes errors in the `fn_cdc_get_all_changes` function.
49-
* Fixing bug. The _sdc_lsn_operation should be 2 for Inserts not Deletes.
50-
* Fixing bug. pymssql 2.2.8 does not work. Excluding this version from pyproject.toml.
51-
* Feature. Adding a logo to the tap.
52-
* Resolving PR: https://github.com/wintersrd/pipelinewise-tap-mssql/pull/36
53-
* Resolving PR: https://github.com/wintersrd/pipelinewise-tap-mssql/pull/16
54-
55-
# tap-mssql 2.0.0 2022-08-15
56-
57-
This is the first properly productionalized release of tap-mssql and brings a number of significant enhancements to both extend capability and improve future maintainability
58-
59-
## Features
60-
* Adds capability to use log based CDC
61-
* Adds a number of new data types:
62-
* Additional datetime options
63-
* Small money
64-
* Numeric
65-
* Allows configuration on how to use datetime
66-
67-
## Development enhancements
68-
* Adds a tox test environment plus dockerized MSSQL server to allow proper testing
69-
* Rebuilds all tests to match SQL server
70-
* Replaces setup.py with pyproject.toml + poetry.lock and extends requirements to include full testing suite
71-
* Adds a robust pre-commit for development purposes
72-
* Configures bumpversion
73-
74-
## Other under the hood
75-
* Reformats all code to black (100 character lines)
76-
* Corrects all open flake8 errors
77-
* Removes unused packages
1+
# tap-mssql 2.4.0 2024-08-08
2+
3+
* Moving to patched version of Singer Framework plus using msgspec rather than orjson for JSON serialization speed.
4+
* Adjusting tox file for test pipeline to point to correct location of the sqlcmd in the latest MSSQL docker image.
5+
* Adjusting tox file. Added -No switch calling sqlcmd to allow user/password connection to MSSQL.
6+
* Patching dependencies.
7+
* Pinned Requests to "2.31.0". This is required as the tox pipeline fails with a Docker API error with requests "2.32.x".
8+
* Explicitly setting python version 3.8 -> 3.12
9+
* Moving to using recommended poetry-core for build. https://python-poetry.org/docs/pyproject/#poetry-and-pep-517
10+
* NOTE: Recommend refactoring to avoid using tox-poetry-installer as restrictive dependencies are stopping tox and docker updates. This may resolve the requests/docker api issue.
11+
12+
# tap-mssql 2.3.1 2024-07-22
13+
14+
* Bug Fix. Issue #62 - change to column selection using lists instead of sets to preserve column ordering
15+
16+
# tap-mssql 2.3.0 2024-04-18
17+
18+
* Bug Fix. Change pendulum DateTime type to datetime.datetime as pymssql 2.3.0 is no longer compatible with query parameters as pendulum DateTime (https://github.com/pymssql/pymssql/issues/889)
19+
20+
# tap-mssql 2.2.3 2024-02-26
21+
22+
* Bug Fix. Enhancing singer-decimal output for numeric data to correctly output the correct datatype - string and precision.
23+
24+
# tap-mssql 2.2.1 2023-10-30
25+
26+
* Bug Fix. Removing test code which slipped into the release, and adjusting offending code with correct dynamic column name.
27+
28+
# tap-mssql 2.2.0 2023-08-23
29+
30+
This feature increases support for SQL Datatypes.
31+
32+
* Adds support for datetimeoffset - https://github.com/wintersrd/pipelinewise-tap-mssql/issues/50
33+
* Resolves issue with datetime2 rounding with high precision timestamps https://github.com/wintersrd/pipelinewise-tap-mssql/issues/51
34+
* Resolves the max and min range for tinyints. The current min and max are correct for MariaDB and MySQL only. MSSQL Server only supports
35+
positive integers (unsigned tinyint). https://github.com/wintersrd/pipelinewise-tap-mssql/issues/2
36+
37+
# tap-mssql 2.1.0 2023-08-01
38+
39+
This is a number of new enhancements to extend capability and resolving a few bugs.
40+
41+
## Features
42+
* Adds `use_singer_decimal` setting, capability i.e. the ability to output decimal, float, and integer data as strings to avoid loss of precision. A target like this custom [target-snowflake](https://github.com/mjsqu/pipelinewise-target-snowflake) can decode this back into numeric data.
43+
* Ordering columns by their ordinal position so the schema matches the database definition
44+
* Support for new `cursor_array_size` parameter. This allows fetching records in batches
45+
speeding up network traffic. Recommend a setting of `10000` records.
46+
* Fixing bug. We have encountered a scenario where a stream starts. It failed and there
47+
was no bookmark written.
48+
When the job is re-run the state file had currently_syncing for the given stream but with
49+
no bookmark written it didn't appear in the streams_with_state list. This small change will
50+
ensure that the currently_syncing_list will be populated whether there is a bookmark or not
51+
by filtering against the ordered_streams list which includes both streams_without_state and
52+
streams_with_state.
53+
* Removed test for "INCREMENTAL" and not primary_keys.
54+
INCREMENTAL loads can be performed without primary keys as long as there
55+
is a replication key.
56+
* Updating the CDC documentation
57+
* Fixing bug. Where encountered a scenario where state lsn == max lsn.
58+
If the lsn coming from state is equal to the maximum lsn, then we should not be incrementing
59+
this lsn as it causes errors in the `fn_cdc_get_all_changes` function.
60+
* Fixing bug. The _sdc_lsn_operation should be 2 for Inserts not Deletes.
61+
* Fixing bug. pymssql 2.2.8 does not work. Excluding this version from pyproject.toml.
62+
* Feature. Adding a logo to the tap.
63+
* Resolving PR: https://github.com/wintersrd/pipelinewise-tap-mssql/pull/36
64+
* Resolving PR: https://github.com/wintersrd/pipelinewise-tap-mssql/pull/16
65+
66+
# tap-mssql 2.0.0 2022-08-15
67+
68+
This is the first properly productionalized release of tap-mssql and brings a number of significant enhancements to both extend capability and improve future maintainability
69+
70+
## Features
71+
* Adds capability to use log based CDC
72+
* Adds a number of new data types:
73+
* Additional datetime options
74+
* Small money
75+
* Numeric
76+
* Allows configuration on how to use datetime
77+
78+
## Development enhancements
79+
* Adds a tox test environment plus dockerized MSSQL server to allow proper testing
80+
* Rebuilds all tests to match SQL server
81+
* Replaces setup.py with pyproject.toml + poetry.lock and extends requirements to include full testing suite
82+
* Adds a robust pre-commit for development purposes
83+
* Configures bumpversion
84+
85+
## Other under the hood
86+
* Reformats all code to black (100 character lines)
87+
* Corrects all open flake8 errors
88+
* Removes unused packages

0 commit comments

Comments
 (0)