Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strip White Space from Cell Values #215

Closed
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
0d0dbea
feat(logic): strip white space;
JVickery-TBS May 7, 2024
616c948
fix(logic): strip white space for load table;
JVickery-TBS May 7, 2024
860ca9e
fix(logic): strip white space for load table;
JVickery-TBS May 7, 2024
88f96a8
fix(logic): strip white space;
JVickery-TBS May 8, 2024
341ec1e
Merge branch 'master' into feature/strip-white-space
JVickery-TBS May 8, 2024
21a1ece
fix(tests): new code for tests;
JVickery-TBS May 8, 2024
a6ab0a0
fix(logic): load csv white space;
JVickery-TBS May 12, 2024
54f87e0
feat(logic): added `strip_extra_white` info field;
JVickery-TBS May 14, 2024
50080ea
feat(logic): added `strip_extra_white` field;
JVickery-TBS May 14, 2024
116c29f
fix(logic): minor logic fixes;
JVickery-TBS May 14, 2024
43b9f94
feat(tests,i18n): updated tests;
JVickery-TBS Jul 15, 2024
06ee48a
Merge branch 'master' into feature/strip-white-space
JVickery-TBS Jul 15, 2024
c00fb5a
fix(tests,logic): misc fixes;
JVickery-TBS Jul 15, 2024
669930e
fix(tests,logic): new output and parody;
JVickery-TBS Jul 15, 2024
3263bab
fix(logic): ckan versioning;
JVickery-TBS Jul 16, 2024
7cb6a84
fix(logic): ckan versioning;
JVickery-TBS Jul 16, 2024
bf2e939
feat(templates,logic): pre-datadictionary implement;
JVickery-TBS Jul 16, 2024
d6de1b1
feat(tests): add coverage;
JVickery-TBS Jul 22, 2024
8564cdd
Merge branch 'master' into feature/strip-white-space
JVickery-TBS Jul 23, 2024
9b65844
Merge branch 'master' into feature/strip-white-space
JVickery-TBS Dec 9, 2024
bf7efc6
fix(tests): ds keys;
JVickery-TBS Dec 9, 2024
4a35fc8
fix(tests): datadictionary;
JVickery-TBS Dec 9, 2024
4886902
debug tests...
JVickery-TBS Dec 9, 2024
6587421
debug tests...
JVickery-TBS Dec 9, 2024
20a33f6
fix(logic): loader;
JVickery-TBS Dec 9, 2024
e4aac5d
fix(logic): datadict versions;
JVickery-TBS Dec 9, 2024
dd137f3
fix(templates): selected value;
JVickery-TBS Dec 9, 2024
c7fb399
feat(misc): readme, changelog;
JVickery-TBS Dec 9, 2024
ec8ef04
Merge branch 'master' into feature/strip-white-space
JVickery-TBS Dec 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions ckanext/xloader/loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -169,11 +169,17 @@ def load_csv(csv_filepath, resource_id, mimetype='text/csv', logger=None):
try:
with UnknownEncodingStream(csv_filepath, file_format, decoding_result,
skip_rows=skip_rows) as stream:
stream.save(**save_args)
for row in stream:
for _index, _cell in enumerate(row):
row[_index] = str(_cell).strip() # strip white space around cell values
stream.save(**save_args) # have to save inside of the tabulator stream iterator
except (EncodingError, UnicodeDecodeError):
with Stream(csv_filepath, format=file_format, encoding=SINGLE_BYTE_ENCODING,
skip_rows=skip_rows) as stream:
stream.save(**save_args)
for row in stream:
for _index, _cell in enumerate(row):
row[_index] = str(_cell).strip() # strip white space around cell values
stream.save(**save_args) # have to save inside of the tabulator stream iterator
csv_filepath = f_write.name

# datastore db connection
Expand Down
2 changes: 2 additions & 0 deletions ckanext/xloader/parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ def convert_types(self, extended_rows):
for cell_index, cell_value in enumerate(row):
if cell_value is None:
row[cell_index] = ''
cell_value = str(cell_value).strip() # strip white space around cell values
JVickery-TBS marked this conversation as resolved.
Show resolved Hide resolved
row[cell_index] = str(cell_value).strip() # strip white space around cell values
if not cell_value:
continue
cell_type = self.types[cell_index] if self.types else None
Expand Down
Loading