-
Notifications
You must be signed in to change notification settings - Fork 62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Map item catch #365
Merged
Merged
Map item catch #365
Changes from 5 commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
292d51b
use iterate_item to check map_item returns something if not warn admi…
dale-wahl deb7791
apparently something with spacy and typing_extensions is broken
dale-wahl 6b5e31d
remove old debug print
dale-wahl 635e205
wrap map_item with processor.get_mapped_item as well as check for map…
dale-wahl c4d6d41
conform iterate_mapped_items to new method
dale-wahl 0fb19cc
get_columns needs to detect non CSV and NDJSON
dale-wahl 3944165
Check Instagram items for ads
dale-wahl 5ffd6dc
warn on instagram ads; add customizable warning message
dale-wahl 6367e31
add healthcheck from master
dale-wahl 1725a92
Merge branch 'master' into map_item_catch
dale-wahl b32ddca
do not always warn on map_item error (for example when getting datase…
dale-wahl 2184e2e
ensure processor exists prior to checking map_item (processors may be…
dale-wahl a1d99d5
no mapping when there is no processor
dale-wahl f71110e
Merge branch 'master' into map_item_catch
dale-wahl 5d9c5d5
do not warn unmappable for previews or creating CSV extract (warning …
dale-wahl 92440f5
only warn admins once per dataset
dale-wahl 1cab055
fix warning when processor is None
dale-wahl dd1027f
Count number of unmappable items
dale-wahl File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dale-wahl is there any datasource that does this at the moment?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's some old voodoo (I think it is yours to be honest!)... I extracted that code and comment into a new method to use as a check to see if
map_item
should be run, but I did not modify the code. I believe it has to do with custom imports of, say, a Twitter dataset. You could convert the dataset type from custom to a Twitter datasource type, but it would not be an NDJSON as expected. That would also presumably apply to any ZeeSchuimer datasource (a custom Doutin/Instagram CSV uploaded would not be able to usemap_item
). In practice, I am not exactly sure how often we ran into this problem since users cannot change datatypes only admins (am I wrong about that?).