Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Be more robust in handling a user-specified list of fields #46

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Be more robust in handling a user-specified list of fields #46

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions es2csv.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,22 @@
import progressbar
from functools import wraps

def normalize_fields(fields):
'''Normalize different possible ways to specify the document fields

Both space and comma are invalid characters to use in field names so we can split on those safely.

>>> normalize_fields([])
[]
>>> normalize_fields(['a,b,c', 'd', 'e f g'])
['a', 'b', 'c', 'd', 'e', 'f', 'g']
>>> normalize_fields(['a,b,c'])
['a', 'b', 'c']
>>> normalize_fields(['a','b','c'])
['a', 'b', 'c']
'''
return [f for e in fields for f in e.replace(' ', ',').split(',')]

FLUSH_BUFFER = 1000 # Chunk of docs to flush in temp file
CONNECTION_TIMEOUT = 120
TIMES_TO_TRY = 3
Expand Down Expand Up @@ -121,6 +137,7 @@ def next_scroll(scroll_id):
self.opts.query, '(%s)' % ' AND '.join(self.opts.tags))
search_args['q'] = query

self.opts.fields = normalize_fields(self.opts.fields)
if '_all' not in self.opts.fields:
search_args['_source_include'] = ','.join(self.opts.fields)
self.csv_headers.extend([field for field in self.opts.fields if '*' not in field])
Expand Down