fix: Reconfigure the encoding of standard input according to the --en…

…coding option, closes #1038
wireservice · Oct 18, 2023 · 76996a8 · 76996a8
1 parent 12be2ff
commit 76996a8
Show file tree

Hide file tree

Showing 2 changed files with 2 additions and 0 deletions.
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -10,6 +10,7 @@ Unreleased
 * :doc:`/scripts/csvstat` adds a :code:`--non-nulls` option to only output counts of non-null values.
 * :doc:`/scripts/csvstat` adds a :code:`--max-precision` option to only output the most decimal places.
 * feat: Add a :code:`--null-value` option to commands with the :code:`--blanks` option, to convert additional values to NULL.
+* fix: Reconfigure the encoding of standard input according to the :code:`--encoding` option, which defaults to ``utf-8-sig``. Affected users no longer need to set the ``PYTHONIOENCODING`` environment variable.
 * fix: Prompt the user if additional input is expected (i.e. if no input file or piped data is provided) in :doc:`/scripts/csvjoin`, :doc:`/scripts/csvsql` and :doc:`/scripts/csvstack`.
 * fix: No longer errors if a NUL byte occurs in an input file.
 * Add Python 3.12 support.

diff --git a/csvkit/cli.py b/csvkit/cli.py
@@ -238,6 +238,7 @@ def _open_input_file(self, path):
         Open the input file specified on the command line.
         """
         if not path or path == '-':
+            sys.stdin.reconfigure(encoding=self.args.encoding)
             f = sys.stdin
         else:
             extension = splitext(path)[1]