Add python 3.10 in image #4
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… in the queue and not rows.
According to uber#279 an issue with pickling `ParseResult` accross pyspark driver-executor border may result in a corrupt `ParseResult` instance that is lacking `hostname` attribute. We try to fix the issue by marshalling url string instead of the `ParseResult` instance.
Observed the following failure with pyarrow 0.14.1 and python 2.7:
```
from petastorm.reader import Reader, make_batch_reader
path = '/tmp/train'
schema_fields = ['features']
batch_reader = make_batch_reader('file://'+path, schema_fields=schema_fields)
print(batch_reader.next())
Traceback (most recent call last):
File "test.py", line 9, in <module>
print(batch_reader.next())
File "/home/.../lib/python3.7/site-packages/petastorm/reader.py", line 616, in next
return self.__next__()
File "/home/.../lib/python3.7/site-packages/petastorm/reader.py", line 610, in __next__
return self._results_queue_reader.read_next(self._workers_pool, self.schema, self.ngram)
File "/home/.../lib/python3.7/site-packages/petastorm/arrow_reader_worker.py", line 79, in read_next
return schema.make_namedtuple(**result_dict)
File "/home/.../lib/python3.7/site-packages/petastorm/unischema.py", line 286, in make_namedtuple
return self._get_namedtuple()(**typed_dict)
TypeError: __new__() got an unexpected keyword argument 'datestr'
```
Drop columns we did not explicitly request. This may happen when a table is partitioned. Besides columns
requested, pyarrow will also return partition values. Having these unexpected fields will break some
downstream code.
Allows user to query the codec type: png/jpeg
This bug manifests in either: - `pyarrow.lib.ArrowIOError: The file only has <X> row groups, requested metadata for row group: <Y>` (where <X> and <Y> are integers) exception, or: - silently - some rows in a dataset might have been silently ignored. This issue occured when: - using `make_batch_reader` - number of row-groups in different parquet files (pieces) is different. The issue was in introduced in petastorm 0.7.7
Using values property instead of the deprecated as_matrix function
New image includes latest packages (inc. pandas 1.0) and python 3.7
…eader Creating a new PR for this as PR uber#445 is having conflicts with master, probably due to my following the wrong git advice on the internet... Adding two tests, for selecting just an invalid column and both invalid and valid columns, for parquet reader.
The warning was occuring when both legacy and the new rule produced the same overall set of fields, but maybe in different order. Although, both rules would behave the same from the user perspective, they would see a warning. The warning would be bogus because it wouldn't contain any field names.
* Get rid of RuntimeWarning when using process pool Restructure process_pool implementation code in a way that resolves `RuntimeWarning: 'petastorm.workers_pool.exec_in_new_process' found in sys.modules after import of package 'petastorm.workers_pool', but prior to execution of 'petastorm.workers_pool.exec_in_new_process'; this may result in unpredictable behaviou when using process pool` warning. In this diff we break main entry point from `exec_in_new_process.py` file into two a separate `exec_in_new_process_entrypoint.py`. Resolves uber#585
Allow more than two namenodes in hdfs configuration file. Resolves uber#724. Hadoop 3 configuration file allows more than two namenodes to be available. Petastorm code was assuming up to two namenodes in the configuration file. This PR allows for unbound number of namenodes to be present.
Support passing multiple url files to make_reader function. Resolves uber#728.
Use pickle 5 in IPC
Co-authored-by: Yevgeni Litvin <[email protected]>
This is the recommended way of checking for process existence by pid.
…r#762) Avoid creating many copies in _make_batch(). Reuse same rowgroup to generate all available batches.
Readthedocs build hit failure when using Python 3.6 with latest PyArrow.
Also use a shuffle seed to generate reproducible shuffle results across: single row shuffle, row groups shuffle in ventilator and sharding.
…ber#771) Add this change into v0.12.0 release.
* Support results_queue_size parameter in make_batch_reader api * Test results_queue_size is propagated properly to the workers pool * Update release notes * Reformat unit test
…dataframe path (uber#787) Signed-off-by: Weichen Xu [email protected] Make make_spark_converter supports creating converter from a saved dataframe path. In this case, we can skip the step of materializing spark dataframe that might be slow.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.