Add python 3.10 in image#4

Open

selitvin wants to merge 612 commits intomasterfrom python3_10_in_image

+16,598-2,675

This pull request is big! We're only showing the most recent 250 commits

Commits on Dec 4, 2019

0.8.0rc0
Yevgeni Litvin
committed

Commits on Dec 5, 2019

0.8.0 release
Yevgeni Litvin
committed

Commits on Dec 6, 2019

Clarify that results_queue_size affects number of row-groups stored in the queue and not rows.

Yevgeni Litvin
authored and
selitvin
committed

Commits on Dec 14, 2019

update materialize_dataset documentation for filesystem factory arg

jakelarkn
authored and
selitvin
committed

Commits on Jan 10, 2020

Fix missing hostname attribute in ParseResult

Yevgeni Litvin
authored and
selitvin
committed

Commits on Jan 18, 2020

Fix failure due to partition fields values returned by pyarrow read.

Yevgeni Litvin
authored and
selitvin
committed

Commits on Jan 21, 2020

Adding image_codec accessor to CompressedImageCodec

Yevgeni Litvin
authored and
selitvin
committed

Commits on Jan 25, 2020

Add GCSFSWrapper

megaserg
authored and
selitvin
committed
Use GCSFSWrapper

megaserg
authored and
selitvin
committed
Add GCSFS tests

megaserg
authored and
selitvin
committed
Update docs for GCS

megaserg
authored and
selitvin
committed
Fix lint: Parameters differ from overridden 'walk' method

megaserg
authored and
selitvin
committed
0.8.1rc0
Yevgeni Litvin
committed

Commits on Jan 29, 2020

Minor fixup to materialize_dataset docstring

Yevgeni Litvin
authored and
selitvin
committed
Fixing MemeoryError when building docker image on docker.io

Yevgeni Litvin
authored and
selitvin
committed
Fix incorrect counting of number of row-groups per piece.

Yevgeni Litvin
authored and
selitvin
committed
0.8.1rc1
Yevgeni Litvin
committed

Commits on Jan 31, 2020

0.8.1
Yevgeni Litvin
committed
A monitor thread added to workers

ingolfured
authored and
selitvin
committed

Commits on Feb 1, 2020

Fixing pandas 1.0 compatibility issue (as_matrix)

Yevgeni Litvin
authored and
selitvin
committed
0.8.2rc0
Yevgeni Litvin
committed

Commits on Feb 2, 2020

Update to Python 3.7 in CI Dockerfile

Yevgeni Litvin
authored and
selitvin
committed
Update travis to use new CI image

Yevgeni Litvin
authored and
selitvin
committed

Commits on Feb 3, 2020

Resurrect codecov integration

Yevgeni Litvin
authored and
selitvin
committed
0.8.2
Yevgeni Litvin
committed

Commits on Feb 8, 2020

New PR for adding tests for requesting invalid columns from parquet reader

gregw18
authored and
selitvin
committed
Null change to trigger build

gregw18
authored and
selitvin
committed

Commits on Feb 25, 2020

assign transform result in _load_rows_with_predicate

jgblight
authored and
selitvin
committed
add test case

jgblight
authored and
selitvin
committed

Commits on Mar 2, 2020

Fix spurious legacy-regex rule warning

Yevgeni Litvin
authored and
selitvin
committed

Commits on Mar 12, 2020

[ML-9631] Convert Spark DataFrame into TensorFlow dataset via petastorm without lazy loading (#1 )

liangz1
authored and
selitvin
committed
[ML-9741] make_spark_converter() uses parquet format and always saves in new sub-directory (#2 )

liangz1
authored and
selitvin
committed
[ML-9741] Use atexit handler to delete cached folder

liangz1
authored and
selitvin
committed
[ML-9890] Add tests for all primitive types (uber#5 )

liangz1
authored and
selitvin
committed
[ML-9750] make_spark_converter() takes advanced params "compression" (uber#8 )

WeichenXu123
authored and
selitvin
committed
[ML-9742] make_spark_converter() can reuse previously saved dataframe (uber#6 )

WeichenXu123
authored and
selitvin
committed
[ML-9941] Support both hdfs:/ and file:/// (uber#9 )

liangz1
authored and
selitvin
committed
cleanup (uber#10 )

authored and
selitvin
committed
Cleanup-2 (uber#12 )

liangz1
authored and
selitvin
committed
Add Test pickling remotely (uber#14 )

WeichenXu123
authored and
selitvin
committed
Always respect private cache dir, and refactor some var names (uber#15 )

WeichenXu123
authored and
selitvin
committed
Log delete cache failure warning when exit

WeichenXu123
authored and
selitvin
committed
update doc

WeichenXu123
authored and
selitvin
committed
address comments

WeichenXu123
authored and
selitvin
committed
update tests

WeichenXu123
authored and
selitvin
committed
fix pylint

WeichenXu123
authored and
selitvin
committed
fix pylint

WeichenXu123
authored and
selitvin
committed
fix test

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
change to pytest

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
address comments

WeichenXu123
authored and
selitvin
committed
add test_change_cache_dir_raise_error

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
refine cache dir name format

WeichenXu123
authored and
selitvin
committed
update

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
add readme

WeichenXu123
authored and
selitvin
committed

Commits on Mar 18, 2020

init

WeichenXu123
authored and
selitvin
committed
update

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
add test

WeichenXu123
authored and
selitvin
committed
update test

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
address comments

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed

Commits on Mar 19, 2020

init

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
add test

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
add make_reader test

WeichenXu123
authored and
selitvin
committed
add test

WeichenXu123
authored and
selitvin
committed
fix test

WeichenXu123
authored and
selitvin
committed
fix test

WeichenXu123
authored and
selitvin
committed
fix test

WeichenXu123
authored and
selitvin
committed
remove make_reader support

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
fix

WeichenXu123
authored and
selitvin
committed
update

WeichenXu123
authored and
selitvin
committed

Commits on Mar 20, 2020

Start validating scalar types to be actual scalars (non-list) (uber#498 )
selitvin
authored

Commits on Mar 21, 2020

Simplify data conversion from Spark to TensorFlow: support tensorflow dataset advance arguments (uber#506 )
WeichenXu123
authored

Commits on Mar 26, 2020

Simplify data conversion from Spark to PyTorch DataLoader (uber#505 )
liangz1
authored

Commits on Mar 30, 2020

Commits on Apr 1, 2020

Simplify data conversion from Spark: support vector type and precision cast (uber#522 )
liangz1
authored

Commits on Apr 2, 2020

Commits on Apr 4, 2020

Add spark dataset converter mnist example scripts (uber#530 )
liangz1
authored

Commits on Apr 9, 2020

Commits on Apr 10, 2020

Fix: The "median size too small" warning is too frequent (uber#538 )
liangz1
authored

Commits on Apr 11, 2020

Use parquet.summary.metadata.level to control _summary file creation. (uber#529 )
selitvin
authored

Commits on Apr 12, 2020

Replace deprecated Schema.add_metadata with Schema.with_metadata call. (uber#528 )
selitvin
authored

Commits on Apr 13, 2020

Fix bug: respect dynamically changed parent cache dir conf (uber#537 )

liangz1
and
WeichenXu123
authored

Commits on Apr 14, 2020

Fix Fix: The 'median size too small' warning is too frequent uber#538 (uber#541 )
liangz1
authored

Commits on Apr 15, 2020

CI with Tensorflow 2.1 (uber#536 )
WeichenXu123
authored

Commits on Apr 18, 2020

New PyTorch BatchedDataLoader implementation using batched operations (uber#540 )
Felipe Petroski Such
authored

Commits on Apr 20, 2020

Add ngrams support to make_petastorm_dataset function. (uber#533 )
selitvin
authored

Commits on Apr 21, 2020

Commits on Apr 22, 2020

Fix DataLoader iter(dataloader) cannot be called more than once (uber#544 )
WeichenXu123
authored
0.9.0rc0
Yevgeni Litvin
committed

Commits on Apr 23, 2020

Mention PR 513: field order in 0.9.0 release notes.
Yevgeni Litvin
committed

Commits on Apr 24, 2020

0.9.0
Yevgeni Litvin
committed

Commits on Apr 30, 2020

Adding new SparkDatasetConverter documentation to the API section of autogeneraed docs. (uber#552 )
selitvin
authored

Commits on May 2, 2020

handle names like [path]/part/ (uber#535 )
xiaohanhuang
authored

Commits on May 12, 2020

Added last_row_consumed property to WeightedSamplingReader (uber#554 )
selitvin
authored
0.9.1rc0
Yevgeni Litvin
committed

Commits on May 13, 2020

0.9.1
Yevgeni Litvin
committed

Commits on Jun 2, 2020

Remove driver param for hdfs.connect when using pyarrow 0.17 and above (uber#556 )
tgaddair
authored

Commits on Jun 3, 2020

0.9.2rc0
Yevgeni Litvin
committed

Commits on Jun 4, 2020

0.9.2
Yevgeni Litvin
committed

Commits on Jun 25, 2020

Retire support for Python 2. (uber#562 )
selitvin
authored

Commits on Jul 5, 2020

Remove python 2.7 support from petastorm docker image (uber#565 )
selitvin
authored

Commits on Jul 28, 2020

Change definition of UnischemaField to be PY3.6 compatible. (uber#577 )
selitvin
authored
Release 0.9.4rc0 (uber#579 )
selitvin
authored
0.9.4
Yevgeni Litvin
committed

Commits on Aug 3, 2020

Add Python 3.6 to travis CI docker image (uber#578 )
selitvin
authored

Commits on Aug 25, 2020

Move gcsfs library to testing dependencies (uber#586 )
selitvin
authored
0.9.5rc0
Yevgeni Litvin
committed

Commits on Aug 27, 2020

0.9.5
Yevgeni Litvin
committed

Commits on Sep 10, 2020

Commits on Sep 14, 2020

Parameterize factory methods with s3 configs (uber#594 )
David McGuire
authored

Commits on Sep 15, 2020

Add a flag to factory methods to allow zmq copy buffers to be disabled (uber#596 )
David McGuire
authored

Commits on Sep 23, 2020

Commits on Sep 24, 2020

Fix docker file (uber#606 )
selitvin
authored
0.9.6rc0
Yevgeni Litvin
committed
Improve release notes formatting
Yevgeni Litvin
committed

Commits on Sep 26, 2020

0.9.6
Yevgeni Litvin
committed

Commits on Oct 1, 2020

Bugfix: S3FSWrapper is deprecated at s3fs 0.5.0 (uber#611 )
dmcguire81
authored

Commits on Oct 22, 2020

Raising an error if an empty list of columns is selected by a user (uber#615 )
selitvin
authored

Commits on Oct 26, 2020

Commits on Oct 27, 2020

0.9.7rc0
Yevgeni Litvin
committed

Commits on Oct 28, 2020

v0.9.7
Yevgeni Litvin
committed

Commits on Nov 9, 2020

Commits on Dec 2, 2020

bug with weighted_sampling_reader and pytorch (uber#627 )
gueguenster
authored

Commits on Dec 7, 2020

Fixed docstring in NGram class: timestamp_field argument illustration fixed (uber#628 )
ritwikbera
authored

Commits on Dec 10, 2020

Support on-premise s3-compatible storage. (uber#625 )
acmore
authored

Commits on Dec 12, 2020

In-memory cache (uber#555 )
abditag2
authored
0.9.8rc0
Yevgeni Litvin
committed

Commits on Dec 15, 2020

0.9.8
Yevgeni Litvin
committed

Commits on Jan 20, 2021

Commits on Jan 23, 2021

docs: Add comments for development.rst (uber#639 )
chongxiaoc
authored

Commits on Feb 3, 2021

Tests: fix the unit test of batched dataloader with in-memory cache (uber#643 )
chongxiaoc
authored

Commits on Mar 20, 2021

Fixing Dockerfile (uber#661 )
selitvin
authored

Commits on Apr 14, 2021

Reader: shuffle row groups before sharding. (uber#662 )
chongxiaoc
authored

Commits on Apr 26, 2021

Commits on Apr 27, 2021

Commits on Apr 28, 2021

Commits on May 3, 2021

Facelift to Dockerfile (uber#670 )
selitvin
authored

Commits on May 5, 2021

tf_dataset: add unit test to verify repeat() works after cache() (uber#674 )
chongxiaoc
authored

Commits on May 7, 2021

Commits on May 11, 2021

Commits on May 13, 2021

Commits on May 14, 2021

Bump up to v0.11.0
chongxiaoc
committed

Commits on May 27, 2021

Commits on May 28, 2021

0.11.1rc0

Yevgeni Litvin
authored and
Yevgeni Litvin
committed

Commits on Jun 2, 2021

0.11.1
Yevgeni Litvin
committed

Commits on Jun 28, 2021

Add python 3.9 to the dockerfile used in testing (uber#695 )
selitvin
authored
Fix build badge on the README.rst page (uber#688 )

selitvin
and
Yevgeni Litvin
authored

Commits on Jun 29, 2021

Python 3.9 isAlive -> is_alive compatibility fix (uber#696 )
selitvin
authored

Commits on Jul 25, 2021

Unbreak "C901 ... too complex" flake failures in CI (uber#701 )
selitvin
authored

Commits on Jul 28, 2021

Commits on Jul 30, 2021

v0.11.2rc0
Yevgeni Litvin
committed
v0.11.2rc1
Yevgeni Litvin
committed

Commits on Aug 3, 2021

0.11.2
Yevgeni Litvin
committed

Commits on Aug 19, 2021

Fix package versions in Dockerfile (uber#715 )
chongxiaoc
authored

Commits on Aug 25, 2021

Added fsspec support for _default_delete_dir_handler (uber#717 )
Manjur Ansari
authored

Commits on Sep 3, 2021

0.11.3rc0
Yevgeni Litvin
committed
release-notes fix: fails rst compilation.
Yevgeni Litvin
committed

Commits on Oct 22, 2021

fix typo "suffling" -> "shuffling" (uber#722 )
noxthot
authored

Commits on Oct 27, 2021

Use assertEqual instead of assertEquals for Python 3.11 compatibility. (uber#723 )
tirkarthi
authored

Commits on Jan 8, 2022

Get rid of RuntimeWarning when using process pool (uber#732 )
selitvin
authored

Commits on Jan 10, 2022

Commits on Feb 15, 2022

0.11.4rc0
Yevgeni Litvin
committed

Commits on Feb 19, 2022

0.11.4
Yevgeni Litvin
committed

Commits on Mar 24, 2022

Use highest available pickle protocol when serializing (uber#737 )
rbetz
authored

Commits on Apr 14, 2022

Import ABC from collections.abc for Python 3.10 compatibility (uber#746 )

tirkarthi
and
selitvin
authored

Commits on Apr 21, 2022

Commits on Jun 24, 2022

Replace process_iter by pid_exists (uber#757 )
MostafaFarahani
authored

Commits on Jul 26, 2022

PyTorch: improve memory-efficiency in batched non-shuffle buffer (uber#762 )
chongxiaoc
authored

Commits on Jul 27, 2022

Bump v0.11.5rc0
chongxiaoc
committed

Commits on Jul 28, 2022

Commits on Aug 23, 2022

Commits on Sep 14, 2022

Remove LocalDiskArrowTableCache and use latest pickle protocol for local caching (uber#777 )
selitvin
authored

Commits on Dec 13, 2022

Support results_queue_size parameter in make_batch_reader api (uber#783 )
s-udhaya
authored

Commits on Dec 16, 2022

0.12.1rc1
Yevgeni Litvin
committed
v0.12.1
Yevgeni Litvin
committed

Commits on Jan 30, 2023

Make make_spark_converter supports creating converter from a saved dataframe path (uber#787 )
WeichenXu123
authored

Commits on Feb 3, 2023

0.12.2rc0
Yevgeni Litvin
committed

Commits on Feb 20, 2023

Merge branch 'master' into python3_10_in_image
selitvin
authored