You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the -s option allows one to specify the shard files to generate. However, this is not feasible when creating hundreds of shard files.
Instead have the following two options:
--num_shards <int>: how many shard files to generate, default is -1, so no sharding
--format <format>: the format for the file names (see below)
--width <int>: the number of digits to use to the index/total number of shards in the file name (left-padded with 0s), default is 5
The shard format string supports the following two placeholders:
{INDEX} - the 1-based index for the shard file
{TOTAL} - the total number of shard files that get generated (see --num_shards)
This allows the specification of formats:
--format "train.record-{INDEX}-{TOTAL}" (see here)
--format "train-{INDEX}-{TOTAL}.tfrecords"
The text was updated successfully, but these errors were encountered:
Currently, the
-s
option allows one to specify the shard files to generate. However, this is not feasible when creating hundreds of shard files.Instead have the following two options:
--num_shards <int>
: how many shard files to generate, default is -1, so no sharding--format <format>
: the format for the file names (see below)--width <int>
: the number of digits to use to the index/total number of shards in the file name (left-padded with 0s), default is 5The shard format string supports the following two placeholders:
{INDEX}
- the 1-based index for the shard file{TOTAL}
- the total number of shard files that get generated (see--num_shards
)This allows the specification of formats:
--format "train.record-{INDEX}-{TOTAL}"
(see here)--format "train-{INDEX}-{TOTAL}.tfrecords"
The text was updated successfully, but these errors were encountered: