Skip to content

Commit

Permalink
Update mimicit_format.md
Browse files Browse the repository at this point in the history
  • Loading branch information
Luodian authored Nov 8, 2023
1 parent 1c056f1 commit 1cef803
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion docs/mimicit_format.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Breaking Down the MIMIC-IT Format

❗❗❗We changed previous `images.json` to `images.parquet`. They are all containing multiple `key:base64` pairs but the later one would consume far less CPU memory and faster during loading with `pandas.Dataframe`. It enables us to train with larger datasets more conviently.

We mainly use one integrate dataset format and we refer it to MIMIC-IT format since.

The mimic-it format contains the following data yaml file. Within this data yaml file, you could assign the path of the instruction json file and the image parquet file, and also the number of samples you want to use. The number of samples within each group will be uniformly sampled, and the `number_samples / total_numbers`` will decide sampling ratio of each dataset.
Expand Down Expand Up @@ -86,4 +88,4 @@ parquet_file_path = os.path.join(
parquet_root_path, os.path.basename(json_file_path).split(".")[0].replace("_image", "") + ".parquet"
)
df.to_parquet(parquet_file_path, engine="pyarrow")
```
```

0 comments on commit 1cef803

Please sign in to comment.