Skip to content

Commit

Permalink
update preparation
Browse files Browse the repository at this point in the history
  • Loading branch information
pufanyi committed Jun 19, 2024
1 parent 19f9bd6 commit 7f1159a
Showing 1 changed file with 43 additions and 1 deletion.
44 changes: 43 additions & 1 deletion tools/make_image_hf_dataset.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,55 @@
"source": [
"# Make Image dataset on Hugging Face Datasets\n",
"\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/EvolvingLMMs-Lab/lmms-eval/blob/main/tools/make_image_hf_dataset.ipynb)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/EvolvingLMMs-Lab/lmms-eval/blob/pufanyi/hf_dataset_docs/tools/make_image_hf_dataset.ipynb)\n",
"\n",
"This notebook will guide you to make correct format of Huggingface dataset, in proper parquet format and visualizable in Huggingface dataset hub.\n",
"\n",
"We will take the example of the dataset [`pufanyi/VQAv2_Example`](https://huggingface.co/datasets/lmms-lab/VQAv2) and convert it to the proper format."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Preparation\n",
"\n",
"We need to install `datasets` library to create the dataset and `Pillow` to handle images."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "bat"
}
},
"outputs": [],
"source": [
"!pip install datasets Pillow"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And we need to login into Hugging Face to upload the dataset. You should goto the [Hugging Face website](https://huggingface.co/settings/tokens) to get your API token."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "bat"
}
},
"outputs": [],
"source": [
"!huggingface-cli login --token hf_YOUR_HF_TOKEN # replace hf_YOUR_HF_TOKEN to your own Hugging Face token."
]
},
{
"cell_type": "markdown",
"metadata": {
Expand Down

0 comments on commit 7f1159a

Please sign in to comment.