Script to adapt Yahoo Q&A datasets

[Shadowner]

Hello I've made a python script to adapt the Yahoo Q&A datasets to the expected Open Assistant Datasets format.

Because the datasets doesn't include any toxicity parameter, i've added the possibility to generate a toxicity parameter using the detoxify package has it is Apache 2 License and has pretty good results.

Exemple of gems found in the datasets that would need a small toxicity parameter :

  {
    "INSTRUCTION": "Why won't drivers use thier turn signals?",
    "RESPONSE": "Simple math: one hand on cell phone, other hand on coffee mug, right foot on gas pedal, left knee for steering.  What do you want them to use to activate the turn signal?",
    "SOURCE": "yahoo QA",
    "METADATA": {
      "category": "Cars & Transportation"
    }
  }

I could have add every parameter of Detoxify to the metadata, but i thought it was too much.

[github-actions]

❌ pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md

[olliestanley]

Looks good - could you also add an arg push-to-hf to enable pushing the Dataset to HuggingFace in Parquet format? If you then run that code and push it as a Dataset to your HuggingFace account, you can include the dataset path here as part of the PR.