How to process the data in tai_safety_research

[CaffreyR]

Excellent guide in ipynb, but may I ask how to process the data with multi-feature like tai_safety_research? @uvafan

[uvafan]

Hey, it's been a while but I believe it should work out of the box for multi-feature datasets if you adjust the TASK; are you getting an error?

[CaffreyR]

Well my point is the starter kit notebook, I just want to learn how to generate multi-feature in to the tokenizer, like tai_safety_research

[uvafan]

It should just work, and you should be able to modify which fields are included if you wish via https://github.com/oughtinc/raft-baselines/blob/master/src/raft_baselines/data/prompt_construction_settings.jsonl