Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issues training with custom dataset #102

Open
Tostino opened this issue Jan 9, 2024 · 0 comments
Open

Issues training with custom dataset #102

Tostino opened this issue Jan 9, 2024 · 0 comments

Comments

@Tostino
Copy link

Tostino commented Jan 9, 2024 •
edited
Loading

Hi there. I've read through some of the other threads about custom datasets, and am not quite sure what I am doing wrong.

I attempt to train using a custom jsonl file using the Relevancy Judged Dataset (Tevatron/msmarco-passage) format, but get errors.

My training call looks like:

accelerate launch examples/colbert/train_colbert.py \
  --output_dir jina_claim_contract_0.1 \
  --model_name_or_path jinaai/jina-embeddings-v2-base-en \
  --save_steps 20000 \
  --dataset_name Tevatron/msmarco-passage \
  --train_dir dataset/contract_claim_dataset.jsonl \
  --passage_field_separator "" \
  --fp16 \
  --per_device_train_batch_size 4 \
  --train_n_passages 4 \
  --learning_rate 5e-6 \
  --q_max_len 4095 \
  --p_max_len 4096 \
  --num_train_epochs 3 \
  --add_pooler \
  --projection_in_dim 768 \
  --projection_out_dim 768 \
  --logging_steps 50 \
  --overwrite_output_dir

The error message I am getting is:

Traceback (most recent call last):
  File "/home/user/IdeaProjects/tevatron/venv/lib/python3.10/site-packages/datasets/builder.py", line 1973, in _prepare_split_single
    for _, table in generator:
  File "/home/user/IdeaProjects/tevatron/venv/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 85, in _generate_tables
    parquet_file = pq.ParquetFile(f)
  File "/home/user/IdeaProjects/tevatron/venv/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 341, in __init__
    self.reader.open(
  File "pyarrow/_parquet.pyx", line 1250, in pyarrow._parquet.ParquetReader.open
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

A single line of the data in my jsonl looks like:

{"query_id": "8ece333c-e092-4359-b16d-2f803a02abae", "query": "<#meta#>\n- Task: match_claim_contract\n<#system#>\nYour purpose is to find the correct contract based on the json data in the user_context section.\n<#chat#>\n<#user#>\nPlease use the invoice data below to find the corresponding contract.\n<#user_context#>\n{\"client_name\":\"blah\",\"invoice_number\":\"641782\",\"invoice_date\":\"2016-03-05\",\"start_date\":\"2016-02-27\",\"end_date\":\"2016-03-04\",\"amount_requested\":630.31,\"company_number\":\"12257\",\"company_name\":\"SHAMROCK FS-Colorado\",\"country\":\"USA\",\"region\":\"CO\",\"city\":\"AURORA, CO 80011\",\"claims\":{\"claim_state\":\"COMPLETE\",\"claim_tag\":\"Country Buffet\",\"claim_alt_payee\":null,\"claim_lumpsums\":null,\"claim_adhoc_products\":null,\"claim_products\":[{\"product_number\":\"100009\",\"uom_type_cd\":\"CS\",\"distributor_company_number\":\"12257\",\"resolve_date\":\"2016-03-03\",\"claimant_contract_name\":\"Country Buffet\",\"distributor_invoice_number\":null},{\"product_number\":\"9619596\",\"uom_type_cd\":\"CS\",\"distributor_company_number\":\"12257\",\"resolve_date\":\"2016-03-03\",\"claimant_contract_name\":\"Country Buffet\",\"distributor_invoice_number\":null},{\"product_number\":\"9619596\",\"uom_type_cd\":\"CS\",\"distributor_company_number\":\"12257\",\"resolve_date\":\"2016-03-03\",\"claimant_contract_name\":\"Country Buffet\",\"distributor_invoice_number\":null}]}}\n<#bot#>\n", "positive_passages": [{"docid": "1000058", "title": "", "text": "{\"client_name\":\"blah\",\"contract_title\":\"Dynamic Foods\",\"contract_type\":\"STANDARD\",\"cost_basis_type\":\"FOB\",\"catch_all\":false,\"creator_email\":\"[email protected]\",\"contractee_type_description\":\"Operator\",\"contractee_name\":\"Ovation Brands\",\"contractee_number\":\"96\",\"contractee_country\":\"USA\",\"contractee_region\":\"Jay\",\"contractee_city\":\"happy\",\"contractee_address_1\":\"1329 test lane\",\"contract_template_description\":\"Default\",\"contracts\":[{\"start_date\":\"2016-04-21\",\"end_date\":\"2016-10-31\",\"contract_version_id\":\"1000058.R3.A0\",\"contract_name\":\"Dynamic Foods\",\"contract_state\":\"CANCELLED\",\"contract_state_last_modified\":\"2016-06-01\",\"contract_products\":[\"9618296\",\"9618496\",\"7112696\",\"9610996\",\"6395100\",\"70003234\",\"9618396\",\"6602352\",\"S8148805\",\"7000496\",\"5448525\",\"9691096\",\"7111196\",\"100009\",\"9619596\",\"6516096\",\"7110496\"],\"contract_product_groupings\":null,\"contract_lumpsums\":null},{\"start_date\":\"2016-04-21\",\"end_date\":\"2016-07-24\",\"contract_version_id\":\"1000058.R4.A0\",\"contract_name\":\"Dynamic Foods\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-07-08\",\"contract_products\":[\"70003234\",\"7110496\",\"7111196\",\"7000496\",\"5448525\",\"9691096\",\"6516096\",\"9618296\",\"100009\",\"9619596\",\"6602352\",\"9610996\",\"9618396\",\"6395100\",\"9618496\",\"S8148805\",\"7112696\"],\"contract_product_groupings\":null,\"contract_lumpsums\":null},{\"start_date\":\"2016-04-21\",\"end_date\":\"2016-07-24\",\"contract_version_id\":\"1000058.R4.A1\",\"contract_name\":\"Dynamic Foods\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-09-08\",\"contract_products\":[\"7111196\",\"6516096\",\"7112696\",\"9610996\",\"6395100\",\"5448525\",\"9619596\",\"70003234\",\"7000496\",\"S8148805\",\"9618496\",\"9618296\",\"6602352\",\"100009\",\"9618396\",\"9691096\",\"7110496\"],\"contract_product_groupings\":null,\"contract_lumpsums\":null},{\"start_date\":\"2015-04-08\",\"end_date\":\"2015-10-21\",\"contract_version_id\":\"1000058.R0.A0\",\"contract_name\":\"Ovation's\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2015-10-23\",\"contract_products\":[\"9618396\",\"7112696\",\"9618296\",\"6395100\",\"6602352\",\"7111196\",\"100009\",\"9618496\",\"S8148805\",\"9619596\",\"7110496\",\"5448525\",\"9610996\",\"7000496\",\"6516096\",\"9691096\"],\"contract_product_groupings\":null,\"contract_lumpsums\":null},{\"start_date\":\"2015-11-22\",\"end_date\":\"2016-04-20\",\"contract_version_id\":\"1000058.R2.A0\",\"contract_name\":\"Dynamic Foods\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-08-03\",\"contract_products\":[\"100009\",\"S8148805\",\"7110496\",\"7111196\",\"9618396\",\"5448525\",\"9619596\",\"6516096\",\"6395100\",\"7000496\",\"9610996\",\"7112696\",\"9618496\",\"9618296\",\"9691096\",\"6602352\"],\"contract_product_groupings\":null,\"contract_lumpsums\":null},{\"start_date\":\"2015-10-22\",\"end_date\":\"2015-11-21\",\"contract_version_id\":\"1000058.R1.A0\",\"contract_name\":\"Ovation's\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2016-01-04\",\"contract_products\":[\"9619596\",\"9691096\",\"S8148805\",\"7111196\",\"7110496\",\"7112696\",\"5448525\",\"7000496\",\"9618396\",\"6516096\",\"9618496\",\"6395100\",\"9610996\",\"6602352\",\"100009\",\"9618296\"],\"contract_product_groupings\":null,\"contract_lumpsums\":null},{\"start_date\":\"2015-11-22\",\"end_date\":\"2016-04-20\",\"contract_version_id\":\"1000058.R2.A1\",\"contract_name\":\"Dynamic Foods\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2016-08-03\",\"contract_products\":[\"7000496\",\"9618296\",\"9610996\",\"9691096\",\"6602352\",\"7112696\",\"9618496\",\"100009\",\"9618396\",\"5448525\",\"S8148805\",\"7110496\",\"7111196\",\"9619596\",\"6516096\",\"6395100\"],\"contract_product_groupings\":null,\"contract_lumpsums\":null},{\"start_date\":\"2016-04-21\",\"end_date\":\"2016-07-24\",\"contract_version_id\":\"1000058.R4.A2\",\"contract_name\":\"Dynamic Foods\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2016-09-08\",\"contract_products\":[\"9610996\",\"100009\",\"6516096\",\"7111196\",\"70003234\",\"5448525\",\"6395100\",\"9619596\",\"7000496\",\"7112696\",\"9618296\",\"9618496\",\"6602352\",\"7110496\",\"9618396\",\"S8148805\",\"9691096\"],\"contract_product_groupings\":null,\"contract_lumpsums\":null}]}"}], "negative_passages": [{"docid": "1000018", "title": "", "text": "{\"client_name\":\"blah\",\"contract_title\":\"SAMPLES\",\"contract_type\":\"STANDARD\",\"cost_basis_type\":\"FOB\",\"catch_all\":false,\"creator_email\":\"[email protected]\",\"contractee_type_description\":\"Distributor\",\"contractee_name\":\"All FoodService Distributors\",\"contractee_number\":\"alldist\",\"contractee_country\":null,\"contractee_region\":null,\"contractee_city\":null,\"contractee_address_1\":null,\"contract_template_description\":\"Default\",\"contracts\":[{\"start_date\":\"2015-01-01\",\"end_date\":\"2017-12-31\",\"contract_version_id\":\"1000018.R0.A2\",\"contract_name\":\"SAMPLES\",\"contract_state\":\"TERMINATED\",\"contract_state_last_modified\":\"2015-10-12\",\"contract_products\":null,\"contract_product_groupings\":[\"allfs\"],\"contract_lumpsums\":null},{\"start_date\":\"2015-01-01\",\"end_date\":\"2017-12-31\",\"contract_version_id\":\"1000018.R0.A0\",\"contract_name\":\"KEEP:  Samples Contract\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2015-10-12\",\"contract_products\":null,\"contract_product_groupings\":[\"allfs\"],\"contract_lumpsums\":null},{\"start_date\":\"2015-01-01\",\"end_date\":\"2017-12-31\",\"contract_version_id\":\"1000018.R0.A1\",\"contract_name\":\"Samples Contract\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2015-10-12\",\"contract_products\":null,\"contract_product_groupings\":[\"allfs\"],\"contract_lumpsums\":null}]}"}, {"docid": "1000025", "title": "", "text": "{\"client_name\":\"blah\",\"contract_title\":\"IMA 2% Service Fee and Marketing\",\"contract_type\":\"STANDARD\",\"cost_basis_type\":\"FOB\",\"catch_all\":false,\"creator_email\":\"[email protected]\",\"contractee_type_description\":\"Distributor\",\"contractee_name\":\"IMA\",\"contractee_number\":\"004\",\"contractee_country\":null,\"contractee_region\":null,\"contractee_city\":null,\"contractee_address_1\":null,\"contract_template_description\":\"Default\",\"contracts\":[{\"start_date\":\"2015-01-01\",\"end_date\":\"2015-12-31\",\"contract_version_id\":\"1000025.R0.A0\",\"contract_name\":\"IMA 2% Service Fee and Marketing\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2015-12-31\",\"contract_products\":null,\"contract_product_groupings\":[\"fghcbk\",\"fgc5-10\",\"fgpcfs\",\"fgmbg\",\"fg32b-fs\"],\"contract_lumpsums\":[{\"lumpsum_cd\":\"m\",\"contract_lumpsum_description\":\"Point of Sale and Training Video\"},{\"lumpsum_cd\":\"m\",\"contract_lumpsum_description\":\"Marketing Program 2015\"}]},{\"start_date\":\"2017-01-01\",\"end_date\":\"2017-12-31\",\"contract_version_id\":\"1000025.R2.A0\",\"contract_name\":\"IMA 2% Service Fee and Marketing\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2017-01-09\",\"contract_products\":[\"14234\",\"6399925\",\"14225\",\"PFC6528\",\"14235\",\"14230\",\"100399\",\"14228\",\"S8P00800\",\"S8003801\",\"14229\",\"14237\",\"PFC6548\",\"PFC6508\",\"PFC0796\",\"PRG5096C\",\"14232\",\"14231\",\"14223\",\"7046896\",\"6092633\",\"9616596\",\"14226\",\"S8P00801\",\"14224\",\"14236\",\"14233\",\"14227\",\"14222\",\"PFC6538\",\"C9621196\"],\"contract_product_groupings\":[\"FSA\",\"fgmbk\",\"fgpbk\",\"fgmbg\",\"fg32b-fs\",\"fgc5-10\",\"FGHCFDSVC\",\"fgpcfs\"],\"contract_lumpsums\":[{\"lumpsum_cd\":\"m\",\"contract_lumpsum_description\":\"Marketing Program 2016\"},{\"lumpsum_cd\":\"m\",\"contract_lumpsum_description\":\"Point of Sale and Training Video\"}]},{\"start_date\":\"2016-01-01\",\"end_date\":\"2016-12-31\",\"contract_version_id\":\"1000025.R1.A1\",\"contract_name\":\"IMA 2% Service Fee and Marketing\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2016-12-31\",\"contract_products\":[\"14223\",\"14231\",\"9616596\",\"14226\",\"7046896\",\"14224\",\"S8P00801\",\"PFC6548\",\"S8P00800\",\"14229\",\"S8003801\",\"14228\",\"14237\",\"14235\",\"PFC6528\",\"PFC0796\",\"PFC6508\",\"14225\",\"6399925\",\"14234\",\"14230\",\"100399\",\"14222\",\"14227\",\"14233\",\"14236\",\"C9621196\",\"PFC6538\",\"PRG5096C\",\"6092633\",\"14232\"],\"contract_product_groupings\":[\"fgmbk\",\"FGHCFDSVC\",\"fg32b-fs\",\"fgmbg\",\"FSA\",\"fgpbk\",\"fgpcfs\",\"fgc5-10\"],\"contract_lumpsums\":[{\"lumpsum_cd\":\"m\",\"contract_lumpsum_description\":\"Marketing Program 2016\"},{\"lumpsum_cd\":\"m\",\"contract_lumpsum_description\":\"Point of Sale and Training Video\"}]},{\"start_date\":\"2018-01-01\",\"end_date\":\"2023-12-31\",\"contract_version_id\":\"1000025.R3.A2\",\"contract_name\":\"IMA 2% Service Fee and Marketing\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2024-01-01\",\"contract_products\":[\"14234\",\"9619995\",\"14231\",\"14230\",\"14224\",\"PFC6538\",\"14232\",\"C9621196\",\"9616596\",\"14225\",\"7046896\",\"9615133\",\"PFC0796\",\"14226\",\"S8P00800\",\"14229\",\"6092633\",\"14236\",\"PRG5096C\",\"22818\",\"PFC6508\",\"14228\",\"S8003801\",\"6399925\",\"9616796\",\"14227\",\"PFC6528\",\"14223\",\"PFC6548\",\"100399\",\"14233\",\"14235\",\"14237\",\"14222\",\"S8P00801\"],\"contract_product_groupings\":[\"FGHCFDSVC\",\"fg32b-fs\",\"fgpbk\",\"fgmbg\",\"fgc5-10\",\"fgmbk\",\"FSA\",\"fgpcfs\"],\"contract_lumpsums\":null},{\"start_date\":\"2016-01-01\",\"end_date\":\"2016-12-31\",\"contract_version_id\":\"1000025.R1.A0\",\"contract_name\":\"IMA 2% Service Fee and Marketing\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-05-02\",\"contract_products\":null,\"contract_product_groupings\":[\"fghcbk\",\"fgpcfs\",\"fgc5-10\",\"fg32b-fs\",\"fgmbg\"],\"contract_lumpsums\":[{\"lumpsum_cd\":\"m\",\"contract_lumpsum_description\":\"Marketing Program 2016\"},{\"lumpsum_cd\":\"m\",\"contract_lumpsum_description\":\"Point of Sale and Training Video\"}]},{\"start_date\":\"2018-01-01\",\"end_date\":\"2023-12-31\",\"contract_version_id\":\"1000025.R3.A0\",\"contract_name\":\"IMA 2% Service Fee and Marketing\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2023-06-07\",\"contract_products\":[\"14226\",\"6399925\",\"14227\",\"14222\",\"14235\",\"14230\",\"14233\",\"S8003801\",\"6092633\",\"14237\",\"S8P00800\",\"S8P00801\",\"C9621196\",\"14229\",\"7046896\",\"PFC6508\",\"PFC6528\",\"14223\",\"14225\",\"14236\",\"PRG5096C\",\"PFC0796\",\"PFC6548\",\"14228\",\"100399\",\"14224\",\"14231\",\"14232\",\"9616596\",\"PFC6538\",\"14234\"],\"contract_product_groupings\":[\"fgpcfs\",\"fgmbg\",\"fg32b-fs\",\"FGHCFDSVC\",\"FSA\",\"fgmbk\",\"fgc5-10\",\"fgpbk\"],\"contract_lumpsums\":null},{\"start_date\":\"2018-01-01\",\"end_date\":\"2023-12-31\",\"contract_version_id\":\"1000025.R3.A1\",\"contract_name\":\"IMA 2% Service Fee and Marketing\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2023-06-07\",\"contract_products\":[\"14228\",\"14224\",\"14237\",\"6399925\",\"PFC6528\",\"14226\",\"14232\",\"S8P00801\",\"14233\",\"14236\",\"6092633\",\"PFC6508\",\"14230\",\"14231\",\"S8P00800\",\"7046896\",\"PFC0796\",\"14225\",\"14227\",\"PRG5096C\",\"S8003801\",\"PFC6548\",\"14234\",\"14223\",\"9616596\",\"PFC6538\",\"100399\",\"14235\",\"C9621196\",\"14222\",\"14229\"],\"contract_product_groupings\":[\"fgmbk\",\"fgpbk\",\"FGHCFDSVC\",\"fgmbg\",\"fgpcfs\",\"FSA\",\"fg32b-fs\",\"fgc5-10\"],\"contract_lumpsums\":null}]}"}, {"docid": "1000026", "title": "", "text": "{\"client_name\":\"blah\",\"contract_title\":\"FSA Corporate EMI Azar brand\",\"contract_type\":\"STANDARD\",\"cost_basis_type\":\"FOB\",\"catch_all\":false,\"creator_email\":\"[email protected]\",\"contractee_type_description\":\"Distributor\",\"contractee_name\":\"FSA CORP\",\"contractee_number\":\"fs\",\"contractee_country\":null,\"contractee_region\":null,\"contractee_city\":null,\"contractee_address_1\":null,\"contract_template_description\":\"Default\",\"contracts\":[{\"start_date\":\"2016-02-01\",\"end_date\":\"2016-12-31\",\"contract_version_id\":\"1000026.R2.A4\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-08-04\",\"contract_products\":[\"14234\",\"644\",\"14237\",\"14235\",\"27115\",\"14233\",\"14226\",\"14230\",\"17558\",\"6092633\",\"27112\",\"14225\",\"14223\",\"14232\",\"14236\",\"27110\",\"640\",\"PFC6508\",\"27114\",\"27108\",\"PFC6538\",\"642\",\"27106\",\"14224\",\"27104\",\"27113\",\"14229\",\"PFC6528\",\"14231\",\"14222\",\"14228\"],\"contract_product_groupings\":[\"fgc5-10\",\"fgb5alob\",\"FGMGLBLKFS\",\"FGHCFDSVC\",\"FBBKFSFS\",\"fg32b-fs\",\"fgcpc\",\"fglfgfs\",\"fgmbk\",\"fgpcfs\",\"fgcpfsn\"],\"contract_lumpsums\":null},{\"start_date\":\"2018-03-01\",\"end_date\":\"2018-05-31\",\"contract_version_id\":\"1000026.R4.A1\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2018-04-27\",\"contract_products\":[\"14229\",\"14237\",\"27110\",\"PFC6508\",\"27106\",\"14230\",\"642\",\"27115\",\"14228\",\"14235\",\"644\",\"00644\",\"14225\",\"14222\",\"27114\",\"PFC6528\",\"6092633\",\"14226\",\"14223\",\"14236\",\"PFC6538\",\"17558\",\"00640\",\"14234\",\"00642\",\"27104\",\"27108\",\"14233\",\"640\",\"14231\",\"27113\",\"14224\",\"14232\",\"27112\"],\"contract_product_groupings\":[\"FGHCFDSVC\",\"fgc5-10\",\"FBBKFSFS\",\"ALLPVTLBL\",\"fglfgfs\",\"fg32b-fs\",\"fgcpfsn\",\"fgb5alob\",\"fgmbk\",\"fgpcfs\",\"fgpbr\",\"fgpbk\",\"fgcpc\",\"FGMGLBLKFS\"],\"contract_lumpsums\":null},{\"start_date\":\"2015-09-01\",\"end_date\":\"2016-01-31\",\"contract_version_id\":\"1000026.R1.A5\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2016-08-10\",\"contract_products\":[\"14229\",\"17558\",\"14225\",\"14224\",\"14230\",\"14231\",\"642\",\"14228\",\"27104\",\"14236\",\"27114\",\"27115\",\"PFC6528\",\"14233\",\"14234\",\"14223\",\"27112\",\"640\",\"27110\",\"27106\",\"27108\",\"14226\",\"14237\",\"14232\",\"644\",\"PFC6508\",\"14222\",\"27113\",\"14235\",\"PFC6538\",\"6092633\"],\"contract_product_groupings\":[\"fgb5alob\",\"FGHCFDSVC\",\"fgmbk\",\"fgpcfs\",\"fgcpc\",\"fglfgfs\",\"fgcpfsn\",\"FGMGLBLKFS\",\"FBBKFSFS\",\"fg32b-fs\",\"fgc5-10\"],\"contract_lumpsums\":null},{\"start_date\":\"2017-01-01\",\"end_date\":\"2018-02-28\",\"contract_version_id\":\"1000026.R3.A1\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2018-02-22\",\"contract_products\":[\"27114\",\"14234\",\"27112\",\"14230\",\"27113\",\"27110\",\"PFC6528\",\"27115\",\"14233\",\"27106\",\"14229\",\"6092633\",\"PFC6508\",\"14225\",\"14237\",\"14232\",\"14235\",\"PFC6538\",\"642\",\"644\",\"14236\",\"14223\",\"640\",\"27104\",\"14222\",\"14231\",\"00640\",\"14228\",\"14226\",\"27108\",\"14224\",\"17558\",\"00644\",\"00642\"],\"contract_product_groupings\":[\"fgb5alob\",\"fgpbk\",\"fg32b-fs\",\"fgc5-10\",\"fgpbr\",\"FBBKFSFS\",\"fgcpfsn\",\"fgpcfs\",\"FGHCFDSVC\",\"fgmbk\",\"ALLPVTLBL\",\"fglfgfs\",\"fgcpc\",\"FGMGLBLKFS\"],\"contract_lumpsums\":null},{\"start_date\":\"2018-07-31\",\"end_date\":\"2021-06-30\",\"contract_version_id\":\"1000026.R6.A1\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2021-07-01\",\"contract_products\":[\"6092633\",\"14226\",\"14228\",\"14223\",\"27115\",\"14232\",\"PFC6508\",\"14234\",\"14229\",\"640\",\"14237\",\"00644\",\"PFC6528\",\"27106\",\"644\",\"14236\",\"27108\",\"27112\",\"14235\",\"14222\",\"PFC6538\",\"00640\",\"14233\",\"14225\",\"00642\",\"14231\",\"642\",\"27104\",\"17558\",\"14230\",\"14224\",\"27110\",\"6484800\",\"6485350\",\"27113\",\"27114\"],\"contract_product_groupings\":[\"FBBKFSFS\",\"fgc5-10\",\"fgpbr\",\"fglfgfs\",\"fgpcfs\",\"FGHCFDSVC\",\"fgb5alob\",\"fgmbk\",\"fg32b-fs\",\"ALLPVTLBL\",\"FGMGLBLKFS\",\"fgcpc\",\"fgpbk\",\"fgcpfsn\"],\"contract_lumpsums\":null},{\"start_date\":\"2016-02-01\",\"end_date\":\"2016-12-31\",\"contract_version_id\":\"1000026.R2.A7\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2016-12-31\",\"contract_products\":[\"27114\",\"14226\",\"14232\",\"6092633\",\"27113\",\"27106\",\"14222\",\"17558\",\"27112\",\"14231\",\"PFC6538\",\"14225\",\"644\",\"642\",\"14230\",\"14236\",\"640\",\"14229\",\"27104\",\"14235\",\"14223\",\"PFC6528\",\"27110\",\"27115\",\"14234\",\"14237\",\"PFC6508\",\"14228\",\"14233\",\"14224\",\"27108\"],\"contract_product_groupings\":[\"fgpbk\",\"fg32b-fs\",\"fgpcfs\",\"fgb5alob\",\"ALLPVTLBL\",\"fgmbk\",\"FGHCFDSVC\",\"FGMGLBLKFS\",\"fgpbr\",\"fgcpfsn\",\"FBBKFSFS\",\"fgc5-10\",\"fgcpc\",\"fglfgfs\"],\"contract_lumpsums\":null},{\"start_date\":\"2016-02-01\",\"end_date\":\"2016-12-31\",\"contract_version_id\":\"1000026.R2.A1\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-05-18\",\"contract_products\":[\"14233\",\"14224\",\"14223\",\"17558\",\"27110\",\"27114\",\"PFC6528\",\"27104\",\"640\",\"PFC6538\",\"14231\",\"644\",\"27113\",\"14225\",\"PFC6508\",\"14230\",\"14237\",\"642\",\"27106\",\"27115\",\"14226\",\"14228\",\"27112\",\"14229\",\"14235\",\"14222\",\"14232\",\"27108\",\"14236\",\"14234\"],\"contract_product_groupings\":[\"fgb5alob\",\"fgcpc\",\"fg32b-fs\",\"FGHCFDSVC\",\"fgc5-10\",\"fgcpfsn\",\"fglfgfs\",\"fgpcfs\",\"FBBKFSFS\"],\"contract_lumpsums\":null},{\"start_date\":\"2018-07-31\",\"end_date\":\"2021-06-30\",\"contract_version_id\":\"1000026.R6.A0\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2019-05-07\",\"contract_products\":[\"640\",\"14222\",\"27104\",\"14228\",\"27110\",\"14230\",\"14224\",\"14225\",\"27113\",\"642\",\"644\",\"6092633\",\"14223\",\"14235\",\"27108\",\"14234\",\"14232\",\"27115\",\"PFC6528\",\"14231\",\"00640\",\"27114\",\"14237\",\"14236\",\"17558\",\"00644\",\"14229\",\"27106\",\"27112\",\"PFC6508\",\"14226\",\"14233\",\"00642\",\"PFC6538\"],\"contract_product_groupings\":[\"fg32b-fs\",\"fglfgfs\",\"fgpcfs\",\"fgcpc\",\"ALLPVTLBL\",\"fgb5alob\",\"fgcpfsn\",\"fgc5-10\",\"fgmbk\",\"fgpbr\",\"FBBKFSFS\",\"fgpbk\",\"FGMGLBLKFS\",\"FGHCFDSVC\"],\"contract_lumpsums\":null},{\"start_date\":\"2016-02-01\",\"end_date\":\"2016-12-31\",\"contract_version_id\":\"1000026.R2.A8\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"CANCELLED\",\"contract_state_last_modified\":\"2016-10-17\",\"contract_products\":[\"14224\",\"27108\",\"27115\",\"14236\",\"PFC6508\",\"14234\",\"14231\",\"27106\",\"14235\",\"27114\",\"6092633\",\"14232\",\"27104\",\"27110\",\"14237\",\"14228\",\"14223\",\"PFC6538\",\"14222\",\"27112\",\"14226\",\"14225\",\"642\",\"14230\",\"14233\",\"17558\",\"14229\",\"27113\",\"644\",\"PFC6528\",\"640\"],\"contract_product_groupings\":[\"fgpcfs\",\"fgpbk\",\"fgmbk\",\"fgpbr\",\"fg32b-fs\",\"fgcpfsn\",\"fgcpc\",\"FGHCFDSVC\",\"FGMGLBLKFS\",\"fgb5alob\",\"FBBKFSFS\",\"fgcps\",\"fgc5-10\",\"ALLPVTLBL\",\"fglfgfs\"],\"contract_lumpsums\":null},{\"start_date\":\"2016-02-01\",\"end_date\":\"2016-12-31\",\"contract_version_id\":\"1000026.R2.A6\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-08-10\",\"contract_products\":[\"14225\",\"14229\",\"27115\",\"27114\",\"14226\",\"27110\",\"644\",\"14237\",\"14228\",\"17558\",\"27104\",\"27108\",\"14223\",\"642\",\"14234\",\"PFC6508\",\"640\",\"14224\",\"27106\",\"14222\",\"14230\",\"PFC6538\",\"PFC6528\",\"14231\",\"14233\",\"6092633\",\"14236\",\"27112\",\"14235\",\"27113\",\"14232\"],\"contract_product_groupings\":[\"FBBKFSFS\",\"FGMGLBLKFS\",\"fglfgfs\",\"fgpcfs\",\"fgc5-10\",\"fgpbk\",\"ALLPVTLBL\",\"fg32b-fs\",\"fgb5alob\",\"fgcpc\",\"FGHCFDSVC\",\"fgcpfsn\",\"fgpbr\",\"fgmbk\"],\"contract_lumpsums\":null},{\"start_date\":\"2015-09-01\",\"end_date\":\"2016-01-31\",\"contract_version_id\":\"1000026.R1.A0\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-05-16\",\"contract_products\":[\"14234\",\"14229\",\"14233\",\"14226\",\"14231\",\"14223\",\"14228\",\"14225\",\"644\",\"27115\",\"PFC6528\",\"14235\",\"27113\",\"27110\",\"640\",\"PFC6508\",\"17558\",\"27106\",\"27112\",\"14224\",\"27108\",\"642\",\"14230\",\"14236\",\"PFC6538\",\"27114\",\"14222\",\"27104\",\"14232\",\"14237\"],\"contract_product_groupings\":[\"FGHCFDSVC\",\"fg32b-fs\",\"fgcpfsn\",\"fgcpc\",\"fgpcfs\",\"fglfgfs\",\"fgb5alob\",\"fgc5-10\",\"FBBKFSFS\"],\"contract_lumpsums\":null},{\"start_date\":\"2015-01-01\",\"end_date\":\"2015-08-31\",\"contract_version_id\":\"1000026.R0.A2\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-08-10\",\"contract_products\":[\"14228\",\"14232\",\"14234\",\"14233\",\"14223\",\"14226\",\"27106\",\"27104\",\"14236\",\"27108\",\"14224\",\"17558\",\"14231\",\"14229\",\"27112\",\"14237\",\"14230\",\"27113\",\"14235\",\"14222\",\"27110\",\"14225\",\"PFC6508\",\"644\",\"27114\",\"642\",\"PFC6538\",\"640\",\"27115\",\"PFC6528\"],\"contract_product_groupings\":[\"fgb5alob\",\"fgcpc\",\"fgpcfs\",\"fgcpfsn\",\"FGHCFDSVC\",\"fglfgfs\",\"fgc5-10\",\"fg32b-fs\",\"fgmbk\",\"FBBKFSFS\"],\"contract_lumpsums\":null},{\"start_date\":\"2015-09-01\",\"end_date\":\"2016-01-31\",\"contract_version_id\":\"1000026.R1.A1\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-05-18\",\"contract_products\":[\"14223\",\"27110\",\"642\",\"14233\",\"640\",\"27108\",\"14236\",\"14226\",\"14228\",\"27104\",\"14231\",\"644\",\"PFC6538\",\"27106\",\"PFC6528\",\"14224\",\"27115\",\"14234\",\"27112\",\"14237\",\"14235\",\"14225\",\"27114\",\"14222\",\"14229\",\"PFC6508\",\"14230\",\"17558\",\"27113\",\"14232\"],\"contract_product_groupings\":[\"fg32b-fs\",\"fgpcfs\",\"fglfgfs\",\"fgcpc\",\"fgb5alob\",\"FBBKFSFS\",\"fgc5-10\",\"FGHCFDSVC\",\"fgmbk\",\"fgcpfsn\"],\"contract_lumpsums\":null},{\"start_date\":\"2018-06-01\",\"end_date\":\"2018-07-30\",\"contract_version_id\":\"1000026.R5.A0\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2018-08-01\",\"contract_products\":[\"PFC6538\",\"27112\",\"14228\",\"27110\",\"27108\",\"14229\",\"00640\",\"14232\",\"00644\",\"14235\",\"642\",\"14223\",\"27113\",\"00642\",\"14236\",\"14226\",\"6092633\",\"27114\",\"14224\",\"27115\",\"17558\",\"PFC6508\",\"14230\",\"644\",\"14222\",\"14237\",\"14231\",\"27106\",\"640\",\"27104\",\"14234\",\"PFC6528\",\"14233\",\"14225\"],\"contract_product_groupings\":[\"fglfgfs\",\"fg32b-fs\",\"ALLPVTLBL\",\"fgmbk\",\"fgpcfs\",\"fgcpc\",\"fgc5-10\",\"fgpbk\",\"fgb5alob\",\"fgpbr\",\"FBBKFSFS\",\"fgcpfsn\",\"FGHCFDSVC\",\"FGMGLBLKFS\"],\"contract_lumpsums\":null},{\"start_date\":\"2015-09-01\",\"end_date\":\"2016-01-31\",\"contract_version_id\":\"1000026.R1.A2\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-06-15\",\"contract_products\":[\"27110\",\"14226\",\"14231\",\"14224\",\"14230\",\"14229\",\"14228\",\"17558\",\"14233\",\"27104\",\"14232\",\"640\",\"644\",\"14223\",\"642\",\"PFC6528\",\"27114\",\"14234\",\"14237\",\"27115\",\"14225\",\"14222\",\"PFC6538\",\"27112\",\"27106\",\"PFC6508\",\"14235\",\"27108\",\"14236\",\"27113\"],\"contract_product_groupings\":[\"fgb5alob\",\"fglfgfs\",\"FGHCFDSVC\",\"fgc5-10\",\"fg32b-fs\",\"fgpcfs\",\"fgcpc\",\"fgmbk\",\"fgcpfsn\",\"FBBKFSFS\"],\"contract_lumpsums\":null},{\"start_date\":\"2017-01-01\",\"end_date\":\"2018-02-28\",\"contract_version_id\":\"1000026.R3.A2\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2018-02-22\",\"contract_products\":[\"14234\",\"14225\",\"14233\",\"27110\",\"27106\",\"640\",\"PFC6528\",\"14231\",\"14232\",\"17558\",\"14230\",\"27112\",\"27108\",\"PFC6508\",\"27114\",\"642\",\"6092633\",\"27104\",\"644\",\"00644\",\"14224\",\"27115\",\"14235\",\"27113\",\"00642\",\"14228\",\"14236\",\"00640\",\"14222\",\"14226\",\"14229\",\"14237\",\"PFC6538\",\"14223\"],\"contract_product_groupings\":[\"ALLPVTLBL\",\"fgpbr\",\"fgpbk\",\"fgb5alob\",\"fgc5-10\",\"fgcpfsn\",\"FGHCFDSVC\",\"fglfgfs\",\"FGMGLBLKFS\",\"fgmbk\",\"fg32b-fs\",\"fgpcfs\",\"fgcpc\",\"FBBKFSFS\"],\"contract_lumpsums\":null},{\"start_date\":\"2015-09-01\",\"end_date\":\"2016-01-31\",\"contract_version_id\":\"1000026.R1.A3\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-08-04\",\"contract_products\":[\"14235\",\"14223\",\"14236\",\"27112\",\"14226\",\"PFC6528\",\"27115\",\"27110\",\"27106\",\"27108\",\"14228\",\"14237\",\"14225\",\"14233\",\"14231\",\"640\",\"642\",\"27104\",\"27114\",\"PFC6538\",\"17558\",\"27113\",\"14234\",\"PFC6508\",\"14222\",\"644\",\"14224\",\"14229\",\"14230\",\"14232\"],\"contract_product_groupings\":[\"fgcpfsn\",\"fgcpc\",\"fgc5-10\",\"FGHCFDSVC\",\"fglfgfs\",\"fg32b-fs\",\"FBBKFSFS\",\"fgmbk\",\"fgb5alob\",\"fgpcfs\"],\"contract_lumpsums\":null},{\"start_date\":\"2016-02-01\",\"end_date\":\"2016-12-31\",\"contract_version_id\":\"1000026.R2.A0\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-04-25\",\"contract_products\":[\"14229\",\"14228\",\"14233\",\"14226\",\"PFC6528\",\"27106\",\"14234\",\"14235\",\"27113\",\"27110\",\"640\",\"17558\",\"14225\",\"644\",\"PFC6508\",\"14231\",\"14223\",\"14236\",\"PFC6538\",\"14232\",\"27104\",\"14237\",\"27112\",\"14224\",\"27108\",\"642\",\"14230\",\"14222\",\"27115\",\"27114\"],\"contract_product_groupings\":[\"fgpcfs\",\"fgcpc\",\"fgb5alob\",\"fgbfs\",\"fglfgfs\",\"fg32b-fs\",\"fgcpfsn\",\"fgc5-10\",\"fghcbk\"],\"contract_lumpsums\":null},{\"start_date\":\"2018-03-01\",\"end_date\":\"2018-05-31\",\"contract_version_id\":\"1000026.R4.A0\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2018-04-27\",\"contract_products\":[\"14233\",\"27112\",\"00642\",\"644\",\"14229\",\"14228\",\"6092633\",\"27113\",\"14231\",\"27114\",\"14226\",\"642\",\"27108\",\"14234\",\"17558\",\"14222\",\"14237\",\"14225\",\"14235\",\"27106\",\"27110\",\"00644\",\"PFC6538\",\"14236\",\"00640\",\"14224\",\"27104\",\"14232\",\"14223\",\"14230\",\"640\",\"27115\",\"PFC6528\",\"PFC6508\"],\"contract_product_groupings\":[\"fgpbk\",\"fgb5alob\",\"FGMGLBLKFS\",\"fgc5-10\",\"fg32b-fs\",\"fgpcfs\",\"FBBKFSFS\",\"fglfgfs\",\"fgmbk\",\"FGHCFDSVC\",\"fgcpfsn\",\"fgpbr\",\"ALLPVTLBL\",\"fgcpc\"],\"contract_lumpsums\":null},{\"start_date\":\"2016-02-01\",\"end_date\":\"2016-12-31\",\"contract_version_id\":\"1000026.R2.A3\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-08-04\",\"contract_products\":[\"14223\",\"27106\",\"PFC6508\",\"27104\",\"14232\",\"644\",\"14230\",\"14237\",\"27115\",\"14235\",\"14233\",\"14226\",\"17558\",\"14224\",\"14234\",\"14228\",\"642\",\"14222\",\"PFC6528\",\"27114\",\"27110\",\"640\",\"27108\",\"PFC6538\",\"27112\",\"14231\",\"14225\",\"27113\",\"14236\",\"14229\"],\"contract_product_groupings\":[\"fgcpc\",\"fgcpfsn\",\"FBBKFSFS\",\"fglfgfs\",\"fgpcfs\",\"fgb5alob\",\"fg32b-fs\",\"fgmbk\",\"FGHCFDSVC\",\"fgc5-10\"],\"contract_lumpsums\":null},{\"start_date\":\"2015-01-01\",\"end_date\":\"2015-08-31\",\"contract_version_id\":\"1000026.R0.A0\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-05-16\",\"contract_products\":null,\"contract_product_groupings\":[\"fgpcfs\",\"fgb5alob\",\"fghcbk\",\"fgcpc\",\"fgbfs\",\"fg32b-fs\",\"fgcpfsn\",\"fglfgfs\",\"fgc5-10\"],\"contract_lumpsums\":null},{\"start_date\":\"2016-02-01\",\"end_date\":\"2016-12-31\",\"contract_version_id\":\"1000026.R2.A2\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-07-11\",\"contract_products\":[\"14222\",\"644\",\"14229\",\"14234\",\"27113\",\"14233\",\"27108\",\"27112\",\"14226\",\"14231\",\"14223\",\"PFC6508\",\"14230\",\"27114\",\"PFC6528\",\"642\",\"14236\",\"27106\",\"14225\",\"14237\",\"14235\",\"27110\",\"27104\",\"14224\",\"14228\",\"14232\",\"17558\",\"640\",\"27115\",\"PFC6538\"],\"contract_product_groupings\":[\"fgc5-10\",\"fglfgfs\",\"fgcpfsn\",\"fgpcfs\",\"fgcpc\",\"fgb5alob\",\"FGHCFDSVC\",\"fg32b-fs\",\"FBBKFSFS\"],\"contract_lumpsums\":null},{\"start_date\":\"2015-01-01\",\"end_date\":\"2015-08-31\",\"contract_version_id\":\"1000026.R0.A3\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2016-08-10\",\"contract_products\":[\"14236\",\"27108\",\"PFC6508\",\"14224\",\"14231\",\"14229\",\"17558\",\"27112\",\"14237\",\"14230\",\"644\",\"27113\",\"27104\",\"14235\",\"27114\",\"PFC6538\",\"14222\",\"27110\",\"642\",\"14225\",\"14232\",\"14234\",\"640\",\"14228\",\"27115\",\"14233\",\"14223\",\"14226\",\"PFC6528\",\"27106\",\"6092633\"],\"contract_product_groupings\":[\"fgc5-10\",\"fglfgfs\",\"fgb5alob\",\"fgcpfsn\",\"fgcpc\",\"FBBKFSFS\",\"fg32b-fs\",\"FGHCFDSVC\",\"fgmbk\",\"fgpcfs\"],\"contract_lumpsums\":null},{\"start_date\":\"2018-06-01\",\"end_date\":\"2018-07-30\",\"contract_version_id\":\"1000026.R5.A1\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"EXPIRED\",\"contract_state_last_modified\":\"2018-08-01\",\"contract_products\":[\"640\",\"27106\",\"14224\",\"27108\",\"14235\",\"14222\",\"14233\",\"14234\",\"642\",\"PFC6528\",\"14236\",\"27115\",\"17558\",\"27110\",\"PFC6508\",\"00640\",\"27112\",\"14231\",\"27113\",\"27104\",\"14225\",\"14237\",\"6092633\",\"644\",\"PFC6538\",\"00644\",\"14228\",\"14230\",\"14223\",\"14226\",\"14232\",\"27114\",\"00642\",\"14229\"],\"contract_product_groupings\":[\"FGHCFDSVC\",\"fg32b-fs\",\"fgpcfs\",\"fgb5alob\",\"FGMGLBLKFS\",\"fgmbk\",\"ALLPVTLBL\",\"fgc5-10\",\"FBBKFSFS\",\"fgcpfsn\",\"fgcpc\",\"fgpbr\",\"fgpbk\",\"fglfgfs\"],\"contract_lumpsums\":null},{\"start_date\":\"2015-09-01\",\"end_date\":\"2016-01-31\",\"contract_version_id\":\"1000026.R1.A4\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-08-10\",\"contract_products\":[\"27104\",\"27114\",\"PFC6538\",\"17558\",\"27113\",\"14234\",\"PFC6508\",\"14222\",\"644\",\"14232\",\"14224\",\"14229\",\"14230\",\"14235\",\"14223\",\"14236\",\"27112\",\"14226\",\"PFC6528\",\"27115\",\"27110\",\"27106\",\"27108\",\"14228\",\"14237\",\"14225\",\"14233\",\"14231\",\"640\",\"642\"],\"contract_product_groupings\":[\"fgcpc\",\"fgmbk\",\"FGMGLBLKFS\",\"fglfgfs\",\"fgc5-10\",\"FBBKFSFS\",\"fgcpfsn\",\"FGHCFDSVC\",\"fgb5alob\",\"fgpcfs\",\"fg32b-fs\"],\"contract_lumpsums\":null},{\"start_date\":\"2016-02-01\",\"end_date\":\"2016-12-31\",\"contract_version_id\":\"1000026.R2.A5\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-08-05\",\"contract_products\":[\"14234\",\"27112\",\"14225\",\"644\",\"14223\",\"14232\",\"14236\",\"14237\",\"27110\",\"640\",\"14235\",\"27115\",\"14233\",\"PFC6508\",\"27114\",\"27108\",\"PFC6538\",\"14226\",\"14230\",\"642\",\"27106\",\"14224\",\"27104\",\"27113\",\"14229\",\"PFC6528\",\"17558\",\"14231\",\"14222\",\"6092633\",\"14228\"],\"contract_product_groupings\":[\"fgcpfsn\",\"FGMGLBLKFS\",\"fgc5-10\",\"fgb5alob\",\"FBBKFSFS\",\"fg32b-fs\",\"fgmbk\",\"fgcpc\",\"FGHCFDSVC\",\"fglfgfs\",\"fgpcfs\",\"fgpbr\"],\"contract_lumpsums\":null},{\"start_date\":\"2015-01-01\",\"end_date\":\"2015-08-31\",\"contract_version_id\":\"1000026.R0.A1\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2016-05-16\",\"contract_products\":[\"17558\",\"14237\",\"14235\",\"27104\",\"14226\",\"14234\",\"27108\",\"14236\",\"PFC6528\",\"644\",\"27114\",\"14230\",\"642\",\"PFC6508\",\"640\",\"14224\",\"14228\",\"14231\",\"14225\",\"14232\",\"27115\",\"27112\",\"27110\",\"14233\",\"14223\",\"PFC6538\",\"14222\",\"14229\",\"27113\",\"27106\"],\"contract_product_groupings\":[\"fgcpc\",\"fg32b-fs\",\"fglfgfs\",\"fghcbk\",\"fgc5-10\",\"fgb5alob\",\"fgbfs\",\"fgpcfs\",\"fgcpfsn\"],\"contract_lumpsums\":null},{\"start_date\":\"2017-01-01\",\"end_date\":\"2018-02-28\",\"contract_version_id\":\"1000026.R3.A0\",\"contract_name\":\"FSA Corporate EMI Azar brand\",\"contract_state\":\"AMENDED\",\"contract_state_last_modified\":\"2017-02-02\",\"contract_products\":[\"27113\",\"27106\",\"14237\",\"14230\",\"27108\",\"14224\",\"14228\",\"644\",\"PFC6528\",\"27114\",\"PFC6538\",\"14236\",\"PFC6508\",\"17558\",\"14233\",\"14229\",\"14232\",\"6092633\",\"642\",\"27115\",\"14222\",\"640\",\"14235\",\"27110\",\"14225\",\"14226\",\"27104\",\"27112\",\"14234\",\"14231\",\"14223\"],\"contract_product_groupings\":[\"ALLPVTLBL\",\"fgcpc\",\"fgcpfsn\",\"FGMGLBLKFS\",\"fgpbr\",\"FBBKFSFS\",\"fgb5alob\",\"fgc5-10\",\"fglfgfs\",\"fg32b-fs\",\"FGHCFDSVC\",\"fgpcfs\",\"fgmbk\",\"fgpbk\"],\"contract_lumpsums\":null}]}"}]}

If I go and convert the jsonl file to a parquet file and try and load that instead, the train/split generation of the dataset starts...but then I get a different error:

Traceback (most recent call last):
  File "/home/user/IdeaProjects/tevatron/examples/colbert/train_colbert.py", line 102, in <module>
    main()
  File "/home/user/IdeaProjects/tevatron/examples/colbert/train_colbert.py", line 79, in main
    train_dataset = HFTrainDataset(tokenizer=tokenizer, data_args=data_args,
  File "/home/user/IdeaProjects/tevatron/src/tevatron/datasets/dataset.py", line 24, in __init__
    self.dataset = load_dataset(data_args.dataset_name,
  File "/home/user/IdeaProjects/tevatron/venv/lib/python3.10/site-packages/datasets/load.py", line 2549, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/user/IdeaProjects/tevatron/venv/lib/python3.10/site-packages/datasets/builder.py", line 1005, in download_and_prepare
    self._download_and_prepare(
  File "/home/user/IdeaProjects/tevatron/venv/lib/python3.10/site-packages/datasets/builder.py", line 1118, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/home/user/IdeaProjects/tevatron/venv/lib/python3.10/site-packages/datasets/utils/info_utils.py", line 92, in verify_splits
    raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))
datasets.utils.info_utils.ExpectedMoreSplits: {'dl20', 'dl19', 'dev'}

Would really appreciate some help with what is going on here.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Tostino