[BFCL] Multi-turn query missing info & incorrect label

[mingzhu0527]

Describe the issue
Hi, recently we reviewed some multi-turn data from BFCL-v3, and noticed a few cases where there's not enough information in the query to complete the task, or the groundtruth is incorrect.

ID datapoint

ID	Query	GroundTruth Answer	Comment
multi_turn_base_1	I am alex. Check if the current directory is under my name and list all the visible and hidden contents in the current directory now, please.	cd(folder='workspace')ls(a=True)	'workspace' not mentioned in query
multi_turn_base_2	Go into document folder and Could you draft up a create a document titled 'TeamNotes.txt' for keeping track of all the fresh ideas?	cd(folder='documents')touch(file_name='TeamNotes.txt')	Query use 'document' but ground
multi_turn_base_3	As part of my latest photography project, I need to gather files that have 'test' in their name from a specific folder in my current directory. Could you help me locate those?	find(path='.',name='test')	Query missing info
multi_turn_base_4	Could you kindly place the report in the /tmp directory or show me the list of files in tmp if already there?	pwd()ls()	GroundTruth error
multi_turn_base_11	Display all the available files located within the '/temp' directory using the terminal interface.	pwd()ls(a=True)	GroundTruth error
multi_turn_base_18	Copy the txt contents of the 'Quarter1_Reports' directory and place it in a new directory naming it 'Archived_Quarter1.	mkdir(dir_name='Archived_Quarter1')cp(source='report1.txt',destination='Archived_Quarter1')cp(source='report2.txt',destination='Archived_Quarter1')cp(source='History101.txt',destination='Archived_Quarter1')cp(source='History202.txt',destination='Archived_Quarter1')	GroundTruth Error

[Fanjia-Yan]

Hi,

Thank you for raising the issues above. We might have addressed some of the issues above in #740 but let me break it down individually:

1. multi_turn_base_1: This has been fixed in [BFCL Dataset Revamp 1/n] Multi-Turn (Part 1) #740, the new ground truth is ["ls(a=True)"]. The model needs to list the current path and after ensuring that current dir is Alex, the model will proceed to list everything in current directory.
2. multi_turn_base_2: Inside current working directory, there is a documents folder while the question mentions document only. We expect model to list the current directory and understand the possible options then make a choice.
3. multi_turn_base_3: This is our data labelling issue, we will modify the question with an explicit directory name
4. multi_turn_base_4: The model should make some exploration to file system and discover that the current directory is already the tmp folder
5. multi_turn_base_11: /temp should signify root directory, which is where the user initially configured to reside
6. multi_turn_base_18: I do see some ambiguity here. Our current working directory is Quarter1_Reports with 4 txt files within and the question asks to copy all text file to a new directory but not explicitly mentioning where it resides and ground truth assumes it's the current working directory. Is this the ground truth error you are talking about?

Thank you!

[mingzhu0527]

Thank you for your reply!
6. multi_turn_base_18: The groundtruth error I'm talking about is that, the query says "Copy the txt contents of the 'Quarter1_Reports' directory", but the groundtruth did not first navigate to Quarter1_Reports or check what is the current directory.
You mentioned that "Our current working directory is Quarter1_Reports", but this information is not included in either the query or the system prompt, and because of which the model could not figure out. I wonder if it is supposed to be this way.