Add ability to specify batch size for bulk operations

[YodaDaCoda]

I'm performing some operations using the Bulk API, using a CSV of >6 million records.

Currently, I can pass that CSV to sfdx and the bulk operation will be split into batches automagically if the number of records in the CSV is greater than the default batch size of 10,000. For some operations (and due to things outside my control) the default batch size easily reaches various Salesforce limits (e.g. too many soql queries).

I'd like to be able to specify the batch size in order to work around these limitations.

My only alternatives at this time are to use the force cli (which has a -chunk option), or to split the CSV manually and create lots of jobs. I'd prefer not to have to use a separate tool, and splitting the CSV is a bit of a pain, so it'd be great if sfdx could handle this for me.

[aheber]

It is my understanding that the batch size used in the Bulk API (10,000) is only the file chunking size and internally it will still split to the 200 records per-transaction. I don't believe that per-transaction size can be adjusted on the bulk API. Unfortunately sending in file sizes smaller than 10,000 wouldn't help the transaction size where you're likely running into the limitation problems.

Batches are processed in chunks. The chunk size depends on the API version. 
In API version 20.0 and earlier, the chunk size is 100 records. 
In API version 21.0 and later, the chunk size is 200 records.

https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/asynch_api_concepts_limits.htm

If you need smaller than 200 chunks per-transaction then you need to use a different API.

[YodaDaCoda]

Having the ability to specify a batch size <200 would be beneficial in my case. I know this would result in more batches and would therefore approach the limits much more quickly, but it'd avoid me having to manually split my CSVs into smaller chunks.

[codeulike]

@YodaDaCoda I agree, several times I've been in a situation where the triggers on an object were so busy that the default transaction chunk size of 200 was hitting SOQL query limits. The way to get around that with bulk api v1 was to reduce batch size to something less than 200 (e.g. 50) to force the transaction chunks to be 50.

Apparently Bulk API 2.0 doesnt let you specify batch size. Seems strange to not allow that. I've also had situations where e.g. a batch size of 2000 hits the batch time limit, so you have to go down to 1000 etc.

I'm thinking of writing something that can chunk a CSV up and convert it to the JSON format used by force:data:tree:import

[codeulike]

Is this Closed because it is implemented?

[tsadigovAgmail]

The message saying closed as completed creates wrong impression. I would argue that closing similar tickets is a good idea at all. Need is still there and you end up not tracking that AT ALL. I mean shouldnt you have a backlog you want to check when you have resources?