[enhancement] Improve the performance of incremental loading

[Lloyd-Pottiger]

Now we only load incremental files one by one. However, files that have the same schema can be loaded in batch to improve the performance.

[breezewish]

What about updates to the same key?

[Lloyd-Pottiger]

What about updates to the same key?

Yes, it is the most directly way.

[breezewish]

I mean, will there be issues when multiple files contain updates to the same primary key and we concurrently load them?

[Lloyd-Pottiger]

I mean, will there be issues when multiple files contain updates to the same primary key and we concurrently load them?

Not a problem. There are rows with the same primary key in the same file. We have handled it.