Pure python solution

[abhishekkrthakur]

Here is a solution that I thought of. Runs in 5.5 seconds on my machine:

import glob
import time

start = time.time()

path = "../data/"
all_files = glob.glob(path + "/*.csv")

cols = ["f_0", "f_1", "f_2", "f_3", "f_4", "f_5", "f_6", "f_7", "f_8", "f_9", "target"]
with open("out.csv", "w") as out:
    out.write("%s\n" % ",".join(cols))
    for filename in all_files:
        with open(filename) as f:
            for idx, line in enumerate(f):
                if idx > 0:
                    out.write(line)
end = time.time()
print(end - start)

[sbarthwal]

Could you (@abhishekkrthakur ) please provide your machine configuration?
Because it is taking 124.09613084793091 s on my machine.
My machine configuration:
MacBook Pro (15-inch, 2019)
2.3 GHz Intel Core i9
16 GB 2400 MHz DDR4
Radeon Pro 560X 4 GB
Intel UHD Graphics 630 1536 MB

Thank you

[abhishekkrthakur]

core i7, 32gb ram but that shouldnt matter. 124s and 5s is huge!!! something else is wrong.