OError: [Errno 24] Too many open files. Too many reads?

[ViriatoII]

Hey,

This tool looks great, and so does the explanation. But I'm immediately running into a problem. I have a very big FASTQ file with dozens of thousands of PacBio reads. The tool fails after processing the first 1021 reads.

"Traceback (most recent call last):
File "/home/guerreiro/bin/FALCON-formatter", line 6, in
main()
File "/home/guerreiro/.local/lib/python2.7/site-packages/FALCON_formatter/init.py", line 29, in main
parseFastq(infile, OF, args.w, args.o)
File "/home/guerreiro/.local/lib/python2.7/site-packages/FALCON_formatter/init.py", line 62, in parseFastq
writeFA(fName, faHeader, seq, OF, wrap)
File "/home/guerreiro/.local/lib/python2.7/site-packages/FALCON_formatter/init.py", line 79, in writeFA
OF[fName] = open(fName,'w')
IOError: [Errno 24] Too many open files: 'ERR/ERR1713481.1021 "

[zyndagj]

Hello,

It looks like you ran into the open file limit on your system. File open and close operations in python hammer my distributed metadata servers, so I usually try keep a file handle open until I am done writing. Since you seem to have many different machine runs in your input file, I would suggest trying the following:

1. Have you tried using the raw file with the latest release of Falcon? It has been about a year and a half since I have interacted with Falcon, so this limitation may not exist any longer.
2. Is it possible to split your file instead of having a single monolithic input? The FOFN input can accept multiple files
   https://pb-falcon.readthedocs.io/en/latest/tutorial.html#create-fofn
   • Splitting your input will also allow you to [de]compress your input in parallel.
3. If neither of the first suggestions work, I can try to modify the code to limit the number of simultaneous file handles.

-Greg