Stuck at the simulation step

[HLHsieh]

Hi there,

Thank you for developing this handy tool; I've been using it for a long time. Recently, I wanted to perform a more challenging test on my tool by adjusting the read length of the simulated reads as follows:

simulator.py genome -rg test.fasta -c /home/hsinlun/bin/NanoSim/pre-trained_models/human_giab_hg002_sub1M_kitv14_dorado/hg002_nanosim_sub1M -t 24 -med 10000 -sd 2 -n 27737 -o test_2x --seed 100 -b guppy

However, it took about five days and still failed. I noticed that it got stuck at the simulation step. I then tried the following command, which completed the entire process in just 10 minutes:

simulator.py genome -rg test.fasta -c /home/hsinlun/bin/NanoSim/pre-trained_models/human_giab_hg002_sub1M_kitv14_dorado/hg002_nanosim_sub1M -t 24 -n 27737 -o test_2x --seed 100 -b guppy

I am seeking your suggestions for optimizing my workflow. Additionally, is there any parameter or recommendation for controlling the mean read length instead of the median?

PS. My version is 3.1.0.

Best regards,
Hsin

[kmnip]

I think your simulation is stuck in an infinite loop here:
https://github.com/bcgsc/NanoSim/blob/master/src/simulator.py#L1237
It is trying to simulate certain read lengths, but it cannot find the "suitable" combination of lengths for the unaligned head/tail and the aligned segments.

Can you add the --perfect flag to your command? Basically, it simulates perfect reads. I just want see whether it gets stuck (i.e. running longer than 10 minutes), too.

[kmnip]

@HLHsieh , please checkout the changes in my branch:
https://github.com/bcgsc/NanoSim/tree/kmnip_patch1
It may potentially handle the infinite loop. This is not an ideal solution, but the user is informed of the issue at the very least.
@SaberHQ , what do you think?

[HLHsieh]

Hi @kmnip,

Thank you for your suggestions. First, I tried to add --perfect to my command, but it got error:

Process Process-2:
Traceback (most recent call last):
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hsinlun/bin/NanoSim/src/simulator.py", line 1240, in simulation_aligned_genome
    np.random.lognormal(np.log(median_l), sd_l, remaining_segments)
  File "mtrand.pyx", line 3015, in numpy.random.mtrand.RandomState.lognormal
  File "_common.pyx", line 598, in numpy.random._common.cont
ValueError: maximum supported dimension for an ndarray is 32, found 357424
Process Process-3:
Traceback (most recent call last):
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hsinlun/bin/NanoSim/src/simulator.py", line 1240, in simulation_aligned_genome
    np.random.lognormal(np.log(median_l), sd_l, remaining_segments)
  File "mtrand.pyx", line 3015, in numpy.random.mtrand.RandomState.lognormal
  File "_common.pyx", line 598, in numpy.random._common.cont
ValueError: maximum supported dimension for an ndarray is 32, found 357424
Process Process-8:
Traceback (most recent call last):
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hsinlun/bin/NanoSim/src/simulator.py", line 1240, in simulation_aligned_genome
    np.random.lognormal(np.log(median_l), sd_l, remaining_segments)
  File "mtrand.pyx", line 3015, in numpy.random.mtrand.RandomState.lognormal
  File "_common.pyx", line 598, in numpy.random._common.cont
ValueError: maximum supported dimension for an ndarray is 32, found 357426
Process Process-1:
Traceback (most recent call last):
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hsinlun/bin/NanoSim/src/simulator.py", line 1240, in simulation_aligned_genome
    np.random.lognormal(np.log(median_l), sd_l, remaining_segments)
  File "mtrand.pyx", line 3015, in numpy.random.mtrand.RandomState.lognormal
  File "_common.pyx", line 598, in numpy.random._common.cont
ValueError: maximum supported dimension for an ndarray is 32, found 357424
Process Process-7:
Traceback (most recent call last):
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hsinlun/bin/NanoSim/src/simulator.py", line 1240, in simulation_aligned_genome
    np.random.lognormal(np.log(median_l), sd_l, remaining_segments)
  File "mtrand.pyx", line 3015, in numpy.random.mtrand.RandomState.lognormal
  File "_common.pyx", line 598, in numpy.random._common.cont
ValueError: maximum supported dimension for an ndarray is 32, found 357424
Process Process-4:
Traceback (most recent call last):
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hsinlun/bin/NanoSim/src/simulator.py", line 1240, in simulation_aligned_genome
    np.random.lognormal(np.log(median_l), sd_l, remaining_segments)
  File "mtrand.pyx", line 3015, in numpy.random.mtrand.RandomState.lognormal
  File "_common.pyx", line 598, in numpy.random._common.cont
ValueError: maximum supported dimension for an ndarray is 32, found 357424
Process Process-5:
Traceback (most recent call last):
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hsinlun/bin/NanoSim/src/simulator.py", line 1240, in simulation_aligned_genome
    np.random.lognormal(np.log(median_l), sd_l, remaining_segments)
  File "mtrand.pyx", line 3015, in numpy.random.mtrand.RandomState.lognormal
  File "_common.pyx", line 598, in numpy.random._common.cont
ValueError: maximum supported dimension for an ndarray is 32, found 357424
Process Process-6:
Traceback (most recent call last):
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/nfs/turbo/umms-kinfai/hsinlun/miniconda3/envs/nanosim/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hsinlun/bin/NanoSim/src/simulator.py", line 1240, in simulation_aligned_genome
    np.random.lognormal(np.log(median_l), sd_l, remaining_segments)
  File "mtrand.pyx", line 3015, in numpy.random.mtrand.RandomState.lognormal
  File "_common.pyx", line 598, in numpy.random._common.cont
ValueError: maximum supported dimension for an ndarray is 32, found 357424

2024-07-12 14:56:04: Finished!

When the --perfect flag was removed, the error did not occur but stuck somewhere.

Secondly, I applied the latest script which was changed two days ago.

Here is the log:

Simulated 356652 `aligned` reads...s simulated >> 2260001
Simulated 356652 `aligned` reads...s simulated >> 2270001
Simulated 356652 `aligned` reads...s simulated >> 2300001
Simulated 356654 `aligned` reads...s simulated >> 2330001
Simulated 356652 `aligned` reads...

But, it was still stuck until the time limit.

Best,
Hsin-Lun

[SaberHQ]

Thank you @HLHsieh for your interest in using NanoSim and reporting this issue and thank you @kmnip for your solution.

I am currently away from work with limited access to internet. I will look into it deeper once I return in about 2 weeks. For now, what I can say is that the root of the problem (based on your log file provided) comes down to the following line of the code:

np.random.lognormal(np.log(median_l), sd_l, remaining_segments)

Where it tries to create an array with X (X here is the number of sequences you are trying to simulate, say 27,737 as in your first post) log-normally distributed samples based on a median of 10,000 and a standard deviation of 2 (based on your first post).

From what I see, the problem does not look like an infinite loop, it is rather related to the NumPy function and I suspect that it might be an environment and version issue. It could be that there is a memory limitation or a bug in the version of NumPy you are using. Ensure that you are using a recent version of NumPy and that your environment has sufficient memory available. (You can update NumPy using pip install NumPy --upgrade.) It could also be a memory problem since creating a huge NumPy array might break your code. Make sure your system isn’t running out of memory when this operation is performed.

To further look into it, if you are comfortable with Python coding, write a print function just before line 1239 to check the values for median_l, sd_l, and remaining_segments.

Now, there is obviously room for improvement on the code to avoid such problems and I might look into this once I have free time as I mentioned earlier. Since the log file you created does not contain Warning messages Ka Ming incorporated in the patch mode, I suspect again that it is not a infinite loop problem (At least by looking into the log file you just provided).

Thanks and I will keep you updated.

[HLHsieh]

Hi @SaberHQ ,

Thank you for taking the time to look into my issue. Additionally, I found something strange. I executed the previous simulator.py today around 12pm, and the following are the output files:

-rw-rw-r-- 1 hsinlun kinfai_root 929996253 Jul 12 12:03 test_2x_aligned_reads0.fasta
-rw-rw-r-- 1 hsinlun kinfai_root 929370674 Jul 12 12:03 test_2x_aligned_reads1.fasta
-rw-rw-r-- 1 hsinlun kinfai_root 932867870 Jul 12 12:03 test_2x_aligned_reads2.fasta
-rw-rw-r-- 1 hsinlun kinfai_root 931018020 Jul 12 12:03 test_2x_aligned_reads3.fasta
-rw-rw-r-- 1 hsinlun kinfai_root 930458715 Jul 12 12:03 test_2x_aligned_reads4.fasta
-rw-rw-r-- 1 hsinlun kinfai_root 541488149 Jul 12 12:00 test_2x_aligned_reads5.fasta
-rw-rw-r-- 1 hsinlun kinfai_root 931178341 Jul 12 12:03 test_2x_aligned_reads6.fasta
-rw-rw-r-- 1 hsinlun kinfai_root 932068207 Jul 12 12:03 test_2x_aligned_reads7.fasta
-rw-rw-r-- 1 hsinlun kinfai_root 587778206 Jul 12 12:03 test_2x_error_profile0
-rw-rw-r-- 1 hsinlun kinfai_root 587177640 Jul 12 12:03 test_2x_error_profile1
-rw-rw-r-- 1 hsinlun kinfai_root 588649987 Jul 12 12:03 test_2x_error_profile2
-rw-rw-r-- 1 hsinlun kinfai_root 587296727 Jul 12 12:03 test_2x_error_profile3
-rw-rw-r-- 1 hsinlun kinfai_root 588038704 Jul 12 12:03 test_2x_error_profile4
-rw-rw-r-- 1 hsinlun kinfai_root 341204833 Jul 12 12:00 test_2x_error_profile5
-rw-rw-r-- 1 hsinlun kinfai_root 587499390 Jul 12 12:03 test_2x_error_profile6
-rw-rw-r-- 1 hsinlun kinfai_root 589159695 Jul 12 12:03 test_2x_error_profile7

After executing for 6 hours, there were no changes to these files, and I found it can not generate the 650001th reads.

2024-07-12 11:56:58: Number of reads simulated >> 1^M2024-07-12 11:57:04: Number of reads simulated >> 10001^M2024-07-12 11:57:10: Number of reads simulated >> 20001^M2024-07-12 11:57:16: Number of reads simulated >> 30001^M2024-07-12 11:57:22: Number of reads simulated >> 40001^M2024-07-12 11:57:28: Number of reads simulated >> 50001^M2024-07-12 11:57:34: Number of reads simulated >> 60001^M2024-07-12 11:57:39: Number of reads simulated >> 70001^M2024-07-12 11:57:45: Number of reads simulated >> 80001^M2024-07-12 11:57:51: Number of reads simulated >> 90001^M2024-07-12 11:57:57: Number of reads simulated >> 100001^M2024-07-12 11:58:03: Number of reads simulated >> 110001^M2024-07-12 11:58:09: Number of reads simulated >> 120001^M2024-07-12 11:58:15: Number of reads simulated >> 130001^M2024-07-12 11:58:20: Number of reads simulated >> 140001^M2024-07-12 11:58:26: Number of reads simulated >> 150001^M2024-07-12 11:58:32: Number of reads simulated >> 160001^M2024-07-12 11:58:38: Number of reads simulated >> 170001^M2024-07-12 11:58:44: Number of reads simulated >> 180001^M2024-07-12 11:58:50: Number of reads simulated >> 190001^M^M2024-07-12 11:58:55: Number of reads simulated >> 200001^M2024-07-12 11:59:01: Number of reads simulated >> 210001^M2024-07-12 11:59:07: Number of reads simulated >> 220001^M2024-07-12 11:59:13: Number of reads simulated >> 230001^M2024-07-12 11:59:19: Number of reads simulated >> 240001^M2024-07-12 11:59:25: Number of reads simulated >> 250001^M2024-07-12 11:59:31: Number of reads simulated >> 260001^M2024-07-12 11:59:37: Number of reads simulated >> 270001^M2024-07-12 11:59:43: Number of reads simulated >> 280001^M2024-07-12 11:59:49: Number of reads simulated >> 290001^M2024-07-12 11:59:55: Number of reads simulated >> 300001^M2024-07-12 12:00:01: Number of reads simulated >> 310001^M2024-07-12 12:00:06: Number of reads simulated >> 320001^M2024-07-12 12:00:12: Number of reads simulated >> 330001^M2024-07-12 12:00:18: Number of reads simulated >> 340001^M2024-07-12 12:00:24: Number of reads simulated >> 350001^M2024-07-12 12:00:30: Number of reads simulated >> 360001^M2024-07-12 12:00:36: Number of reads simulated >> 370001^M2024-07-12 12:00:42: Number of reads simulated >> 380001^M2024-07-12 12:00:48: Number of reads simulated >> 390001^M2024-07-12 12:00:54: Number of reads simulated >> 400001^M2024-07-12 12:01:00: Number of reads simulated >> 410001^M2024-07-12 12:01:07: Number of reads simulated >> 420001^M2024-07-12 12:01:13: Number of reads simulated >> 430001^M2024-07-12 12:01:20: Number of reads simulated >> 440001^M2024-07-12 12:01:27: Number of reads simulated >> 450001^M2024-07-12 12:01:33: Number of reads simulated >> 460001^M2024-07-12 12:01:40: Number of reads simulated >> 470001^M2024-07-12 12:01:47: Number of reads simulated >> 480001^M2024-07-12 12:01:53: Number of reads simulated >> 490001^M2024-07-12 12:02:00: Number of reads simulated >> 500001^M2024-07-12 12:02:07: Number of reads simulated >> 510001^M2024-07-12 12:02:14: Number of reads simulated >> 520001^M2024-07-12 12:02:21: Number of reads simulated >> 530001^M2024-07-12 12:02:28: Number of reads simulated >> 540001^M2024-07-12 12:02:34: Number of reads simulated >> 550001^M2024-07-12 12:02:41: Number of reads simulated >> 560001^M2024-07-12 12:02:48: Number of reads simulated >> 570001^M2024-07-12 12:02:55: Number of reads simulated >> 580001^M2024-07-12 12:03:02: Number of reads simulated >> 590001^M2024-07-12 12:03:09: Number of reads simulated >> 600001^M2024-07-12 12:03:15: Number of reads simulated >> 610001^M2024-07-12 12:03:22: Number of reads simulated >> 620001^M2024-07-12 12:03:29: Number of reads simulated >> 630001^M2024-07-12 12:03:35: Number of reads simulated >> 640001^M2024-07-12 12:03:43: Number of reads simulated >> 650001

Many thanks,
Hsin