Abnormal size of *.0.hic

[linshengnan09]

Hi,
I run 3d-dna with a 4.17G genome, but I got a abnormal size of *.0.hic, The log file is as follows：

...visualizing round 0 results:
:) -p flag was triggered. Running with GNU Parallel support parameter set to true.
:) -q flag was triggered, starting calculations for 1 threshold mapping quality
:) -i flag was triggered, building mapq without
:) -c flag was triggered, will remove temporary files after completion
...Remapping contact data from the original contig set to assembly
...Building track files
...Building the hic file
Not including fragment map
Start preprocess
Writing header
Writing body
.Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3181)
at java.util.ArrayList.grow(ArrayList.java:267)
at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:241)
at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:233)
at java.util.ArrayList.add(ArrayList.java:464)
at com.google.common.base.Splitter.splitToList(Splitter.java:422)
at juicebox.tools.utils.original.AsciiPairIterator.advance(AsciiPairIterator.java:93)
at juicebox.tools.utils.original.AsciiPairIterator.next(AsciiPairIterator.java:194)
at juicebox.tools.utils.original.Preprocessor.writeBody(Preprocessor.java:387)
at juicebox.tools.utils.original.Preprocessor.preprocess(Preprocessor.java:283)
at juicebox.tools.clt.old.PreProcessing.run(PreProcessing.java:108)
at juicebox.tools.HiCTools.main(HiCTools.java:86)
Sep 14, 2023 10:10:19 PM java.util.prefs.FileSystemPreferences syncWorld
WARNING: Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.
###############
Starting polish:
:) -p flag was triggered. Running with GNU Parallel support parameter set to true.
:) -q flag was triggered, performing polishing taking into account signal with minimum 1 mapping quality
...-j flag was triggered, will use Juicebox map Aa_merge_curated70.filter.resolved.hic
...-a flag was triggered, will use scaffold annotation file Aa_merge_curated70.filter.resolved_asm.scaffold_track.txt
...-b flag was triggered, will use superscaffold annotation file Aa_merge_curated70.filter.resolved_asm.superscaf_track.txt
...-s flag was triggered, will ignore all scaffolds shorter than 1000000 for polishing
:) -c flag was triggered, starting calculations with 5% saturation level
:) -w flag was triggered, performing cursory search for mismatches at 50000 resolution
:) -d flag was triggered, depletion score will be averaged across a region bounded by 1500000 superdiagonal
:) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
:) -n flag was triggered, performing mismatch region thinning at 500 resolution
:) -p flag was triggered. Running with GNU Parallel support parameter set to true.
:) -c flag was triggered, starting calculations with 5% saturation level
:) -w flag was triggered, performing cursory search for mismatches at 50000 resolution
:) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
:) -d flag was triggered, depletion score will be averaged across a region bounded by 1500000 superdiagonal
:) -n flag was triggered, performing mismatch region thinning at 500 resolution
...Dumping 50000 resolution matrix
HiC file version: 8
ERROR [2023-09-14 22:10:45,183] [DatasetReaderV2.java:229] [main] Error reading dataset
java.io.EOFException
at htsjdk.tribble.util.LittleEndianInputStream.readString(LittleEndianInputStream.java:119)
at juicebox.data.DatasetReaderV2.readFooter(DatasetReaderV2.java:435)
at juicebox.data.DatasetReaderV2.read(DatasetReaderV2.java:225)
at juicebox.data.HiCFileTools.extractDatasetForCLT(HiCFileTools.java:64)
at juicebox.tools.clt.old.Dump.readArguments(Dump.java:353)
at juicebox.tools.HiCTools.main(HiCTools.java:85)
Could not read hic file: null
Sep 14, 2023 10:10:46 PM java.util.prefs.FileSystemPreferences syncWorld
WARNING: Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.
:( Juicebox dump is empty! Perhaps something is wrong with the hic file or the requested resolution is too high. Exiting!
mv: cannot stat ‘depletion_score_wide.wig’: No such file or directory
mv: cannot stat ‘depletion_score_narrow.wig’: No such file or directory
mv: cannot stat ‘mismatch_wide.bed’: No such file or directory
mv: cannot stat ‘mismatch_narrow.bed’: No such file or directory
awk: /public/home/linshengnan/02biosoft/3d-dna-201008/edit/overlay-edits.awk:37: fatal: cannot open file `Aa_merge_curated70.filter.resolved.po
...-p flag was triggered. Running with GNU Parallel support parameter set to true.
...-r flag was triggered, output will be labeled as .polish.
...applying edits to cprops file
...applying edits to mnd file
:) -p flag was triggered. Running LIGer with GNU Parallel support parameter set to true.
:) -q flag was triggered, starting calculations with 1 threshold mapping quality
:) -s flag was triggered, starting calculations with 1000000 threshold starting contig/scaffold size
...Using cprops file: Aa_merge_curated70.filter.resolved.polish.cprops
...Using merged_nodups file: Aa_merge_curated70.filter.resolved.polish.mnd.txt
...Explicit scaffold set has been listed as input. Using set as a first iteration.
.............
#######
I have tried updating the Java version and modifying memory for the juicebox_tools.sh script, but it didn't work. Could you give some advise?
Thank you!

[luca-Wang]

Hello, I encountered the same error as you, did you solve it?

[linshengnan09]

I have solved it with increasing -Xmx memory of juicebox_tools.sh script.

[zhangna123-ux]

I have solved it with increasing -Xmx memory of juicebox_tools.sh script.

Hi, @linshengnan09
When I deal with a 3.8G genome with run-asm-pipeline.sh. There is an issue like '.Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded'. What would be a suitable value should I set for the parameter -Xmx in juicer_tool.sh to resolve this issue and will not cause my server to crash. Here is the memory status of my server (Linux):

Thanks in advance!

[linshengnan09]

Hi, @zhangna123-ux , I increased the default memory by 10 times, i.e. set it to: -Xms491520m. I think you can try setting it to 5 times. If that doesn’t work, increase it a little more.

[zhangna123-ux]

Hi, @zhangna123-ux , I increased the default memory by 10 times, i.e. set it to: -Xms491520m. I think you can try setting it to 5 times. If that doesn’t work, increase it a little more.

Hi, @linshengnan09
Thank you very much for your prompt reply. I will go and try your suggestion right away.