Making database for Culex Tarsalis ends without after processing queries with no warning or error #83
Comments
Afei99357
changed the title
|
Can you run the test file? |
|
I tried your docker installation and run on the test file, everything looks okay until the same issue as above. suddenly crashed and stop here: I am watching it do something like: processing database part 1 (size ~0.25 GB): 100.00/100.00% * I remember it finished at least part 220, and somehow must be crashed somehow processing other part of the databases! here is the terminal screen, somehow it does not show the processing database part when it crashed. I have no name!@bc8b48e0e31e:/Users/ericliao/Desktop/WNV_project_files/sift_docker/SIFT4G_Create_Genomic_DB$ perl make-SIFT-db-all.pl -config ./test_files/homo_sapiens-test.txt
Here is the configuration for the homo_sapiens_small test file: GENETIC_CODE_TABLE=1 PARENT_DIR=/Users/ericliao/Desktop/WNV_project_files/sift_docker/SIFT4G_Create_Genomic_DB/test_files/homo_sapiens_small #Running SIFT4G, this path works for the Dockerfile #PROTEIN_DB needs to be uncompressed Sub-directories, don't need to changeGENE_DOWNLOAD_DEST=gene-annotation-src Doesn't need to changeFASTA_LOG=fasta.log do you know what is going on here? why everytime it crash here? |
|
What's your docker command? Did you mount all the volumes? Also, I tested this on linux, I've never tested it on Windows. |
|
Thank you for your quick responds! I modified the comments above a little bit if you haven't seen that yet. Also even the first one, i was originally running it on docker because my labtop is Macbook with apple m1 chip. So today i tried your new way to do it with your docker command. here is the command docker build --platform linux/amd64 -t sift4g_db . docker run -it --user |
|
my initial thought for this problem is in the make-SIFT-db-all.pl file, line 122 to line 124. some how the $sift4g_command does not execute right. and it just exit without any error and warning? my $sift4g_command = $meta_hash{"SIFT4G_PATH"} . " -d " . $meta_hash{"PROTEIN_DB"} . " -q " . $all_prot_fasta . " --subst " . $meta_hash{"PARENT_DIR"} . "/" . $meta_hash{"SUBST_DIR"} . " --out " . $meta_hash{"PARENT_DIR"} . "/" . $meta_hash{"SIFT_SCORE_DIR"} . " --sub-results " ; print $sift4g_command . "\n"; # if ($? != 0) { Right now, I am running the $sift4g_command in myy terminal just use sift4g based on the results i get from previous steps, this error must somewhere from sift4g if it stop the same as before. |
|
Go into your container and then check that all the paths are there
Do |
|
i think the mounting so far is right, because i am able to run the sift4g on my docker container right now. it is still processing but this is the results after run your recommendations. I have no name!@bc8b48e0e31e:/$ /sift4g/bin/sift4g and I am curretnly run sift4g on my docker to see if the error reproducible as last night. I have no name!@bc8b48e0e31e:/$ ./sift4g/bin/sift4g -d /Users/ericliao/Desktop/WNV_project_files/sift_docker/bigdrive/SIFT_databases/uniref90.fasta -q /Users/ericliao/Desktop/WNV_project_files/sift_docker/SIFT4G_Create_Genomic_DB/test_files/homo_sapiens_small/all_prot.fasta --subst /Users/ericliao/Desktop/WNV_project_files/sift_docker/SIFT4G_Create_Genomic_DB/test_files/homo_sapiens_small/subst --out /Users/ericliao/Desktop/WNV_project_files/sift_docker/SIFT4G_Create_Genomic_DB/test_files/homo_sapiens_small/singleRecords_with_scores --sub-results
** Searching database for candidate sequences **
|
|
Hello! I got the error after i run the sift4g separately, any idea where this error from? memory issue? I have no name!@bc8b48e0e31e:/$ ./sift4g/bin/sift4g -d /Users/ericliao/Desktop/WNV_project_files/sift_docker/bigdrive/SIFT_databases/uniref90.fasta -q /Users/ericliao/Desktop/WNV_project_files/sift_docker/SIFT4G_Create_Genomic_DB/test_files/homo_sapiens_small/all_prot.fasta --subst /Users/ericliao/Desktop/WNV_project_files/sift_docker/SIFT4G_Create_Genomic_DB/test_files/homo_sapiens_small/subst --out /Users/ericliao/Desktop/WNV_project_files/sift_docker/SIFT4G_Create_Genomic_DB/test_files/homo_sapiens_small/singleRecords_with_scores --sub-results
** Searching database for candidate sequences ** |
|
Looks like you ran out of memory |
|
Thank you so much. I realize it is the memory issue, then I raise the memory limit for docker to 14 gb out of 16gb total. But it ends with crashing at different step. still no idea what causes this crash.. here is the terminal output: I have no name!@1945c502387a:/sift4g_run_test/SIFT4G_Create_Genomic_DB$ perl make-SIFT-db-all.pl -config /sift4g_run_test/SIFT4G_Create_Genomic_DB/test_files/homo_sapiens-test.txt
** Searching database for candidate sequences **
** Aligning queries with candidate sequences ** |
|
There is no error message here. |
|
Yes. That is the problem as the very beginning. It just stopped without any error... is it potentially another memory? how much memory does it need to run the test files?? I can only give 14 gb to the docker.. I am running my own files on a more powerful machine now, hopefully it is the memory issue? It will be nice if they give some errors if the processing suddenly stopped? |
Have you solved your problem? ** Searching database for candidate sequences ** Could you give me some suggestions and help? best |
|
sorry. I just saw it. For me, it is the memory issue. This app requires a lot of memory due to the way they process the data. So try to run on a machine with bigger memory. |
|
it's ok |
Mine is 64GB and and that is enough for me! maybe your data is larger than mine! don't remember exactly the size of my data since more than a year ago. I remember I went inside their code. and run the step one by one, until the memory error shows up and i realize it is the memory issue for me! |
|
OK I get it |
Hello. thank you for help me with the previous issue. After address the issue, the processing continue to this then stop for no reasons? and I check the database in the <Organ_version> folder, there is no files there at all. Could you please help me with this issue?
here is the running code##
####### my config file########
GENE_DOWNLOAD_SITE=ftp://ftp.ensemblgenomes.org/pub/bacteria/release-34/gtf//bacteria_11_collection/candidatus_carsonella_ruddii_pv/Candidatus_carsonella_ruddii_pv.ASM1036v1.34.gtf.gz
PEP_FILE=ftp://ftp.ensemblgenomes.org/pub/bacteria/release-34/fasta//bacteria_11_collection/candidatus_carsonella_ruddii_pv/pep/Candidatus_carsonella_ruddii_pv.ASM1036v1.34.pep.all.fa.gz
CHR_DOWNLOAD_SITE=ftp://ftp.ensemblgenomes.org/pub/bacteria/release-34/fasta//bacteria_11_collection/candidatus_carsonella_ruddii_pv/dna/
GENETIC_CODE_TABLE=1
GENETIC_CODE_TABLENAME=Standard
MITO_GENETIC_CODE_TABLE=1
MITO_GENETIC_CODE_TABLENAME=Standard
PARENT_DIR=/sift4g/run_sift
ORG=Culex Tarsalis
ORG_VERSION=v1.0.a1
#Running SIFT 4G
SIFT4G_PATH=/sift4g/bin/sift4g
PROTEIN_DB=/landscape_genetics/SIFT/uniref90.fasta
Sub-directories, don't need to change
LOGFILE=Log.txt
ZLOGFILE=Log2.txt
GENE_DOWNLOAD_DEST=gene-annotation-src
CHR_DOWNLOAD_DEST=chr-src
FASTA_DIR=fasta
SUBST_DIR=subst
SIFT_SCORE_DIR=SIFT_predictions
SINGLE_REC_BY_CHR_DIR=singleRecords/
SINGLE_REC_WITH_SIFTSCORE_DIR=singleRecords_with_scores
DBSNP_DIR=dbSNP
Doesn't need to change
FASTA_LOG=fasta.log
INVALID_LOG=invalid.log
PEPTIDE_LOG=peptide.log
ENS_PATTERN=ENS
SINGLE_RECORD_PATTERN=:change:_aa1valid_dbsnp.singleRecord
The text was updated successfully, but these errors were encountered: