Bulk download failing #3

bschilder · 2021-04-10T15:11:34Z

Originally posted by @jdblischak in Issue #1 :

Caveat: I didn't actually perform the bulk download from within the app. It always crashed at 87%. I extracted the code from app.R to combine all the results.

bschilder · 2021-04-10T15:30:25Z

Doesn't crash per se on Macbook with Chrome, but at 87% it returns an html file instead.

<!doctype html>
--
<!--[if lt IE 7 ]>
<html lang="en"   class="no-js ie6"> <![endif]-->
<!--[if IE 7 ]>
<html lang="en"   class="no-js ie7"> <![endif]-->
<!--[if IE 8 ]>
<html lang="en"   class="no-js ie8"> <![endif]-->
<!--[if (gte IE   9)\|!(IE)]><!-->
<html lang="en"   class="no-js"> <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv='X-UA-Compatible'   content='IE=edge
<meta name="viewport"   content="width=device-width
<!-- HTML5 shim and Respond.js IE8   support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script   src="//oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script   src="//oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
<![endif]-->
<link   href="/__static__/components/@icon/themify-icons/themify-icons.css?v=b282775482260fd5fe87b8f92db29099"   rel="stylesheet"/>
<link   href="/__static__/frontend/css/style.css?v=6c1baf841c4c1ca84c5eb0f1c98ec4d5"   rel="stylesheet"/>
<!-- block head -->
 
<style>
.spinner {
clear: both;
margin: 0 auto;
}
</style>
 
<!-- block head -->
</head>
<body>
<header>
<!-- block header -->
 
<!-- block header -->
</header>
<div id="content">
<!-- block content -->
 
<div class="container   vertical-table">
<div class="row   vertical-align-middle">
<div class="col-lg-12">
<div class="spinner   text-center">
<h3>Please Wait</h3>
<img alt="loading"   src="/__static__/frontend/images/spinner.gif?v=4889784689c1b8109f97a0eecf9265f4"/>
</div>
</div>
</div>
</div>
 
<!-- block content -->
</div>
<footer>
<!-- block footer -->
 
<!-- block footer -->
</footer>
</body>
<script   src="/__static__/components/jquery/dist/jquery.min.js?v=220afd743d9e9643852e31a135a9f3ae"   type="text/javascript"></script>
<script   src="/__static__/components/bootstrap/dist/js/bootstrap.min.js?v=2f34b630ffe30ba2ff2b91e3f3c322a1"   type="text/javascript"></script>
<script   src="/__static__/components/iframe-resizer/js/iframeResizer.contentWindow.js?v=381efb0877cc316e6c007f402203d7f0"   type="text/javascript"></script>
<script   src="/__static__/frontend/scripts/shinyapps.js?v=cec01ed9ffe3cf03c95e164a1911f274"   type="text/javascript"></script>
<script   src="/__static__/frontend/scripts/shinyapps.frame.content.js?v=3c17be5413e55ee145876df176d5eefc"   type="text/javascript"></script>
<!-- block scripts -->
 
<script>
var app_url =   'https://rajlab.shinyapps.io/Fine_Mapping_Shiny';
var healthcheck_endpoint =   '/__health-check__/';
var started_at = new Date().getTime();
var finished_at = null;
var count = 0;
var timeout = null;
function debug(message) {
if (window.console &&   window.console.log) {
window.console.log(message);
}
}
function reload() {
window.location.reload(true);
}
function wait() {
// wait and try again
count += 1;
var snooze = Math.min(3000
debug("Application not loaded.   Retrying in " + snooze + "ms");
setTimeout(function () {
check();
}
}
function check() {
var url = app_url + healthcheck_endpoint;
$.ajax({url: url
.done(function (data
if (xhr.status === 200) {
// log time elapsed
finished_at = new Date().getTime();
var elapsed = finished_at - started_at;
debug("Application loaded in "   + elapsed + "ms (" + count + " checks). Reloading.");
if (timeout != null) {
clearTimeout(timeout);
}
reload();
} else {
wait();
}
})
.fail(function() {
wait();
});
}
$(document).ready(function () {
debug("Waiting for application to   load ...");
check();
// reload regardless of success after 10   seconds
timeout = setTimeout(function () {
debug("Application failed to load   after 10 seconds. Reloading.");
reload();
}
});
</script>
 
<!-- block scripts -->
</html>

bschilder · 2021-04-10T16:23:43Z

When running Shiny App locally, produces csv fine. This suggests it's an issue on shinyapps.io's server. Are there caps on the total file size that can be downloaded? echolocatoR_Finemapping_Portal.all_data.multi_finemap.csv is 686.6Mb, which is large but not crazy large.

bschilder · 2021-04-10T17:31:32Z

Reading some shiny documentation and found that this may be related to how I'm calling the download. Switched:

 datasetInput_bulk <- reactive({
    withProgress(message = 'Gathering and merging all results', value = 0, {
      dat_paths <- subset(all_paths, file_type=="multi_finemap")
      merged_DT <- lapply(1:nrow(dat_paths), function(i){
        ROW <- dat_paths[i,]
        dat <- data.table::fread(ROW$file_path, nThread = 1)
        dat <- cbind(study=ROW$study, study_type=ROW$study_type, LD_ref=ROW$LD_ref, unique(dat))
        incProgress(1/nrow(dat_paths), detail = paste(round(i/nrow(dat_paths),2)*100,"%"))
        return(dat)
      }) %>% data.table::rbindlist(fill = T)
    })
    return(merged_DT)
  })

datasetInput_bulk <- function(){
    withProgress(message = 'Gathering and merging all results', value = 0, {
      dat_paths <- subset(all_paths, file_type=="multi_finemap") 
      merged_DT <- lapply(1:nrow(dat_paths), function(i){
        ROW <- dat_paths[i,]
        dat <- data.table::fread(ROW$file_path, nThread = 1)
        dat <- cbind(study=ROW$study, study_type=ROW$study_type, LD_ref=ROW$LD_ref, unique(dat))
        incProgress(1/nrow(dat_paths), detail = paste(round(i/nrow(dat_paths),2)*100,"%"))
        return(dat)
      }) %>% data.table::rbindlist(fill = T)
    })
    return(merged_DT)
  }

As before, this works fine locally.

However, after rebuilding the image and pushing the new one to shinyapps.io (takes a while), I found this made no difference. Produces the same html file instead of csv. Also noticed the "Disconnected from server" message in the bottom left. This wasn't due to lack of activity, as I tried interacting with the app the whole time to ensure it didn't go to sleep.

Potentially related issue, will try the suggested solution:
rstudio/shiny#2152

jdblischak · 2021-04-12T16:23:36Z

Doesn't crash per se on Macbook with Chrome, but at 87% it returns an html file instead.

@bschilder To clarify, this is the same behavior I observed. I was also using Chrome on macOS. At 87%, it downloads an HTML file and then shows the message "Disconnected from server".

bschilder · 2021-04-25T10:44:29Z

@jdblischak thanks for confirming. Still trying to figure out a fix for this. In the meantime, I could provide a Google Drive link to the merged summary stats.

bschilder · 2021-04-28T12:11:35Z

Just added a link to the Pre-merged bulk download here:

jdblischak · 2021-06-03T14:51:51Z

Just added a link to the Pre-merged bulk download here:

I just confirmed that I am able to successfully download the pre-merged bulk data from Google Drive. Thanks!

Note however that the bulk data would be more useful if every loci was processed with the same version of echolocatoR. As we discussed in #2, I found it difficult to compare the performance of each method across loci since the potential values in each column are not consistent across loci.

This was referenced Apr 10, 2021

Include LD reference panel in bulk download #1

Merged

Multi-finemap column descriptions #4

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bulk download failing #3

Bulk download failing #3

bschilder commented Apr 10, 2021

bschilder commented Apr 10, 2021 •

edited

Loading

bschilder commented Apr 10, 2021 •

edited

Loading

bschilder commented Apr 10, 2021 •

edited

Loading

jdblischak commented Apr 12, 2021

bschilder commented Apr 25, 2021

bschilder commented Apr 28, 2021

jdblischak commented Jun 3, 2021

Bulk download failing #3

Bulk download failing #3

Comments

bschilder commented Apr 10, 2021

bschilder commented Apr 10, 2021 • edited Loading

bschilder commented Apr 10, 2021 • edited Loading

bschilder commented Apr 10, 2021 • edited Loading

jdblischak commented Apr 12, 2021

bschilder commented Apr 25, 2021

bschilder commented Apr 28, 2021

jdblischak commented Jun 3, 2021

bschilder commented Apr 10, 2021 •

edited

Loading

bschilder commented Apr 10, 2021 •

edited

Loading

bschilder commented Apr 10, 2021 •

edited

Loading