1 million targets challenge

[wlandau]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I don't know if it's possible, but I want to push the limits of scale in targets even further (see @stemangiola's comment in #1329 (reply in thread)).

I usually profile targets without any parallel computing, but at this scale, I think we will eventually need crew to be fast. Test case:

# _targets.R file
library(targets)
library(crew)
tar_option_set(garbage_collection = 10000L)
autometric::log_phase_set("pipeline")
autometric::log_start(
  path = "pipeline.log",
  seconds = 15
)
tar_option_set(
  controller = crew_controller_local(
    name = "crew",
    workers = 1, # Planning to vary this to explore sources of bottlenecks.
    options_metrics = crew_options_metrics(
      path = "workers",
      seconds_interval = 15
    )
  )
)
list(
  tar_target(x, seq_len(1e6)),
  tar_target(y, x, pattern = map(x))
)

This pipeline runs unexpectedly slowly. I profiled the first 5 minutes:

proffer::pprof(targets::tar_make(callr_function = NULL), seconds_timeout = 300)

And the flame graph shows surprising bottlenecks:

The data blames targets' internal priority queue most of all, and its internal graph structure next. But I'm not sure whether take the evidence at face value: the overhead might actually come from crew, in which case the data just shows targets what it does while it waits. Maybe % CPU usage on the various processes would reveal where the work is actually happening. I will also need to profile crew at this scale and compare.

But if the priority queue and graph really are slow, then this is the most exciting low-hanging fruit I have found in quite some time.