[help] How to Efficiently Combine and Save Large Dataframes

[grobins]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I need to combine and process data efficiently due to memory constraints. Basically I want to run 80 queries to get GIS data (20 locations over 4 date ranges). I have the targets for that, and to process the downloaded data per iteration. But I need to group the data per location, and passing in the previously mapped targets still has 80 branches. The data is too big to read it all into one target and split it out.

list(
tar_targets(locations, [read in the locations]),
tar_targets(mydates, [table of dates from and to]),
tar_targets(combos, expand.grid(locations, dates)),
tar_targets(gis_data, run_query(location = combos$location, combos$from, combos$to), map(combos)),

** so now I have a 80 branches each with a ton of data. **

tar_target(summarise, gis_data %>% group_by(location) %>% summarise(count = sum(something)), [WHAT_TO_PUT_HERE]

** now I'm expecting 20 branches each with the summary of it's dates **

)

)

[wlandau]

You could statically branch over locations and dynamically branch over factors within location, similar to https://books.ropensci.org/targets/static.html#dynamic-within-static-branching. That way, groupings automatically happen within location. Here is a sketch that covers the structure of the pipeline, even if it doesn't have the GIS details:

# _targets.R file
library(targets)
library(tarchetypes)
list(
  tar_map(
    values = list(location = c(0.3, 0.4)),
    tar_target(from, c(10, 20)),
    tar_target(gis_data, rbinom(from, 1, location), pattern = map(from)),
    tar_target(summaries, data.frame(count = sum(gis_data)))
  )
)

tar_visnetwork()

tar_make(reporter = "silent")
tar_read(summaries_0.3)
#>   count
#> 1     8
tar_read(summaries_0.4)
#>    count
#> 1    12

[grobins]

ah right... so if I read that correct, I can have a tar_map for the 20 locations. Then map over the dates for each of them, and summarise so I have 20 summaries, one for each location...

Thanks I'll give that a got

[grobins]

Can I pull the values from an existing target?

library(targets)
library(tarchetypes)
list( 
tar_target(mylocations, [thing that generates the locations]),
 tar_group(location_groups, mylocations, location),
  tar_map(
    values = location_groups$location,
    tar_target(mydates, [table of dates from and to]),
    tar_target(get_data, runsqlquery(mydates$from, mydates$to, location), pattern = map(mydates)),
    tar_target(summaries, summariseData(get_data))
  )
)`

[wlandau]

Not for tar_map() because it does static branching (meaning the branching has to be figured out before any targets start running).