You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Problem:
Oftentimes, travel cost matrices can get really big in large study areas. This creates a challenge of dealing with larger-than-memory data. The current version of {accessibility} cannot cope with this problem. Although the package is super efficient because it is built on top od {data.table}, the package still requires users to load the travel_matrix and land_use_data input to RAM memory, which can be a constraint for these larger-than-memory cases.
Proposed solution:
One solution to this would be to make the {accessibility} compatible with {arrow} (link). There are different ways to do this. The alternative I would suggest would use the following workfkow:
If the user passes travel_matrix and land_use_data inputs as data.tables or data.frames, then the package follows the current natural flow.
If the user passes a string path pointing to the location of .parquet or .csv files that contain the travel_matrix and land_use_data inputs, then the package follows a different flow to calculate the indicador using a {dplyr} + {arrow} syntax
Implications:
For users, this only means more flexibility to work with larger-than-memory data and greater computation speed.
For developers, this means we would need to make a few changes to check inputs, but most of the work would be translating the accessibility functions into a {dplyr} syntax, which is generally super simple.
The text was updated successfully, but these errors were encountered:
Problem:
Oftentimes, travel cost matrices can get really big in large study areas. This creates a challenge of dealing with larger-than-memory data. The current version of
{accessibility}
cannot cope with this problem. Although the package is super efficient because it is built on top od{data.table}
, the package still requires users to load thetravel_matrix
andland_use_data
input to RAM memory, which can be a constraint for these larger-than-memory cases.Proposed solution:
One solution to this would be to make the
{accessibility}
compatible with{arrow}
(link). There are different ways to do this. The alternative I would suggest would use the following workfkow:travel_matrix
andland_use_data
inputs as data.tables or data.frames, then the package follows the current natural flow..parquet
or.csv
files that contain thetravel_matrix
andland_use_data
inputs, then the package follows a different flow to calculate the indicador using a{dplyr}
+{arrow}
syntaxImplications:
{dplyr}
syntax, which is generally super simple.The text was updated successfully, but these errors were encountered: