You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Parallel processing files is currently awkward -- it must either be done by creating map and extradata files from object files or creating parallel iterations.
Suggest true/false_taskNameBasedFiles_ parameter, default to current behavior. Setting it to true would include task name in maps and extradata files as is done with object files. Expectation would be that IC would manage these files manually/separately. This most useful for bibs and holdings (MARC and CSV based) tasks.
Aside from simplifying parallel processes, the feature would simplify combining task output from different sources requiring different maps.
The text was updated successfully, but these errors were encountered:
Currently, object and extradata files (I mistitled this) all contain the task name, but maps do not -- this is based on migrationTaskType.
This makes it awkward to run overlapping processes for the same task type (e.g. different source systems, vertically sharding out processes for Instance/Holdings/Items with especially large systems) -- you basically have to run parallel iterationIdentifiers or directories which is doable, but clunky.
Parallel processing files is currently awkward -- it must either be done by creating map and extradata files from object files or creating parallel iterations.
Suggest true/false_taskNameBasedFiles_ parameter, default to current behavior. Setting it to true would include task name in maps and extradata files as is done with object files. Expectation would be that IC would manage these files manually/separately. This most useful for bibs and holdings (MARC and CSV based) tasks.
Aside from simplifying parallel processes, the feature would simplify combining task output from different sources requiring different maps.
The text was updated successfully, but these errors were encountered: