You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Roddy checks input files for existence, maybe in checkExecutibility() from workflows. Provide a way to transparently check input, no matter whether file or S3 object.
Three alternative access modes: (1) URI (file://..., s3://..., others are in principle possible like hdfs://...) and let the job handle it, (2) pre-download the object to a temporary local file before the actual jobs starts, (3) provide a stream to download the complete file (similar to decompression but not refering to file:// but s3://; decompression may additionally be necessary).
Automatically upload declared output files (those that are job-outputs) to bucket.
Related existing concepts:
File: May also be an S3 object under the hood (generalize to DataResource or so)
FileSystemAccessProvider: Could also be S3 (generalize to DataResourceAccessProvider or so)
TODO:
Collect more issues, like those above
Identify composition into minimally viable products (MVPs)
Create issues to create MVPs
The text was updated successfully, but these errors were encountered:
Read input from S3 and write output to S3
Issues:
checkExecutibility()
from workflows. Provide a way to transparently check input, no matter whether file or S3 object.Related existing concepts:
TODO:
The text was updated successfully, but these errors were encountered: