Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Support placeholders for Map state #101

Open
mochacafe opened this issue Oct 21, 2020 · 2 comments · May be fixed by #158
Open

Feature Request: Support placeholders for Map state #101

mochacafe opened this issue Oct 21, 2020 · 2 comments · May be fixed by #158

Comments

@mochacafe
Copy link

Currently I am not able to pass execution input to a Map state. If I try to pass execution input to map state it throws 'Object of type 'ExecutionInput' is not JSON serializable' . So is it possible to add palceholder support to Map state.

input_data_check_map_step= steps.states.Map(state_id='input_data_check_map',
                                           iterator=input_data_check_lambda_task,
                                            input_path=execution_input,
                                           items_path=execution_input['input_data_dirs'],
                                           parameters={'input_dir':'$.Map.Item.Value','fcst_date':"$.fcst_date"},
                                           max_concurrency=5,
                                           comment='Run input data checks in dynamic parllel state',
                                           result_path='$.InputDataCheckRes')
@yoodan93
Copy link
Contributor

yoodan93 commented Nov 6, 2020

Step Functions recently added support for execution input in InputPath and ItemsPath for Map state! https://docs.aws.amazon.com/step-functions/latest/dg/input-output-contextobject.html

The python SDK still does not support this, but we have plans of keeping the python SDK up to date to current Amazon States Language specs.

@wong-a
Copy link
Contributor

wong-a commented Sep 10, 2021

"Support placeholders for Map state" is a broad topic - let's clarify what this means and enumerate the specific use cases and feature gaps.

In @mochacafe's code example there are two places where placeholders are not trying to be used but not supported:

1. InputPath should accept Placeholders (and also OutputPath and ResultPath)

input_path=execution_input

The generated ASL should be the following.

"InputPath": "$$.Execution.Input"

Note that unlike Placeholders in Parameters (the only place they are supported today) the original key should not be suffixed by ".$". All the "Path" ASL fields are just JSONPath strings.

This allows you to specify InputPath without writing a literal JSONPath string by hand. Today you can use the context object (or any JSONPath) in input_path but you need to pass a JSONPath string (e.g. input_path='$$.Execution.Input'') like how result_path is set in the example. The Placeholder class is just a pythonic way to construct the dynamic references without writing JSONPath strings by hand (plus some optional static validation if schemas are used).

This applies to all states that support the InputPath field, not just Map states. OutputPath and ResultPath can have the same functionality too. One caveat is that ResultPath shouldn't accept ExecutionInput, since the context object is non-writable.

2. ItemsPath should accept Placeholders

Similar to above, make it easier to construct the JSONPath using Placeholders. The workaround today is to just write the string:

items_path='$$.Execution.Input.input_data_dirs'

Only Map state has this field.

3. Add Placeholders for Map Index and Map Value

This one is not actually called out in @mochacafe 's example but would be nice to have too.

For Map states, the Context Object has special fields to reference the iteration's Index and Value. See: https://docs.aws.amazon.com/step-functions/latest/dg/input-output-contextobject.html#contextobject-map

Today you can use them in a Map state's Parameters by writing the strings like so:

parameters={
  'field_in_iterator_value': '$$.Map.Item.Value["myField"]',
  'index':'$$.Map.Item.Index'', 
  'static_value': [1, 2, 3],
  ...
},

This generates the following ASL:

"Parameters": {
  "field_in_iterator_value.$":  "$$.Map.Item.Value['myField']",
  "index.$": "$$.Map.Item.Index",
  "static_value": [1, 2, 3]
}

Supporting placeholders means you can use a Placeholder class to build the dynamic reference without writing JSONPath. (Borrowing the proposed class names from #158:

parameters={
  'field_in_iterator_value': MapItemValue()['myField'],
  'index':MapItemIndex(), 
  'static_value': [1, 2, 3],
  ...
},

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants