[BUG] Notebook task run as a AWS Map Task throw exception

[jw0515]

Describe the bug

import os
import pathlib

from flytekitplugins.papermill import NotebookTask
from flytekitplugins.awsbatch import AWSBatchConfig

config = AWSBatchConfig(
    platformCapabilities="EC2",
)

backtest_wages = NotebookTask(
    name="test-notebook",
    notebook_path=os.path.join(
        pathlib.Path(__file__).parent.parent.absolute(), "notebooks/test-notebook.ipynb"
    ),
    render_deck=True,
    task_config=config
)

Tried to use AWS batch job to run the notebook task using the code example above, got an error:

logger.debug(f"Starting _dispatch_execute for {task_def.name}")
AttributeError: 'function' object has no attribute 'name' 

When I looked at the batch job docker command:
["pyflyte-execute","--inputs","s3://fiolito-eks-cluster/metadata/propeller/liquidity-forecast-development-alg58fr86fmkgh9nqnf4/n0/data/inputs.pb","--output-prefix","s3://fiolito-eks-cluster/metadata/propeller/liquidity-forecast-development-alg58fr86fmkgh9nqnf4/n0/data/0/0","--raw-output-data-prefix","s3://fiolito-eks-cluster/99/alg58fr86fmkgh9nqnf4-n0-0","--resolver","flytekit.core.python_auto_container.default_task_resolver","--","task-module","flytekitplugins.papermill.task","task-name","_dummy_task_func"]

It seems that the _dummy_task_func is wrong which should be test-notebook and it comes from:
https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-papermill/flytekitplugins/papermill/task.py#L145

Expected behavior

Notebook task can run as an AWS Map Task without throwing exception.

Additional context to reproduce

Python3.10
flytekit 1.7.0
flytekitplugin-awsbatch 1.7.0
flytekitplugin-papermil 1.7.0

Screenshots

in aws batch console:

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[Ronanlfy]

Same issue here.

[jw0515]

@eapolinario any update on this?

[kumare3]

@eapolinario did you take a look at this?
@jw0515 and @Ronanlfy would you be open to contributing? we can help.

[jw0515]

Yes, I am open to contributing, if it's only flytekit needs some update (python code) @kumare3

[pingsutw]

Could you try not to pass the aws batch config to the notebook task? And update the task-plugin config for container_ array.

default-for-task-types:
    # Set it as the default handler for array/map tasks.
    container_array: aws_array

[jw0515]

I believe the config is already like this, but then the notebook task will run a normal python pod task if I don't pass the was config. @pingsutw

[kumare3]

this has been merged and works. closing