Propagate custom_info Dict through agent Resource

[ddl-ebrown]

See also flyteorg/flyte#5391

Tracking issue

Why are the changes needed?

• The agent defines a Resource return type with values:

  • outputs
  • message
  • log_links
  • phase

  These are all a part of the underlying protobuf contract defined in
  flyteidl.

  However, the message field custom_info from the protobuf is not here

  google.protobuf.Struct custom_info

  https://github.com/flyteorg/flyte/blob/519080b6e4e53fc0e216b5715ad9b5b5270f35c0/flyteidl/protos/flyteidl/admin/agent.proto#L140

  This field was added in Agents protocol refactor flyte#4874
  but never made it into the corresponding flytekit PR
  Add SyncAgentBase #2146

• It's useful for agents to return additional metadata about the job,
  and it looks like custom_info is the intended location

• Make a minor refactor to how the agent responds to requests that
  return Resource by implementing to_flyte_idl / from_flyte_idl
  directly

What changes were proposed in this pull request?

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[codecov]

Codecov Report

Attention: Patch coverage is 50.00000% with 10 lines in your changes are missing coverage. Please review.

Project coverage is 77.62%. Comparing base (70332db) to head (2fd654a).
Report is 1 commits behind head on master.

Files	Patch %	Lines
flytekit/extend/backend/base_agent.py	52.94%	8 Missing ⚠️
flytekit/extend/backend/agent_service.py	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2426      +/-   ##
==========================================
+ Coverage   72.10%   77.62%   +5.51%     
==========================================
  Files         181      238      +57     
  Lines       18395    20841    +2446     
  Branches     3601     3600       -1     
==========================================
+ Hits        13264    16178    +2914     
+ Misses       4506     4062     -444     
+ Partials      625      601      -24     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ddl-rliu]

Tracking issue: flyteorg/flyte#5603

[pingsutw]

@ddl-rliu Could you rebase the PR to fix the CI error?

[pingsutw]

something wrong in the github

[ddl-rliu]

Fixed!

[pingsutw]

@ddl-rliu overall lgtm, some tests are failing though.

[ddl-rliu]

test errors should be fixed now 👍