Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Add invocation_stacktrace to Tasks and Actors. #48920

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

rynewang
Copy link
Contributor

@rynewang rynewang commented Nov 25, 2024

Adds a new field string invocation_stacktrace to TaskSpec. It's populated from language frontend (_raylet.pyx), if RAY_enable_invocation_stacktrace is set to true. The field is propagated to TaskEvent and ActorTableData and through Dashboard APIs.

Users can use ray list task --detail as well as the Web UI. It works for Tasks, Actors (creation) and Actor Methods.

The flag RAY_enable_invocation_stacktrace is disabled by default, user can enable it if they want; so by default there should be no performance costs.

Screenshots:

  • Task
image
  • Actor
image
  • Actor Method
image

@rynewang rynewang marked this pull request as ready for review November 25, 2024 09:39
@rynewang rynewang requested review from a team, pcmoritz and raulchen as code owners November 25, 2024 09:39
@Moonquakes
Copy link

Hi @rynewang , it would be great to print out the parameters received by this ray task/actor!

Copy link
Contributor

@alanwguo alanwguo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dashboard UI code lgtm.

@jjyao jjyao added the go add ONLY when ready to merge, run all tests label Nov 27, 2024
Copy link
Collaborator

@jjyao jjyao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Could you add public doc about this feature?

@rynewang
Copy link
Contributor Author

Q: do we want to default enable this feature?

@jjyao
Copy link
Collaborator

jjyao commented Nov 27, 2024

Q: do we want to default enable this feature?

No. It will use more memory.

@Moonquakes
Copy link

No. It will use more memory.

May I know how much more memory will be consumed after turning on this feature? @jjyao

Copy link
Collaborator

@MengjinYan MengjinYan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only minor comment on the code logic.

At the same time, wondering if it is possible to add a test for setting the stacktrace in addition to the manual tests?

@@ -591,6 +594,9 @@ message TaskInfoEntry {
// If the task/actor is created within a placement group,
// this value is configured.
optional bytes placement_group_id = 26;
// Human readable stacktrace of the task invocation, or actor creation. The exact data
// format depends on the language. Only populated if the flag is enabled.
optional string invocation_stacktrace = 27;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering would it be helpful also add the field to the TaskInfoEntry in export_task_event.proto and expose the field to the export API users? cc: @nikitavemuri

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes but I am not familiar with that API. let's do this in another PR.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we create an issue for that so that the item can be tracked?

src/ray/protobuf/common.proto Show resolved Hide resolved
@rynewang
Copy link
Contributor Author

rynewang commented Dec 2, 2024

No. It will use more memory.

May I know how much more memory will be consumed after turning on this feature? @jjyao

We did not profile it, you can have a try after it's merged. Target end of this week.

Signed-off-by: Ruiyang Wang <[email protected]>
Signed-off-by: Ruiyang Wang <[email protected]>
Signed-off-by: Ruiyang Wang <[email protected]>
Signed-off-by: Ruiyang Wang <[email protected]>
@rynewang rynewang requested a review from a team as a code owner December 2, 2024 19:20
Copy link
Collaborator

@MengjinYan MengjinYan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core logic & tests looks good to me!

Signed-off-by: Ruiyang Wang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
go add ONLY when ready to merge, run all tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants