Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

is the test suite something that could be factored out/generalized? #40

Open
Gankra opened this issue Jul 7, 2022 · 5 comments
Open

Comments

@Gankra
Copy link
Contributor

Gankra commented Jul 7, 2022

You've done some incredible stuff in there making it possible to generate dumps, use dump_syms, and then load dumps again. I've been wanting a tool that is two binaries, minidump-harness and futility.

futility

(it seems like src/bin/test.rs is basically this)

The futile utility application, providing a "crash suite"

  • takes a simple cli arg choosing which func to call that will die in an interesting way
  • we can build for whatever platform
  • run dump_syms on

minidump-harness

(it seems like tests/ is kinda this)

Orchestrates building/analyzing/torturing futility and storing/diffing the results. it would serve three roles:

  • regression tests (running in ci)
  • getting a particular intermediate result of the pipeline for inspection
  • enabling us to quickly test/diff a change across our entire pipeline

The first is straight-forward and you mostly have that setup (although #30 still needs to be done). The second is more subtle. You can see my initial attempts at this problem with my socc-pair utility which largely automates diffing/benching a local version of minidump-stackwalk against what mozilla has in production. This is great for testing if I fixed a bug in production, but it's still more manual than I'd like.

The Dream

$ minidump-harness run base-trial
<spews a bunch of shit into a dir called base-trial>

<now i make some changes like [patch]ing local changes to rust-minidump/symbolic/dump_syms...>

$ minidump-harness run patched-trial
<spews a bunch of shit into a dir called patched-trial>

$ minidump-harness diff base-trial patched-trial
<spits out a report of what changed>

The Design

minidump-harness run $DIR would:

  • build futility, putting the target in $DIR/target
  • run dump-syms on it, putting the symbols in $DIR/symbols
  • run every crash suite it has, putting the dumps in $DIR/dumps/$suite.dmp
  • run minidump-stackwalk --cyborg --verbose=trace --symbols-path=$DIR/symbols on each minidump, putting:
    • json in $DIR/results/$suite.json
    • human in $DIR/results/$suite.txt
    • logs in $DIR/results/$suite.log.txt

minidump-harness diff $DIR1 $DIR2 would:

  • spit out a diff of particular details (default would probably be what socc-pair does?)
  • but ideally would streamline other diff/filter queries like "did the human output change" or "just tell me about this suite"
  • probably also want to be able to cargo-insta (snapshot) some of these results?

Right now you have some of these things, but a lot of the test implementations are in the harness and kind of manually adhoced. I don't fully understand your design, and all the constraints there are. I'm happy to do a lot of legwork here, but you've clearly done a lot stuff that's worth using/extending instead of me kicking off a green-field project.

@Gankra
Copy link
Contributor Author

Gankra commented Jul 7, 2022

cc @mstange and @Jake-Shadle

@Jake-Shadle
Copy link
Collaborator

I think something like this would be great. The MacOS dump_syms stuff I added was loosely based on the end-to-end testing I did in https://github.com/EmbarkStudios/crash-handling/tree/main/minidumper-test though there I at the moment just doing the basic "does this minidump contain the expected crash reason" without checking the symbolicated stack traces for the thread(s) also match, but that was something I wanted to add eventually.

I guess the one big difference to me is that the dream scenario you specified above feels a bit too manual for me, but maybe this is just a different workflow for doing changes before making automated tests?

@Gankra
Copy link
Contributor Author

Gankra commented Jul 7, 2022

yeah i made the dream slightly manual just so we could do absolutely whatever we want and extract only the parts we care about for individual runs.

i.e. "this old dump_syms version had this problem, I want a sym file showing that for my tests, so I will [patch] symbolic back to that version and do minidump-harness run and grab the sym file it spits out for my test-suite"

making automated tests I think would look closer to cargo insta snapshots, where we have a copy of the outputs we care about saved in one dir, and the tests just minidump-harness run and diff against the snapshot.

@Gankra
Copy link
Contributor Author

Gankra commented Jul 7, 2022

basically it would be useful as a

  • test-harness
  • artifacts-for-other-tests/projects generator
  • debugger

@Gankra
Copy link
Contributor Author

Gankra commented Jul 14, 2022

Ok here's a rough diagram of what a full deployment architecture looks like.

I've broken the architecture into 3 major sections:

  • your build
  • your client (user)
  • your server (you)

The client and server are further broken up into separate processes.

Rough atlas:

  • solid arrows: code invoking
  • dashed arrows: data flow
  • diamond boxes: data (files/structs)
  • square boxes: specific rust-minidump/crash-handling crates
  • round boxes: abstract services that we don't provide but you "need" to glue things together. I have labeled some of them with mozilla-specific names because those are what I know, but a lot of these could be labeled "Sentry". Many of these are the things a test harness would need to fill in.

Note that Mozilla (and Sentry?) don't use any of the "client" stuff right now except for minidump-writer in some specific situations. Migrating to minidump-writer completely is desirable for Mozilla (kill breakpad), but the other stuff is less clearly important to us.

graph TD;
subgraph build
  cargo
  dump_syms
  http-symbol-sender(http-symbol-sender)
  binary{.exe/.pdb}
  sym{.sym}
end

subgraph client
  subgraph app
    YOUR_ACTUAL_APP(YOUR ACTUAL APP)
    minidumper::Client
    crash-handler
  end

  subgraph crash-monitor
    minidumper::Service
    minidump-writer
    http-dump-sender(http-dump-sender)
    context{CrashContext}
    dump{.dump}
  end

  
end

subgraph server
  subgraph crash-server
    http-dump-receiver(http-dump-receiver)
    minidump-processor
    minidump
    structMinidump{Minidump}
    report{report.json}
  end
  subgraph symbol-server
    tecken(tecken)
    http-symbol-receiver(http-symbol-receiver)
  end
  subgraph crash-stats
    socorro(socorro)
  end
end

cargo -.-> binary
binary -.-> dump_syms
binary -.-> deploy(deploy)
dump_syms -.-> sym
sym -.-> http-symbol-sender

crash-handler ---> minidumper::Client
minidumper::Client ---> minidumper::Service
minidumper::Service ---> minidump-writer
minidump-writer -.-> dump
dump -.-> http-dump-sender
minidumper::Service ---> http-dump-sender 
minidumper::Service -.-> context
context -.-> minidump-writer

http-dump-receiver ---> minidump-processor
http-dump-receiver ---> minidump
minidump -.-> structMinidump
structMinidump -.-> minidump-processor
tecken -. .sym .-> minidump-processor
minidump-processor -.-> report
report -.-> http-dump-receiver
http-dump-receiver -. report.json .-> socorro

http-symbol-sender -. .sym .-> http-symbol-receiver
http-dump-sender -. .dump .-> http-dump-receiver
Loading
graph source
graph TD;
subgraph build
  cargo
  dump_syms
  http-symbol-sender(http-symbol-sender)
  binary{.exe/.pdb}
  sym{.sym}
end

subgraph client
  subgraph app
    YOUR_ACTUAL_APP(YOUR ACTUAL APP)
    minidumper::Client
    crash-handler
  end

  subgraph crash-monitor
    minidumper::Service
    minidump-writer
    http-dump-sender(http-dump-sender)
    context{CrashContext}
    dump{.dump}
  end

  
end

subgraph server
  subgraph crash-server
    http-dump-receiver(http-dump-receiver)
    minidump-processor
    minidump
    structMinidump{Minidump}
    report{report.json}
  end
  subgraph symbol-server
    tecken(tecken)
    http-symbol-receiver(http-symbol-receiver)
  end
  subgraph crash-stats
    socorro(socorro)
  end
end

cargo -.-> binary
binary -.-> dump_syms
binary -.-> deploy(deploy)
dump_syms -.-> sym
sym -.-> http-symbol-sender

crash-handler ---> minidumper::Client
minidumper::Client ---> minidumper::Service
minidumper::Service ---> minidump-writer
minidump-writer -.-> dump
dump -.-> http-dump-sender
minidumper::Service ---> http-dump-sender 
minidumper::Service -.-> context
context -.-> minidump-writer

http-dump-receiver ---> minidump-processor
http-dump-receiver ---> minidump
minidump -.-> structMinidump
structMinidump -.-> minidump-processor
tecken -. .sym .-> minidump-processor
minidump-processor -.-> report
report -.-> http-dump-receiver
http-dump-receiver -. report.json .-> socorro

http-symbol-sender -. .sym .-> http-symbol-receiver
http-dump-sender -. .dump .-> http-dump-receiver

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants