Parallelize decryption of reports

[andyleiserson]

No description provided.

[akoshelev]

did you have a chance to test it and compare with previous implementation? The way I was doing it was changing the e2e hybrid test to use 2 shards and running it on 1M input. You could observe CPU utilization by helper binaries and see if it is better with this change

[andyleiserson]

I did the CPU utilization experiment you describe. It's not super compelling, since two shards * three helpers is already 6 cores, i.e., most of the performance cores on my laptop (and CPU utilization can get somewhat above 100% per shard even before this change), but there was an improvement.

I also logged the current tokio task ID when decrypting to confirm that decryption was being performed on multiple tasks.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.18%. Comparing base (291efab) to head (5511e63).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1512      +/-   ##
==========================================
- Coverage   93.20%   93.18%   -0.02%     
==========================================
  Files         241      241              
  Lines       44054    44051       -3     
==========================================
- Hits        41061    41050      -11     
- Misses       2993     3001       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[akoshelev]

most of the performance cores on my laptop (and CPU utilization can get somewhat above 100% per shard even before this change), but there was an improvement.

Have you had a chance to compare it with main? The reason I am asking is because I ran this experiment on main a few days ago and it was also showing > 100% CPU utliziation

[andyleiserson]

I see the CPU utilization increase from ~110% per helper to ~150% per helper comparing main to this change.

[andyleiserson]

But that said, I would be more confident in this change if we see a performance improvement on the real cluster. I don't think it will be harmful, but it is rearranging the order in which flattening of the stream and decryption happen. (Doing it in the preferred order ran afoul of 100013.) So there's a small risk.