[Epic] 10.1 Overhaul bad token handling

[MartinquaXD]

Epic label: E:badTokenDetection
Planned start date:
Due date:

Summary

Batch auctions are especially sensitive to problematic orders because a single such order could lead reverts of solutions that settle many orders just fine. Historically these orders usually involved tokens that take a fee on transfer because early solvers have not been implemented with support of that in mind.
The current approach of dealing with that is highly centralized and still leads to many false negatives (tokens that we don't expect to cause issues but still do).
The goal of this epic is to overhaul the way the protocol deals with "bad" tokens to make it more robust, less centralized and more accurate leading to overall better UX.

This Epic should happen in 3 phases:

1. Initial Implementation
2. Test and Refine
3. Clean up

Acceptance Criteria

• bad token detection no longer happens in the autopilot but rather in the individual drivers
• bad token detection happens on a per-solver basis because different solvers have different capabilities
• new bad token detection should be based on observing settlement simulations rather than use the current trace_callMany based bad token test
• if a driver is able to find a working solution for obscure/weird tokens users should be able to trade it
• solution would ideally work autonomously but needs to have "escape hatches" which allow explicitly marking tokens as supported or unsupported
• solution should have helpful logging and metrics to make it debuggable

Tasks

Initial Implementation

• implement BadTokenMonitor in the driver
• set up necessary logging / metrics / alerting
• add configuration options for BadTokenMonitor in services repo
• setup necessary configurations in infra repo

Test and Refine

• enable new solution in logOnly mode, debug, fix discovered issues
• switch to new solution, debug, fix discovered issues

Clean up

• remove legacy implementation
• unimplemented refactoring / cleanup ideas we accumulated along the way