bug: We don't /notify solvers on some errors

[MartinquaXD]

Problem

It looks like there are some errors for which we don't correctly call the /notify endpoint to give the solver debugging information.

Impact

External solver maintainers have a hard time debugging issues with their solutions.

To reproduce

Check these logs to see that we don't have a single notification for:

• simulation errors
• negative scores
• encoding errors

Expected behaviour

A solver should receive a /notify request for every solution it provided. Either a reason why it was discarded or the rank it achieved in the competition.

[MartinquaXD]

cc @ColePBryan (user which reported that they are missing /notifys for many auctions)

[sunce86]

I see that we have notifies for simulation errors: https://production-6de61f.kb.eu-central-1.aws.cloud.es.io/app/r/s/Gk4lS

[sunce86]

And for negative scores, I don't know what is the exact case here, but we usually report either ObjectiveValueNonPositive for that https://production-6de61f.kb.eu-central-1.aws.cloud.es.io/app/r/s/5kiFQ

NonPositiveScore notify is only sent if the solver reports a 0 score.

[squadgazzz]

Some of the simulation errors are ignored:

services/crates/driver/src/infra/simulator/mod.rs

Lines 169 to 202 in b5a258a

	fn with<E>(tx: eth::Tx, block: eth::BlockNo) -> impl FnOnce(E) -> Error
	where
	E: Into<SimulatorError>,
	{
	move |err| {
	let err: SimulatorError = err.into();
	let tx = match &err {
	SimulatorError::Tenderly(tenderly::Error::Http(_)) => None,
	SimulatorError::Tenderly(tenderly::Error::Revert(_)) => Some(tx),
	SimulatorError::Blockchain(blockchain::Error::Method(error))
	if matches!(error.inner, ExecutionError::Revert(_)) =>
	{
	Some(tx)
	}
	SimulatorError::Blockchain(blockchain::Error::Method(_)) => None,
	SimulatorError::Blockchain(blockchain::Error::Web3(inner)) => {
	let error = ExecutionError::from(inner.clone());
	if matches!(error, ExecutionError::Revert(_)) {
	Some(tx)
	} else {
	None
	}
	}
	SimulatorError::Blockchain(blockchain::Error::Gas(_)) => None,
	SimulatorError::Blockchain(blockchain::Error::Response(_)) => None,
	SimulatorError::Enso(enso::Error::Http(_)) => None,
	SimulatorError::Enso(enso::Error::Revert(_)) => Some(tx),
	};
	match tx {
	Some(tx) => Error::Revert(RevertError { err, tx, block }),
	None => Error::Basic(err),
	}
	}
	}

But it needs to be confirmed if the following errors are properly built from responses:

services/crates/driver/src/infra/simulator/tenderly/mod.rs

Lines 115 to 118 in 492605c

	pub enum Error {
	Http(#[from] reqwest::Error),
	Revert(SimulationId),
	}

Further investigation is required on what errors are not received by solvers exactly.

[fleupold]

Do we have a concrete solver that didn't receive a notification for a specific auction?

Unlike stated in the issue description, I see logs for simulation failures, non-positive score (encoding errors are mostly also simulation failures).

We don't notify runner ups with their rank, but this is by design (rank can be fetched from the solver competition endpoint)

[MartinquaXD]

The specific example that sparked this issue is here where the odos solver provided some solution which failed to simulate with execution reverted but did not receive any /notify call about it.
The odos maintainer says there are lots and lots of auctions where they participated in the shadow competition and didn't receive a notification.

[fleupold]

In this specific auction, the error was in estimating the access list, which currently is classified as a "Basic" Simulation error (for which we don't notify solvers). I will make a PR to address this.