[Bug]: Precision loss when going from position -> fair-price -> position causes crossed market

[wwestgarth]

Problem encountered

The market-sim run in this comment:
#11583 (comment)

is failing with: panic: re submit special order would cross

The reason is because during some operations in TradableVolumeInRange() we lose precision causing an AMM to trade 1 less volume that it should.

There are two ways to view the state of an AMM:

• as prices, where the AMM has an upper and lower price bound, and its fair-price exists between it
• as positions, where the AMM's price bounds map to a maximum long/short positions and the AMM's position exists between those

The AMM equations allow us to jump between the two views, but when we do so we lose some precision. This is because a position maps to a decimal price. Equally a price maps to a decimal position. At some point we have to round the results and round the position/price into integers and lose some precision.

In the case of TradableVolumeInRange() we go from position -> fair-price -> position and expect the position to be the same. What in fact happens is:

position: 3062 -> fair-price-decimal: 492602461141146970.48778761474 -> fair-price 492602461141146970 -> position-decimal: 3061.9999999999918262 -> position: 3061

and we lose a 1.

Solution:
TradableVolumeInRange() needs to be changed to avoid doing this round trip, we should map the input st, nd ranges into theoretical positions at entry and from there only work in position-space.

Observed behaviour

Markets cross

Expected behaviour

Markets not crossed.

Steps to reproduce

1. Submit `xyz` transaction
2. With this payload
3. Query `abc` API
4. Observe the error

Software version

develop

Failing test

No response

Jenkins run

No response

Configuration used

No response

Relevant log output

No response