help: foundry assertions

[0xteddybear]

I'm currently having the following issue, which I cannot explain, and would like to see if you have any input or could replicate it:

context

I'm trying to make medusa tests also runnable in foundry. Due to the limitations already discussed off-repo, I can only meaningfully test as pure properties two of the 15 already tested ones

• Medusa tests are passing (at least for a few minutes worth of fuzzing)
• The medusa assertions tests (which technically include the properties also tested in foundry, de-duplicating that is a WIP), don't cause any failures when run from within foundry foundry test --mc MedusaAdaptoor --mt invariant_handlerAssertions, with any combination of runs+depth
• When trying to assert the two testable properties from the foundry invariants contract, I get the following failure (see logs below)

this looks like a typical case of not exposing the same data internally vs externally, but the logs + traces seem to confirm that the assertion should indeed pass, but it doesn't

things I tried

• using assert(trackedSupply == fundsInTransit + totalSupply) instead of DSTest's assertEq

notes

I had to lower the depth and increase the runs number because foundry doesn't seem to be shrinking the call sequence effectively (doesn't even remove reverted calls)

running foundry forge 0.2.0 (d28a337 2024-08-27T00:20:10.476146991Z)

[N] ~/w/e/o/p/contracts-bedrock (help/foundry-assertions){44}
> forge test --mc MedusaAdaptooor  --mt InTransit -vvv
[⠆] Compiling...
No files changed, compilation skipped

Ran 1 test for test/invariants/OptimismSuperchainERC20.t.sol:MedusaAdaptooor
[FAIL. Reason: assertion failed: 1721466380621 != 0]
	[Sequence]
		sender=0x0000000000000000000000000000000000001ECd addr=[test/properties/medusa/Protocol.t.sol:ProtocolProperties]0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f calldata=handler_MintSupertoken(uint256,uint96) args=[257005939243648192387814660777860 [2.57e32], 1721466380621 [1.721e12]]
		sender=0x0000000000000000000000000000000000000104 addr=[test/properties/medusa/Protocol.t.sol:ProtocolProperties]0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f calldata=fuzz_DeployNewSupertoken((uint8,uint8,uint8,uint8),uint256) args=[TokenDeployParams({ remoteTokenIndex: 129, nameIndex: 88, symbolIndex: 89, decimalsIndex: 0 }), 5872]
 invariant_totalSupplyAcrossChainsEqualsMintsMinusFundsInTransit() (runs: 7190, calls: 28760, reverts: 15299)
Logs:
  Bound result 0
  Bound result 0
  Bound result 0
  Bound result 0
  Bound result 0
  Bound result 0
  trackedSupply    : 1721466380621
  totalSupply      : 1721466380621
  fundsInTransit   : 0

Traces:
  [235077] ProtocolProperties::handler_MintSupertoken(257005939243648192387814660777860 [2.57e32], 1721466380621 [1.721e12])
    ├─ [0] console::log("Bound result", 0) [staticcall]
    │   └─ ← [Stop]
    ├─ [0] VM::prank(0x4200000000000000000000000000000000000010)
    │   └─ ← [Return]
    ├─ [52871] ERC1967Proxy::mint(0x0000000000000000000000000000000000001ECd, 1721466380621 [1.721e12])
    │   ├─ [47996] OptimismSuperchainERC20ForToBProperties::mint(0x0000000000000000000000000000000000001ECd, 1721466380621 [1.721e12]) [delegatecall]
    │   │   ├─ emit Transfer(src: 0x0000000000000000000000000000000000000000, dst: 0x0000000000000000000000000000000000001ECd, wad: 1721466380621 [1.721e12])
    │   │   ├─ emit Mint(account: 0x0000000000000000000000000000000000001ECd, tokenId: 1721466380621 [1.721e12])
    │   │   └─ ← [Stop]
    │   └─ ← [Return]
    ├─ [2542] 0x4200000000000000000000000000000000000023::superTokenInitDeploySalts(ERC1967Proxy: [0x2fDe76d4b5df4882efB6Ae68987645D4C8102da1]) [staticcall]
    │   └─ ← [Return] 0x9157c5fb7181d9b8766f308fdfb258798589babd769ef73fb492ef13933b73eb
    ├─ [542] 0x4200000000000000000000000000000000000023::superTokenInitDeploySalts(ERC1967Proxy: [0x2fDe76d4b5df4882efB6Ae68987645D4C8102da1]) [staticcall]
    │   └─ ← [Return] 0x9157c5fb7181d9b8766f308fdfb258798589babd769ef73fb492ef13933b73eb
    └─ ← [Stop]

  [9079256848778899524] ProtocolProperties::fuzz_DeployNewSupertoken(TokenDeployParams({ remoteTokenIndex: 129, nameIndex: 88, symbolIndex: 89, decimalsIndex: 0 }), 5872)
    ├─ [0] console::log("Bound result", 0) [staticcall]
    │   └─ ← [Stop]
    ├─ [0] console::log("Bound result", 0) [staticcall]
    │   └─ ← [Stop]
    ├─ [0] console::log("Bound result", 0) [staticcall]
    │   └─ ← [Stop]
    ├─ [0] console::log("Bound result", 0) [staticcall]
    │   └─ ← [Stop]
    ├─ [0] console::log("Bound result", 0) [staticcall]
    │   └─ ← [Stop]
    ├─ [0] → new ERC1967Proxy@0x2fDe76d4b5df4882efB6Ae68987645D4C8102da1
    │   └─ ← [CreateCollision] EvmError: CreateCollision
    └─ ← [Revert] EvmError: Revert

  [56224] MedusaAdaptooor::invariant_totalSupplyAcrossChainsEqualsMintsMinusFundsInTransit()
    ├─ [2546] ProtocolProperties::deploySaltsLength() [staticcall]
    │   └─ ← [Return] 1
    ├─ [5122] ProtocolProperties::totalSupplyAcrossChainsAtIndex(0) [staticcall]
    │   └─ ← [Return] 0x9157c5fb7181d9b8766f308fdfb258798589babd769ef73fb492ef13933b73eb, 1721466380621 [1.721e12]
    ├─ [4965] ProtocolProperties::tokensInTransitForDeploySalt(0x9157c5fb7181d9b8766f308fdfb258798589babd769ef73fb492ef13933b73eb) [staticcall]
    │   └─ ← [Return] 0
    ├─ [291] ProtocolProperties::MAX_CHAINS() [staticcall]
    │   └─ ← [Return] 4
    ├─ [2557] 0x4200000000000000000000000000000000000023::superTokenAddresses(0, 0x9157c5fb7181d9b8766f308fdfb258798589babd769ef73fb492ef13933b73eb) [staticcall]
    │   └─ ← [Return] ERC1967Proxy: [0x2fDe76d4b5df4882efB6Ae68987645D4C8102da1]
    ├─ [7220] ERC1967Proxy::totalSupply() [staticcall]
    │   ├─ [2348] OptimismSuperchainERC20ForToBProperties::totalSupply() [delegatecall]
    │   │   └─ ← [Return] 1721466380621 [1.721e12]
    │   └─ ← [Return] 1721466380621 [1.721e12]
    ├─ [291] ProtocolProperties::MAX_CHAINS() [staticcall]
    │   └─ ← [Return] 4
    ├─ [2557] 0x4200000000000000000000000000000000000023::superTokenAddresses(1, 0x9157c5fb7181d9b8766f308fdfb258798589babd769ef73fb492ef13933b73eb) [staticcall]
    │   └─ ← [Return] 0x0000000000000000000000000000000000000000
    ├─ [291] ProtocolProperties::MAX_CHAINS() [staticcall]
    │   └─ ← [Return] 4
    ├─ [2557] 0x4200000000000000000000000000000000000023::superTokenAddresses(2, 0x9157c5fb7181d9b8766f308fdfb258798589babd769ef73fb492ef13933b73eb) [staticcall]
    │   └─ ← [Return] 0x0000000000000000000000000000000000000000
    ├─ [291] ProtocolProperties::MAX_CHAINS() [staticcall]
    │   └─ ← [Return] 4
    ├─ [2557] 0x4200000000000000000000000000000000000023::superTokenAddresses(3, 0x9157c5fb7181d9b8766f308fdfb258798589babd769ef73fb492ef13933b73eb) [staticcall]
    │   └─ ← [Return] 0x0000000000000000000000000000000000000000
    ├─ [291] ProtocolProperties::MAX_CHAINS() [staticcall]
    │   └─ ← [Return] 4
    ├─ [0] console::log("trackedSupply    :", 1721466380621 [1.721e12]) [staticcall]
    │   └─ ← [Stop]
    ├─ [0] console::log("totalSupply      :", 1721466380621 [1.721e12]) [staticcall]
    │   └─ ← [Stop]
    ├─ [0] console::log("fundsInTransit   :", 0) [staticcall]
    │   └─ ← [Stop]
    ├─ [0] VM::assertEq(1721466380621 [1.721e12], 1721466380621 [1.721e12]) [staticcall]
    │   └─ ← [Return]
    ├─ [546] ProtocolProperties::deploySaltsLength() [staticcall]
    │   └─ ← [Return] 1
    └─ ← [Stop]

Suite result: FAILED. 0 passed; 1 failed; 0 skipped; finished in 3.45s (3.45s CPU time)

Ran 1 test suite in 4.13s (3.45s CPU time): 0 tests passed, 1 failed, 0 skipped (1 total tests)

Failing tests:
Encountered 1 failing test in test/invariants/OptimismSuperchainERC20.t.sol:MedusaAdaptooor
[FAIL. Reason: assertion failed: 1721466380621 != 0]
	[Sequence]
		sender=0x0000000000000000000000000000000000001ECd addr=[test/properties/medusa/Protocol.t.sol:ProtocolProperties]0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f calldata=handler_MintSupertoken(uint256,uint96) args=[257005939243648192387814660777860 [2.57e32], 1721466380621 [1.721e12]]
		sender=0x0000000000000000000000000000000000000104 addr=[test/properties/medusa/Protocol.t.sol:ProtocolProperties]0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f calldata=fuzz_DeployNewSupertoken((uint8,uint8,uint8,uint8),uint256) args=[TokenDeployParams({ remoteTokenIndex: 129, nameIndex: 88, symbolIndex: 89, decimalsIndex: 0 }), 5872]
 invariant_totalSupplyAcrossChainsEqualsMintsMinusFundsInTransit() (runs: 7190, calls: 28760, reverts: 15299)

Encountered a total of 1 failing tests, 0 tests succeeded

[simon-something]

Replicating, quite wtf...
This could have been some shadowing but that would have been too easy ig

[simon-something]

real issue is the revert on create2 conflict happening in fuzz_DeployNewSupertoken. It seems like foundry considers the assertEq as a failing property erroneously (funny enough, the trace is correct, while the assert "summary" ([FAIL. Reason: assertion failed: 1721466380621 != 0]) is not, don't know where this 0 comes from tbh).

Fixed it by wrapping the create2 deployment in a try/catch (invariant then holds/test passes) - this looks like a bug tho

    function fuzz_DeployNewSupertoken(
        TokenDeployParams memory params,
        uint256 chainId
    )
        external
        validateTokenDeployParams(params)
    {
        chainId = bound(chainId, 0, MAX_CHAINS - 1);

        OptimismSuperchainERC20 supertoken;
        try this.deploy(params, chainId) returns(OptimismSuperchainERC20 _supertoken) {
            supertoken = _supertoken;
        } catch {
        }

        // 14
        compatibleAssert(supertoken.totalSupply() == 0);
    }

    function deploy(
        TokenDeployParams memory params,
        uint256 chainId
    )
        public
        returns(OptimismSuperchainERC20)
    {
        return _deploySupertoken(
            remoteTokens[params.remoteTokenIndex],
            WORDS[params.nameIndex],
            WORDS[params.symbolIndex],
            DECIMALS[params.decimalsIndex],
            chainId
        );
    }

[0xteddybear]

thanks for your help dr! moved forward with #40