500 Error on a emitted mempool tx

[wileyj]

stacks-node: 3.0.0.0.3 && 3.1.0.0.0
api version: 8.2.2 && 8.3.0-beta.2

stacks-node log:

WARN [1733809037.560269] [testnet/stacks-node/src/event_dispatcher.rs:502] [relayer-http://0.0.0.0:20443/] Event dispatcher: connection or request failed to stacks-blockchain-api:3700 - Custom { kind: Other, error: "HTTP 'POST /new_mempool_tx' did not succeed (500 != 200). Response body = JSON(Object {\"error\": Object {}})" }, backoff: 900s, attempts: 22

api log:

{"level":"error","time":"2024-12-10T05:40:36.511Z","pid":19,"hostname":"afab7069d49f","name":"stacks-blockchain-api","component":"core-api","err":{"type":"Error","message":"Failed to decode transaction: DeserializeError { error: \"Failed to parse multisig spending condition: got 4 sigs, expected 3\" }\n","stack":"Error: Failed to decode transaction: DeserializeError { error: \"Failed to parse multisig spending condition: got 4 sigs, expected 3\" }\n\n    at /app/src/event-stream/event-server.ts:140:39\n    at Array.map (<anonymous>)\n    at handleMempoolTxsMessage (/app/src/event-stream/event-server.ts:139:29)\n    at /app/src/event-stream/event-server.ts:697:54\n    at observeEvent (/app/src/event-stream/event-server.ts:651:13)\n    at /app/src/event-stream/event-server.ts:697:20\n    at run (/app/node_modules/p-queue/dist/index.js:157:104)\n    at PQueue._tryToStartAnother (/app/node_modules/p-queue/dist/index.js:105:17)\n    at /app/node_modules/p-queue/dist/index.js:171:18\n    at new Promise (<anonymous>)"},"msg":"error processing core-node /new_mempool_tx"}

i'm haven't found which tx causes this, but i've tried the following to no success:

• upgrade the stacks-node from 3.0.0.0.3 to 3.1.0.0.0
• upgrade the API to the beta release 8.3.0-beta.2
• remove entires from the stacks-node mempool table for the past 12 hours
  • remove the mempool DB entirely

nothing else is notable in the stacks-node logs or the api logs, and i'm so far unable to find which tx causes this - but i've seen it across several remote instances so far (Hiro's api seems unaffected, so i think it's coming from a remote peer that Hiro may not be connected with).

[wileyj]

Update: deleting the event_observer.sqlite allows the node to move past the error.
i think it's this row that causes the issue:

1149964|http://stacks-blockchain-api:3700/new_mempool_tx|["0x0000000001040506cb614d936db3a4c31b1a73e516e88d1f7579040000000000000015000000000006c34000000005000212e37e65b9741f09eeed659ceafe577020f15032d14f6de394c93ac2aeae2cd802013e4081ba21cf55a79d44364979bd69b1ff752af92a0402ca12cb9af1a46f0a11442b4fab5a47b8b25fda073e4e1f2782eba1de1dce18564ece471eb8693280b30200942b48e62e7c90aeb3c37f94022c60b8ecc1cf54d666812f6cbd267b12dc99df07c2955aabb9e136a6616eb20fe2bc5719cf3168644e379182ccbf51418de9ea02010df1a1161a65deafc79c170f5ef05707c399802c6257a1f1fb600414ed6ee08772b12a031e67682aeb179320da141b4679be718aa0545efca4714b3aa63db1660201f865ae066057deeeafaa4b7ad796b94f5d965e3de92699aedf49d9756b00c3617703979b67d9c0daa7f30184f722f7fa75edb0c6f4041a28d8e619323915a32000030302000000000216495ca29b5e23da51265dfc0388ea91539e5b67c510706f6e7469732d6272696467652d7635116d696e742d6274632d66726f6d2d627463000000060616495ca29b5e23da51265dfc0388ea91539e5b67c512706f6e7469732d6272696467652d7042544301000000000000000000000000004c4b400516d105ee658ba40a2e24662498dddc41a3d0452ab00200000021904e124b3d17611f769d641e4f366697ea850fe94c630e916b90feb6521f613d0102000000200000000000000000000145608686892318974350a6a12c218318f104f066f00c010000000000000000000000000004c51e"]|300000

the rest i have in that db all seem "normal", and this one is the outlier

[jcnelson]

This particular transaction uses the new OrderIndependentMultisigSpendingCondition authorization header, and it has four signatures whereas only three are required. This is allowed by the consensus rules.

[zone117x]

Should be fixed in https://github.com/hirosystems/stacks-blockchain-api/releases/tag/v8.2.3

[wileyj]

Deployed - i think it's fine to close, if i see it again i'll reopen with new info. thanks for the quick turnaround!