[Bug]: Data-node is getting SQL error when replaying from block 0

[daniel1302]

Problem encountered

I am replaying node from block 0 and It is happening every 300-500k blocks. Validators have the same issue multiple times

Observed behaviour

Error in data-node logs during replay

Expected behaviour

Vega should be able to replay without restarting every a few times.

Steps to reproduce

1. Replay node from block 0

Software version

v0.71.5

Failing test

No response

Jenkins run

No response

Configuration used

I have tested multiple configurations:

1. PostgreSQL on the same node
2. PostgreSQL on the separated node

Both nodes are pretty big:
1. 128GB RAM, 16 cores, 4TB NVME
2. PostgreSQL 64GB RAM, 8 cores, 2 TB SSD, Vega+Data-node 32 GB RAM, 6 cores 2 TB SSD.

Relevant log output

Feb 18 17:29:17 data-node visor[13892]: 2024-02-18T17:29:17.348Z        ERROR        datanode.start.runNode        start/node.go:175        Vega data node stopped with error        {"error": "failed to flush subscriber:flushing margin levels: flushing margin levels: failed to copy margin_levels entries into database:ERROR: could not open relation with OID 206443 (SQLSTATE XX000)"}

[daniel1302]

I have also got the following error today:

Feb 22 18:26:35 data-node2 visor[3077636]: 2024-02-22T18:26:35.854Z        ERROR        datanode.start.runNode        start/node.go:175        Vega data node stopped with error        {"error": "failed to flush subscriber:failed to copy orders entries into database:ERROR: deadlock detected (SQLSTATE 40P01)"}
Feb 22 18:26:35 data-node2 visor[3077636]: vega data node stopped with error: failed to flush subscriber:failed to copy orders entries into database:ERROR: deadlock detected (SQLSTATE 40P01)