e2e flake: error unmarshalling: invalid character 'o' in literal null (expecting 'u')

[evan-forbes]

breaking apart this issues originally from #4028

New state sync test
2024/11/03 18:22:12 failed to get header: error in json rpc client, with http response metadata: (Status: 503 Service Unavailable, Protocol HTTP/1.1). error unmarshalling: invalid character 'o' in literal null (expecting 'u')
exit status 1

we should post links to specific runs here each time we see this flake

[evan-forbes]

this is actually two flakes in one, which might mean they are related

test-e2e2024/12/13 15:14:02 waiting for node to sync name val1
test-e2e2024/12/13 15:14:02 Creating HTTP client for node rpc_address [http://151.115.15.246:80/minorversioncompatibility-20241213-151305-val1-26657](http://151.115.15.246/minorversioncompatibility-20241213-151305-val1-26657)
test-e2e2024/12/13 15:14:02 error getting status, retrying... name val1 attempt 0 error error in json rpc client, with http response metadata: (Status: 404 Not Found, Protocol HTTP/1.1). error unmarshalling: invalid character 'p' after top-level value
test-e2e2024/12/13 15:14:03 node has synced name val1 attempts 1 latest_block_height 2
test-e2e2024/12/13 15:14:03 waiting for node to sync name val2
test-e2e2024/12/13 15:14:03 Creating HTTP client for node rpc_address http://151.115.15.246:80/minorversioncompatibility-202[41](https://github.com/celestiaorg/celestia-app/actions/runs/12318189915/job/34382358823#step:5:42)213-151305-val2-26657
test-e2e2024/12/13 15:14:03 node has synced name val2 attempts 0 latest_block_height 2
test-e2e2024/12/13 15:14:03 waiting for node to sync name val3
test-e2e2024/12/13 15:14:03 Creating HTTP client for node rpc_address http://151.115.15.246:80/minorversioncompatibility-20241213-151305-val3-26657
test-e2e2024/12/13 15:14:03 node has synced name val3 attempts 0 latest_block_height 2
test-e2e2024/12/13 15:14:04 txsim started name txsim
test-e2e2024/12/13 15:14:25 Creating HTTP client for node rpc_address http://151.115.15.2[46](https://github.com/celestiaorg/celestia-app/actions/runs/12318189915/job/34382358823#step:5:47):80/minorversioncompatibility-20241213-151305-val1-26657
test-e2e2024/12/13 15:14:25 Upgrading node node 2 version v2.2.0-rc0
2024/12/13 15:14:26 failed to upgrade node: error deploying resources for instance '%s'
exit status 1
make: *** [Makefile:160: test-e2e] Error 1

I still think there are issues using https in the cluster or something similar @smuu

[evan-forbes]

it might be better to migrate these issues to knuu once we know they are knuu specific and not strictly related to our testing logic