fix(rpc): actually use RPC in CLI tests; add icmd and assertCmd for better error reporting

[willemneal]

What

Correctly name the env vars for setting the network configuration when invoking the CLI.

Also fix auth bug for deploys.

runCLICommand now uses icmd to execute CLI commands and shlex to parse the input string. 'assertCmd' wraps a call to runCLICommand and require.NoError, passing the stderr writer to lazily write with error, returning stdout as a string.

Why

It currently is using the sandbox.

Known limitations

[TODO or N/A]

[willemneal]

I'm currently getting when trying to deploy.

        	            	error: transaction submission timeout

[2opremio]

I have been looking into this and it seems soroban-rpc receives a "PENDING" status when submitting the contract creation transaction but core never sends back meta for it. This causes the CLI to receive a "NOT_FOUND" status when querying the transaction for 10 seconds, leading to a timeout.

I see three possibilities:

1. The transaction fails but Core is responding with "PENDING" instead of "FAIL"
2. The transaction fails later on and Core fails to send the corresponding meta.
3. Core does emit the meta later on but soroban-rpc fails to obtain it.

[2opremio]

Here are the three transactions sent to the standalone network, the third of which doesn't result in any transaction being ingested by soroban-rpc:

1. Account creation:

AAAAAgAAAABzdv3ojkzWHMD7KUoXhrPx0GH18vHKV0ZfqpMiEblG1gAAAGQAAAAAAAAAAQAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAADRjwIQ/2zB8tzxMB+71MMO4RoHWCBoTUcd+J0PEBHqKAAAAOjUpRAAAAAAAAAAAAERuUbWAAAAQKAEpum2TGh/P2K0/eOxeXw1eGEG5fl/Ft2a/j7YUN+H3XNjkFAfYnJvfpmvTsNYqPsoHKufgRpDmJuAhd0xJgk=

3. Contract upload:

AAAAAgAAAADRjwIQ/2zB8tzxMB+71MMO4RoHWCBoTUcd+J0PEBHqKAABSZ8AAAANAAAAAQAAAAAAAAAAAAAAAQAAAAEAAAAA0Y8CEP9swfLc8TAfu9TDDuEaB1ggaE1HHfidDxAR6igAAAAYAAAAAgAAFsQAYXNtAQAAAAGFARZgAn5+AX5gBH5+fn4BfmADfn5+AX5gAX4BfmAFf39/f38AYAN/f38AYAJ/fwF+YAR/f39+AGABfwF+YAABfmAAAGABfgBgAX8Bf2ABfwBgBX9/f39/AX5gA39/fwF+YAN/fn4Bf2ADf35+AX5gBH9+fn4BfmACf34BfmACf38AYAF+AX8CNwkBYgFpAAABYgFqAAABdgFnAAABeAFfAAEBeAEyAAABbAFfAAIBbAEwAAABbAExAAABYQEwAAMDSEcEBAUGBQcFBQUGBQMDAwgAAAkKCQoDAwsFBQQFBQwNCgMGBgYGDg8PBQ0FDw0QEQUREggPDw8OERIRERMFFAUVFQMIDRQFFAQFAXABAwMFAwEAEQYZA38BQYCAwAALfwBB1IHAAAt/AEHggcAACwd4DAZtZW1vcnkCAAVoZWxsbwAUA25vdAAWBGF1dGgAGA5pbmNfcGVyc2lzdGVudAAaB2luY190bXAAHA5tdWx0aV93b3JkX2NtZAAeA2xvZwAfBXdvcmxkABQBXwAoCl9fZGF0YV9lbmQDAQtfX2hlYXBfYmFzZQMCCQgBAEEBCwJMTwrLI0d2AQF/I4CAgIAAQRBrIgUkgICAgAAgBSACNgIMIAUgATYCCCAAIAVBCGoQpoCAgAAiAjYCGCAAQQA2AhAgAEEMaiAENgIAIAAgAzYCCCAAIAUpAwg3AgAgACACIAQgA2siBCACIARJGzYCFCAFQRBqJICAgIAAC3kBAX8jgICAgABBEGsiBSSAgICAACAFIAI2AgwgBSABNgIIIAAgBUEIahCmgICAACICNgIYIABBADYCECAAQQxqIAQ2AgAgACADNgIIIAAgBSkDCDcCACAAIAIgBCADa0EDdiIEIAIgBEkbNgIUIAVBEGokgICAgAALXwEBfgJAAkACQCABIAEgAhCMgICAACIDQgEQtoCAgAANAEEAIQEMAQsgASADQgEQt4CAgAAiA0L/AYNCBFINASADQiCIpyECQQEhAQsgACACNgIEIAAgATYCAA8LAAALRQIBfwF+I4CAgIAAQRBrIgIkgICAgAAgAiAAIAEQs4CAgAACQCACKQMAp0UNAAAACyACKQMIIQMgAkEQaiSAgICAACADCxAAIAAgASACQgEQjoCAgAALJwAgACAAIAEQjICAgAAgAiAAEKqAgIAAIAMQuoCAgAAQqYCAgAAaC18BAX4CQAJAAkAgASABIAIQjICAgAAiA0IAELaAgIAADQBBACEBDAELIAEgA0IAELeAgIAAIgNC/wGDQgRSDQEgA0IgiKchAkEBIQELIAAgAjYCBCAAIAE2AgAPCwAACxAAIAAgASACQgAQjoCAgAAL5QEBA38jgICAgABBMGsiAySAgICAACADIAEgAhCSgICAADcDACADQYCAwIAAEMuAgIAANwMIIANBEGogA0EIaiADQQhqQQhqIAMgA0EIahCjgICAAEEAIAMoAiQiAiADKAIgIgRrIgUgBSACSxshAiADKAIQIARBA3QiBWohBCADKAIYIAVqIQUCQANAIAJFDQEgBCAFIAEQq4CAgAA3AwAgAkF/aiECIARBCGohBCAFQQhqIQUMAAsLIAAgASADQQhqQQEQsICAgAAQqYCAgAA3AwggAEIANwMAIANBMGokgICAgAALGgAgACABKAIAIAEoAgQQr4CAgAAQyoCAgAALpQIDAX8BfgJ/I4CAgIAAQcAAayIDJICAgIAAIAEgAhCMgICAACEEIAMgASACQQhqEIyAgIAANwMIIAMgBDcDAEEAIQJBgIDAgAAQy4CAgAAhBAJAA0AgAkEQRg0BIANBEGogAmogBDcDACACQQhqIQIMAAsLIANBIGogA0EQaiADQRBqQRBqIAMgA0EQahCjgICAAEEAIAMoAjQiAiADKAIwIgVrIgYgBiACSxshAiADKAIgIAVBA3QiBmohBSADKAIoIAZqIQYCQANAIAJFDQEgBSAGIAEQq4CAgAA3AwAgAkF/aiECIAVBCGohBSAGQQhqIQYMAAsLIAAgASADQRBqQQIQsICAgAAQqYCAgAA3AwggAEIANwMAIANBwABqJICAgIAAC2kBAX8jgICAgABBMGsiASSAgICAACABIAA3AxggAUEIaiABQShqIAFBGGoQpYCAgAACQCABKQMIp0UNAAAACyABIAEpAxAQlYCAgAA3AyAgAUEgahDLgICAACEAIAFBMGokgICAgAAgAAuVAgEEfyOAgICAAEHQAGsiASSAgICAACABIAA3AxAgAUKO6PHYugI3AwhBACECQYCAwIAAEMuAgIAAIQACQANAIAJBEEYNASABQRhqIAJqIAA3AwAgAkEIaiECDAALCyABQShqIAFBGGogAUEYakEQaiABQQhqIAFBCGpBEGoQioCAgABBACABKAI8IgIgASgCOCIDayIEIAQgAksbIQIgASgCKCADQQN0IgRqIQMgASgCMCAEaiEEAkADQCACRQ0BIAMgAUHIAGogBBCMgICAADcDACACQX9qIQIgA0EIaiEDIARBCGohBAwACwsgAUHIAGogAUEYakECELCAgIAAEKmAgIAAIQAgAUHQAGokgICAgAAgAAtiAQJ/I4CAgIAAQRBrIgEkgICAgAACQEEBIACnQf8BcSICQQBHQQF0IAJBAUYbIgJBAkcNAAAACyABIAJBAEcQl4CAgAA3AwggAUEIahDLgICAACEAIAFBEGokgICAgAAgAAvmAQIDfwF+I4CAgIAAQcAAayIBJICAgIAAIAEgAEEBczoADyABQYCAwIAAEMuAgIAANwMQIAFBGGogAUEQaiABQRBqQQhqIAFBD2ogAUEPakEBahCJgICAAEEAIAEoAiwiACABKAIoIgJrIgMgAyAASxshACABKAIgIAJqIQMgASgCGCACQQN0aiECAkADQCAARQ0BIAIgAyABQThqEKyAgIAANwMAIABBf2ohACACQQhqIQIgA0EBaiEDDAALCyABQThqIAFBEGpBARCwgICAABCpgICAACEEIAFBwABqJICAgIAAIAQLmgEBAX8jgICAgABBwABrIgIkgICAgAAgAiABNwMoIAIgADcDICACQRBqIAJBOGogAkEgahCxgICAAAJAIAIpAxCnDQAgAikDGCEBIAIgAkE4aiACQShqEKWAgIAAIAIpAwCnDQAgAiABIAIpAwgQmYCAgAA3AzAgAkEwaiACQThqEK2AgIAAIQEgAkHAAGokgICAgAAgAQ8LAAALogEBAX8jgICAgABBwABrIgIkgICAgAAgAiAANwMYIAJBGGoQsoCAgAAgAiABNwMoIAJBBDYCJCACQdWAwIAANgIgIAJBCGogAkE4aiACQSBqEJGAgIAAIAIgAikDEDcDMCACQThqIAJBMGoQu4CAgAAgAkE4aiACQShqEIyAgIAAELmAgIAAEKmAgIAAGiACKQMYIQAgAkHAAGokgICAgAAgAAsQABCbgICAAEICEMqAgIAAC84BAQJ/I4CAgIAAQSBrIgAkgICAgAAgAEEYahC1gICAACAAIABBGGpB4IDAgAAQi4CAgAAgACAAKAIEQQAgACgCABs2AgwgACAAQQxqIABBGGoQqoCAgAA3AxAgAEEYakHogMCAAEEJIABBEGpBARCugICAABoCQCAAKAIMQQFqIgFFDQAgACABNgIMIABBGGoQtYCAgAAgAEEYakHggMCAACAAQQxqEI2AgIAAIABBIGokgICAgAAPC0GQgcCAAEEcQfSAwIAAEM6AgIAAAAsQABCdgICAAEICEMqAgIAAC84BAQJ/I4CAgIAAQSBrIgAkgICAgAAgAEEYahC1gICAACAAIABBGGpB4IDAgAAQj4CAgAAgACAAKAIEQQAgACgCABs2AgwgACAAQQxqIABBGGoQqoCAgAA3AxAgAEEYakHogMCAAEEJIABBEGpBARCugICAABoCQCAAKAIMQQFqIgFFDQAgACABNgIMIABBGGoQtYCAgAAgAEEYakHggMCAACAAQQxqEJCAgIAAIABBIGokgICAgAAPC0GQgcCAAEEcQayBwIAAEM6AgIAAAAtSAQF/I4CAgIAAQSBrIgEkgICAgAAgASAANwMQIAEgAUEYaiABQRBqELiAgIAAAkAgASgCAEUNAAAAC0ICEMqAgIAAIQAgAUEgaiSAgICAACAAC14BAX8jgICAgABBIGsiASSAgICAACABIAA3AxAgASABQRhqIAFBEGoQpYCAgAACQCABKQMAp0UNAAAACyABKQMIEKCAgIAAQgIQyoCAgAAhACABQSBqJICAgIAAIAAL4wECAX8CfiOAgICAAEHAAGsiASSAgICAACABIAA3AxAgAUE4akHQgMCAAEEFELSAgIAAIQIgAUE4akG8gcCAAEEAELSAgIAAIQMgASAANwMoIAEgAzcDICABIAI3AxggASABQThqIAFBGGoQk4CAgAAgASABKQMINwMwIAFBOGogAUEwahC7gICAACABQThqIAFBKGoQjICAgAAQuYCAgAAQqYCAgAAaIAEgAUE4aiABQRBqEIyAgIAANwMYIAFBOGpBvIHAgABBCCABQRhqQQEQroCAgAAaIAFBwABqJICAgIAACx8AIAAgAjUCAEIghkIEhBDKgICAADcDCCAAQgA3AwALGQAgACACMQAAEMqAgIAANwMIIABCADcDAAtOACAAQQA2AhAgACADNgIIIAAgAjYCBCAAIAE2AgAgAEEMaiAENgIAIAAgAiABa0EDdiICNgIYIAAgAiAEIANrQQN2IgMgAiADSRs2AhQLagICfwF+I4CAgIAAQRBrIgMkgICAgAAgAyACKAIAIgQgAigCBCICEMeAgIAAAkACQCADKAIADQAgAykDCCEFDAELIAEgBCACEL2AgIAAIQULIAAgBTcDCCAAQgA3AwAgA0EQaiSAgICAAAs1AQJ+AkAgAikDACIDEMmAgIAAIgJFDQAgAxDKgICAACEECyAAIAQ3AwggACACQQFzrTcDAAsQACAAKAIEIAAoAgBrQQN2CwQAAAALAgALBAAgAAs3AgF/AX4jgICAgABBEGsiAiSAgICAACACIAIgABChgICAACACKQMIIQMgAkEQaiSAgICAACADCwcAIAApAwALNwIBfwF+I4CAgIAAQRBrIgIkgICAgAAgAiACIAAQooCAgAAgAikDCCEDIAJBEGokgICAgAAgAwsKACAAEMuAgIAACxIAIAAgASACIAMgBBC/gICAAAsOACAAIAEgAhC8gICAAAsOACAAIAEgAhC+gICAAAsgAQF+IAAgAikDACIDNwMIIAAgA0L/AYNCzQBSrTcDAAsQACAAIAApAwAQxICAgAAaCxYAIAAgAhDLgICAADcDCCAAQgA3AwALSwIBfwF+I4CAgIAAQSBrIgMkgICAgAAgAyACNgIcIAMgATYCGCADQQhqIAAgA0EYahCkgICAACADKQMQIQQgA0EgaiSAgICAACAECwIACxQAIAAgASACEMKAgIAAEMiAgIAACw4AIAAgASACEMOAgIAACyABAX4gACACKQMAIgM3AwggACADQv8Bg0LJAFKtNwMACw4AIAAgASACEMCAgIAACxAAIAAgASACIAMQwYCAgAALBwAgACkDAAsaACABrUIghkIEhCACrUIghkIEhBCAgICAAAsaACABrUIghkIEhCACrUIghkIEhBCBgICAAAsaACABrUIghkIEhCACrUIghkIEhBCCgICAAAssACABrUIghkIEhCACrUIghkIEhCADrUIghkIEhCAErUIghkIEhBCDgICAAAsMACABIAIQhICAgAALDgAgASACIAMQhYCAgAALDAAgASACEIaAgIAACwwAIAEgAhCHgICAAAsKACABEIiAgIAAC64BAgJ/AX4jgICAgABBEGsiAySAgICAAEEAIQRCACEFAkACQANAAkAgAiAERw0AIABBADYCACAAIAVCCIZCDoQ3AwgMAwsgBEEJRg0BIAMgASAEai0AABDGgICAAAJAIAMoAgANACAEQQFqIQQgAykDCCAFQgaGhCEFDAELCyAAIAMpAgQ3AgQgAEEBNgIADAELIABCATcDACAAQQhqIAI2AgALIANBEGokgICAgAALhgECAX4Bf0IBIQICQAJAIAFB3wBGDQACQCABQVBqQQpJDQACQCABQb9/akEaSQ0AAkAgAUGff2pBGkkNAEEBIQMgAEEBNgIEIABBCGogATYCAAwECyABrUJFfCECDAILIAGtQkt8IQIMAQsgAa1CUnwhAgsgACACNwMIQQAhAwsgACADNgIACw4AIAAgASACEMWAgIAACwcAIABCAVELFwEBfyAAp0H/AXEiAUEORiABQcoARnILBAAgAAsHACAAKQMACwIAC0wBAX8jgICAgABBIGsiAiSAgICAACACIAA2AhQgAkHEgcCAADYCDCACQcSBwIAANgIIIAJBAToAGCACIAE2AhAgAkEIahCngICAAAALVAEBfyOAgICAAEEgayIDJICAgIAAIANBDGpCADcCACADQQE2AgQgA0HEgcCAADYCCCADIAE2AhwgAyAANgIYIAMgA0EYajYCACADIAIQzYCAgAAACyEAIABCpdHlybrfzo7dADcDCCAAQuWr4ril5v/neDcDAAsL3gEBAEGAgMAAC9QBAgAAAAAAAABjbWQvY3JhdGVzL3Nvcm9iYW4tdGVzdC90ZXN0cy9maXh0dXJlcy90ZXN0LXdhc21zL2hlbGxvX3dvcmxkL3NyYy9saWIucnNoZWxsb2F1dGgAAAAAAAAADh30ZaDmAABjb3VudDoge30AAAAIABAASAAAACYAAAAJAAAAAAAAAAAAAAAAAAAAYXR0ZW1wdCB0byBhZGQgd2l0aCBvdmVyZmxvdwgAEABIAAAAMQAAAAkAAABoZWxsbyB7fQEAAAAAAAAAAQAAAAIAAAAAHhFjb250cmFjdGVudm1ldGF2MAAAAAAAAAAUAAAAOQBvDmNvbnRyYWN0bWV0YXYwAAAAAAAAAAVyc3ZlcgAAAAAAAAYxLjcyLjAAAAAAAAAAAAAIcnNzZGt2ZXIAAAAuMC45LjIjMThiOGZhMWEzNThhYTg0YWZkMTk2ZTJmNmQ0NDk0Mjc5OGM4ZjMzNQAAAMMDDmNvbnRyYWN0c3BlY3YwAAAAAAAAAAAAAAAFaGVsbG8AAAAAAAABAAAAAAAAAAV3b3JsZAAAAAAAABEAAAABAAAD6gAAABEAAAAAAAAAAAAAAAV3b3JsZAAAAAAAAAEAAAAAAAAABWhlbGxvAAAAAAAAEQAAAAEAAAPqAAAAEQAAAAAAAAAAAAAAA25vdAAAAAABAAAAAAAAAAdib29sZWFuAAAAAAEAAAABAAAD6gAAAAEAAAAAAAAAAAAAAARhdXRoAAAAAgAAAAAAAAAEYWRkcgAAABMAAAAAAAAABXdvcmxkAAAAAAAAEQAAAAEAAAATAAAAAAAAAAAAAAAOaW5jX3BlcnNpc3RlbnQAAAAAAAAAAAAAAAAAAAAAAAAAAAAHaW5jX3RtcAAAAAAAAAAAAAAAAAAAAAAAAAAADm11bHRpX3dvcmRfY21kAAAAAAABAAAAAAAAAA5jb250cmFjdF9vd25lcgAAAAAAEAAAAAAAAAAAAAAAJUxvZ3MgYSBzdHJpbmcgd2l0aCBgaGVsbG8gYCBpbiBmcm9udC4AAAAAAAADbG9nAAAAAAEAAAAAAAAAA3N0cgAAAAARAAAAAAAAAAAAAAABAAAAAAAAAAAAAAABAAAABxby6OgWX9//BhHjHPIxoimYLbIWzFlbsBNcg3Zy4FsXAAQhvgAAADAAABb4AAAAAAAATlgAAAABEBHqKAAAAEAzHyYM+hWR+Zsn0CtdZpoaBfRxzh0vtbcc4VKXDl6hy3Fse9rCRcAED06QSZN4VX3wS9sH7bGMs4ZnWMgROFYC

4. Contract creation:

AAAAAgAAAADRjwIQ/2zB8tzxMB+71MMO4RoHWCBoTUcd+J0PEBHqKAABHvkAAAANAAAAAgAAAAAAAAAAAAAAAQAAAAAAAAAYAAAAAQAAAAAAAAAAAAAAANGPAhD/bMHy3PEwH7vUww7hGgdYIGhNRx34nQ8QEeooAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFvLo6BZf3/8GEeMc8jGiKZgtshbMWVuwE1yDdnLgWxcAAAABAAAAAAAAAAEAAAAAAAAAAAAAAADRjwIQ/2zB8tzxMB+71MMO4RoHWCBoTUcd+J0PEBHqKAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABby6OgWX9//BhHjHPIxoimYLbIWzFlbsBNcg3Zy4FsXAAAAAAAAAAEAAAAAAAAAAQAAAAcW8ujoFl/f/wYR4xzyMaIpmC2yFsxZW7ATXIN2cuBbFwAAAAEAAAAGAAAAAQasoE+Ur1tXsNLuo9BRkUwdsK6/9D6UEMF/WSEhBvK4AAAAFAAAAAEAAcLfAAAXWAAAAGgAAAAAAABOWAAAAAEQEeooAAAAQBL4AUjw+kKDiktiO7hKmrrJFU2M87/6se4PMkn+stpqCAn9Zk0j+D205x74irmNLEX6DnNd+7Sekw2GCOk4Hgc=

I didn't see any meaningful information in the core logs: core-logs.txt

[2opremio]

It seems to have worked once! https://github.com/stellar/soroban-tools/actions/runs/6149908369/job/16686748093?pr=942

So maybe it is a concurrency problem.

But it still fails locally for me.

[2opremio]

Here are the Core debug logs: core-logs-debug.txt

[2opremio]

Here is the meaningful part of the logs:

2023-09-11T19:40:28.293 GD5KD [Tx INFO] applying ledger 17 (classic phase: 0 component(s): [], soroban phase: 1 component(s): [{discounted txs:1, ops:1, base_fee:100}])
2023-09-11T19:40:28.294 GD5KD [Tx DEBUG]  tx#0 = 935a1d ops=1 txseq=55834574850 (@ GDIY6)
2023-09-11T19:40:28.314 GD5KD [Ledger DEBUG] Ledger: 17 txs: 1, ops: 1, prefetch hit rate (%): 0
2023-09-11T19:40:28.319 GD5KD [Bucket DEBUG] Adopting bucket file buckets/tmp/bucket-0af91ebc98c2ef6d/tmp-bucket-1f6ead688f743a2e.xdr as buckets/bucket-0415c30a7fa5582dd7a27a505282cb339c8a30625055b1e9375f10ac9267c631.xdr
2023-09-11T19:40:28.321 GD5KD [Bucket DEBUG] Deleting bucket file buckets/tmp/bucket-0af91ebc98c2ef6d/tmp-bucket-54dd1ef87af15e42.xdr that is redundant with existing bucket
2023-09-11T19:40:28.321 GD5KD [Perf DEBUG] Bucket merge on level 0 finished in 0.001333583 seconds (0.13335829999999999% of available time)
2023-09-11T19:40:28.323 GD5KD [Ledger DEBUG] Advancing LCL: [seq=16, hash=278066] -> [seq=17, hash=b69244]
2023-09-11T19:40:28.328 GD5KD [Perf DEBUG] Applied ledger in 0.036461917 seconds
2023-09-11T19:40:28.328 GD5KD [Ledger INFO] Closed ledger: [seq=17, hash=b69244]
2023-09-11T19:40:28.328 GD5KD [Ledger DEBUG] LedgerManager::valueExternalized LCL advanced 16 -> 17
2023-09-11T19:40:28.328 GD5KD [Tx DEBUG] Dropping de59fa transaction
2023-09-11T19:40:28.328 GD5KD [Tx DEBUG] Ban applied transaction de59fa

It seems the transaction is dropped/banned (Whatever that means) The affected transaction has hash 935a1d827f01e8cb5b379537f24ad4fcc4e407695926334e203e9b3c33d93870

[2opremio]

OK, I think I know what is happening.

After the last transaction is sent, soroban-rpc stops ingesting ledgers. I don't know if it's due to a problem in soroban-rpc or due to a problem in captive core.

The last ingestion logs are:

time="2023-09-11T22:10:33.601+02:00" level=info msg="Ledger: Closed ledger: [seq=16, hash=ae68cb]" pid=66079 subservice=stellar-core
time="2023-09-11T22:10:33.602+02:00" level=debug msg="Ingested ledger 16" pid=66079
time="2023-09-11T22:10:33.602+02:00" level=info msg="Ingesting ledger 17" pid=66079

(After this there aren't any more Ingested ledger or Ledger: Closed ledger: logs

But ... the last transaction is included in ledger 17 acording to the Core logs.

My guess is that, for some reason, stellar-core stops producing meta.

[2opremio]

I can confirm that we get stuck at s.ledgerBackend.GetLedger(ctx, 16) sometimes at s.ledgerBackend.GetLedger(ctx, 17)

[2opremio]

It turns out the problem was caused by a mismatch in the commit of Core vs Captive Core.