feat(provider): improve pattern detection (#217)

[andy108369]

Updated run.sh script logic worked: The provider-services got restarted as expected upon hitting account sequence mismatch.

Pod logs got preserved by the K8s throughout pod's lifetime as expected.

• "account sequence mismatch" was detected and provider-services got restarted within the pod:

# kubectl -n akash-services logs akash-provider-0 --tail=1000 -f
...
D[2023-11-23|13:18:36.030] submitting fulfillment                       module=bidengine-order cmp=provider order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13808453/1/1 price=0.896297000000000000uakt
E[2023-11-23|13:18:36.034] transaction broadcast failed                 cmp=provider cmp=client/broadcaster err="rpc error: code = Unknown desc = rpc error: code = Unknown desc = account sequence mismatch, expected 80132, got 80130: incorrect account sequence [cosmos/[email protected]/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '47692' : unknown request"
Pattern 'account sequence mismatch' found. Restarting provider-services...

• logs preserved (since Pod wasn't restarted itself):

# crictl ps |grep provider
7c7afee8edef7       a3b0b89a6048e       6 minutes ago        Running             provider                   1                   cd8c53467d2cc       akash-provider-0

# ls -latr /var/log/pods/akash-services_akash-provider-0_8b31c6f6-f3c5-4cd6-b052-12dd8a171bb4/provider/*
-rw-r----- 1 root root 147727 Nov 23 13:18 /var/log/pods/akash-services_akash-provider-0_8b31c6f6-f3c5-4cd6-b052-12dd8a171bb4/provider/0.log
-rw-r----- 1 root root 303228 Nov 23 13:26 /var/log/pods/akash-services_akash-provider-0_8b31c6f6-f3c5-4cd6-b052-12dd8a171bb4/provider/1.log

# cat /var/log/pods/akash-services_akash-provider-0_8b31c6f6-f3c5-4cd6-b052-12dd8a171bb4/provider/0.log | tail -2
2023-11-23T13:18:36.034485314Z stdout F E[2023-11-23|13:18:36.034] transaction broadcast failed                 cmp=provider cmp=client/broadcaster err="rpc error: code = Unknown desc = rpc error: code = Unknown desc = account sequence mismatch, expected 80132, got 80130: incorrect account sequence [cosmos/[email protected]/x/auth/ante/sigverify.go:265] With gas wanted: '0' and gas used: '47692' : unknown request"
2023-11-23T13:18:36.034509194Z stdout F Pattern 'account sequence mismatch' found. Restarting provider-services...

• pod briefly went into the Error state as "account sequence mismatch" was detected and thus provider-services service was restarted:

$ kubectl -n akash-services get pods
NAME                                        READY   STATUS    RESTARTS        AGE
...
akash-provider-0                            0/1     Error     0               2m45s

$ kubectl -n akash-services get pods
NAME                                        READY   STATUS    RESTARTS          AGE
...
akash-provider-0                            1/1     Running   1 (9m31s ago)     12m

• pod's exit code was "2" as expected - the "run.sh" logic exits with "2" when finds provider hits "account sequence mismatch"

$ kubectl -n akash-services describe pod akash-provider-0
...
    State:          Running
      Started:      Thu, 23 Nov 2023 14:18:39 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Thu, 23 Nov 2023 14:16:07 +0100
      Finished:     Thu, 23 Nov 2023 14:18:39 +0100