Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge main #118

Merged
merged 98 commits into from
Sep 8, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
984bde7
Apply custom filter to logs in log recoverer getLogTriggerCheckData (…
infiloop2 Aug 29, 2023
20e2692
[Functions] Fix allowlist V1 updates (#10375)
bolekk Aug 29, 2023
e439b0d
[TT-555] Fix Soak Reporting Bug in OCR (#10357)
kalverra Aug 29, 2023
d6ea7b6
vrfv2+: various audit fixes (#10328)
makramkd Aug 29, 2023
e8dc3ea
chore: skip test (#10384)
makramkd Aug 29, 2023
b30f585
(chore): Update Functions Scripts oracle.toml template (#10387)
justinkaseman Aug 29, 2023
c9e87f3
Use correct error type in DecryptionQueue (#10389)
bolekk Aug 29, 2023
567bc03
update to toml path in integration tests (#10374)
AnieeG Aug 29, 2023
093f570
moving pg and mockserver to CTF (#10372)
AnieeG Aug 29, 2023
c1e0ff6
(fix): Functions OracleWithdrawAll uses transmitters (#10392)
justinkaseman Aug 30, 2023
dcfb03f
[AUTO-4758] Update Automation benchmark test to support v2.1 (#10365)
anirudhwarrier Aug 30, 2023
2448724
Revert "Scroll update toml config (#10243)" (#10380)
mohamed-mehany Aug 30, 2023
65c60be
Transmit event provider: fix CPU utilization (#10385)
amirylm Aug 30, 2023
90d6307
Use -1 as sentinel value for missing Link/Native fee (#10383)
samsondav Aug 30, 2023
3414320
move len check to match previous behavior (#10405)
krehermann Aug 30, 2023
64395bc
update VL contract to check upkeep id with check data (#10233)
FelixFan1992 Aug 30, 2023
b41e626
BCF 2440 remove chainsets from core (#10349)
krehermann Aug 30, 2023
76999e5
Read logs for block number and trigger config (#10354)
ferglor Aug 30, 2023
34dfb21
.github/workflows: fix ci-core slack race notification url (#10408)
jmank88 Aug 30, 2023
b218430
Update solana to take BCF-2562: Relayer dependent on chain not chains…
krehermann Aug 30, 2023
a17b55e
core: logger name cleanup (#10414)
jmank88 Aug 31, 2023
3a7fc9d
add ineligibility reason for simulation failure (#10407)
FelixFan1992 Aug 31, 2023
92845b4
Bump sonarsource/sonarqube-scan-action from 2.0.0 to 2.0.1 (#10390)
dependabot[bot] Aug 31, 2023
02ffee4
Bump actions/checkout from 3.5.3 to 3.6.0 (#10346)
dependabot[bot] Aug 31, 2023
1a9154b
github.com/prometheus/prometheus v0.46.0 (#10418)
jmank88 Aug 31, 2023
aa888f1
chore: remove the word chainset from variables and comments (#10421)
krehermann Aug 31, 2023
876e81d
golangci-lint (#10420)
jmank88 Aug 31, 2023
4e45352
Fix recoverer cleanup (#10417)
amirylm Aug 31, 2023
f2f6c09
Cleanup warn log in registry check pipeline (#10412)
infiloop2 Aug 31, 2023
d4609e9
evm_transfer cli: wait for and output TxAttempt (#10371)
aalu1418 Aug 31, 2023
14edc7c
track deleted and revoked job proposal counts (#10424)
eutopian Aug 31, 2023
74596ea
Chore/vrf 584 add vrfv2plus ctf test native billing (#10386)
iljapavlovs Aug 31, 2023
37d6bda
fix test & linter issue with loop variable. (#10426)
krehermann Aug 31, 2023
38b462f
Optimise log poller replay (#10428)
infiloop2 Aug 31, 2023
68e2197
support operator forwarder for ocr2 feeds manager (#10425)
eutopian Aug 31, 2023
8728bb2
remove wait since offchain can handle earlier kicks (#10369)
shileiwill Aug 31, 2023
98cb209
rename FeedLookup to StreamsLookup (#10388)
FelixFan1992 Sep 1, 2023
feb86e4
refactor link transfer (#10373)
shileiwill Sep 1, 2023
4292b6a
Update ocr2keepers & small change to reduce LogBackfillBuffer (#10439)
infiloop2 Sep 1, 2023
08096d2
Move Uptime feeds contracts to a new folder (#10404)
simsonraj Sep 1, 2023
c206d9c
fix llo-feeds imports (#10441)
RensR Sep 1, 2023
0ab6fef
Disable unused telemetry types (#10443)
skubakdj Sep 1, 2023
748538f
[VRF-592] Add LINK token check (#10422)
vreff Sep 1, 2023
fc522f9
Update all deps to take removal of relayer.ChainSet interface (#10445)
krehermann Sep 1, 2023
a65e238
Tt 562 functions + gateway performance tests, chaos suite (#10397)
skudasov Sep 1, 2023
11490c6
Add regex to include config toml files (#10440)
davidcauchi Sep 1, 2023
50d5570
[TT-549] Renable Live Test Runs (#10379)
kalverra Sep 1, 2023
ceea5fb
remove check for finality depth in bhs and bhf. Do not update interna…
jinhoonbang Sep 2, 2023
2d574be
Upgrade forge-std to 1.6.0 (#10459)
justinkaseman Sep 2, 2023
5b78a81
add foundry refresh make command (#10442)
RensR Sep 2, 2023
6954c90
Add debug logs when updating lastPoll / lastRepoll block (#10461)
infiloop2 Sep 2, 2023
2837baf
bump pyroscope; use new import paths (#10463)
jmank88 Sep 3, 2023
528dfc6
[AUTO-4790] increase delta stage for integration-tests (#10398)
anirudhwarrier Sep 4, 2023
39f7cde
chore: skip ocr2vrf tests (#10448)
makramkd Sep 4, 2023
fe6fd64
Fix bug in updating last poll block for filter (#10462)
infiloop2 Sep 4, 2023
f07c491
chore: fix shared snapshot (#10471)
makramkd Sep 4, 2023
200970a
Show job runs for disabled chains (#10401)
george-dorin Sep 4, 2023
2afd9c2
Return all logs even if seen before (#10473)
ferglor Sep 4, 2023
c38b862
Add RequestID field to Transactions (#10363)
amit-momin Sep 5, 2023
0e2d15a
Increase block time sample size (#10482)
infiloop2 Sep 5, 2023
08e1bae
Small cleanup in event provider for caching behaviour (#10483)
infiloop2 Sep 5, 2023
5aadc9c
bump deps google/uuid, shirou/gopsutil/v3, avast/retry-go/v4, jackc/p…
jmank88 Sep 5, 2023
c40169c
fix/gh-actions-setup-postgres: fixing script to be resilient to array…
patrickhuie19 Sep 6, 2023
19eb5fc
core: use loop.Keystore, support arbitrarily prefixed Cosmos addresse…
cfal Sep 6, 2023
43030bd
shadow: declaration of err shadows declaration at line * (govet) (#10…
jmank88 Sep 6, 2023
3bf936a
HasFilter to check whether some filter was loaded by the poller (#10469)
amirylm Sep 6, 2023
94d05b7
Allow burst in log recoverer when it lags (#10479)
infiloop2 Sep 6, 2023
e43bdf4
Update ocr2keepers v0.7.20 (#10486)
amirylm Sep 6, 2023
3c9cf01
Remove explicit checkblock too old check and rely on rpc, add a check…
infiloop2 Sep 6, 2023
37f3880
Using minmax heap for Mercury transmission queue to evict oldest tran…
sdrug Sep 6, 2023
1176a8c
Upkeep filters life cycle: avoid replying logs for existing filters (…
amirylm Sep 6, 2023
2ff674b
Additional even emitting + sanity checks + version (#10419)
Fletch153 Sep 6, 2023
da3e611
[VRF] opt out of snapshot checks (#10491)
vreff Sep 6, 2023
a53b972
BCF-2564: take new loop relayer interface (#10430)
krehermann Sep 6, 2023
b5e6207
golangci-lint tweaks (#10510)
jmank88 Sep 6, 2023
a15bcd8
[Automation] Update docker scripts (#10516)
vreff Sep 6, 2023
ad22c6e
Seed order to manage logs overflow (#10485)
amirylm Sep 6, 2023
6f8a302
BCF-2497: Refactoring sessions orm query pattern to avoid a confusing…
patrickhuie19 Sep 6, 2023
0e3b379
Switch integration tests to nightly (#10520)
davidcauchi Sep 6, 2023
a0e97cf
TT-562 Improved load test client contract (#10504)
KuphJr Sep 6, 2023
c3d94fa
Added log for decryption queue error (#10517)
KuphJr Sep 6, 2023
2c0cb9c
(test): Add Functions Subscriptions foundry tests (#10435)
justinkaseman Sep 6, 2023
3a98db6
Add batch size backoff in log poller insert logs (#10525)
infiloop2 Sep 6, 2023
a1040eb
address lingering clean up and todos after BCF-2441 & BCF-2564 (#10521)
krehermann Sep 6, 2023
a87a7e6
Resolve TODOs in registry and provider life cycle (#10477)
infiloop2 Sep 6, 2023
55be162
(feat): Add Gateways to Functions core/scripts oracle.toml template (…
justinkaseman Sep 6, 2023
3dfb527
Update Simulated Backend Client (#10403)
EasterTheBunny Sep 6, 2023
e5487f0
Create CI checks for GQL schema changes (#7492)
HenryNguyen5 Sep 6, 2023
a7dc6f1
(test): Add ToS Allow List foundry tests (#10438)
justinkaseman Sep 6, 2023
8f61ff2
Fix data race in PersistenceManager tests (#10394)
martin-cll Sep 7, 2023
9586f07
Upsert only if we filter store had a registered filter before (#10531)
infiloop2 Sep 7, 2023
f6586c0
chore: add subscription api forge tests (#10446)
makramkd Sep 7, 2023
dcf265d
add more info to pipeline transmitter error (#10537)
infiloop2 Sep 7, 2023
e728e46
Merge remote-tracking branch 'upstream/develop' into merge-main
RensR Sep 7, 2023
847fd9e
fix Ptr ref & toml
RensR Sep 7, 2023
f605991
fixing smoke tests
AnieeG Sep 7, 2023
842ddd2
turn off broken functions tests
RensR Sep 8, 2023
c1eecab
turn off GQL checks that are missing permissions
RensR Sep 8, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Read logs for block number and trigger config (#10354)
* Read logs for block number and trigger config

* Use indexed logs

* Cleanup

* Reduce confs

* Update test coverage

* Continue if id cannot be parsed

* Please the linter

* Batch the log trigger upkeep refresh

* Batch the upkeep refresh

* No longer use errgroup

* Dont error on a single batch, add a processing delay, add comments

* Update tests

* Add a todo

* Refresh active upkeeps before batching

* update comment

* Only take max values

* Filter log upkeeps at a higher level

* Refresh after collecting the log upkeep IDs, generate hashes
  • Loading branch information
ferglor authored Aug 30, 2023
commit 76999e58a4b2f52a5a21f842bd015d7094c2168e
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ var (
)

func (p *logEventProvider) RefreshActiveUpkeeps(ids ...*big.Int) ([]*big.Int, error) {
// Exploratory: investigate how we can batch the refresh
if len(ids) == 0 {
return nil, nil
}
Expand Down
107 changes: 92 additions & 15 deletions core/services/ocr2/plugins/ocr2keeper/evm21/registry.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ import (
"github.com/ethereum/go-ethereum/accounts/abi/bind"
"github.com/ethereum/go-ethereum/common"
coreTypes "github.com/ethereum/go-ethereum/core/types"
"github.com/pkg/errors"

"github.com/patrickmn/go-cache"
"github.com/pkg/errors"
ocr2keepers "github.com/smartcontractkit/ocr2keepers/pkg/v3/types"
"go.uber.org/multierr"

"github.com/smartcontractkit/chainlink/v2/core/chains/evm"
"github.com/smartcontractkit/chainlink/v2/core/chains/evm/client"
Expand All @@ -37,6 +37,9 @@ const (
defaultAllowListExpiration = 20 * time.Minute
// allowListCleanupInterval decides when the expired items in allowList cache will be deleted.
allowListCleanupInterval = 5 * time.Minute
// TODO decide on a value for this
indexedLogsConfirmations = 10
logTriggerRefreshBatchSize = 32
)

var (
Expand Down Expand Up @@ -270,34 +273,108 @@ func (r *EvmRegistry) refreshActiveUpkeeps() error {
}
r.active.Reset(ids...)

return r.refreshLogTriggerUpkeeps(ids)
}

// refreshLogTriggerUpkeeps refreshes the active upkeep ids for log trigger upkeeps
//
// TODO: check for updated config for log trigger upkeeps and update it, currently we ignore them.
func (r *EvmRegistry) refreshLogTriggerUpkeeps(ids []*big.Int) error {
logTriggerIDs := make([]*big.Int, 0)
var logTriggerIDs []*big.Int
for _, id := range ids {
uid := &ocr2keepers.UpkeepIdentifier{}
if ok := uid.FromBigInt(id); !ok {
r.lggr.Warnf("failed to parse upkeep id %s", id.String())
continue
}
switch core.GetUpkeepType(*uid) {
case ocr2keepers.LogTrigger:
logTriggerIDs = append(logTriggerIDs, id)
default:
}
}

newUpkeeps, err := r.logEventProvider.RefreshActiveUpkeeps(logTriggerIDs...)
if err != nil {
return fmt.Errorf("failed to refresh active upkeep ids in log event provider: %w", err)
}

return r.refreshLogTriggerUpkeeps(newUpkeeps)
}

// refreshLogTriggerUpkeeps refreshes the active upkeep ids for log trigger upkeeps
//
// TODO: check for updated config for log trigger upkeeps and update it, currently we ignore them.
func (r *EvmRegistry) refreshLogTriggerUpkeeps(ids []*big.Int) error {
var err error
for i := 0; i < len(ids); i += logTriggerRefreshBatchSize {
end := i + logTriggerRefreshBatchSize
if end > len(ids) {
end = len(ids)
}
idBatch := ids[i:end]

if batchErr := r.refreshLogTriggerUpkeepsBatch(idBatch); batchErr != nil {
multierr.AppendInto(&err, batchErr)
}

time.Sleep(500 * time.Millisecond)
}

return err
}

func (r *EvmRegistry) refreshLogTriggerUpkeepsBatch(logTriggerIDs []*big.Int) error {
var logTriggerHashes []common.Hash
for _, id := range logTriggerIDs {
logTriggerHashes = append(logTriggerHashes, common.BigToHash(id))
}

unpausedLogs, err := r.poller.IndexedLogs(iregistry21.IKeeperRegistryMasterUpkeepUnpaused{}.Topic(), r.addr, 1, logTriggerHashes, indexedLogsConfirmations, pg.WithParentCtx(r.ctx))
if err != nil {
return err
}
configSetLogs, err := r.poller.IndexedLogs(iregistry21.IKeeperRegistryMasterUpkeepTriggerConfigSet{}.Topic(), r.addr, 1, logTriggerHashes, indexedLogsConfirmations, pg.WithParentCtx(r.ctx))
if err != nil {
return err
}

logs := append(unpausedLogs, configSetLogs...)

configSetBlockNumbers := map[string]uint64{}
unpausedBlockNumbers := map[string]uint64{}
perUpkeepConfig := map[string][]byte{}

for _, log := range logs {
rawLog := log.ToGethLog()
abilog, err := r.registry.ParseLog(rawLog)
if err != nil {
return err
}
switch l := abilog.(type) {
case *iregistry21.IKeeperRegistryMasterUpkeepTriggerConfigSet:
if rawLog.BlockNumber > configSetBlockNumbers[l.Id.String()] {
configSetBlockNumbers[l.Id.String()] = rawLog.BlockNumber
perUpkeepConfig[l.Id.String()] = l.TriggerConfig
}
case *iregistry21.IKeeperRegistryMasterUpkeepUnpaused:
if rawLog.BlockNumber > unpausedBlockNumbers[l.Id.String()] {
unpausedBlockNumbers[l.Id.String()] = rawLog.BlockNumber
}
}
}

var merr error
for _, id := range newUpkeeps {
// TODO: find the ConfigSet/UpkeepUnpaused events for this upkeep and pass cfg and block number
// block number should be taken from UpkeepUnpaused if it's block is higher than ConfigSet
if err := r.updateTriggerConfig(id, nil, 0); err != nil {
for _, id := range logTriggerIDs {
logBlock, ok := configSetBlockNumbers[id.String()]
if !ok {
r.lggr.Warnf("unable to find config set block number for %s", id.String())
continue
}

config, ok := perUpkeepConfig[id.String()]
if !ok {
r.lggr.Warnf("unable to find per upkeep config for %s", id.String())
continue
}

// In case an upkeep was paused then unpaused after a config set event, start the config from the unpaused block number
if unpausedBlockNumbers[id.String()] > logBlock {
logBlock = unpausedBlockNumbers[id.String()]
}
if err := r.updateTriggerConfig(id, config, logBlock); err != nil {
merr = goerrors.Join(merr, fmt.Errorf("failed to update trigger config for upkeep id %s: %w", id.String(), err))
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -208,12 +208,17 @@ func TestRegistry_VerifyCheckBlock(t *testing.T) {
type mockLogPoller struct {
logpoller.LogPoller
GetBlocksRangeFn func(ctx context.Context, numbers []uint64, qopts ...pg.QOpt) ([]logpoller.LogPollerBlock, error)
IndexedLogsFn func(eventSig common.Hash, address common.Address, topicIndex int, topicValues []common.Hash, confs int, qopts ...pg.QOpt) ([]logpoller.Log, error)
}

func (p *mockLogPoller) GetBlocksRange(ctx context.Context, numbers []uint64, qopts ...pg.QOpt) ([]logpoller.LogPollerBlock, error) {
return p.GetBlocksRangeFn(ctx, numbers, qopts...)
}

func (p *mockLogPoller) IndexedLogs(eventSig common.Hash, address common.Address, topicIndex int, topicValues []common.Hash, confs int, qopts ...pg.QOpt) ([]logpoller.Log, error) {
return p.IndexedLogsFn(eventSig, address, topicIndex, topicValues, confs, qopts...)
}

func TestRegistry_VerifyLogExists(t *testing.T) {
lggr := logger.TestLogger(t)
upkeepId := ocr2keepers.UpkeepIdentifier{}
Expand Down
Loading