Add Topic1, Topic2, Topic3, LogsPerBlock fields to logpoller.Filter and log_poller_filters schema

[reductionista]

Also in this PR:

• Converted all cases where a logpoller.Filter object is constructed over to using field names instead of positional arguments. This should make it much easier to add more fields in the future, and allow default values for uninteresting fields during tests to be filled in automatically.
• Handle NULL HashArray and AddressArray reads from db by setting dest to nil instead of erroring out.
• Bug fix to SQL QUERY logging (handles NULL and empty array values for BYTEA[] now, instead of pancing)
• Minor improvements to SQL QUERY logging, so it's closer to an actual SQL query you can run via cut and paste
• Add test for writing and reading back in filters from db, with various combinations of topic filters set

[github-actions]

I see that you haven't updated any README files. Would it make sense to do so?

[mateusz-sekara]

Correct me if I'm wrong, but I don't see any usage of these filters during PollAndSave cycle. IIUC goal of this PR is to support filters with extended definition, but probably code responsible for using them during PollAndSave would be in a separate PR, right?

[mateusz-sekara]

Does Goose run all these DDLs in a single TX? If not, there is an edge in which you drop constraint, malicious TX violating that constraint gets in, and then you bring back a new constraint. Not sure what's the probability of this happening, but probably sth like this could be safer

ALTER TABLE evm.log_poller_filters
    ADD COLUMN topic2 BYTEA CHECK (octet_length(topic2) = 32),
    ADD COLUMN topic3 BYTEA CHECK (octet_length(topic3) = 32),
    ADD COLUMN topic4 BYTEA CHECK (octet_length(topic4) = 32),
    ADD COLUMN max_logs_kept NUMERIC(78,0),
    ADD COLUMN logs_per_block NUMERIC(78,0);

CREATE UNIQUE INDEX evm_log_poller_filters_name_chain_address_event_topics_key ON evm.log_poller_filters (hash_record_extended((name, evm_chain_id, address, event, topic2, topic3, topic4), 0));

ALTER TABLE evm.log_poller_filters
    DROP CONSTRAINT evm_log_poller_filters_name_evm_chain_id_address_event_key;

[reductionista]

I don't think the db migrations run concurrently with any other tx's, it should just run in isolation before the node starts up. But sure, I can change the order seems like good practice.

[mateusz-sekara]

I totally understand the max_logs_kept argument. It might be useful for some filters for CCIP in which we are interested in the latest logs, so it's better to rely on the number of logs instead of duration retention. However, I wonder what's the use case

[reductionista]

All of the changes in this PR are aimed at addressing security concerns with Automation's new LogTriggers feature. They allow external users to track any logs on any smart contract, so without this it would open the door to a DoS attack where someone tells us to save every log from an ulta high volume smart contract, filling up the disk and halting operation.

The LogsPerBlock limit is less important, but it will be there as a safeguard against overloading the network connection with a high burst of logs all in one block. (They verified by writing a test contract that there is no limit on the number of logs you can emit at once, they showed me an example where they emitted 16,000 logs in one tx.)

[mateusz-sekara]

10^77 of logs? Does it make sense? Isn't regular bigint just good enough? I can't imagine a case in which someone defines that he/she wants to keep 10^18 logs or sth :D

[mateusz-sekara]

I believe this is gonna be used during PruneOldLogs, right? Counting during delete might be heavy, have you thought about the delete query that would respect that value?

[reductionista]

I haven't thought about that yet, but I suspect we can find a way to make it efficient. If pruning all filters at once is a problem, one thing we could do is do one filter at a time... so that only the rows for a single filter are locked at once while we count & prune them.

[reductionista]

Correct me if I'm wrong, but I don't see any usage of these filters during PollAndSave cycle. IIUC goal of this PR is to support filters with extended definition, but probably code responsible for using them during PollAndSave would be in a separate PR, right?

That's right. There are two followup PR's to make use of these extra fields. One of them changes the way in which logs are requested (instead of a single request with a big combined filter, it's a batch of requests sent all at once, normally 1 batch element per filter but with some optimizations for combining requests. This one is in progress, and there will be another PR after that for implementing the rate limiting & MaxLogsKept feature. Since both of these PR's required extending the logpoller.Filter object and migrating the db, I figured it made sense to combine that part of it as a common precursor to these changes.

[mateusz-sekara]

lgtm!

[cl-sonarqube-production]

Quality Gate passed

Issues
1 New issue
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
90.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube