Conform grafting auto-resolve feature to latest main

[tilacog]

No description provided.

[tilacog]

This is an update on the current status of this PR, which is still a work in progress.

This branch derives from #481, and the very first set of commits had the purpose of bringing it back to the latest main.

Changes so far

The main changes are the introduction of two new modules in indexer-common:

• grafting.ts
• subgraph-deployment.ts

The latter exposes a deploySubgraph function, which aims to replace calls to the GraphNode.ensure method. This method is called only if the new function detects that an incoming subgraph deployment does not have graft bases.

That's for the public items of this PR.

Internally, the grafting module contains types and functions:

• iteratively fetch subgraph manifests on IPFS.
• represent graft bases and subgraph graft lineage.
• resolve the (possible) indexing status of such graft bases in Graph-Node.
• produce deployment decisions out of a subgraph lineage.
• create and remove temporary indexing rules to support syncing graft bases.

All of this is used by the public deploySubgraph function.

Usage

This PR only replaced the call to GraphNode.ensure() for deploySubgraph in the reconcileDeployments function.

It seems like the use of ensure in specific cases, like deploying the Network Subgraph, is intentional. Our primary focus was on the reconciliation loop itself.

Current Status and Observations

deploySubgraph is being called as expected, and grafted subgraphs had their subgraph lineage properly resolved.

The current design might contain flaws because the target (grafted) subgraph reentries the reconciliation loop after having its graft bases resolved. Also, all graft bases reenter the reconciliation loop, as new (temporary) indexing rules are created for them during deploySubgraph execution.

Besides that, I found a bug in which the program tried to deploy a grafted subgraph (still unsure if it was the target or one of its bases) while its base was still syncing behind its expected block height.

The strategy so far has been inspecting logs segmented by each reconciliation loop as to observe changes on the selected subgraph deployment sets.

My last effort was focused on upgrading our logs for the SubgraphDeploymentID type because those don't reveal their IPFS hash when printed to the console, but a different representation is shown.

Problems with the current Indexing Rules design

We need a value for the temporary indexing rule's protocolNetwork field.

Ideally, we would use the same protocol network as the target subgraph, but that information is erased in the previous steps of the reconciliation loop.

Possible fixes are:

• Using a placeholder CAIP2-ID for offchian indexing rules, such as offchain:0, and adjusting the input validation components to allow for that exception.
• Remove the NOT NULL constraint from the protocolNetwork column in the IndexingRules table so off-chain indexing rules can be excused from having such information. I've sketched a migration in 03e8b19 but removed it in ae4a62d, as further work was necessary to accommodate those changes.
• Refactor the reconciliation loop operations to carry the original protocolNetwork over to the deployment reconciliation routine, but that might not be semantically accurate.

Alternative Designs

This might not be the best design to solve this problem.

The reconciliation loop holds no state in the application memory, and all data is collected fresh at every iteration. This turned out to be a problem, as the target (grafted) subgraph deployment indexing rule stays active until all the graft base resolution and the same is true for all graft bases except for the root.

Ideally, the Agent could use a high-level component that would be aware of grafted subgraph deployment requests and state-fully orchestrate the deployment of the subgraph lineage without resorting to the indexing rules system.

[fordN]

We may revive this later, but ideally this will be solved in graph-node.