| CHIP Number | 0022 |
|---|---|
| Title | Enhanced Harvester Protocol |
| Description | A protocol for communication between closed-source harvesters and the open-source farmer |
| Author | Harold Brenes |
| Editor | Dan Perry |
| Comments-URI | CHIPs repo, PR #88 |
| Status | Final |
| Category | Standards Track |
| Sub-Category | Core |
| Created | 2023-11-15 |
| Requires | None |
| Replaces | None |
| Superseded-By | None |
Currently, in order for proprietary Chia software to modify the farmer reward address, a closed-source harvester and farmer are both required. This CHIP introduces a new version of the harvester protocol which will allow closed-source harvesters to interact with the open-source reference farmer. This enhanced protocol will allow proprietary harvesters to modify the farmer reward address by calculating a hash based on the proof of space and the subslot challenge, and comparing it against a fee threshold. In addition, the reference farmer can verify the same calculation the harvester used when determining whether to modify the farmer reward address, thus allowing the farmer to verify the accuracy of the harvester's fee claim.
Throughout this document, we'll use the following terms:
- harvester -- retrieves proofs from plot files; communicates with a farmer
- farmer -- earns block rewards by finding valid proofs; communicates with one or more harvesters and one trusted full node
- Chia farmer -- when
Chiaappears beforefarmer, it indicates a person who runs a Chia farm, as opposed to the farming software itself - full node -- maintains a copy of the blockchain; communicates with one farmer and up to one timelord
- timelord -- creates Proofs of Time and infuses Proofs of Space into a Verifiable Delay Function (VDF); communicates with exactly one trusted full node
- VDF -- a Proof of Time -- a sequential function executed a certain number of times
- plot NFT -- a singleton that allows users to manage their membership in a pool
- singleton -- a coin that keeps track of the state of something on the blockchain; can be spent and recreated with different rules
This section will discuss why the enhanced harvester protocol is needed. It will begin with an overview of CNI's role in an expanding ecosystem, then it will delve into the process the network uses to create blocks. Later it will discuss Chia's official pooling protocol, as well as the architecture of some newer pools on Chia's blockchain.
When the Chia blockchain was launched in March 2021, Chia Network, Inc. was the sole software developer for the network's full nodes, harvesters, farmers, wallets, etc. Each of these components is open-source and fully auditable. Today, CNI continues to develop this software in order to support its large community of farmers and other users.
The following is a truncated description of how a block is created (see the included diagram for more details):
- A harvester finds a valid Proof of Space, and sends it to its farmer.
- The farmer validates the proof, then requests signage-point signatures from the harvester.
- The harvester responds with the signed signage points, which the farmer then sends to its trusted full node.
- The full node creates and propagates a new block which contains the valid proof and signatures.
- A timelord validates the block and infuses it into the VDF.
The reward for creating a block is split into two pieces:
- Farmer reward -- 1/8 of the block reward plus transaction fees, sent directly to the farmer's wallet
- Pool reward -- 7/8 of the block reward, sent to one of two places:
- If the winning plot was created using the original plot format, the reward is sent to the farmer's public pool key.
- If the winning plot was created using the new plot NFT format introduced around July 2021, a pool address was encoded into the plot. The reward is sent to this address, which is a singleton that can be updated to pay to a pool, or directly to the farmer.
The plot NFT format allows for decentralized pooling. The pool receives 7/8 of the block reward, which it is responsible for dividing among its members. The pool does not create blocks on behalf of the farmers.
On most other blockchains, valid proofs are sent directly to the pool, which creates the blocks and distributes the rewards. The pools have the power to choose which transactions go into a given block, along with the order in which those transactions are included. The pools also centralize the hash rate (for Proof-of-Work chains) or stake (for Proof-of-Stake chains). For example, two or three of Bitcoin's pools typically account for more than half of that network's hash rate (source: blockchain.com).
In contrast, Chia's official pooling protocol eliminates the risk of centralized pools by placing the power to create blocks directly with the farmers.
In the time since Chia's mainnet launch, multiple closed-source proprietary pools have been introduced. These pools do not conform to Chia's official pooling protocol. In certain cases, the pools require proprietary plots, thus disallowing users to change pools without replotting. In other cases, the pools create and sign the blocks, while allowing the use of official plots. The thing these pools have in common is that they require a proprietary farmer, which can change the farmer reward address in order to charge a developer fee.
While these proprietary pools do not violate Chia's consensus (they create valid blocks), they are a step backwards in terms of security and decentralization. For example, if a centralized pool stopped creating blocks (due to a number of potential reasons, including simply going out of business), then that pool's portion of the netspace would be taken offline until its users could replot their space (for pools that require proprietary plots), or change to another pool (for pools that allow official plots).
Contrast this with the official pooling protocol, where farmers can change pools with the click of a button. In fact, under the official pooling protocol, even if a pool does go offline and some of its users do not update their plot NFTs to point to a new pool, their farms will continue to function. The pool will no longer divide its rewards, but after a certain amount of time, the individual farmers can claim their own rewards for creating blocks. The network's total netspace will not drop. This is true because the pools are only responsible for distributing rewards, and not for creating blocks.
This CHIP will add a series of extensions to the official harvester protocol to allow closed-source harvesters to interact with the open-source reference farmer. The proprietary harvester can modify the farmer reward address by calculating a hash based on the proof of space and the subslot challenge, and comparing it against a predetermined fee threshold. The reference farmer can then verify the same calculation the harvester used when determining whether to modify the farmer reward address, thus allowing the farmer to verify the accuracy of the harvester's fee claim.
If this CHIP is accepted, Chia farmers will be able to use the farmer included with the reference software, even when using a proprietary harvester. This solution will likely be popular among all parties:
- Chia farmers will have multiple secure options when selecting a harvester and a pool, each of which will contribute to network decentralization.
- Software vendors will be able to collect fees for their efforts without compromising on network decentralization.
- Chia users and developers will remain confident in the decentralization of the network.
The enhanced harvester protocol is not backwards compatible with the original harvester protocol. In addition, this CHIP extends the existing farmer protocol unconditionally.
However, the changes from this CHIP:
- do not modify the consensus
- do not require a code fork
- do not require Chia farmers to replot
This CHIP simply introduces a new way for harvesters to interact with farmers.
Several Chia plot formats have been developed over the years, each of which comes with tradeoffs:
The original plot format encodes a pool public key into the plots, but this key has never been used. Instead, the pool coin (7/8 of the block reward) is sent to an address that is typically derived from the farmer's wallet key. In some cases, it is sent directly to a centralized pool. Decentralized pooling had yet to be developed when this format was released.
The plot NFT format encodes the pool address directly into the plots. This address uses a pay-to-singleton puzzle, which allows the payout address to be updated. This format marked the introduction of decentralized pooling. However, prior to this CHIP, when the reference farmer was used, the farmer coin had to be sent to the same address for each block. The farmer coin could only be sent to a different address as a developer fee when a proprietary farmer was used.
Proprietary plot formats have also been developed, which also allow developers to take a fee. These formats currently require the use of proprietary farmers, and they lock the Chia farmer into a specific pool.
We feel that developers should be able to get paid for their efforts. However, this should not come at a compromise to security or decentralization. Therefore, we are proposing a new open-source harvester protocol that will enable Chia farmers to use the open-source farmer, while a closed-source harvester can change the farmer reward in order to collect a developer fee. In addition, the updated reference farmer will validate the methodology the harvester used for deciding whether to change the farmer reward address upon a block's creation.
We have maintained an open channel of communication with various software vendors in order to ensure a broad level of support for the harvester protocol proposed in this CHIP.
The new harvester protocol includes four primary improvements:
-
Allow a proprietary harvester to request the source data to be included along with the hashes in a signature request.
- Prior to this CHIP, the hashes had to be sent without the source data.
-
Allow the farmer to include the requested source data, which the harvester can then use to validate the hashes.
- Prior to this CHIP, the harvester had to blind-sign the hashes due to not having access to the source data.
-
Allow the harvester to specify a farmer reward address that overrides the original address. The harvester will decide whether to override the address by performing the following two actions:
- Calculate a
uint32using the following formula:(sha256(proof of space | challenge) mod 2^32), wherechallengespecifically refers to the plot challenge (the value from the ProofOfSpace data structure, which is used to look up proofs). - Compare this
uint32against a pre-determined fee threshold.
If the
uint32is less than or equal to the fee threshold, then the harvester will override the farmer reward address, typically with a vendor's address, which will act as a developer fee.- Prior to this CHIP, the harvester had to specify the original address. A proprietary farmer could override the farmer reward address, but the open-source farmer could not.
- Calculate a
-
Allow the reference farmer to validate the aforementioned calculation.
- Note 1: The farmer will not have access to the fee threshold used by the harvester in its calculation. Therefore, the farmer will not be able to determine whether the harvester modified this threshold.
- Note 2: Because the farmer will not know the fee threshold, it will not decline to create a block based on the farmer reward address having been modified. Instead, the farmer is recommended to log both the calculation and the modified address.
- Prior to this CHIP, the harvester did not pass any additional information to the farmer, so this validation could not be performed.
Note that this CHIP does not add any new state such as signage points, proofs, etc to the farmer.
From the diagram included with this CHIP:
- In
new_signage_point, the node unconditionally sends the source data for the signage point hashes - In
new_proof_of_space, the reward address is overwritten - In
request_signatures, the source data is included - In
declare_proof_of_space, the farmer overwrites the original reward address with the new reward address, which the harvester specified
The headers from the classes that contain the new source data can be found in farmer_protocol.py:
@streamable
@dataclass(frozen=True)
class SPSubSlotSourceData(Streamable):
cc_sub_slot: ChallengeChainSubSlot
rc_sub_slot: RewardChainSubSlot
@streamable
@dataclass(frozen=True)
class SPVDFSourceData(Streamable):
cc_vdf: ClassgroupElement
rc_vdf: ClassgroupElement
@streamable
@dataclass(frozen=True)
class SignagePointSourceData(Streamable):
sub_slot_data: Optional[SPSubSlotSourceData]
vdf_data: Optional[SPVDFSourceData]This data is then passed to the harvester in SignatureRequestSourceData in harvester_protocol.py:
@streamable
@dataclass(frozen=True)
class SignatureRequestSourceData(Streamable):
foliage_block_data: Optional[FoliageBlockData] = None
foliage_transaction_block: Optional[FoliageTransactionBlock] = None
cc_vdf: Optional[ClassgroupElement] = None
rc_vdf: Optional[ClassgroupElement] = None
cc_sub_slot: Optional[ChallengeChainSubSlot] = None
rc_sub_slot: Optional[RewardChainSubSlot] = None
partial: Optional[PostPartialPayload] = NoneFor complete details of the new code, see the reference implementation.
The following test case pertains to the enhanced harvester protocol:
- test_farmer_harvester.py
- test_farmer_handles_farmer_reward_address_override
This CHIP is implemented in the following Pull Requests in the chia-blockchain GitHub repository:
Chia Network, Inc. has conducted an internal review of the code involved with this CHIP.
- chia_farming_sequence.png -- a diagram of the processes involved with farming a new Chia block
Copyright and related rights waived via CC0.