Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

candidate-validation: RFC103 implementation #5847

Merged
merged 164 commits into from
Oct 31, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
164 commits
Select commit Hold shift + click to select a range
793142e
WIP primitives
sandreim Jul 17, 2024
3a29fdf
WIP
sandreim Aug 5, 2024
4a53577
Working version.
sandreim Aug 7, 2024
8285ae7
Better version
sandreim Aug 8, 2024
2831c5e
Add missing primitives and fix things
sandreim Aug 8, 2024
c767d60
Implement v2 receipts in polkadot-runtime-parachains
sandreim Aug 8, 2024
96999e3
add missing stuff
sandreim Aug 12, 2024
c5f2dc3
Switch parachains runtime to use new primitives
sandreim Aug 12, 2024
dbb0160
use vstaging primitives
sandreim Aug 12, 2024
5efab68
update rococo and westend
sandreim Aug 12, 2024
c2232e4
client keeps using the old primitives
sandreim Aug 12, 2024
87b079f
no unsafe pls
sandreim Aug 12, 2024
00e8c13
move async backing primtiives to own file
sandreim Aug 12, 2024
cd4d02f
fix
sandreim Aug 12, 2024
5509e33
fix test build
sandreim Aug 12, 2024
f8b86d2
fix test-runtime
sandreim Aug 12, 2024
fe2fbfb
self review feedback
sandreim Aug 13, 2024
975e13b
review feedback
sandreim Aug 13, 2024
1c7ac55
feedback
sandreim Aug 13, 2024
653873b
feedback
sandreim Aug 13, 2024
dc98149
clippy
sandreim Aug 13, 2024
0a6bce3
chores
sandreim Aug 13, 2024
5e4dac2
Filter v2 candidate descriptors
sandreim Aug 14, 2024
f12ca7a
fix
sandreim Aug 14, 2024
13734de
fix prospective parachains tests
sandreim Aug 14, 2024
effb1cc
fix fix
sandreim Aug 14, 2024
3f75cba
fmt
sandreim Aug 14, 2024
75a47bb
fix comment
sandreim Aug 14, 2024
12ed853
another one
sandreim Aug 14, 2024
f2c0882
fix build
sandreim Aug 15, 2024
768e034
.
sandreim Aug 15, 2024
4bf0706
improve test and add comment
sandreim Aug 15, 2024
0c83201
add log
sandreim Aug 15, 2024
4296942
simplify check()
sandreim Aug 19, 2024
e1a7509
Merge branch 'sandreim/rfc103-primitives' of github.com:paritytech/po…
sandreim Aug 20, 2024
6fb7790
impl<H>
sandreim Aug 20, 2024
e6add9c
Merge branch 'sandreim/rfc103-primitives' of github.com:paritytech/po…
sandreim Aug 20, 2024
d0b3961
comment
sandreim Aug 20, 2024
66f7a96
add some tests
sandreim Aug 20, 2024
5c0c919
update
sandreim Aug 20, 2024
38ce589
prdoc
sandreim Aug 21, 2024
9f1d611
can't be happy if CI is sad
sandreim Aug 21, 2024
a6a7329
Merge branch 'master' of github.com:paritytech/polkadot-sdk into sand…
sandreim Aug 21, 2024
663817d
remove newlines
sandreim Aug 21, 2024
a1dacc1
match rfc 103 reserved field naming
sandreim Aug 21, 2024
33b80ea
remove default cq offset
sandreim Aug 21, 2024
d5b165f
Merge branch 'sandreim/rfc103-primitives' of github.com:paritytech/po…
sandreim Aug 21, 2024
29e4b47
Ignore UMP signals when checking and processing UMP queue
sandreim Aug 16, 2024
ab85fe3
wip
sandreim Aug 20, 2024
7d5636b
refactor a bit
sandreim Aug 20, 2024
2954bba
use descriptor core_index in `map_candidates_to_cores`
sandreim Aug 20, 2024
e7abe8b
nits
sandreim Aug 20, 2024
1db5eb0
Para Inherent: filter v2 candidate descriptors (#5362)
sandreim Aug 22, 2024
cdb49a6
increase test coverage
sandreim Aug 22, 2024
f6f714a
Merge branch 'sandreim/rfc103-primitives' of github.com:paritytech/po…
sandreim Aug 22, 2024
aa925cd
Improve usability of primitives
sandreim Aug 23, 2024
00d7c71
use committed core index if available in v1 receipts
sandreim Aug 23, 2024
af9f561
typo
sandreim Aug 23, 2024
fb2cefb
fix check
sandreim Aug 23, 2024
b53787d
typo
sandreim Aug 23, 2024
e2ef46e
add test for mixed v1 v2 scenario
sandreim Aug 26, 2024
2dfc542
comment
sandreim Aug 26, 2024
a38a243
add ump test
sandreim Aug 26, 2024
da381da
avoid one storage read
sandreim Aug 26, 2024
ca5c618
store claim queue snapshot in allowed relay parent info
sandreim Aug 27, 2024
4266665
check v2 receipts using claim queue snapshots
sandreim Aug 27, 2024
e93b983
typo
sandreim Aug 27, 2024
e01bf53
it was a bad idea to process commitments of v1 receipts
sandreim Aug 28, 2024
fb9fbe6
fmt
sandreim Aug 28, 2024
c507488
remove unused
sandreim Aug 28, 2024
178e201
Validate session index
sandreim Aug 28, 2024
984e8e1
add unknown version
sandreim Aug 29, 2024
fab215d
add check for unknown version and test
sandreim Aug 29, 2024
7300552
Merge branch 'sandreim/rfc103-primitives' of github.com:paritytech/po…
sandreim Aug 29, 2024
9bbe2cc
typo
sandreim Aug 29, 2024
4dda9df
adjust comments
sandreim Aug 29, 2024
cd3eb5f
Merge branch 'master' of github.com:paritytech/polkadot-sdk into sand…
sandreim Aug 30, 2024
f8ef4ce
fix merge damage
sandreim Aug 30, 2024
04e31a1
unused
sandreim Aug 30, 2024
5fd1279
fix
sandreim Aug 30, 2024
19d6f32
fix benchmark build
sandreim Sep 2, 2024
552078a
Merge branch 'sandreim/rfc103-primitives' of github.com:paritytech/po…
sandreim Sep 2, 2024
4ec3fc8
typos
sandreim Sep 2, 2024
2ba0a27
fmt
sandreim Sep 2, 2024
e468d62
fix comment
sandreim Sep 2, 2024
3fe368f
Merge branch 'master' of github.com:paritytech/polkadot-sdk into sand…
sandreim Sep 3, 2024
18a0496
mixed v1, v2, v2 without select core tests,
sandreim Sep 4, 2024
d320269
Add allowed relay parents storage migration
sandreim Sep 4, 2024
8490488
fix migration
sandreim Sep 5, 2024
db67486
fix
sandreim Sep 5, 2024
03cf8c1
clippy
sandreim Sep 5, 2024
43f6de7
feedback
sandreim Sep 5, 2024
70e48d2
sir, make it faster
sandreim Sep 5, 2024
1e26c73
fix
sandreim Sep 5, 2024
f4e3fb5
one last fix
sandreim Sep 5, 2024
2e87ad3
fixes
sandreim Sep 5, 2024
54432be
remove println
sandreim Sep 5, 2024
cfbecb0
add prdoc
sandreim Sep 6, 2024
3a518f2
fix comment
sandreim Sep 6, 2024
54106e2
refactor map_candidates_to_cores
sandreim Sep 6, 2024
b44a604
doc updates
sandreim Sep 9, 2024
4c5c707
Merge branch 'master' of github.com:paritytech/polkadot-sdk into sand…
sandreim Sep 9, 2024
0cc2e6c
switch nodes to vstaging primitives
sandreim Sep 11, 2024
f9ddb15
test fixes
sandreim Sep 11, 2024
add4592
fmt
sandreim Sep 11, 2024
ef51144
extract test helpers in MutateDescriptorV2
sandreim Sep 12, 2024
3981425
fix tests
sandreim Sep 12, 2024
32af174
more fixes
sandreim Sep 12, 2024
9eba1ef
more wip
sandreim Sep 13, 2024
caff543
feedback
sandreim Sep 13, 2024
218f530
refactor
sandreim Sep 13, 2024
216937a
fix try-runtime
sandreim Sep 13, 2024
e169d0d
Finally all tests compile
sandreim Sep 16, 2024
ceac8b2
:facepalm:
sandreim Sep 16, 2024
c0aee8c
check ump signal count and test
sandreim Sep 16, 2024
1ef7952
remove unused
sandreim Sep 18, 2024
5790b8e
fix prdoc
sandreim Sep 19, 2024
ba9d3ff
more tests cases
sandreim Sep 19, 2024
888880f
Merge branch 'sandreim/runtime_v2_descriptor_support' into sandreim/n…
sandreim Sep 19, 2024
0dcf739
workaround #64
sandreim Sep 20, 2024
604273b
stricter UMP signal checks and tests
sandreim Sep 23, 2024
9c4e2ae
stricter UMP signal checks and tests
sandreim Sep 23, 2024
43bbb9d
type alias
sandreim Sep 24, 2024
64febf4
Merge branch 'sandreim/runtime_v2_descriptor_support' of github.com:p…
sandreim Sep 24, 2024
a493225
remove unused
sandreim Sep 24, 2024
d7e57fd
Merge branch 'master' into sandreim/runtime_v2_descriptor_support
sandreim Sep 24, 2024
7e3a943
Merge branch 'sandreim/runtime_v2_descriptor_support' of github.com:p…
sandreim Sep 24, 2024
7352577
add prdoc
sandreim Sep 24, 2024
c9e44d1
fix merge damage
sandreim Sep 24, 2024
bf948a7
fix
sandreim Sep 24, 2024
b6ba4b0
more test helpers
sandreim Sep 26, 2024
589a0d0
fmt
sandreim Sep 26, 2024
8a96c42
impl rfc 103 for candidate validation subsystem
sandreim Sep 26, 2024
b9452e5
typo
sandreim Sep 26, 2024
fd9dfee
fix some builds
sandreim Sep 26, 2024
5bd465b
fix the fix
sandreim Sep 26, 2024
868cc15
refactor
sandreim Sep 27, 2024
cd3547c
more refactor
sandreim Sep 27, 2024
5fe5738
fix
sandreim Sep 27, 2024
53bb5b8
toml fixes
sandreim Sep 27, 2024
4942227
Update from sandreim running command 'prdoc --audience node_dev --bum…
actions-user Oct 11, 2024
555c91b
remove bogus code
sandreim Oct 17, 2024
90932b3
Merge branch 'master' of github.com:paritytech/polkadot-sdk into sand…
sandreim Oct 17, 2024
4b2c06f
fix build
sandreim Oct 17, 2024
320318a
fmt and feedback
sandreim Oct 17, 2024
dfde71f
more fixes
sandreim Oct 21, 2024
cb93bb6
delete PRDoc
sandreim Oct 21, 2024
a402509
Update from sandreim running command 'prdoc --audience node_dev runti…
actions-user Oct 21, 2024
6371454
adjust generated PRDoc
sandreim Oct 21, 2024
9c5097d
fix build again
sandreim Oct 22, 2024
abfd836
Merge branch 'sandreim/node_v2_descriptors' of github.com:paritytech/…
sandreim Oct 22, 2024
669662f
Merge branch 'sandreim/node_rfc103_part_1' of github.com:paritytech/p…
sandreim Oct 22, 2024
b2818f3
fix test
sandreim Oct 22, 2024
f647249
fix example
sandreim Oct 23, 2024
82da09a
fmt
sandreim Oct 24, 2024
db77530
Merge branch 'sandreim/node_v2_descriptors' of github.com:paritytech/…
sandreim Oct 24, 2024
36ef936
review
sandreim Oct 24, 2024
2ac194f
update bumps
sandreim Oct 24, 2024
7a025fe
fix audience
sandreim Oct 24, 2024
d89517e
taplo format
sandreim Oct 24, 2024
d1bad21
Merge branch 'master' of github.com:paritytech/polkadot-sdk into sand…
sandreim Oct 25, 2024
67e0dc7
feedback
sandreim Oct 30, 2024
b696eaa
more feedback
sandreim Oct 30, 2024
9f12e39
update prdoc
sandreim Oct 31, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions polkadot/node/core/candidate-validation/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,5 @@ polkadot-node-subsystem-test-helpers = { workspace = true }
sp-maybe-compressed-blob = { workspace = true, default-features = true }
sp-core = { workspace = true, default-features = true }
polkadot-primitives-test-helpers = { workspace = true }
rstest = { workspace = true }
polkadot-primitives = { workspace = true, features = ["test"] }
166 changes: 137 additions & 29 deletions polkadot/node/core/candidate-validation/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ use polkadot_node_subsystem::{
overseer, FromOrchestra, OverseerSignal, SpawnedSubsystem, SubsystemError, SubsystemResult,
SubsystemSender,
};
use polkadot_node_subsystem_util as util;
use polkadot_node_subsystem_util::{self as util, runtime::ClaimQueueSnapshot};
use polkadot_overseer::ActiveLeavesUpdate;
use polkadot_parachain_primitives::primitives::ValidationResult as WasmValidationResult;
use polkadot_primitives::{
Expand All @@ -46,8 +46,9 @@ use polkadot_primitives::{
DEFAULT_LENIENT_PREPARATION_TIMEOUT, DEFAULT_PRECHECK_PREPARATION_TIMEOUT,
},
vstaging::{
CandidateDescriptorV2 as CandidateDescriptor, CandidateEvent,
transpose_claim_queue, CandidateDescriptorV2 as CandidateDescriptor, CandidateEvent,
Overkillus marked this conversation as resolved.
Show resolved Hide resolved
CandidateReceiptV2 as CandidateReceipt,
CommittedCandidateReceiptV2 as CommittedCandidateReceipt,
},
AuthorityDiscoveryId, CandidateCommitments, ExecutorParams, Hash, PersistedValidationData,
PvfExecKind as RuntimePvfExecKind, PvfPrepKind, SessionIndex, ValidationCode,
Expand Down Expand Up @@ -148,6 +149,25 @@ impl<Context> CandidateValidationSubsystem {
}
}

// Returns the claim queue at relay parent and logs a warning if it is not available.
async fn claim_queue<Sender>(relay_parent: Hash, sender: &mut Sender) -> Option<ClaimQueueSnapshot>
where
Sender: SubsystemSender<RuntimeApiMessage>,
{
match util::runtime::fetch_claim_queue(sender, relay_parent).await {
Ok(maybe_cq) => maybe_cq,
Err(err) => {
gum::warn!(
target: LOG_TARGET,
?relay_parent,
?err,
"Claim queue not available"
);
None
},
}
}

fn handle_validation_message<S>(
mut sender: S,
validation_host: ValidationHost,
Expand All @@ -167,24 +187,40 @@ where
exec_kind,
response_sender,
..
} => async move {
let _timer = metrics.time_validate_from_exhaustive();
let res = validate_candidate_exhaustive(
validation_host,
validation_data,
validation_code,
candidate_receipt,
pov,
executor_params,
exec_kind,
&metrics,
)
.await;
} =>
async move {
let _timer = metrics.time_validate_from_exhaustive();
let relay_parent = candidate_receipt.descriptor.relay_parent();

let maybe_claim_queue = claim_queue(relay_parent, &mut sender).await;

let maybe_expected_session_index =
match util::request_session_index_for_child(relay_parent, &mut sender)
.await
.await
{
Ok(Ok(expected_session_index)) => Some(expected_session_index),
_ => None,
};

let res = validate_candidate_exhaustive(
maybe_expected_session_index,
validation_host,
validation_data,
validation_code,
candidate_receipt,
pov,
executor_params,
exec_kind,
&metrics,
maybe_claim_queue,
)
.await;

metrics.on_validation_event(&res);
let _ = response_sender.send(res);
}
.boxed(),
metrics.on_validation_event(&res);
let _ = response_sender.send(res);
}
.boxed(),
CandidateValidationMessage::PreCheck {
relay_parent,
validation_code_hash,
Expand Down Expand Up @@ -637,6 +673,7 @@ where
}

async fn validate_candidate_exhaustive(
maybe_expected_session_index: Option<SessionIndex>,
mut validation_backend: impl ValidationBackend + Send,
persisted_validation_data: PersistedValidationData,
validation_code: ValidationCode,
Expand All @@ -645,18 +682,41 @@ async fn validate_candidate_exhaustive(
executor_params: ExecutorParams,
exec_kind: PvfExecKind,
metrics: &Metrics,
maybe_claim_queue: Option<ClaimQueueSnapshot>,
) -> Result<ValidationResult, ValidationFailed> {
let _timer = metrics.time_validate_candidate_exhaustive();

let validation_code_hash = validation_code.hash();
let relay_parent = candidate_receipt.descriptor.relay_parent();
let para_id = candidate_receipt.descriptor.para_id();

gum::debug!(
target: LOG_TARGET,
?validation_code_hash,
?para_id,
"About to validate a candidate.",
);

// We only check the session index for backing.
match (exec_kind, candidate_receipt.descriptor.session_index()) {
(PvfExecKind::Backing | PvfExecKind::BackingSystemParas, Some(session_index)) => {
let Some(expected_session_index) = maybe_expected_session_index else {
let error = "cannot fetch session index from the runtime";
gum::warn!(
target: LOG_TARGET,
?relay_parent,
error,
);

return Err(ValidationFailed(error.into()))
};

if session_index != expected_session_index {
return Ok(ValidationResult::Invalid(InvalidCandidate::InvalidSessionIndex))
}
},
(_, _) => {},
};

if let Err(e) = perform_basic_checks(
&candidate_receipt.descriptor,
persisted_validation_data.max_pov_size,
Expand Down Expand Up @@ -754,15 +814,21 @@ async fn validate_candidate_exhaustive(
gum::info!(target: LOG_TARGET, ?para_id, "Invalid candidate (para_head)");
Ok(ValidationResult::Invalid(InvalidCandidate::ParaHeadHashMismatch))
} else {
let outputs = CandidateCommitments {
head_data: res.head_data,
upward_messages: res.upward_messages,
horizontal_messages: res.horizontal_messages,
new_validation_code: res.new_validation_code,
processed_downward_messages: res.processed_downward_messages,
hrmp_watermark: res.hrmp_watermark,
let committed_candidate_receipt = CommittedCandidateReceipt {
descriptor: candidate_receipt.descriptor.clone(),
commitments: CandidateCommitments {
head_data: res.head_data,
upward_messages: res.upward_messages,
horizontal_messages: res.horizontal_messages,
new_validation_code: res.new_validation_code,
processed_downward_messages: res.processed_downward_messages,
hrmp_watermark: res.hrmp_watermark,
},
};
if candidate_receipt.commitments_hash != outputs.hash() {

if candidate_receipt.commitments_hash !=
committed_candidate_receipt.commitments.hash()
{
gum::info!(
target: LOG_TARGET,
?para_id,
Expand All @@ -773,7 +839,48 @@ async fn validate_candidate_exhaustive(
// invalid.
Ok(ValidationResult::Invalid(InvalidCandidate::CommitmentsHashMismatch))
} else {
Ok(ValidationResult::Valid(outputs, (*persisted_validation_data).clone()))
let core_index = candidate_receipt.descriptor.core_index();
sandreim marked this conversation as resolved.
Show resolved Hide resolved

match (core_index, exec_kind) {
// Core selectors are optional for V2 descriptors, but we still check the
Overkillus marked this conversation as resolved.
Show resolved Hide resolved
// descriptor core index.
(
Some(_core_index),
PvfExecKind::Backing | PvfExecKind::BackingSystemParas,
sandreim marked this conversation as resolved.
Show resolved Hide resolved
) => {
let Some(claim_queue) = maybe_claim_queue else {
let error = "cannot fetch the claim queue from the runtime";
gum::warn!(
target: LOG_TARGET,
?relay_parent,
error
);

return Err(ValidationFailed(error.into()))
};

if let Err(err) = committed_candidate_receipt
.check_core_index(&transpose_claim_queue(claim_queue.0))
{
gum::warn!(
target: LOG_TARGET,
?err,
candidate_hash = ?candidate_receipt.hash(),
"Candidate core index is invalid",
);
return Ok(ValidationResult::Invalid(
InvalidCandidate::InvalidCoreIndex,
))
}
},
// No checks for approvals and disputes
(_, _) => {},
}

Ok(ValidationResult::Valid(
committed_candidate_receipt.commitments,
(*persisted_validation_data).clone(),
))
}
},
}
Expand Down Expand Up @@ -1003,6 +1110,7 @@ fn perform_basic_checks(
return Err(InvalidCandidate::CodeHashMismatch)
}

// No-op for `v2` receipts.
if let Err(()) = candidate.check_collator_signature() {
return Err(InvalidCandidate::BadSignature)
}
Expand Down
Loading
Loading