Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft of state archival simulation support #1475

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
+839 −249
Draft

Draft of state archival simulation support #1475

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
83f363c
Draft of state archival simulation support
dmkozh Oct 17, 2024
a2ca6da
Rebase
SirTyson Nov 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 5 additions & 2 deletions soroban-simulation/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,15 @@ testutils = ["soroban-env-host/testutils"]
[dependencies]
anyhow = { version = "1.0.75", features = [] }
thiserror = "1.0.40"
soroban-env-host = { workspace = true, features = ["recording_mode", "unstable-next-api"]}
soroban-env-host = { workspace = true, features = ["recording_mode", "unstable-next-api", "next"]}
stellar-xdr = { workspace = true, features = ["next"]}
static_assertions = "1.1.0"
rand = "0.8.5"


[dev-dependencies]
soroban-env-host = { workspace = true, features = ["recording_mode", "testutils", "unstable-next-api"]}
soroban-env-host = { workspace = true, features = ["recording_mode", "testutils", "unstable-next-api", "next"]}
stellar-xdr = { workspace = true, features = ["next"]}
soroban-test-wasms = { package = "soroban-test-wasms", path = "../soroban-test-wasms" }
pretty_assertions = "1.4"
tap = "1.0.1"
Expand Down
3 changes: 2 additions & 1 deletion soroban-simulation/src/network_config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@ fn load_configuration_setting(
let key = Rc::new(LedgerKey::ConfigSetting(LedgerKeyConfigSetting {
config_setting_id: setting_id,
}));
let (entry, _) = snapshot
let entry = snapshot
.get_including_archived(&key)?
.entry
.ok_or_else(|| anyhow!("setting {setting_id:?} is not present in the snapshot"))?;
if let LedgerEntry {
data: LedgerEntryData::ConfigSetting(cs),
Expand Down
79 changes: 48 additions & 31 deletions soroban-simulation/src/resources.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
use super::snapshot_source::SnapshotSourceWithArchive;
use crate::network_config::NetworkConfig;
use crate::simulation::{SimulationAdjustmentConfig, SimulationAdjustmentFactor};
use anyhow::{anyhow, ensure, Context, Result};
use crate::snapshot_source::LedgerEntryArchivalState;
use anyhow::{anyhow, bail, ensure, Context, Result};

use soroban_env_host::xdr::ArchivalProof;
use soroban_env_host::{
e2e_invoke::{extract_rent_changes, LedgerEntryChange},
fees::{
Expand All @@ -12,12 +14,12 @@ use soroban_env_host::{
ledger_info::get_key_durability,
storage::SnapshotSource,
xdr::{
BytesM, ContractDataDurability, DecoratedSignature, Duration, ExtensionPoint, Hash,
LedgerBounds, LedgerFootprint, LedgerKey, Limits, Memo, MuxedAccount, MuxedAccountMed25519,
Operation, OperationBody, Preconditions, PreconditionsV2, ReadXdr, SequenceNumber,
Signature, SignatureHint, SignerKey, SignerKeyEd25519SignedPayload, SorobanResources,
SorobanTransactionData, TimeBounds, TimePoint, Transaction, TransactionExt,
TransactionV1Envelope, Uint256, WriteXdr,
BytesM, ContractDataDurability, DecoratedSignature, Duration, Hash, LedgerBounds,
LedgerFootprint, LedgerKey, Limits, Memo, MuxedAccount, MuxedAccountMed25519, Operation,
OperationBody, Preconditions, PreconditionsV2, ReadXdr, SequenceNumber, Signature,
SignatureHint, SignerKey, SignerKeyEd25519SignedPayload, SorobanResources,
SorobanTransactionData, SorobanTransactionDataExt, TimeBounds, TimePoint, Transaction,
TransactionExt, TransactionV1Envelope, Uint256, WriteXdr,
},
LedgerInfo, DEFAULT_XDR_RW_LIMITS,
};
Expand Down Expand Up @@ -193,7 +195,11 @@ pub(crate) fn simulate_restore_op_resources(
keys_to_restore: &[LedgerKey],
snapshot_source: &impl SnapshotSourceWithArchive,
ledger_info: &LedgerInfo,
) -> Result<(SorobanResources, Vec<LedgerEntryRentChange>)> {
) -> Result<(
SorobanResources,
Vec<LedgerEntryRentChange>,
Option<ArchivalProof>,
)> {
let restored_live_until_ledger = ledger_info
.min_live_until_ledger_checked(ContractDataDurability::Persistent)
.ok_or_else(|| {
Expand All @@ -202,36 +208,42 @@ pub(crate) fn simulate_restore_op_resources(
let mut restored_bytes = 0_u32;
let mut rent_changes: Vec<LedgerEntryRentChange> = Vec::with_capacity(keys_to_restore.len());
let mut restored_keys = Vec::<LedgerKey>::with_capacity(keys_to_restore.len());
let mut restored_keys_needing_proof =
Vec::<Rc<LedgerKey>>::with_capacity(keys_to_restore.len());

for key in keys_to_restore {
let durability = get_key_durability(key);
ensure!(
durability == Some(ContractDataDurability::Persistent),
"Can't restore a ledger entry with key: {key:?}. Only persistent ledger entries with TTL can be restored."
);
let entry_with_live_until = snapshot_source
.get_including_archived(&Rc::new(key.clone()))?
.ok_or_else(|| anyhow!("Missing entry to restore for key {key:?}"))?;
let (entry, live_until) = entry_with_live_until;

let current_live_until_ledger = live_until.ok_or_else(|| {
anyhow!("Internal error: missing TTL for ledger key that must have TTL: `{key:?}`")
})?;

if current_live_until_ledger >= ledger_info.sequence_number {
continue;
let rc_key = Rc::new(key.clone());
let entry_state = snapshot_source.get_including_archived(&rc_key)?;
match &entry_state.state {
LedgerEntryArchivalState::Archived(need_proof) => {
if *need_proof {
restored_keys_needing_proof.push(rc_key.clone());
}
restored_keys.push(key.clone());
}
_ => {
continue;
}
}
restored_keys.push(key.clone());

let entry_size: u32 = entry.to_xdr(DEFAULT_XDR_RW_LIMITS)?.len().try_into()?;
restored_bytes = restored_bytes.saturating_add(entry_size);
rent_changes.push(LedgerEntryRentChange {
is_persistent: true,
old_size_bytes: 0,
new_size_bytes: entry_size,
old_live_until_ledger: 0,
new_live_until_ledger: restored_live_until_ledger,
});
if let Some(entry) = &entry_state.entry {
let entry_size: u32 = entry.to_xdr(DEFAULT_XDR_RW_LIMITS)?.len().try_into()?;
restored_bytes = restored_bytes.saturating_add(entry_size);
rent_changes.push(LedgerEntryRentChange {
is_persistent: true,
old_size_bytes: 0,
new_size_bytes: entry_size,
old_live_until_ledger: 0,
new_live_until_ledger: restored_live_until_ledger,
});
} else {
bail!("missing entry to be restored for key '{key:?}', is the archive incomplete?");
}
}
restored_keys.sort();
let resources = SorobanResources {
Expand All @@ -243,7 +255,12 @@ pub(crate) fn simulate_restore_op_resources(
read_bytes: restored_bytes,
write_bytes: restored_bytes,
};
Ok((resources, rent_changes))
let archival_proof = if !restored_keys_needing_proof.is_empty() {
Some(snapshot_source.generate_restoration_proof(restored_keys_needing_proof.as_slice())?)
} else {
None
};
Ok((resources, rent_changes, archival_proof))
}

fn estimate_max_transaction_size_for_operation(
Expand Down Expand Up @@ -299,7 +316,7 @@ fn estimate_max_transaction_size_for_operation(
write_bytes: 0,
},
resource_fee: 0,
ext: ExtensionPoint::V0,
ext: SorobanTransactionDataExt::V0,
}),
},
signatures: signatures.try_into()?,
Expand Down
34 changes: 24 additions & 10 deletions soroban-simulation/src/simulation.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,11 @@ use soroban_env_host::{
e2e_invoke::LedgerEntryChange,
storage::SnapshotSource,
xdr::{
AccountId, ContractEvent, DiagnosticEvent, HostFunction, InvokeHostFunctionOp, LedgerKey,
OperationBody, ScVal, SorobanAuthorizationEntry, SorobanResources, SorobanTransactionData,
AccountId, ArchivalProof, ContractEvent, DiagnosticEvent, ExtendFootprintTtlOp,
ExtensionPoint, HostFunction, InvokeHostFunctionOp, LedgerEntry, LedgerKey, OperationBody,
ReadXdr, RestoreFootprintOp, ScVal, SorobanAuthorizationEntry, SorobanResources,
SorobanTransactionData, SorobanTransactionDataExt,
},
xdr::{ExtendFootprintTtlOp, ExtensionPoint, LedgerEntry, ReadXdr, RestoreFootprintOp},
HostError, LedgerInfo, DEFAULT_XDR_RW_LIMITS,
};
use std::rc::Rc;
Expand Down Expand Up @@ -116,7 +117,7 @@ pub struct RestoreOpSimulationResult {
/// for failed invocations. It should only fail if ledger is
/// mis-configured (e.g. when computed fees cause overflows).
#[allow(clippy::too_many_arguments)]
pub fn simulate_invoke_host_function_op(
pub fn simulate_invoke_host_function_op<F>(
snapshot_source: Rc<dyn SnapshotSource>,
network_config: &NetworkConfig,
adjustment_config: &SimulationAdjustmentConfig,
Expand All @@ -126,7 +127,11 @@ pub fn simulate_invoke_host_function_op(
source_account: &AccountId,
base_prng_seed: [u8; 32],
enable_diagnostics: bool,
) -> Result<InvokeHostFunctionSimulationResult> {
generate_new_entry_proof: F,
) -> Result<InvokeHostFunctionSimulationResult>
where
F: FnOnce() -> Result<Option<ArchivalProof>>,
{
let snapshot_source = Rc::new(SimulationSnapshotSource::new_from_rc(snapshot_source));
let budget = network_config.create_budget()?;
let mut diagnostic_events = vec![];
Expand Down Expand Up @@ -194,7 +199,11 @@ pub fn simulate_invoke_host_function_op(
&rent_changes,
adjustment_config,
);
simulation_result.transaction_data = Some(create_transaction_data(resources, resource_fee));
simulation_result.transaction_data = Some(create_transaction_data(
resources,
resource_fee,
generate_new_entry_proof()?,
));

Ok(simulation_result)
}
Expand Down Expand Up @@ -243,7 +252,7 @@ pub fn simulate_extend_ttl_op(
adjustment_config,
);
Ok(ExtendTtlOpSimulationResult {
transaction_data: create_transaction_data(resources, resource_fee),
transaction_data: create_transaction_data(resources, resource_fee, None),
})
}

Expand Down Expand Up @@ -271,7 +280,7 @@ pub fn simulate_restore_op(
keys_to_restore: &[LedgerKey],
) -> Result<RestoreOpSimulationResult> {
let snapshot_source = SimulationSnapshotSourceWithArchive::new(snapshot_source);
let (mut resources, rent_changes) =
let (mut resources, rent_changes, archival_proof) =
simulate_restore_op_resources(keys_to_restore, &snapshot_source, ledger_info)?;
let operation = OperationBody::RestoreFootprint(RestoreFootprintOp {
ext: ExtensionPoint::V0,
Expand All @@ -286,7 +295,7 @@ pub fn simulate_restore_op(
adjustment_config,
);
Ok(RestoreOpSimulationResult {
transaction_data: create_transaction_data(resources, resource_fee),
transaction_data: create_transaction_data(resources, resource_fee, archival_proof),
})
}

Expand Down Expand Up @@ -333,11 +342,16 @@ impl SimulationAdjustmentConfig {
fn create_transaction_data(
resources: SorobanResources,
resource_fee: i64,
new_entries_proof: Option<ArchivalProof>,
) -> SorobanTransactionData {
SorobanTransactionData {
resources,
resource_fee,
ext: ExtensionPoint::V0,
ext: if let Some(proof) = new_entries_proof {
SorobanTransactionDataExt::V1(vec![proof].try_into().unwrap())
} else {
SorobanTransactionDataExt::V0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine for now, but in my prototype I've had all SorobanTransactionDataExt default to V1 regardless of if there's a proof or not in p23 (it's just an empty vector if no proof is required). I don't think this is strictly necessary and it does increase TX size overhead slightly, but I thought the status quo for XDR changes like this is to require the new version of the extension. I don't feel strongly about this either way, just wanted to flag it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm actually this is not currently compatible with my prototype, where a soroban tx gets dropped and is not considered valid if we're in p23 and SorobanTransactionDataExt is not v1. I can either change my prototype or we can change it here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not up to core to decide, is it? I think we should keep V0 around for the sake of backwards compatibility. I think for tx envelope changes we should be as backwards compatible as possible and shouldn't deprecate any exts unless they're somehow broken/incompatible with the protocol (which isn't the case here).

},
}
}

Expand Down
Loading
Loading