Benchmarking


How to use the slides - Full screen (new tab)
Slides Content
--- title: FRAME Benchmarking 1 description: How to benchmark Pallets in FRAME. duration: 1 hours ---

FRAME Benchmarking

Lesson 1

Overview

  • Quick Recap of Weights
  • Deep Dive Into Benchmarking

Blockchains are Limited

Blockchain systems are extremely limited environments.

Limited in:

  • Execution Time / Block Time
  • Available Storage
  • Available Memory
  • Network Bandwidth
  • etc...

Performance vs Centralization

Nodes are expected to be decentralized and distributed.

Increasing the system requirements can potentially lead to centralization in who can afford to run that hardware, and where such hardware may be available.

Why do we need benchmarking?

Benchmarking ensures that when users interact with our Blockchain, they are not using resources beyond what is available and expected for our network.

What is Weight?

Weight is a general concept used to track consumption of limited blockchain resources.

What is Weight in Substrate?

We currently track just two main limitations:

  • Execution Time on "Reference Hardware"
  • Size of Data Required to Create a Merkle Proof
#![allow(unused)]
fn main() {
pub struct Weight {
	/// The weight of computational time used based on some reference hardware.
	ref_time: u64,
	/// The weight of storage space used by proof of validity.
	proof_size: u64,
}
}

This was already expanded once, and could be expanded in the future.

Weight limits are specific to each blockchain.

  • 1 second of compute on different computers allows for different amounts of computation.
  • Weights of your blockchain will evolve over time.
  • Higher hardware requirements will result in a more performant blockchain (i.e. TXs per second), but will limit the kinds of validators that can safely participate in your network.
  • Proof size limitations can be relevant for parachains, but ignored for solo-chains.

What can affect relative Weight?

  • Processor
  • Memory
  • Hard Drive
    • HDD vs. SSD vs. NVME
  • Operating System
  • Drivers
  • Rust Compiler
  • Runtime Execution Engine
    • compiled vs. interpreted
  • Database
    • RocksDB vs. ParityDB vs. ?
  • Merkle trie / storage format
  • and more!

Block Import Weight Breakdown

The Benchmarking Framework

The Benchmarking Plan

  • Use empirical measurements of the runtime to determine the time and space it takes to execute extrinsics and other runtime logic.
  • Run benchmarks using worst case scenario conditions.
    • Primary goal is to keep the runtime safe.
    • Secondary goal is to be as accurate as possible to maximize throughput.

The #[benchmarks] Macro

#![allow(unused)]
fn main() {
#[benchmarks]
mod benchmarks {
	use super::*;

	#[benchmark]
	fn benchmark_name() {
		/* setup initial state */

		/* execute extrinsic or function */
		#[extrinsic_call]
		extrinsic_name();

		/* verify final state */
		assert!(true)
	}
}
}

Multiple Linear Regression Analysis

  • We require that no functions in Substrate have superlinear complexity.
  • Ordinary least squared linear regression.
    • linregress crate
  • Supports multiple linear coefficients.
    • Y = Ax + By + Cz + k
  • For constant time functions, we simply use the median value.

The benchmark CLI

Compile your node with --features runtime-benchmarks.

➜  ~ substrate benchmark --help
Sub-commands concerned with benchmarking.
The pallet benchmarking moved to the `pallet` sub-command

Usage: polkadot benchmark <COMMAND>

Commands:
  pallet     Benchmark the extrinsic weight of FRAME Pallets
  storage    Benchmark the storage speed of a chain snapshot
  overhead   Benchmark the execution overhead per-block and per-extrinsic
  block      Benchmark the execution time of historic blocks
  machine    Command to benchmark the hardware
  extrinsic  Benchmark the execution time of different extrinsics
  help       Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help information
  -V, --version  Print version information

pallet Subcommand

  • Benchmark the weight of functions within pallets.
    • Any arbitrary code can be benchmarked.
  • Outputs Autogenerated Weight files.
#![allow(unused)]
fn main() {
pub trait WeightInfo {
   fn transfer() -> Weight;
   fn transfer_keep_alive() -> Weight;
   fn set_balance_creating() -> Weight;
   fn set_balance_killing() -> Weight;
   fn force_transfer() -> Weight;
}
}

Deep Dive

So let’s walk through the steps of a benchmark!

Reference: frame/benchmarking/src/lib.rs

#![allow(unused)]
fn main() {
-> fn run_benchmark(...)
}

The Benchmarking Process

For each component and repeat:

  1. Select component to benchmark
  2. Generate range of values to test (steps)
  3. Whitelist known DB keys
  4. Setup benchmarking state
  5. Commit state to the DB, clearing cache
  6. Get system time (start)
  7. Execute extrinsic / benchmark function
  8. Get system time (end)
  9. Count DB reads and writes
  10. Record Data

Benchmarking Components

  • Imagine a function with 3 components
    • let x in 1..2;
    • let y in 0..5;
    • let z in 0..10;
  • We set number of steps to 3.
  • Vary one component at a time, select high value for the others.
ΔxΔyΔyΔzΔzmax
x122222
y502555
z1010100510

Benchmarks Evaluated Over Components

Whitelisted DB Keys

#![allow(unused)]
fn main() {
/// The current block number being processed. Set by `execute_block`.
#[pallet::storage]
#[pallet::whitelist_storage]
#[pallet::getter(fn block_number)]
pub(super) type Number<T: Config> = StorageValue<_, BlockNumberFor<T>, ValueQuery>;
}
  • Some keys are accessed every block:
    • Block Number
    • Events
    • Total Issuance
    • etc…
  • We don’t want to count these reads and writes in our benchmarking results.
  • Applied to all benchmarks being run.
  • This includes a “whitelisted account” provided by FRAME.

Example Benchmark

The Identity Pallet

Identity Pallet

  • Identity can have variable amount of information
    • Name
    • Email
    • Twitter
    • etc…
  • Identity can be judged by a variable amount of registrars.
  • Identity can have a two-way link to “sub-identities”
    • Other accounts that inherit the identity status of the “super-identity”

Extrinsic: Kill Identity

#![allow(unused)]
fn main() {
pub fn kill_identity(
	origin: OriginFor<T>,
	target: AccountIdLookupOf<T>,
) -> DispatchResultWithPostInfo {
	T::ForceOrigin::ensure_origin(origin)?;

	// Figure out who we're meant to be clearing.
	let target = T::Lookup::lookup(target)?;

	// Grab their deposit (and check that they have one).
	let (subs_deposit, sub_ids) = <SubsOf<T>>::take(&target);
	let id = <IdentityOf<T>>::take(&target).ok_or(Error::<T>::NotNamed)?;
	let deposit = id.total_deposit() + subs_deposit;
	for sub in sub_ids.iter() { <SuperOf<T>>::remove(sub); }

	// Slash their deposit from them.
	T::Slashed::on_unbalanced(T::Currency::slash_reserved(&target, deposit).0);
	Self::deposit_event(Event::IdentityKilled { who: target, deposit });
	Ok(())
}
}

Handling Configurations

  • kill_identity will only execute if the ForceOrigin is calling.
#![allow(unused)]
fn main() {
T::ForceOrigin::ensure_origin(origin)?;
}
  • However, this is configurable by the pallet developer.
  • Our benchmark needs to always work independent of the configuration.
  • We added a special function behind a feature flag:
#![allow(unused)]
fn main() {
/// Returns an outer origin capable of passing `try_origin` check.
///
/// ** Should be used for benchmarking only!!! **
#[cfg(feature = "runtime-benchmarks")]
fn successful_origin() -> OuterOrigin;
}

External Logic / Hooks

#![allow(unused)]
fn main() {
// Figure out who we're meant to be clearing.
let target = T::Lookup::lookup(target)?;
}
  • In general, hooks like these are configurable in the runtime.
  • Each blockchain will have their own logic, and thus their own weight.
  • We run benchmarks against the real runtime, so we get the real results.
  • IMPORTANT! You need to be careful that the limitations of these hooks are well understood by the pallet developer and users of your pallet, otherwise, your benchmark will not be accurate.

Deterministic Storage Reads / Writes

#![allow(unused)]
fn main() {
// Grab their deposit (and check that they have one).
let (subs_deposit, sub_ids) = <SubsOf<T>>::take(&target);
let id = <IdentityOf<T>>::take(&target).ok_or(Error::<T>::NotNamed)?;
}
  • 2 storage reads and writes.
  • The size of these storage items will depends on:
    • Number of Registrars
    • Number of Additional Fields

Variable Storage Reads / Writes

#![allow(unused)]
fn main() {
for sub in sub_ids.iter() { <SuperOf<T>>::remove(sub); }
}

enchmarkinghere you store balances!

  • What happens with slashed funds is configurable too!

Whitelisted Storage

#![allow(unused)]
fn main() {
Self::deposit_event(Event::IdentityKilled { who: target, deposit });
}
  • We whitelist changes to the Events storage item, so generally this is “free” beyond computation and in-memory DB weight.

Preparing to Write Your Benchmark

  • 3 Components

    • R - number of registrars
    • S - number of sub-accounts
    • X - number of additional fields

  • Need to:

    • Set up account with funds.
    • Register an identity with additional fields.
    • Set up worst case scenario for registrars and sub-accounts.
    • Take into account ForceOrigin to make the call.

Kill Identity Benchmark

#![allow(unused)]
fn main() {
#[benchmark]
fn kill_identity(
	r: Linear<1, T::MaxRegistrars::get()>,
	s: Linear<0, T::MaxSubAccounts::get()>,
	x: Linear<0, T::MaxAdditionalFields::get()>,
) -> Result<(), BenchmarkError> {
	add_registrars::<T>(r)?;

	let target: T::AccountId = account("target", 0, SEED);
	let target_origin: <T as frame_system::Config>::RuntimeOrigin = RawOrigin::Signed(target.clone()).into();
	let target_lookup = T::Lookup::unlookup(target.clone());
	let _ = T::Currency::make_free_balance_be(&target, BalanceOf::<T>::max_value());

	let info = create_identity_info::<T>(x);
	Identity::<T>::set_identity(target_origin.clone(), Box::new(info.clone()))?;
	let _ = add_sub_accounts::<T>(&target, s)?;

	// User requests judgement from all the registrars, and they approve
	for i in 0..r {
		let registrar: T::AccountId = account("registrar", i, SEED);
		let balance_to_use =  T::Currency::minimum_balance() * 10u32.into();
		let _ = T::Currency::make_free_balance_be(&registrar, balance_to_use);

		Identity::<T>::request_judgement(target_origin.clone(), i, 10u32.into())?;
		Identity::<T>::provide_judgement( RawOrigin::Signed(registrar).into(), i, target_lookup.clone(), Judgement::Reasonable, T::Hashing::hash_of(&info),
		)?;
	}
	ensure!(IdentityOf::<T>::contains_key(&target), "Identity not set");
	let origin = T::ForceOrigin::successful_origin();

	#[extrinsic_call]
	kill_identity<T::RuntimeOrigin>(origin, target_lookup)

	ensure!(!IdentityOf::<T>::contains_key(&target), "Identity not removed");
	Ok(())
}
}

Benchmarking Components

#![allow(unused)]
fn main() {
fn kill_identity(
	r: Linear<1, T::MaxRegistrars::get()>,
	s: Linear<0, T::MaxSubAccounts::get()>,
	x: Linear<0, T::MaxAdditionalFields::get()>,
) -> Result<(), BenchmarkError> { ... }
}
  • Our components.
    • R = Number of Registrars
    • S = Number of Sub-Accounts
    • X = Number of Additional Fields on the Identity.
  • Note all of these have configurable, known at compile time maxima.
    • Part of the pallet configuration trait.
    • Runtime logic should enforce these limits.

Set Up Logic

#![allow(unused)]
fn main() {
add_registrars::<T>(r)?;

let target: T::AccountId = account("target", 0, SEED);
let target_origin: <T as frame_system::Config>::RuntimeOrigin = RawOrigin::Signed(target.clone()).into();
let target_lookup = T::Lookup::unlookup(target.clone());
let _ = T::Currency::make_free_balance_be(&target, BalanceOf::<T>::max_value());
}
  • Adds registrars to the runtime storage.
  • Set up an account with the appropriate funds.
  • Note this is just like writing runtime tests!

Reusable Setup Functions

#![allow(unused)]
fn main() {
let info = create_identity_info::<T>(x);
Identity::<T>::set_identity(target_origin.clone(), Box::new(info.clone()))?;
let _ = add_sub_accounts::<T>(&target, s)?;
}
  • Using some custom functions defined in the benchmarking file:
  • Give that account an Identity with x additional fields.
  • Give that Identity s sub-accounts.

Set Up Worst Case Scenario

#![allow(unused)]
fn main() {
// User requests judgement from all the registrars, and they approve
for i in 0..r {
	let registrar: T::AccountId = account("registrar", i, SEED);
	let balance_to_use =  T::Currency::minimum_balance() * 10u32.into();
	let _ = T::Currency::make_free_balance_be(&registrar, balance_to_use);

	Identity::<T>::request_judgement(target_origin.clone(), i, 10u32.into())?;
	Identity::<T>::provide_judgement( RawOrigin::Signed(registrar).into(), i, target_lookup.clone(), Judgement::Reasonable, T::Hashing::hash_of(&info),
	)?;
}
}
  • Add r registrars.
  • Have all of them give a judgement to this identity.

Execute and Verify the Benchmark:

#![allow(unused)]
fn main() {
ensure!(IdentityOf::<T>::contains_key(&target), "Identity not set");
let origin = T::ForceOrigin::successful_origin();

#[extrinsic_call]
kill_identity<T::RuntimeOrigin>(origin, target_lookup)

ensure!(!IdentityOf::<T>::contains_key(&target), "Identity not removed");
Ok(())
}
  • First ensure statement verifies the “before” state is as we expect.
  • We need to use our custom origin.
  • Verify block ensures our “final” state is as we expect.

Executing the Benchmark

./target/production/substrate benchmark pallet \
	--chain=dev \				# Configurable Chain Spec
	--steps=50 \				# Number of steps across component ranges
	--repeat=20 \				# Number of times we repeat a benchmark
	--pallet=pallet_identity \	# Select the pallet
	--extrinsic=* \				# Select the extrinsic(s)
	--wasm-execution=compiled \ # Always used `wasm-time`
	--heap-pages=4096 \			# Not really needed, adjusts memory
	--output=./frame/identity/src/weights.rs \	# Output results into a Rust file
	--header=./HEADER-APACHE2 \	# Custom header file to include with template
	--template=./.maintain/frame-weight-template.hbs # Handlebar template

Looking at Raw Benchmarking Data

Results: Extrinsic Time vs. # of Registrars

Notes:

Graph source: https://www.shawntabrizi.com/substrate-graph-benchmarks/old/

Results: Extrinsic Time vs. # of Sub-Accounts

Notes:

Graph source: https://www.shawntabrizi.com/substrate-graph-benchmarks/old/

Results: Extrinsic Time vs. Additional Fields

Notes:

Graph source: https://www.shawntabrizi.com/substrate-graph-benchmarks/old/

Result: DB Operations vs. Sub Accounts

Notes:

Graph source: https://www.shawntabrizi.com/substrate-graph-benchmarks/old/

Final Weight

#![allow(unused)]
fn main() {
// Storage: Identity SubsOf (r:1 w:1)
// Storage: Identity IdentityOf (r:1 w:1)
// Storage: System Account (r:1 w:1)
// Storage: Identity SuperOf (r:0 w:100)
/// The range of component `r` is `[1, 20]`.
/// The range of component `s` is `[0, 100]`.
/// The range of component `x` is `[0, 100]`.
fn kill_identity(r: u32, s: u32, x: u32, ) -> Weight {
	// Minimum execution time: 68_794 nanoseconds.
	Weight::from_ref_time(52_114_486 as u64)
		// Standard Error: 4_808
		.saturating_add(Weight::from_ref_time(153_462 as u64).saturating_mul(r as u64))
		// Standard Error: 939
		.saturating_add(Weight::from_ref_time(1_084_612 as u64).saturating_mul(s as u64))
		// Standard Error: 939
		.saturating_add(Weight::from_ref_time(170_112 as u64).saturating_mul(x as u64))
		.saturating_add(T::DbWeight::get().reads(3 as u64))
		.saturating_add(T::DbWeight::get().writes(3 as u64))
		.saturating_add(T::DbWeight::get().writes((1 as u64).saturating_mul(s as u64)))
}
}

WeightInfo Generation

#![allow(unused)]
fn main() {
/// Weight functions needed for pallet_identity.
pub trait WeightInfo {
	fn add_registrar(r: u32, ) -> Weight;
	fn set_identity(r: u32, x: u32, ) -> Weight;
	fn set_subs_new(s: u32, ) -> Weight;
	fn set_subs_old(p: u32, ) -> Weight;
	fn clear_identity(r: u32, s: u32, x: u32, ) -> Weight;
	fn request_judgement(r: u32, x: u32, ) -> Weight;
	fn cancel_request(r: u32, x: u32, ) -> Weight;
	fn set_fee(r: u32, ) -> Weight;
	fn set_account_id(r: u32, ) -> Weight;
	fn set_fields(r: u32, ) -> Weight;
	fn provide_judgement(r: u32, x: u32, ) -> Weight;
	fn kill_identity(r: u32, s: u32, x: u32, ) -> Weight;
	fn add_sub(s: u32, ) -> Weight;
	fn rename_sub(s: u32, ) -> Weight;
	fn remove_sub(s: u32, ) -> Weight;
	fn quit_sub(s: u32, ) -> Weight;
}
}

WeightInfo Integration

#![allow(unused)]
fn main() {
#[pallet::weight(T::WeightInfo::kill_identity(
	T::MaxRegistrars::get(), // R
	T::MaxSubAccounts::get(), // S
	T::MaxAdditionalFields::get(), // X
))]
pub fn kill_identity(
	origin: OriginFor<T>,
	target: AccountIdLookupOf<T>,
) -> DispatchResultWithPostInfo {

	// -- snip --

	Ok(Some(T::WeightInfo::kill_identity(
		id.judgements.len() as u32,      // R
		sub_ids.len() as u32,            // S
		id.info.additional.len() as u32, // X
	))
	.into())
}
}

Initial Weight

#![allow(unused)]
fn main() {
#[pallet::weight(T::WeightInfo::kill_identity(
	T::MaxRegistrars::get(), // R
	T::MaxSubAccounts::get(), // S
	T::MaxAdditionalFields::get(), // X
))]
}
  • Use the WeightInfo function as the weight definition for your function.
  • Note that we assume absolute worst case scenario to begin since we cannot know these specific values until we query storage.

Final Weight (Refund)

#![allow(unused)]
fn main() {
pub fn kill_identity(...) -> DispatchResultWithPostInfo { ... }
}
#![allow(unused)]
fn main() {
Ok(Some(T::WeightInfo::kill_identity(
	id.judgements.len() as u32,      // R
	sub_ids.len() as u32,            // S
	id.info.additional.len() as u32, // X
))
.into())
}
  • Then we return the actual weight used at the end!
  • We use the same WeightInfo formula, but using the values that we queried from storage as part of executing the extrinsic.
  • This only allows you to decrease the final weight. Nothing will happen if you return a bigger weight than the initial weight.

Questions

In another presentation we will cover some of the things we learned while benchmarking, and best practices.

Benchmarking Exercise