Substrate and FRAME Tips and Tricks


How to use the slides - Full screen (new tab)
Slides Content
--- title: Substrate and FRAME Tips and Tricks description: Substrate and FRAME Tips and Tricks for web3 builders ---

Substrate and FRAME Tips and Tricks

Notes:

  • A random collection of things that you should probably know about.
  • These are relevant for coding in FRAME and Substrate.

Part 1 Substrate Stuff


<Type as Trait>::AssociatedType

  • The single most useful Rust syntactic detail that you MUST know.

Notes:

what is a type? A struct is a type. An unum is a type. all primitives are type. A lot of things are types.

---v

<Type as Trait>::AssociatedType

Example:

#![allow(unused)]
fn main() {
trait Config {
  type Extrinsic
  type Header: HeaderT
}

pub type ExtrinsicFor<C> = <C as Config>::Extrinsic;
fn process_extrinsic<C>(<C as Config>::Extrinsic) { .. }
fn process_extrinsic<C>(BlockFor<C>) { .. }

trait HeaderT {
  type Number;
}

pub type NumberFor<C> = <<C as Config>::Header as HeaderT>::Number;
}

Notes:

turbo fish fully qualified syntax.

---v

Speaking of Traits..

  • What is the difference between generics and associated types?
#![allow(unused)]
fn main() {
trait Block<Extrinsic> {
  fn execute(e: Extrinsic)
}
}

vs

#![allow(unused)]
fn main() {
trait Block {
  type Extrinsic;
  fn execute(e: Self::Extrinsic)
}
}

Notes:

In cambridge, I did this this. But since students should now know traits really well, I will drop it.

trait Engine {
    fn start() {}
}

struct BMW;
impl Engine for BMW {}

trait Brand {
    fn name() -> &'static str;
}

trait Car<E: Engine> {
    type Brand: Brand;
}

struct KianCarCo;
impl Brand for KianCarCo {
  fn name() -> &'static str {
    "KianCarCo!"
    }
}

struct MyCar;
impl<E: Engine> Car<E> for MyCar {
    type Brand = MyBrand;
}

fn main() {
    // Car<E1>, Car<E2> are different traits!

    // Generics can be bounded, or constrained
    // impl<E: Engine> Car<E> {}
    // impl Car<BMW> {}

    // Associated types can:
    // only be bounded when being defined,
    // Can be constrained when being implemented, or when the trait is being used.
    fn some_fn<E: Engine, C: Car<E, Brand = MyBrand>>(car: C) {
      // and we are told associated types are more like output types, lets get the brand of car
      let name = <<C as Car<E>>::Brand as Brand>::name();
    }
    fn other_fn<C: Car<BMW, Brand = MyBrand>>(car: C) {

    }

    // now, check this out
}

---v

Speaking of Traits..

Both generics and associated types can be specified, but the syntax is a bit different.

#![allow(unused)]
fn main() {
trait Block<Extrinsic> {
  type Header
}

fn process_block<B: Block<E1, Header = H1>>(b: B)
}

---v

Speaking of Traits..

  • Anything that can be expressed with associated types can also be expressed with generics.
  • Associated Types << Generics
  • Associated types usually lead to less boilerplate.

The std Paradigm

  • Recap:

    • std is the interface to the common OS-abstractions.
    • core is a subset of std that makes no assumption about the operating system.
  • a no_std crate is one that relies on core rather than std.

---v

Cargo Features

  • Way to compile different code via flags.
  • Crates define some features in their Cargo.toml
  • Crates can conditionally enable features of their dependencies as well.
[dependencies]
other-stuff = { version = "1.0.0" }

[features]
default = [""]
additional-features = ["other-stuff/on-steroids"]

Notes:

imagine that you have a crate that has some additional features that are not always needed. You put that behind a feature flag called additional-features.

---v

Cargo Features: Substrate Wasm Crates

[dependencies]
dep1 = { version = "1.0.0", default-features = false }
dep2 = { version = "1.0.0", default-features = false }

[features]
default = ["std"]
std = [
  "dep1/std"
  "dep2/std"
]

Notes:

every crate will have a feature "std". This is a flag that you are compiling with the standard library. This is the default.

Then, bringing a dependency with default-features = false means by default, don't enable this dependencies "std".

Then, in std = ["dep/std"] you are saying "if my std is enabled, enable my dependencies std as well".

---v

Cargo Features

  • The name "std" is just an idiom in the rust ecosystem.
  • no_std does NOT mean Wasm!
  • std does not mean native!

Notes:

But in substrate, it kinda means like that:

std => native no_std => wasm

---v

The std Paradigm

  • All crates in substrate that eventually compile to Wasm:
#![allow(unused)]
#![cfg_attr(not(feature = "std"), no_std)]
fn main() {
}

---v

The std Paradigm: Adding dependencies

error: duplicate lang item in crate sp_io (which frame_support depends on): panic_impl.
  |
  = Notes:


 the lang item is first defined in crate std (which serde depends on)

error: duplicate lang item in crate sp_io (which frame_support depends on): oom.
  |
  = Notes:


 the lang item is first defined in crate std (which serde depends on)

---v

The std Paradigm

A subset of the standard types in rust that also exist in rust core are re-exported from sp_std.

#![allow(unused)]
fn main() {
sp_std::prelude::*;
}

Notes:

Hashmap not exported due to non-deterministic concerns. floats are usable, but also non-deterministic! (and I think they lack encode, decode impl)

interesting to look at if_std macro in sp_std.


Logging And Prints In The Runtime.

  • First, why bother? let's just add as many logs as we want into the runtime.
  • Size of the wasm blob matters..
  • Any logging increases the size of the Wasm blob. String literals are stored somewhere in your program!

---v

Logging And Prints In The Runtime.

  • wasm2wat polkadot_runtime.wasm > dump | rg stripped

  • Should get you the .rodata (read-only data) line of the wasm blob, which contains all the logging noise.

  • This contains string literals form errors, logs, metadata, etc.

---v

Logging And Prints In The Runtime.

#![allow(unused)]
fn main() {
#[derive(sp_std::fmt::Debug)]
struct LONG_AND_BEAUTIFUL_NAME {
  plenty: u32,
  of: u32,
  fields: u32,
  with: u32,
  different: u32
  names: u32
}
}

will add a lot of string literals to your wasm blob.

---v

Logging And Prints In The Runtime.

sp_std::fmt::Debug vs sp_debug_derive::RuntimeDebug

Notes:

https://paritytech.github.io/substrate/master/sp_debug_derive/index.html

---v

Logging And Prints In The Runtime.

#![allow(unused)]
fn main() {
#[derive(RuntimeDebug)]
pub struct WithDebug {
    foo: u32,
}

impl ::core::fmt::Debug for WithDebug {
    fn fmt(&self, f: &mut ::core::fmt::Formatter) -> ::core::fmt::Result {
        #[cfg(feature = "std)]
        {
          fmt.debug_struct("WithRuntimeDebug")
            .field("foo", &self.foo)
            .finish()
        }
        #[cfg(not(feature = "std))]
        {
          fmt.write("<wasm:stripped>")
        }
    }
}
}

---v

Logging And Prints In The Runtime.

Once types implement Debug or RuntimeDebug, they can be printed. Various ways:

  • If you only want something in tests, native builds etc
#![allow(unused)]
fn main() {
sp_std::if_std! {
  println!("hello world!");
  dbg!(foo);
}
}

---v

Logging And Prints In The Runtime.

  • Or you can use the common log crate
#![allow(unused)]
fn main() {
log::info!(target: "foo", "hello world!");
log::debug!(target: "bar", "hello world! ({})", 10u32);
}

---v

Logging And Prints In The Runtime.

  • But log crate doesn't do much in itself! it needs two additional steps to work:
  1. // $ RUST_LOG=foo=debug,bar=trace cargo run
  2. sp_tracing::try_init_simple()

Notes:

https://paritytech.github.io/substrate/master/sp_tracing/index.html

---v

Logging And Prints In The Runtime.

  • Log statements are only evaluated if the corresponding level and target is met.
#![allow(unused)]
fn main() {
/// only executed if `RUST_LOG=KIAN=trace`
frame_support::log::trace!(target: "KIAN", "({:?})", (0..100000).into_iter().collect());
}

Notes:

log in rust does not do anything -- it only tracks what needs to be logged. Then you need a logger to actually export them. In rust this is often env_logger or sp_tracing in substrate tests.

In the runtime, the log messages are sent via the host functions to the client to be printed.

If the interface is built with disable-logging, it omits all log messages.


Arithmetic Helpers, and the f32, f64 Story.

  • Floating point numbers have different standards, and (slightly) different implementations on different architectures and vendors.

  • If my balance is 10.000000000000001 DOT on one validator and 10.000000000000000 DOT on another validator, game over for your consensus 😮‍💨.

---v

PerThing.

> .2 + .2 + .2 == .6
> false
> a = 10
> b = 0.1
> c = 0.2
> a*(b+c) == a*b + a*c
> false
  • Search "weird float behavior" for more entertainment around this.

---v

PerThing.

  • We store ratios and such in the runtime with "Fixed-Point" arithmetic types.
#![allow(unused)]
fn main() {
struct Percent(u8);

impl Percent {
  fn new(x: u8) {
    Self(x.min(100));
  }
}

impl Mul<u32> for Percent {
  ...
}
}

---v

PerThing.

#![allow(unused)]
fn main() {
use sp_arithmetic::Perbill;

let p = Perbill::from_part_parts(1_000_000_000u32 / 4);
let p = Perbill::from_percent(25);
let p = Perbill::from_rational(1, 4);

> p * 100u32;
> 25u32;
}
  • Some precision concerns exist, but that's a story for another day.

---v

Fixed Point Numbers

Per-thing is great for representing [0, 1] range.

What if we need more?

100 ~ 1
200 ~ 2
300 ~ 3
350 ~ 3.5

---v

Fixed Point Numbers

#![allow(unused)]
fn main() {
use sp_arithmetic::FixedU64;

let x = FixedU64::from_rational(5, 2);
let y = 10u32;
let z = x * y;
> 25
}

---v

Larger Types

#![allow(unused)]
fn main() {
pub struct BigUint {
	/// digits (limbs) of this number (sorted as msb -> lsb).
	pub(crate) digits: Vec<Single>,
}
}

---v

Arithmetic Types


Fallibility: Math Operations

Things like addition, multiplication, division could all easily fail.

  • Panic
    • u32::MAX * 2 / 2 (in debug builds)
    • 100 / 0
  • Overflow
    • u32::MAX * 2 / 2 (in release builds)

---v

Fallibility: Math Operations

  • Checked -- prevention ✋🏻

    if let Some(outcome) = a.checked_mul(b) { ... } else { ... }
    
  • Saturating -- silent recovery 🤫

    let certain_output = a.saturating_mul(b);
    

Notes:

Why would you ever want to saturate? only in cases where you know if the number is overflowing, other aspects of the system is so fundamentally screwed that there is no point in doing any kind of recovery.

There's also wrapping_op and carrying_op etc on all rust primitives, but not quite relevant.

https://doc.rust-lang.org/std/primitive.u32.html#method.checked_add https://doc.rust-lang.org/std/primitive.u32.html#method.saturating_add

---v

Fallibility: Conversion

fn main() {
    let a = 1000u32 as u8;
    println!("{}", a); //
}

Notes:

conversion of primitive number types is also a common point of error. Avoid as.

---v

Fallibility: Conversion

  • Luckily, rust is already pretty strict for the primitive types.
  • TryInto / TryFrom / From / Into
#![allow(unused)]
fn main() {
impl From<u16> for u32 {
  fn from(x: u16) -> u32 {
    x as u32 // ✅
  }
}
}
#![allow(unused)]
fn main() {
impl TryFrom<u32> for u16 {
  fn try_from(x: u32) -> Result<u16, _> {
    if x >= u16::MAX { Err(_) } else { Ok(x as u16) }
  }
}
}

Notes:

Typically you don't implement Into and TryInto, because of blanket impls. See: https://doc.rust-lang.org/std/convert/trait.From.html

For any T and U, impl From<T> for U implies impl Into<U> for T

---v

Fallibility: Conversion

  • struct Foo<T: From<u32>>

T is u32 or larger.

  • struct Foo<T: Into<u32>>

T is u32 or smaller.

  • struct Foo<T: TryInto<u32>>

T can be any of numeric types.

---v

Fallibility: Conversion

  • Substrate also provides a trait for infallible saturated conversion as well.
  • See sp-arithmetic for more handy stuff.
#![allow(unused)]
fn main() {
trait SaturatedConversion {
  fn saturated_into<T>(self) -> T
}

assert_eq!(u128::MAX.saturating_into::<u32>(), u32::MAX);
}

Notes:

https://paritytech.github.io/substrate/master/sp_arithmetic/traits/trait.SaturatedConversion.html


Part 2: FRAME Stuff


trait Get

A very basic, yet very substrate-idiomatic way to pass values through types.

#![allow(unused)]
fn main() {
pub trait Get<T> {
  fn get() -> T;
}
}
#![allow(unused)]
fn main() {
// very basic blanket implementation, which you should be very versed in reading.
impl<T: Default> Get<T> for () {
  fn get() -> T {
    T::default()
  }
}
}
#![allow(unused)]
fn main() {
struct Foo<G: Get<u32>>;
let foo = Foo<()>;
}

Notes:

implementing defaults for () is a very FRAME-idiomatic way of doing things.

---v

trait Get

#![allow(unused)]
fn main() {
parameter_types! {
  pub const Foo: u32 = 10;
}
}
#![allow(unused)]
fn main() {
// expands to:
pub struct Foo;
impl Get<u32> for Foo {
  fn get() -> u32 {
    10;
  }
}
}

Notes:

You have implemented this as a part of your rust exam.


bounded

  • BoundedVec, BoundedSlice, BoundedBTreeMap, BoundedSlice
#![allow(unused)]
fn main() {
#[derive(Encode, Decode)]
pub struct BoundedVec<T, S: Get<u32>>(
  pub(super) Vec<T>,
  PhantomData<S>,
);
}
  • ­ PhantomData?

---v

bounded

  • Why not do a bounded type like this? 🤔
#![allow(unused)]
fn main() {
#[cfg_attr(feature = "std", derive(Serialize))]
#[derive(Encode)]
pub struct BoundedVec<T>(
  pub(super) Vec<T>,
  u32,
);
}

---v

bounded

Get trait is a way to convey values through types. The type system is mostly for compiler, and has minimal overhead at runtime.


trait Convert

#![allow(unused)]
fn main() {
pub trait Convert<A, B> {
	fn convert(a: A) -> B;
}
}
#![allow(unused)]
fn main() {
pub struct Identity;
// blanket implementation!
impl<T> Convert<T, T> for Identity {
	fn convert(a: T) -> T {
		a
	}
}
}

Notes:

this one's much simpler, but good excuse to teach them blanket implementations.

---v

Example of Get and Convert

#![allow(unused)]
fn main() {
/// Some configuration for my module.
trait Config {
  /// Something that gives you a `u32`.
  type MaximumSize: Get<u32>;
  /// Something that is capable of converting `u64` to `u32`,
  /// which is pretty damn hard.
  type Convertor: Convertor<u64, u32>;
}
}
#![allow(unused)]
fn main() {
struct Runtime;
impl Config for Runtime {
  type MaximumSize = ();
  type Convertor = SomeType
}
}
#![allow(unused)]
fn main() {
Runtime as Config>::Convertor::convert(_, _);
}
#![allow(unused)]
fn main() {
fn generic_fn<T: Config>() { <T as Config>::Convertor::convert(_, _)}
}

Notes:

often times, in examples above, you have to use this syntax: https://doc.rust-lang.org/book/ch19-03-advanced-traits.html#fully-qualified-syntax-for-disambiguation-calling-methods-with-the-same-name


Implementing Traits For Tuples

#![allow(unused)]
fn main() {
struct Module1;
struct Module2;
struct Module3;

trait OnInitialize {
  fn on_initialize();
}

impl OnInitialize for Module1 { fn on_initialize() {} }
impl OnInitialize for Module2 { fn on_initialize() {} }
impl OnInitialize for Module3 { fn on_initialize() {} }
}

How can I easily invoke OnInitialize on all 3 of Module1, Module2, Module3?

Notes:

Alternative, but this needs allocation:

struct Module1;
struct Module2;
struct Module3;

trait OnInitializeDyn {
  fn on_initialize(&self);
}

impl OnInitializeDyn for Module1 { fn on_initialize(&self) {} }
impl OnInitializeDyn for Module2 { fn on_initialize(&self) {} }
impl OnInitializeDyn for Module3 { fn on_initialize(&self) {} }

fn main() {
    let x: Vec<Box<dyn OnInitializeDyn>> = vec![Box::new(Module1), Box::new(Module2)];
    x.iter().for_each(|i| i.on_initialize());
}

---v

Implementing Traits For Tuples

  1. on_initialize, in its ideal form, does not have &self, it is defined on the type, not a value.

  2. Tuples are the natural way to group types together (analogous to have a vector is the natural way to group values together..)

#![allow(unused)]
fn main() {
// fully-qualified syntax - turbo-fish.
<(Module1, Module2, Module3) as OnInitialize>::on_initialize();
}

---v

Implementing Traits For Tuples

Only problem: A lot of boilerplate. Macros!

Historically, we made this work with macro_rules!

Notes:

#![allow(unused)]
fn main() {
macro_rules! impl_for_tuples {
    ( $( $elem:ident ),+ ) => {
        impl<$( $elem: OnInitialize, )*> OnInitialize for ($( $elem, )*) {
            fn on_initialize() {
                $( $elem::on_initialize(); )*
            }
        }
    }
}

impl_for_tuples!(A, B, C, D);
impl_for_tuples!(A, B, C, D, E);
impl_for_tuples!(A, B, C, D, E, F);
}

---v

Implementing Traits For Tuples

And then someone made impl_for_tuples crate.

#![allow(unused)]
fn main() {
// In the most basic form:
#[impl_for_tuples(30)]
pub trait OnTimestampSet<Moment> {
	fn on_timestamp_set(moment: Moment);
}
}

Notes:

https://docs.rs/impl-trait-for-tuples/latest/impl_trait_for_tuples/


Defensive Programming

..is a form of defensive design to ensure the continuing function of a piece of software under unforeseen circumstances... where high availability, safety, or security is needed.

  • As you know, you should (almost) never panic in your runtime code.

---v

Defensive Programming

  • First reminder: don't panic, unless if you want to punish someone!
  • .unwrap()? no no

  • be careful with implicit unwraps in standard operations!
    • slice/vector indexing can panic if out of bound
    • .insert, .remove
    • division by zero.

---v

Defensive Programming

  • When using operations that could panic, comment exactly above it why you are sure it won't panic.
#![allow(unused)]
fn main() {
let pos = announcements
  .binary_search(&announcement)
  .ok()
  .ok_or(Error::<T, I>::MissingAnnouncement)?;
// index coming from `binary_search`, therefore cannot be out of bound.
announcements.remove(pos);
}

---v

Defensive Programming: QED

Or when using options or results that need to be unwrapped but are known to be Ok(_), Some(_):

#![allow(unused)]
fn main() {
let maybe_value: Option<_> = ...
if maybe_value.is_none() {
  return "..."
}

let value = maybe_value.expect("value checked to be 'Some'; qed");
}
  • Q.E.D. or QED is an initialism of the Latin phrase "quod erat demonstrandum", meaning "which was to be demonstrated".

---v

Defensive Programming

When writing APIs that could panic, explicitly document them, just like the core rust documentation.

#![allow(unused)]
fn main() {
/// Exactly the same semantics as [`Vec::insert`], but returns an `Err` (and is a noop) if the
/// new length of the vector exceeds `S`.
///
/// # Panics
///
/// Panics if `index > len`.
pub fn try_insert(&mut self, index: usize, element: T) -> Result<(), ()> {
  if self.len() < Self::bound() {
    self.0.insert(index, element);
    Ok(())
  } else {
    Err(())
  }
}
}

---v

Defensive Programming

/// Multiplies the given input by two.
///
/// Some further information about what this does, and where it could be used.
///
/// ```
/// fn main() {
///   let x = multiply_by_2(10);
///   assert_eq!(10, 20);
/// }
/// ```
///
/// ## Panics
///
/// Panics under such and such condition.
fn multiply_by_2(x: u32) -> u32 { .. }

---v

Defensive Programming

  • Try and not be this guy:
#![allow(unused)]
fn main() {
/// This function works with module x and multiples the given input by two. If
/// we optimize the other variant of it, we would be able to achieve more
/// efficiency but I have to think about it. Probably can panic if the input
/// overflows u32.
fn multiply_by_2(x: u32) -> u32 { .. }
}

---v

Defensive Programming

  • The overall ethos of defensive programming is along the lines of:
#![allow(unused)]
fn main() {
// we have good reasons to believe this is `Some`.
let y: Option<_> = ...

// I am really really sure about this
let x = y.expect("hard evidence; qed");

// either return a reasonable default..
let x = y.unwrap_or(reasonable_default);

// or return an error (in particular in dispatchables)
let x = y.ok_or(Error::DefensiveError)?;
}

Notes:

But, for example, you are absolutely sure that Error::DefensiveError will never happen, can we enforce it better?

---v

Defensive Programming

#![allow(unused)]
fn main() {
let x = y
  .ok_or(Error::DefensiveError)
  .map_err(|e| {
    #[cfg(test)]
    panic!("defensive error happened: {:?}", e);

    log::error!(target: "..", "defensive error happened: {:?}", e);
  })?;
}

---v

Defensive Programming

// either return a reasonable default..
let x = y.defensive_unwrap_or(reasonable_default);

// or return an error (in particular in dispatchables)
let x = y.defensive_ok_or(Error::DefensiveError)?;

It adds some boilerplate to:

  1. Panic when debug_assertions are enabled (tests).
  2. append a log::error!.

Additional Resources! 😋

Check speaker notes (click "s" 😉)

Good luck with FRAME!

Notes:

Feedback After Lecture:

  • Lecture is still kinda dense and long, try and trim
  • Update on defensive ops: https://github.com/paritytech/substrate/pull/12967
  • Next time, talk about making a storage struct be <T: Config>.
  • Cargo format
  • SignedExtension should technically be part of the substrate module. Integrate it in the assignment, perhaps.
  • A section about XXXNoBound traits.
  • A section about reading your compiler errors top to bottom, especially with an example in FRAME.