Substrate and FRAME Tips and Tricks

How to use the slides - Full screen (new tab) Slides Content --- title: Substrate and FRAME Tips and Tricks description: Substrate and FRAME Tips and Tricks for web3 builders ---

Substrate and FRAME Tips and Tricks

Notes:

A random collection of things that you should probably know about.
These are relevant for coding in FRAME and Substrate.

Part 1 Substrate Stuff

`<Type as Trait>::AssociatedType`

The single most useful Rust syntactic detail that you MUST know.

Notes:

what is a type? A struct is a type. An unum is a type. all primitives are type. A lot of things are types.

---v

`<Type as Trait>::AssociatedType`

Example:

#![allow(unused)]
fn main() {
trait Config {
  type Extrinsic
  type Header: HeaderT
}

pub type ExtrinsicFor<C> = <C as Config>::Extrinsic;
fn process_extrinsic<C>(<C as Config>::Extrinsic) { .. }
fn process_extrinsic<C>(BlockFor<C>) { .. }

trait HeaderT {
  type Number;
}

pub type NumberFor<C> = <<C as Config>::Header as HeaderT>::Number;
}

Notes:

turbo fish fully qualified syntax.

---v

Speaking of Traits..

Anything that can be expressed with associated types can also be expressed with generics.
Associated Types << Generics
Associated types usually lead to less boilerplate.

The `std` Paradigm

Recap:
- std is the interface to the common OS-abstractions.
- core is a subset of std that makes no assumption about the operating system.
a no_std crate is one that relies on core rather than std.

---v

Cargo Features

Way to compile different code via flags.
Crates define some features in their Cargo.toml
Crates can conditionally enable features of their dependencies as well.

[dependencies]
other-stuff = { version = "1.0.0" }

[features]
default = [""]
additional-features = ["other-stuff/on-steroids"]

Notes:

imagine that you have a crate that has some additional features that are not always needed. You put that behind a feature flag called additional-features.

---v

Cargo Features: Substrate Wasm Crates

[dependencies]
dep1 = { version = "1.0.0", default-features = false }
dep2 = { version = "1.0.0", default-features = false }

[features]
default = ["std"]
std = [
  "dep1/std"
  "dep2/std"
]

Notes:

every crate will have a feature "std". This is a flag that you are compiling with the standard library. This is the default.

Then, bringing a dependency with default-features = false means by default, don't enable this dependencies "std".

Then, in std = ["dep/std"] you are saying "if my std is enabled, enable my dependencies std as well".

---v

Cargo Features

The name "std" is just an idiom in the rust ecosystem.
no_std does NOT mean Wasm!
std does not mean native!

Notes:

But in substrate, it kinda means like that:

std => native no_std => wasm

---v

The `std` Paradigm: Adding dependencies

error: duplicate lang item in crate sp_io (which frame_support depends on): panic_impl.
  |
  = Notes:


 the lang item is first defined in crate std (which serde depends on)

error: duplicate lang item in crate sp_io (which frame_support depends on): oom.
  |
  = Notes:


 the lang item is first defined in crate std (which serde depends on)

---v

The `std` Paradigm

A subset of the standard types in rust that also exist in rust core are re-exported from sp_std.

#![allow(unused)]
fn main() {
sp_std::prelude::*;
}

Notes:

Hashmap not exported due to non-deterministic concerns. floats are usable, but also non-deterministic! (and I think they lack encode, decode impl)

interesting to look at if_std macro in sp_std.

Logging And Prints In The Runtime.

First, why bother? let's just add as many logs as we want into the runtime.

Size of the wasm blob matters..

Any logging increases the size of the Wasm blob. String literals are stored somewhere in your program!

---v

Logging And Prints In The Runtime.

wasm2wat polkadot_runtime.wasm > dump | rg stripped
Should get you the .rodata (read-only data) line of the wasm blob, which contains all the logging noise.
This contains string literals form errors, logs, metadata, etc.

---v

Logging And Prints In The Runtime.

#![allow(unused)]
fn main() {
#[derive(sp_std::fmt::Debug)]
struct LONG_AND_BEAUTIFUL_NAME {
  plenty: u32,
  of: u32,
  fields: u32,
  with: u32,
  different: u32
  names: u32
}
}

will add a lot of string literals to your wasm blob.

---v

Logging And Prints In The Runtime.

sp_std::fmt::Debug vs sp_debug_derive::RuntimeDebug

Notes:

https://paritytech.github.io/substrate/master/sp_debug_derive/index.html

---v

Logging And Prints In The Runtime.

#![allow(unused)]
fn main() {
#[derive(RuntimeDebug)]
pub struct WithDebug {
    foo: u32,
}

impl ::core::fmt::Debug for WithDebug {
    fn fmt(&self, f: &mut ::core::fmt::Formatter) -> ::core::fmt::Result {
        #[cfg(feature = "std)]
        {
          fmt.debug_struct("WithRuntimeDebug")
            .field("foo", &self.foo)
            .finish()
        }
        #[cfg(not(feature = "std))]
        {
          fmt.write("<wasm:stripped>")
        }
    }
}
}

---v

Logging And Prints In The Runtime.

Once types implement Debug or RuntimeDebug, they can be printed. Various ways:

If you only want something in tests, native builds etc

#![allow(unused)]
fn main() {
sp_std::if_std! {
  println!("hello world!");
  dbg!(foo);
}
}

---v

Logging And Prints In The Runtime.

Or you can use the common log crate

#![allow(unused)]
fn main() {
log::info!(target: "foo", "hello world!");
log::debug!(target: "bar", "hello world! ({})", 10u32);
}

---v

Logging And Prints In The Runtime.

But log crate doesn't do much in itself! it needs two additional steps to work:

// $ RUST_LOG=foo=debug,bar=trace cargo run
sp_tracing::try_init_simple()

Notes:

https://paritytech.github.io/substrate/master/sp_tracing/index.html

---v

Logging And Prints In The Runtime.

Log statements are only evaluated if the corresponding level and target is met.

#![allow(unused)]
fn main() {
/// only executed if `RUST_LOG=KIAN=trace`
frame_support::log::trace!(target: "KIAN", "({:?})", (0..100000).into_iter().collect());
}

Notes:

log in rust does not do anything -- it only tracks what needs to be logged. Then you need a logger to actually export them. In rust this is often env_logger or sp_tracing in substrate tests.

In the runtime, the log messages are sent via the host functions to the client to be printed.

If the interface is built with disable-logging, it omits all log messages.

Arithmetic Helpers, and the `f32`, `f64` Story.

Floating point numbers have different standards, and (slightly) different implementations on different architectures and vendors.
If my balance is 10.000000000000001 DOT on one validator and 10.000000000000000 DOT on another validator, game over for your consensus 😮‍💨.

---v

PerThing.

> .2 + .2 + .2 == .6
> false

> a = 10
> b = 0.1
> c = 0.2
> a*(b+c) == a*b + a*c
> false

Search "weird float behavior" for more entertainment around this.

---v

PerThing.

#![allow(unused)]
fn main() {
use sp_arithmetic::Perbill;

let p = Perbill::from_part_parts(1_000_000_000u32 / 4);
let p = Perbill::from_percent(25);
let p = Perbill::from_rational(1, 4);

> p * 100u32;
> 25u32;
}

Some precision concerns exist, but that's a story for another day.

---v

Fallibility: Math Operations

Things like addition, multiplication, division could all easily fail.

Panic
- u32::MAX * 2 / 2 (in debug builds)
- 100 / 0

Overflow
- u32::MAX * 2 / 2 (in release builds)

---v

Why would you ever want to saturate? only in cases where you know if the number is overflowing, other aspects of the system is so fundamentally screwed that there is no point in doing any kind of recovery.

There's also wrapping_op and carrying_op etc on all rust primitives, but not quite relevant.

https://doc.rust-lang.org/std/primitive.u32.html#method.checked_add https://doc.rust-lang.org/std/primitive.u32.html#method.saturating_add

---v

Fallibility: Conversion

fn main() {
    let a = 1000u32 as u8;
    println!("{}", a); //
}

Notes:

conversion of primitive number types is also a common point of error. Avoid as.

---v

Fallibility: Conversion

Luckily, rust is already pretty strict for the primitive types.
TryInto / TryFrom / From / Into

#![allow(unused)]
fn main() {
impl From<u16> for u32 {
  fn from(x: u16) -> u32 {
    x as u32 // ✅
  }
}
}

#![allow(unused)]
fn main() {
impl TryFrom<u32> for u16 {
  fn try_from(x: u32) -> Result<u16, _> {
    if x >= u16::MAX { Err(_) } else { Ok(x as u16) }
  }
}
}

Notes:

Typically you don't implement Into and TryInto, because of blanket impls. See: https://doc.rust-lang.org/std/convert/trait.From.html

For any T and U, impl From<T> for U implies impl Into<U> for T

---v

Fallibility: Conversion

struct Foo<T: From<u32>>

T is u32 or larger.

struct Foo<T: Into<u32>>

T is u32 or smaller.

struct Foo<T: TryInto<u32>>

T can be any of numeric types.

---v

Fallibility: Conversion

Substrate also provides a trait for infallible saturated conversion as well.
See sp-arithmetic for more handy stuff.

#![allow(unused)]
fn main() {
trait SaturatedConversion {
  fn saturated_into<T>(self) -> T
}

assert_eq!(u128::MAX.saturating_into::<u32>(), u32::MAX);
}

Notes:

https://paritytech.github.io/substrate/master/sp_arithmetic/traits/trait.SaturatedConversion.html

Part 2: FRAME Stuff

`trait Get`

A very basic, yet very substrate-idiomatic way to pass values through types.

#![allow(unused)]
fn main() {
pub trait Get<T> {
  fn get() -> T;
}
}

#![allow(unused)]
fn main() {
// very basic blanket implementation, which you should be very versed in reading.
impl<T: Default> Get<T> for () {
  fn get() -> T {
    T::default()
  }
}
}

#![allow(unused)]
fn main() {
struct Foo<G: Get<u32>>;
let foo = Foo<()>;
}

Notes:

implementing defaults for () is a very FRAME-idiomatic way of doing things.

---v

`trait Get`

#![allow(unused)]
fn main() {
parameter_types! {
  pub const Foo: u32 = 10;
}
}

#![allow(unused)]
fn main() {
// expands to:
pub struct Foo;
impl Get<u32> for Foo {
  fn get() -> u32 {
    10;
  }
}
}

Notes:

You have implemented this as a part of your rust exam.

`bounded`

BoundedVec, BoundedSlice, BoundedBTreeMap, BoundedSlice

#![allow(unused)]
fn main() {
#[derive(Encode, Decode)]
pub struct BoundedVec<T, S: Get<u32>>(
  pub(super) Vec<T>,
  PhantomData<S>,
);
}

PhantomData?

---v

`bounded`

Why not do a bounded type like this? 🤔

#![allow(unused)]
fn main() {
#[cfg_attr(feature = "std", derive(Serialize))]
#[derive(Encode)]
pub struct BoundedVec<T>(
  pub(super) Vec<T>,
  u32,
);
}

---v

`bounded`

Get trait is a way to convey values through types. The type system is mostly for compiler, and has minimal overhead at runtime.

`trait Convert`

#![allow(unused)]
fn main() {
pub trait Convert<A, B> {
	fn convert(a: A) -> B;
}
}

#![allow(unused)]
fn main() {
pub struct Identity;
// blanket implementation!
impl<T> Convert<T, T> for Identity {
	fn convert(a: T) -> T {
		a
	}
}
}

Notes:

this one's much simpler, but good excuse to teach them blanket implementations.

---v

Example of `Get` and `Convert`

#![allow(unused)]
fn main() {
/// Some configuration for my module.
trait Config {
  /// Something that gives you a `u32`.
  type MaximumSize: Get<u32>;
  /// Something that is capable of converting `u64` to `u32`,
  /// which is pretty damn hard.
  type Convertor: Convertor<u64, u32>;
}
}

#![allow(unused)]
fn main() {
struct Runtime;
impl Config for Runtime {
  type MaximumSize = ();
  type Convertor = SomeType
}
}

#![allow(unused)]
fn main() {
Runtime as Config>::Convertor::convert(_, _);
}

#![allow(unused)]
fn main() {
fn generic_fn<T: Config>() { <T as Config>::Convertor::convert(_, _)}
}

Notes:

often times, in examples above, you have to use this syntax: https://doc.rust-lang.org/book/ch19-03-advanced-traits.html#fully-qualified-syntax-for-disambiguation-calling-methods-with-the-same-name

Implementing Traits For Tuples

#![allow(unused)]
fn main() {
struct Module1;
struct Module2;
struct Module3;

trait OnInitialize {
  fn on_initialize();
}

impl OnInitialize for Module1 { fn on_initialize() {} }
impl OnInitialize for Module2 { fn on_initialize() {} }
impl OnInitialize for Module3 { fn on_initialize() {} }
}

How can I easily invoke OnInitialize on all 3 of Module1, Module2, Module3?

Notes:

Alternative, but this needs allocation:

struct Module1;
struct Module2;
struct Module3;

trait OnInitializeDyn {
  fn on_initialize(&self);
}

impl OnInitializeDyn for Module1 { fn on_initialize(&self) {} }
impl OnInitializeDyn for Module2 { fn on_initialize(&self) {} }
impl OnInitializeDyn for Module3 { fn on_initialize(&self) {} }

fn main() {
    let x: Vec<Box<dyn OnInitializeDyn>> = vec![Box::new(Module1), Box::new(Module2)];
    x.iter().for_each(|i| i.on_initialize());
}

---v

Implementing Traits For Tuples

on_initialize, in its ideal form, does not have &self, it is defined on the type, not a value.
Tuples are the natural way to group types together (analogous to have a vector is the natural way to group values together..)

#![allow(unused)]
fn main() {
// fully-qualified syntax - turbo-fish.
<(Module1, Module2, Module3) as OnInitialize>::on_initialize();
}

---v

Implementing Traits For Tuples

Only problem: A lot of boilerplate. Macros!

Historically, we made this work with macro_rules!

Notes:

#![allow(unused)]
fn main() {
macro_rules! impl_for_tuples {
    ( $( $elem:ident ),+ ) => {
        impl<$( $elem: OnInitialize, )*> OnInitialize for ($( $elem, )*) {
            fn on_initialize() {
                $( $elem::on_initialize(); )*
            }
        }
    }
}

impl_for_tuples!(A, B, C, D);
impl_for_tuples!(A, B, C, D, E);
impl_for_tuples!(A, B, C, D, E, F);
}

---v

Implementing Traits For Tuples

And then someone made impl_for_tuples crate.

#![allow(unused)]
fn main() {
// In the most basic form:
#[impl_for_tuples(30)]
pub trait OnTimestampSet<Moment> {
	fn on_timestamp_set(moment: Moment);
}
}

Notes:

https://docs.rs/impl-trait-for-tuples/latest/impl_trait_for_tuples/

Defensive Programming

..is a form of defensive design to ensure the continuing function of a piece of software under unforeseen circumstances... where high availability, safety, or security is needed.

As you know, you should (almost) never panic in your runtime code.

---v

Defensive Programming

First reminder: don't panic, unless if you want to punish someone!
.unwrap()? no no

be careful with implicit unwraps in standard operations!
- slice/vector indexing can panic if out of bound
- .insert, .remove
- division by zero.

---v

Defensive Programming

When using operations that could panic, comment exactly above it why you are sure it won't panic.

#![allow(unused)]
fn main() {
let pos = announcements
  .binary_search(&announcement)
  .ok()
  .ok_or(Error::<T, I>::MissingAnnouncement)?;
// index coming from `binary_search`, therefore cannot be out of bound.
announcements.remove(pos);
}

---v

Defensive Programming: QED

Or when using options or results that need to be unwrapped but are known to be Ok(_), Some(_):

#![allow(unused)]
fn main() {
let maybe_value: Option<_> = ...
if maybe_value.is_none() {
  return "..."
}

let value = maybe_value.expect("value checked to be 'Some'; qed");
}

Q.E.D. or QED is an initialism of the Latin phrase "quod erat demonstrandum", meaning "which was to be demonstrated".

---v

Defensive Programming

When writing APIs that could panic, explicitly document them, just like the core rust documentation.

#![allow(unused)]
fn main() {
/// Exactly the same semantics as [`Vec::insert`], but returns an `Err` (and is a noop) if the
/// new length of the vector exceeds `S`.
///
/// # Panics
///
/// Panics if `index > len`.
pub fn try_insert(&mut self, index: usize, element: T) -> Result<(), ()> {
  if self.len() < Self::bound() {
    self.0.insert(index, element);
    Ok(())
  } else {
    Err(())
  }
}
}

---v

Defensive Programming

Speaking of documentation, here's a very good guideline!

/// Multiplies the given input by two.
///
/// Some further information about what this does, and where it could be used.
///
/// ```
/// fn main() {
///   let x = multiply_by_2(10);
///   assert_eq!(10, 20);
/// }
/// ```
///
/// ## Panics
///
/// Panics under such and such condition.
fn multiply_by_2(x: u32) -> u32 { .. }

---v

Defensive Programming

Try and not be this guy:

#![allow(unused)]
fn main() {
/// This function works with module x and multiples the given input by two. If
/// we optimize the other variant of it, we would be able to achieve more
/// efficiency but I have to think about it. Probably can panic if the input
/// overflows u32.
fn multiply_by_2(x: u32) -> u32 { .. }
}

---v

Defensive Programming

The overall ethos of defensive programming is along the lines of:

#![allow(unused)]
fn main() {
// we have good reasons to believe this is `Some`.
let y: Option<_> = ...

// I am really really sure about this
let x = y.expect("hard evidence; qed");

// either return a reasonable default..
let x = y.unwrap_or(reasonable_default);

// or return an error (in particular in dispatchables)
let x = y.ok_or(Error::DefensiveError)?;
}

Notes:

But, for example, you are absolutely sure that Error::DefensiveError will never happen, can we enforce it better?

---v

Defensive Programming

#![allow(unused)]
fn main() {
let x = y
  .ok_or(Error::DefensiveError)
  .map_err(|e| {
    #[cfg(test)]
    panic!("defensive error happened: {:?}", e);

    log::error!(target: "..", "defensive error happened: {:?}", e);
  })?;
}

---v

Defensive Programming

Yes: Defensive traits:

// either return a reasonable default..
let x = y.defensive_unwrap_or(reasonable_default);

// or return an error (in particular in dispatchables)
let x = y.defensive_ok_or(Error::DefensiveError)?;

It adds some boilerplate to:

Panic when debug_assertions are enabled (tests).
append a log::error!.

Additional Resources! 😋

Check speaker notes (click "s" 😉)

Good luck with FRAME!

Notes:

Rust didn't have u128 until not too long ago! https://github.com/paritytech/substrate/pull/163/files
TryFrom/TryInto are also not too old! https://github.com/paritytech/substrate/pull/163/files#r188938077
Remove As, which tried to fill the lack of TryFrom/TryInto https://github.com/paritytech/substrate/pull/2602
Runtime Logging PR: https://github.com/paritytech/substrate/pull/3821
Impl trait for tuples:
std/no_std

Feedback After Lecture:

Lecture is still kinda dense and long, try and trim
Update on defensive ops: https://github.com/paritytech/substrate/pull/12967
Next time, talk about making a storage struct be <T: Config>.
Cargo format
SignedExtension should technically be part of the substrate module. Integrate it in the assignment, perhaps.
A section about XXXNoBound traits.
A section about reading your compiler errors top to bottom, especially with an example in FRAME.

Polkadot Blockchain Academy