Course Materials Book

This book is the home of the majority of materials used within the Core Developer Track of the Polkadot Blockchain Academy.

Read the Book

We suggest the online version for general use, but cloning, installing, and building this book offline is a great option on-the-go.

Hosted Online

The latest version is hosted at: https://polkadot-blockchain-academy.github.io/pba-book/

Build Offline

The Core Developer Track of the Academy is Rust heavy and as such, you need to install rust before you can continue.

In order to make your life easy ๐Ÿ˜‰, there is a set of tasks that use cargo make.

With cargo make installed, you can list all tasks included to facilitate further installation, building, serving, formatting, and more with:

# Run from the top-level working dir of this repo
makers --list-all-steps

The tasks should be self-explanatory, if they are not - please file an issue to help us make them better.

License and Use Policies

All materials found within this repository are licensed under Mozilla Public License Version 2.0 - See the License for details.

In addition to the license, we ask you read and respect the Academy's Code of Conduct and help us foster a healthy and scholarly community of high academic integrity.

Learn more about and apply for the next Academy today!

๐Ÿช„ Using this Book

This book contains all the Academy's content in sequence. It is intended to read from start to finish, in order, and following the index.md page for each module.

๐Ÿ“” How to use mdBook

This book is made with mdBook - a Rust native documentation tool. Please give the official docs there a read to see what nice features are included. To name a few key items:

IconDescription
Opens and closes the chapter listing sidebar.
Opens a picker to choose a different color theme.
Opens a search bar for searching within the book.
Instructs the web browser to print the entire book.
Opens a link to the website that hosts the source code of the book.
Opens a page to directly edit the source of the page you are currently reading.

Pressing the search icon () in the menu bar, or pressing the S key on the keyboard will open an input box for entering search terms. Typing some terms will show matching chapters and sections in real time.

Clicking any of the results will jump to that section. The up and down arrow keys can be used to navigate the results, and enter will open the highlighted section.

After loading a search result, the matching search terms will be highlighted in the text. Clicking a highlighted word or pressing the Esc key will remove the highlighting.

๐ŸŽž๏ธ How-to use reveal.js Slides

Most pages include embedded slides that have a lot of handy features. These are with reveal-md: a tool built with reveal.js to allow for Markdown only slides, with a few extra syntax items to make your slides look and feel awesome with very little effort.

๐Ÿ“ Be use to have the slides iframe on a page active (๐Ÿ–ฑ๏ธ click on it) to use slide keybindings... Otherwise those are captured by the mdbook tools! (s is search for the book, and speaker notes for the slides)

Be a power user of these by using the keybindings to interact with them:

  • Use space to navigate all slides: top to bottom, left to right.
    • Use down/up arrow keys to navigate vertical slides.
    • Use left/right arrow keys to navigate horizontal slides.
  • Press Esc or o to see an overview view that arrow keys can navigate.
  • Press s to open up speaker view.
    ๐Ÿ‘€ Speaker notes include very important information, not to be missed!

๐Ÿ’ซ Slides Demo

Tryout those keybindings (๐Ÿ–ฑ๏ธ click on the slides to start) below:

How to use the slides - Full screen slides (new tab)

(๐Ÿ–ฑ๏ธ expand) Raw Markdown of Slides Content --- title: How-to use Reveal.js Slides description: How to use reveal.js duration: 5 minuets ---

How-to use Reveal.js Slides

These slides are built with reveal.js.

These slides serve as a feature demo of reveal for you! ๐ŸŽ‰


What are we going to see:

  • How to use Reveal.js Features

    • Useful reveal.js tips
    • Speaker Notes

How to use Reveal.js Features

Press the down/up keys to navigate _vertical slides_

Try doing down a slide.

---v

Use the keybindings!

  • Overview mode: โ€œOโ€ to see a birds-eye view of your presentation, โ€œESCโ€ to return to the highlighted slide (you can quickly navigate with arrows)

  • Full-screen: โ€œFโ€, โ€œESCโ€ to exit full-screen mode

  • Speaker mode: โ€œSโ€ it synchronizes 2 windows: one with the presentation, and another with a timer and all speaker notes!

  • Zoom-in: ALT+click make the view zoom at the position of your mouseโ€™s pointer; very useful to look closely at a picture or chart surrounded by too much bullet points.

---v

Speaker Notes & Viewer

Press the s key to bring up a popup window with speaker view

You need to unblock popups to have the window open

Notes: This is a note just for you. Set under a line in your slide starting with "Note:" all subsequent lines are just seen in speaker view.


Enjoy!

๐Ÿ˜Ž

โ˜๏ธ All slides Slides Content is available on all pages. This enables search to work throughout this book to jump-to-location for any keywords you remember related to something covered in a lesson ๐Ÿš€.

๐Ÿ“– Learn More

๐Ÿ“’ Book Overview

This book contains a set of course materials covering both the conceptual underpinnings and hands-on experience in developing blockchain and web3 technologies. Students will be introduced to core concepts in economic, cryptographic, and computer science fields that lay the foundation for approaching web3 development, as well as hands-on experience developing web3 systems in Rust, primarily utilizing the ecosystem of tooling provided by Polkadot and Substrate.

๐Ÿ™‹ This book is designed specifically for use in an in-person course. This provides far more value from these materials than an online only, self-guided experience could provide.

โœ… The Academy encourages everyone to apply to the program Our program is facilitated a few times a year at prestigious places around the world, with on the order of ~50-100 students per cohort.

๐Ÿ‘จโ€๐ŸŽ“ Learning Outcomes

By the end of the Polkadot Blockchain Academy, students will be able to:

  • Apply economic, cryptographic, and computer science concepts to web3 application design
  • Robustly design and evaluate security of web3, both at the protocol and user application level
  • Write a smart contract using one of a number of languages and deploy it to a blockchain
  • Implement a Substrate based blockchain
  • Deploy a parachain utilizing Substrate, Cumulus, and Polkadot
  • Employ FRAME to accelerate blockchain and parachain development
  • Configure XCM for cross-consensus messaging between parachains

๐Ÿ–‹๏ธ Nomenclature

The academy uses explicit terms to describe materials use within as content categories defined here:

  • Lesson: a segment of content (1-2 hours) that is one of:
    • Lecture: An oral presentation that consists primarily of slide based content. Most content in this book is of this type.
      • Exercise: a short (5-10 minutes) exercise for to be completed during a lecture (code snippets, mini-demos, etc.).
    • Workshop: these are step-by-step, longer (0.5-3 hours) guided in-class material (live-coding, competitions, games, etc.). Workshops are instructor lead, and hand-held to get everyone to the same result.
    • Activity: these are self-directed activities for individuals and/or small groups. Activities are not guided or "hand-held" by the instructor like workshops are.
  • Assignment: a graded piece of work, typically one per week is assigned.
    • Assignments are not public - these are only accessible by Academy Faculty, Staff, and (in a derivative form) Students.

๐Ÿชœ Course Sequence

The course is segmented into modules, with the granular lessons intended to be completed in the sequence provided in the left-side navigation bar.

ModuleTopic
๐Ÿ” CryptographyApplied cryptography concepts and introduction to many common tools of the trade for web3 builders.
๐Ÿช™ Economics and Game TheoryApplied economics and game theory fundamental to the architecture and operation of web3 applications.
โ›“๏ธ Blockchains and Smart ContractsBlockchain and applications built on them covered in depth conceptually and hands-on operation and construction.
๐Ÿงฌ SubstrateThe blockchain framework canonical to Polkadot and Parachains covered in depth, at a lower level.
๐Ÿงฑ FRAMEThe primary Substrate runtime framework used for parachain development.
๐ŸŸฃ PolkadotThe Polkadot blockchain covered in depth, focus on high-level design and practically how to utilize its blockspace.
๐Ÿ’ฑ XCMThe cross consensus messaging format covered from first principals to use in protocols.

The lessons include materials used, with links and instructions to required external materials as needed.1

1

Notably, the graded assignments for the Academy and some solutions to public activities and exercises remain closed source, and links are intentionally left out of this book. These materials may be shared as needed with students in person during the Academy.

๐Ÿ” Cryptography

โ€œCryptography rearranges power: it configures who can do what, from whatโ€

Phillip Rogaway, The Moral Character of Cryptographic Work

Applied cryptography concepts and introduction to many common tools of the trade for web3 builders.

Introduction to Cryptography


How to use the slides - Full screen (new tab)
Slides Content
--- title: Introduction to Cryptography description: Cryptographic primitives for web3 builders duration: 1 hour ---

Introduction to Cryptography


Some Useful Equations

Notes:

Just kidding!


Goals for this lesson

  • Understand the goals of cryptography
  • Understand some network and contextual assumptions
  • Learn what expectations cryptography upholds
  • Learn the primitives

Notes:

In this first lesson,


Cryptography Landscape

Notes:

What is covered in this course is all connected subjects. We will not cover any details for hybrid or interactive protocols in the course.


Operating Context

The internet is a public space.

We communicate over public channels. Adversaries may want to:

  • Read messages not intended for them
  • Impersonate others
  • Tamper with messages

Notes:

Use e-mail as an example of an flawed system.

Some examples include:

  • An attacker may impersonate your boss, trying to get you to send them money
  • An attacker may change a message sent over a network, e.g. an instruction to transfer 100 EUR to 10000 EUR

Probably best for the teacher to ask students to participate with examples of application messages, not just person-to-person messages.


Operating Context

Resources are constrained.
  • Network, storage, computation, etc.: We don't want to send, store, or operate on the same data, but we want guarantees about it, e.g. that we agree on a message's contents.
  • Privacy: We must assume that all channels can be monitored, and thus closed channels are heavily constrained (i.e. assumed to not exist).

Open vs. Closed Channels

Cryptography based on public systems is more sound.

Kerckhoff's Principle: Security should not rely on secret methods,
but rather on secret information.

Notes:

There is no such thing as a "closed channel" :)

  • Methods can be reverse engineered. After that, the communication channel is completely insecure. For example, CSS protection for DVDs.
  • We always work with public, open protocols.

Cryptographic Guarantees*

  • Data confidentiality
  • Data authenticity
  • Data integrity
  • Non-repudiation

Notes:

Cryptography is one of the (most important) tools we have to build tools that are guaranteed to work correctly. This is regardless of who (human, machine, or otherwise) is using them and their intentions (good or bad).

Why an asterisk? There generally are no perfect & absolute guarantees here, but for most practical purposes the bounds on where these fail are good enough to serve our needs as engineers and users. Do note the assumptions and monitor their validity over time (like quantum tech).


Important Non-Guarantee

  • Data availability

Cryptography alone cannot make strong guarantees that data is available to people when they want to access it.

Notes:

There are many schemes to get around this, and this topic will come up later in the course. We will touch on erasure coding, which makes data availability more efficient.


Data Confidentiality

A party may gain access to information
if and only if they know some secret (a key).


Confidentiality ensures that a third party cannot read my confidential data.

Notes:

The ability to decrypt some data and reveal its underlying information directly implies knowledge of some secret, potentially unknown to the originator of the information. Supplying the original information (aka plain text message) can be used in a "challenge game" mechanism as one means of proving knowledge of the secret without compromising it.

Mention use of the term "plaintext".

Allegory: A private document stored on server where sysadmin has access can be subpoenaed, violating assumed Attorney-Client Privilege on the document.

---v

Confidentiality in Communication Channels

Suppose Alice and Bob are sending confidential messages back and forth. There are some subtypes of confidentiality here:

  • Forward Secrecy: Even if an adversary temporarily learns Alice's secret, it cannot read future messages after some point.
  • Backwards Secrecy: Even if an adversary temporarily learns Alice's secret, it cannot read past messages beyond some previous point.

Data Authenticity

Users can have the credible expectation that the stated origin of a message is authentic.


Authenticity ensures that a third party cannot pretend I created some data.

Notes:

  • Digital signatures should be difficult (practically speaking: impossible) to forge.
  • Digital signatures should verify that the signer knows some secret, without revealing the secret itself.

Data Integrity

If data is tampered with, it is detectable. In other words, it possible to check if the current state of some data is the consistent with when it was created.


Integrity ensures that if data I create is corrupted, it can be detected.

---v

Physical Signatures

Physical signatures provide weak authenticity guarantees
(i.e. they are quite easy to forge), and no integrity guarantees.

---v

An Ideal Signature

Notes:

For example, if you change the year on your university diploma, the dean's signature is still valid. Digital signatures provide a guarantee that the signed information has not been tampered with.


Non-repudiation

The sender of a message cannot deny that they sent it.


Non-repudiation ensures if Bob sends me some data, I can prove to a third party that they sent it.


One-Way Functions

One-way functions form the basis of both
(cryptographic) hashing and asymmetric cryptography. A function $f$ is one way if:

  • it is reasonably fast to compute
  • it is very, very slow to undo

Notes:

There are a lot of assumptions about why these functions are hard to invert, but we cannot rigorously prove it. We often express inversion problems in terms of mathematical games or oracles.


Hash Functions

Motivation: We often want a succinct, yet unique representation of some (potentially large) data.


A fingerprint, which is much smaller than a person, yet uniquely identifies an individual.

Notes:

The following slides serve as an intro. Many terms may be glossed over, and covered in detail later. There are lessons later in this module dedicated to hashes and hash-based data structures.

---v

Hash Function Applications

Hashes can be useful for many applications:

  • Representation of larger data object
    (history, commitment, file)
  • Keys in a database
  • Digital signatures
  • Key derivation
  • Pseudorandom functions

Symmetric Cryptography

Symmetric encryption assumes all parties begin with some shared secret information, a potentially very difficult requirement.
The shared secret can then be used to protect further communications from others who do not know this secret.

In essence, it gives a way of extending a shared secret over time.

Notes:

Remember that these communications are over an open channel, as we assumed that all channels can be monitored.


Symmetric Encryption

For example, the Enigma cipher in WW2. A channel was initiated by sharing a secret ("key") between two participants. Using the cipher, those participants could then exchange information securely.

However, since the key contained only limited entropy ("information"), enough usage of it eventually compromised the secret and allowed the allies to decode messages. Even altering it once per day was not enough.

Notes:

When communicating over a channel that is protected with only a certain amount of entropy, it is still possible to extend messages basically indefinitely by introducing new entropy that is used to protect the channel sufficiently often.


Asymmetric Cryptography

  • In asymmetric cryptography, we devise a means to transform one value (the "secret") into some corresponding counterpart (the "public" key), preserving certain properties.

  • We believe that this is a one-way function (that there is no easy/fast inverse of this function).

  • Aside from preserving certain properties, we believe this counterpart (the "public key") reveals no information about the secret.


Asymmetric Encryption

Using only the public key, information can be transformed ("encrypted") such that only those with knowledge of the secret are able to inverse and regain the original information.


Digital Signatures

  • Using the secret key, information can be transformed ("signed") such that anyone with knowledge of the information and the counterpart public key is able to affirm the operation.

  • Digital signatures provide message authenticity and integrity guarantees.

  • There are two lessons are dedicated to digital signatures,
    this is strictly an intro.


Digital Signatures

Signing function: a function which operates on some
message data and some secret to yield a signature.

A signature proves that the signer had knowledge of the secret,
without revealing the secret itself.

The signature cannot be used to create other signatures, and is unique to the message.

Notes:

A signing function is a pure function which operates on some message data (which may or may not be small, depending on the function) and some secret (a small piece of information known only to the operator). The result of this function is a small piece of data called a signature.

Pure means that it has no side effects.

It has a special property: it proves (beyond reasonable doubt) that the signer (i.e. operator of the signing function) had knowledge of the secret and utilized this knowledge with the specific message data, yet it does not reveal the secret itself, nor can knowledge of the signature be used to create other signatures (e.g. for alternative message data).


Non-repudiation for Crypgraphic Signatures

There is cryptographic proof that the secret was known to the producer of the signature.


The signer cannot claim that the signature was forged, unless they can defend a claim that the secret was compromised prior to signing.


Practical Considerations

Symmetric cryptography is much faster, but requires more setup (key establishment) and trust (someone else knows the secret).

Asymmetric cryptography is slow, but typically preserves specific algebraic relationships, which then permit more diverse if fragile protocols.


Hybrid Cryptography

Hybrid cryptography composes new mechanisms from different cryptographic primitives.

For example:

  • Symmetric encryption can provide speed, and often confidentiality,
  • Hash functions can reduce the size of data while preserving identity,
  • Asymmetric cryptography can dictate relations among the participants.

Certifications

Certifications are used to make attestations about public key relationships.

Typically in the form of a signature on:

  • One or more cryptographically strong identifiers (e.g. public keys, hashes).
  • Information about its ownership, its use and any other properties that the signer is capable of attesting/authorizing/witnessing.
  • (Meta-)information about this information itself, such as how long it is valid for and external considerations which would invalidate it.

Notes:

  • Real application is the hierarchy of SSL certs.
    • Root keys -> State level entities -> Smaller entities.
  • Web of Trust & GPG cross-signing
  • In the case of signature-based certificates, as long as you have the signature, data, and originating public key, you can trust a certificate no matter where it came from. It could be posted on a public message board, sent to you privately, or etched into stone.

Entropy, Randomness, and Key Size

  • Entropy: Amount of non-redundant information contained within some data.
  • Randomness: Unpredictability of some information. Less random implies lower entropy.
  • Key size: Upper limit of possible entropy contained in a key. Keys with less random (more predictable) data have less entropy than this upper bound.
  • One-time pad: A key of effectively infinite size. If it is perfectly random (i.e. has maximal entropy), then the cipher is theoretically unbreakable.

Notes:

Mention the upcoming "many time pad" activity, that exploits using a one time pad multiple times.


Randomness Generation

#![allow(unused)]
fn main() {
fn roll_die() -> u32 {
  // Guaranteed random: it was achieved through a real-life die-roll.
  4u32
}
}
  • Pseudo-random sequences
  • Physical data collection (e.g. cursor movement, LSB of microphone)
  • Specialised hardware (e.g. low-level noise on silicon gates, quantum-amplifiers)

Notes:

LSB := Least Significant Bit


Summary

Cryptography is much more than encryption.

  • Communicate on public networks, in the open
  • Access information
  • Have expectations about a message's authenticity and integrity
  • Prove knowledge of some secret information
  • Represent large amounts of data succinctly

Questions

What insights did you gain?

Notes:

Class discussion. Last slide.

Addresses and Keys


How to use the slides - Full screen (new tab)
Slides Content
--- title: Addresses and Keys description: Addresses and keys in cryptography duration: 30 min ---

Addresses and Keys


Outline

  1. Binary Formats
  2. Seed Creation
  3. Hierarchical Deterministic Key Derivation

Binary Display Formats

When representing binary data, there are a few different display formats you should be familiar with.

Hex: 0-9, a-f

Base64: A-Z, a-z, 0-9, +, /

Base58: Base64 without 0/O, I/l, +, and /

Notes:

Be very clear that this is a display format that we use to transmit binary data through text. The same data can be encoded with any of these formats, it's just important to know which one you're using to decode. Data is not typically stored in these formats unless it has to be transmitted through text.


Binary Display Formats Example

Every hex character is 4 bits.
Every base64 character is 6 bits.
base58 characters are usually about 6 bits.

binary: 10011111 00001010 10011110 10011000 01001100 11010011 10110010 00000101
hex:    9   f    0   a    9   e    9   8    4    c   d   3    b   2    0   5
base64: n     w      q      e      m     E      z      T      s     g      U=
base58: T     b      u      H      z     e      3      c      t     k      c

hex:    9f0a9e984cd3b205
base64: nwqemEzTsgU=
base58: TbuHze3ctkc

Notes:

It turns out that converting from hex/base64 to base58 can in theory take n^2 time!


Mnemonics and Seed Creation

Notes:

These are all different representation of a secret. Fundamentally doesn't really change anything.


Seeds are secrets

Recall, both symmetric and asymmetric cryptography require a secret.


Mnemonics

Many wallets use a dictionary of words and give people phrases,
often 12 or 24 words, as these are easier to back up/recover than byte arrays.

Notes:

High entropy needed. People are bad at being random. Some people create their own phrases... this is usually stupid.


Dictionaries

There are some standard dictionaries to define which words (and character sets) are included in the generation of a phrase. Substrate uses the dictionary from BIP39.

No.word
1abandon
2ability
3able
4about
5above

The first 5 words of the BIP39 English dictionary


Mnemonic to Secret Key

Of course, the secret key is a point on an elliptic curve, not a phrase.

BIP39 applies 2,048 rounds of the SHA-512 hash function
to the mnemonic to derive a 64 byte key.

Substrate uses the entropy byte array from the mnemonic.


Portability

Different key derivation functions affect the ability to use the same mnemonic in multiple wallets as different wallets may use different functions to derive the secret from the mnemonic.


Cryptography Types

Generally, you will encounter 3 different modern types of cryptography across most systems you use.

  • Ed25519
  • Sr25519
  • ECDSA

We will go more in depth in future lectures!

Notes:

You may have learned RSA in school. It is outdated now, and requires huge keys.


What is an address?

An address is a representation of a public key, potentially with additional contextual information.

Notes:

Having an address for a symmetric cryptography doesn't actually make any sense, because there is no public information about a symmetric key.


Address Formats

Addresses often include a checksum so that a typo cannot change one valid address to another.

Valid address:   5GEkFD1WxzmfasT7yMUERDprkEueFEDrSojE3ajwxXvfYYaF
Invalid address: 5GEkFD1WxzmfasT7yMUERDprk3ueFEDrSojE3ajwxXvfYYaF
                                          ^
                                          E changed to 3

Notes:

It hasn't been covered yet, but some addresses even go extra fancy and include an error correcting code in the address.


SS58 Address Format

SS58 is the format used in Substrate.

It is base58 encoded, and includes a checksum and some context information. Almost always, it is 2 bytes of context and 2 bytes of checksum.

base58Encode( context | public key | checksum )

Notes: | here stands for concatenation.

For ECDSA, the public key is 33 bytes, so we use the hash of it in place of the public key.

There are a lot more variants here, but this is by far the most common one.

reference


HDKD

Hierarchical Deterministic Key Derivation


Hard vs. Soft

Key derivation allows one to derive (virtually limitless)
child keys from one "parent".

Derivations can either be "hard" or "soft".


Hard vs. Soft


Hard Derivation

Hard derivation requires the secret key and derives new child secret keys.

Typical "operational security" usages should favor hard derivation over soft derivation because hard derivations avoid leaking the sibling keys, unless the original secret is compromised.

Always do hard paths first, then conclude in soft paths.


Hard Derivation in Wallets

Wallets can derive keys for use in different consensus systems while only needing to back up one secret plus a pattern for child derivation.


Hard Derivation in Wallets

Let's imagine we want to use this key on multiple networks, but we don't want the public keys to be connected to each other.


Subkey Demo

Hard Derivation

Notes:

Hard keys: Take a path (data like a name/index), concatenate with the original key, and hash it for a new key. They reveal nothing about keys above them, and only with the path between it and children could they be recovered.


Soft Derivation

Soft derivation allows one to create derived addresses from only the public key. Contrary to hard derivation, all keys are related.

Notes:

  • With any key and the paths to children and. or parents, the public and private keys can be recovered.
  • Soft derivations can break some niche advanced protocols, but our sr25519 crate avoids supporting protocols that conflict with soft derivations.

Soft Derivation

  • Note that these generate new addresses, but use the same secret seed.
  • We can also use the same paths, but only using the Account ID from //polkadot. It generates the same addresses!

Soft Derivation in Wallets

Wallets can use soft derivation to link all payments controlled by a single private key, without the need to expose the private key for the address derivation.

Use case: A business wants to generate a new address for each payment, but should be able to automatically give customers an address without the secret key owner deriving a new child.

Notes:

On the use case, taking each payment at a different address could help make the association between payment and customer.

See: https://wiki.polkadot.network/docs/learn-accounts#soft-vs-hard-derivation


Subkey Demo

Soft Derivation

Notes:

See the Jupyter notebook and/or HackMD cheat sheet for this lesson.

Mention that these derivations create entirely new secret seeds.


Questions

Subkey Signature and HDKD (Hierarchical Deterministic Key Derivation) Demo

Here are subkey examples for reference on use. Compliments the formal documentation found here.

Key Generation

subkey generate

Secret phrase:       desert piano add owner tuition tail melt rally height faint thunder immune
  Network ID:        substrate
  Secret seed:       0x6a0ea68072cfd0ffbabb40801570fa5e9f3a88966eaed9dedaeb0cf140b9cd8d
  Public key (hex):  0x7acdc47530002fbc50f413859093b7df90c27874aee732dca940ea4842751d58
  Account ID:        0x7acdc47530002fbc50f413859093b7df90c27874aee732dca940ea4842751d58
  Public key (SS58): 5Eqipnpt5asTm7sCFWQeJjsNJX5cYVJMid3zjKHjDUGKBJTo
  SS58 Address:      5Eqipnpt5asTm7sCFWQeJjsNJX5cYVJMid3zjKHjDUGKBJTo

Sign

echo -n 'Hello Polkadot Blockchain Academy' | subkey sign --suri 'desert piano add owner tuition tail melt rally height faint thunder immune'

Note, this changes each execution, this is one viable signature: f261d56b80e4b53c70dd2ba1de6b9384d85a8f4c6d912fd86acab3439a47992aa85ded04ac55c7525082dcbc815001cd5cc94ec1a907bbd8e3138cfc8a382683

Verify

echo -n 'Hello Polkadot Blockchain Academy' | subkey verify  '0xf261d56b80e4b53c70dd2ba1de6b9384d85a8f4c6d912fd86acab3439a47992aa85ded04ac55c7525082dcbc815001cd5cc94ec1a907bbd8e3138cfc8a382683' \
    '0x7acdc47530002fbc50f413859093b7df90c27874aee732dca940ea4842751d58'

Expect Signature verifies correctly.

Tamper with the Message

Last char in Public key (hex) - AKA URI - is changed:

echo -n 'Hello Polkadot Blockchain Academy' | subkey verify \
	'0xf261d56b80e4b53c70dd2ba1de6b9384d85a8f4c6d912fd86acab3439a47992aa85ded04ac55c7525082dcbc815001cd5cc94ec1a907bbd8e3138cfc8a382683' \
    '0x7acdc47530002fbc50f413859093b7df90c27874aee732dca940ea4842751d59'
Error: SignatureInvalid

Hard Derivation

subkey inspect 'desert piano add owner tuition tail melt rally height faint thunder immune//polkadot' --network polkadot

Secret Key URI `desert piano add owner tuition tail melt rally height faint thunder immune//polkadot` is account:
  Network ID:        polkadot
 Secret seed:       0x3d764056127d0c1b4934725cb9faecf00ed0996daa84d24a903b906f319e06bf
  Public key (hex):  0xce6ccb0af417ade10062ac9b553d506b67d16c61cd2b6ce85330bc023db7e906
  Account ID:        0xce6ccb0af417ade10062ac9b553d506b67d16c61cd2b6ce85330bc023db7e906
  Public key (SS58): 15ffBb8rhETizk36yaevSKM2MCnHyuQ8Dn3HfwQFtLMhy9io
  SS58 Address:      15ffBb8rhETizk36yaevSKM2MCnHyuQ8Dn3HfwQFtLMhy9io
subkey inspect 'desert piano add owner tuition tail melt rally height faint thunder immune//kusama' --network kusama

Secret Key URI `desert piano add owner tuition tail melt rally height faint thunder immune//kusama` is account:
  Network ID:        kusama
 Secret seed:       0xabd92064a63df86174acfd29ab3204897974f0a39f5d61efdd30099aa5f90bd9
  Public key (hex):  0xf62e5d444f89e704bb9b412adc472f990e9a9f40725ac6ff3abee1c9b7625a63
  Account ID:        0xf62e5d444f89e704bb9b412adc472f990e9a9f40725ac6ff3abee1c9b7625a63
  Public key (SS58): J9753RnTdZJct5RmFQ6gFVdKSyrEjzYwvYUBufMX33PB7az
  SS58 Address:      J9753RnTdZJct5RmFQ6gFVdKSyrEjzYwvYUBufMX33PB7az

Soft Derivation from Secret

subkey inspect 'desert piano add owner tuition tail melt rally height faint thunder immune//polkadot/0' --network polkadot

Secret Key URI `desert piano add owner tuition tail melt rally height faint thunder immune//polkadot/0` is account:
  Network ID:        polkadot
 Secret seed:       n/a
  Public key (hex):  0x4e8dfdd8a386ae37b8731dba5480d5cc65739023ea24f1a09d88be1bd9dff86b
  Account ID:        0x4e8dfdd8a386ae37b8731dba5480d5cc65739023ea24f1a09d88be1bd9dff86b
  Public key (SS58): 12mzv68gS8Zu2iEdt4Ktkt48JZSKyFSkAVjvtgYhoa42NLNa
  SS58 Address:      12mzv68gS8Zu2iEdt4Ktkt48JZSKyFSkAVjvtgYhoa42NLNa
subkey inspect 'desert piano add owner tuition tail melt rally height faint thunder immune//polkadot/1' --network polkadot

Secret Key URI `desert piano add owner tuition tail melt rally height faint thunder immune//polkadot/1` is account:
  Network ID:        polkadot
 Secret seed:       n/a
  Public key (hex):  0x2e8b3090b17b12ea63029f03d852af71570e8e526690cc271491318a45785e33
  Account ID:        0x2e8b3090b17b12ea63029f03d852af71570e8e526690cc271491318a45785e33
  Public key (SS58): 1242YwUZGBQ84btGSGdSX4swf1ibfSaCDR1sr1ejC9KQ1NbJ
  SS58 Address:      1242YwUZGBQ84btGSGdSX4swf1ibfSaCDR1sr1ejC9KQ1NbJ

Soft Derivation from Public

Note: We use addresses here because Subkey does not derive paths from a raw public key (AFAIK).

subkey inspect 12mzv68gS8Zu2iEdt4Ktkt48JZSKyFSkAVjvtgYhoa42NLNa/0

Public Key URI `12mzv68gS8Zu2iEdt4Ktkt48JZSKyFSkAVjvtgYhoa42NLNa/0` is account:
  Network ID/Version: polkadot
  Public key (hex):   0x40f22875159420aca51178d1baf2912c18dcb83737dd7bd39dc6743da326dd1c
  Account ID:         0x40f22875159420aca51178d1baf2912c18dcb83737dd7bd39dc6743da326dd1c
  Public key (SS58):  12UA12xuDnEkEsEDrR4T4Cf3S1Hyi2C7B6hJW8LTkcsZy8BX
  SS58 Address:       12UA12xuDnEkEsEDrR4T4Cf3S1Hyi2C7B6hJW8LTkcsZy8BX
subkey inspect 12mzv68gS8Zu2iEdt4Ktkt48JZSKyFSkAVjvtgYhoa42NLNa/1

Public Key URI `12mzv68gS8Zu2iEdt4Ktkt48JZSKyFSkAVjvtgYhoa42NLNa/1` is account:
  Network ID/Version: polkadot
  Public key (hex):   0xc62ec5cd7d83e1f41462d455bb47b6bad9ed5a14741a920ead8366c63746391b
  Account ID:         0xc62ec5cd7d83e1f41462d455bb47b6bad9ed5a14741a920ead8366c63746391b
  Public key (SS58):  15UrNnNSMpX49F3mWcCX7y4kMGcvnQxCabLMT3d8U5abpwr3
  SS58 Address:       15UrNnNSMpX49F3mWcCX7y4kMGcvnQxCabLMT3d8U5abpwr3

Hash Functions


How to use the slides - Full screen (new tab)
Slides Content
--- title: Hash Functions description: Hash functions in Substrate duration: 1 hour ---

Hash Functions


Introduction

We often want a succinct representation of some data
with the expectation that we are referring to the same data.

A "fingerprint"

Hash Function Properties

  1. Accept unbounded size input
  2. Map to a bounded output
  3. Be fast to compute
  4. Be computable strictly one-way
    (difficult to find a pre-image for a hash)
  5. Resist pre-image attacks
    (attacker controls one input)
  6. Resist collisions
    (attacker controls both inputs)

Hash Function API

A hash function should:

  • Accept an unbounded input size ([u8] byte array)
  • Return a fixed-length output (here, a 32 byte array).
#![allow(unused)]
fn main() {
fn hash(s: &[u8]) -> [u8; 32];
}

Example

Short input (5 bytes):

hash('hello') =
 0x1c8aff950685c2ed4bc3174f3472287b56d9517b9c948127319a09a7a36deac8

Large input (1.2 MB):

hash(Harry_Potter_series_as_string) =
 0xc4d194054f03dc7155ccb080f1e6d8519d9d6a83e916960de973c93231aca8f4

Input Sensitivity

Changing even 1 bit of a hash function completely scrambles the output.

hash('hello') =
 0x1c8aff950685c2ed4bc3174f3472287b56d9517b9c948127319a09a7a36deac8
hash('hellp') =
 0x7bc9c272894216442e0ad9df694c50b6a0e12f6f4b3d9267904239c63a7a0807

Rust Demo

Hashing a Message

Notes:

See the Jupyter notebook and/or HackMD cheat sheet for this lesson.

  1. Use a longer message
  2. Hash it
  3. Verify the signature on the hash

Speed

Some hash functions are designed to be slow.

These have applications like password hashing, which would slow down brute-force attackers.

For our purposes, we generally want them to be fast.


Famous Hash Algorithms

  • xxHash a.k.a TwoX (non-cryptographic)
  • MD5
  • SHA1
  • RIPEMD-160
  • SHA2-256 (aka SHA256) &c.
  • SHA3
  • Keccak
  • Blake2

xxHash64 is about 20x faster than Blake2.


Hash Functions in Blockchains

  • Bitcoin: SHA2-256 & RIPMD-160
  • Ethereum: Keccak-256 (though others supported via EVM)
  • Polkadot: Blake2 & xxHash (though others supported via host functions)

Notes:

Substrate also implements traits that provide 160, 256, and 512 bit outputs for each hasher.

Exercise: Write your own benchmarking script that compares the performance of these algorithms with various input sizes.


Hashing Benchmarks

Notes:

Benchmarks for the cryptographic hashing algorithms. Source: https://www.blake2.net/


XXHash - Fast hashing algorithm

Notes:

Benchmarks for the XX-hash algorithms. Source: https://github.com/Cyan4973/xxHash#benchmarks


Non-Cryptographic Hash Functions

Non-cryptographic hash functions provide weaker
guarantees in exchange for performance.

They are OK to use when you know that the input is not malicious.

If in doubt, use a cryptographic hash function.


One Way

Given a hash, it should be difficult to find an input value (pre-image)
that would produce the given hash.

That is, given H(x), it should be difficult to find x.

Notes:

We sometimes add random bytes to pre-images to prevent guesses based on context (e.g., if you are hashing "rock, paper, scissors", then finding a pre-image is trivial without some added randomness.)


Second Pre-Image Attacks

Given a hash and a pre-image, it should be difficult to find another
pre-image that would produce the same hash.

Given H(x), it should be difficult to find any x'

such that H(x) == H(x').

Notes:

Since most signature schemes perform some internal hashing, this second pre-image would also pass signature verification.


Collision Resistance

It should be difficult for someone to find two messages that
hash to the same value.

It should be difficult to find an x and y

such that H(x) == H(y).


Collision Resistance

Difference from second pre-image attack:

In a second pre-image attack, the attacker only controls one input.

In a collision, the attacker controls both inputs.

They may attempt to trick someone into signing one message.

Notes:

Attacker has intention to impersonate the signer with the other. Generally speaking, even finding a single hash collision often results in the hash function being considered unsafe.


Birthday Problem

With 23 people, there is a 6% chance that someone will be born on a specific date, but a 50% chance that two share a birthday.

  • Must compare each output with every other, not with a single one.
  • Number of possible "hits" increases exponentially for more attempts, reducing the expected success to the square-root of what a specific target would be.

Birthday Attack

Thus, with a birthday attack, it is possible to find a collision of a hash function in $\sqrt {2^{n}}=2^{^{\frac{n}{2}}}$, with $\cdot 2^{^{\frac{n}{2}}}$ being the classical preimage resistance security.

So, hash function security is only half of the bit space.

Notes:

e.g., a 256 bit hash output yields 2^128 security


Partial Resistance

It should be difficult for someone to partially (for a substring of the hash output) find a collision or "second" pre-image.

  • Bitcoin PoW is a partial pre-image attack.
  • Prefix/suffix pre-image attack resistance reduces opportunity for UI attacks for address spoofing.
  • Prefix collision resistance important to rationalize costs for some cryptographic data structures.

Hash Function Selection

When users (i.e. attackers) have control of the input, cryptographic hash functions must be used.

When input is not controllable (e.g. a system-assigned index), a non-cryptographic hash function can be used and is faster.

Notes:

Only safe when the users cannot select the pre-image, e.g. a system-assigned index.

Keccak is available for Ethereum compatibility.


Applications


Cryptographic Guarantees

Let's see which cryptographic properties apply to hashes.

---v

Confidentiality

Sending or publically posting a hash of some data $D$ keeps $D$ confidential, as only those who already knew $D$ recognize $H(D)$ as representing $D$.

Both cryptographic and non-cryptographic hashes work for this. only if the input space is large enough.

---v

Confidentiality Bad Example

Imagine playing rock, paper, scissors by posting hashes and then revealing. However, if the message is either "rock", "paper", or "scissors", the output will always be either:

hash('rock') = 0x10977e4d68108d418408bc9310b60fc6d0a750c63ccef42cfb0ead23ab73d102
hash('paper') = 0xea923ca2cdda6b54f4fb2bf6a063e5a59a6369ca4c4ae2c4ce02a147b3036a21
hash('scissors') = 0x389a2d4e358d901bfdf22245f32b4b0a401cc16a4b92155a2ee5da98273dad9a

The other player doesn't need to undo the hash function to know what you played!

Notes:

The data space has to be sufficiently large. Adding some randomness to input of the hash fixes this. Add x bits of randomness to make it x bits of security on that hash.

---v

Authenticity

Anyone can make a hash, so hashes provide no authenticity guarantees.

---v

Integrity

A hash changes if the data changes, so it does provide integrity.

---v

Non-Repudiation

Hashes on their own cannot provide authenticity, and as such cannot provide non-repudiation.

However, if used in another cryptographic primitive that does provide non-repudiation, $H(D)$ provides the same non-repudation as $D$ itself.

Notes:

This is key in digital signatures. However, it's important to realize that if $D$ is kept secret, $H(D)$ is basically meaningless.


Content-Derived Indexing

Hash functions can be used to generate deterministic
and unique lookup keys for databases.

Notes:

Given some fixed property, like an ID and other metadata the user knows beforehand, they can always find the database entry with all of the content they are looking for.


Data Integrity Checks

Members of a peer-to-peer network may host and share
file chunks rather than large files.

In Bittorrent, each file chunk is hash identified so peers can
request and verify the chunk is a member of the larger,
content addressed file.

Notes:

The hash of the large file can also serve as a signal to the protocol that transmission is complete.


Account Abstractions

Public keys can be used to authorize actions by signing of instructions.

The properties of hash functions allow other kinds of representations.


Public Key Representation

Because hashes serve as unique representations of other data,
that other data could include public keys.
A system can map a plurality of key sizes to a fixed length
(e.g. for use as a database key).

For example, the ECDSA public key is 33 bytes:

Public key (hex):
  0x02d82cdc83a966aaabf660c4496021918466e61455d2bc403c34bde8148b227d7a

Hash of pub key:
  0x8fea32b38ed87b4739378aa48f73ea5d0333b973ee72c5cb7578b143f82cf7e9
                                                                    ^^

Commitment Schemes

It is often useful to commit to some information
without storing or revealing it:

  • A prediction market would want to reveal predictions only after the confirming/refuting event occurred.
  • Users of a system may want to discuss proposals without storing the proposal on the system.

However, participants should not be able to modify their predictions or proposals.


Commit-Reveal

  1. Share a hash of data as a commitment ($c$)
  2. Reveal the data itself ($d$)

It is normal to add some randomness to the message
to expand the input set size:

$$ hash(message + randomness) => commitment $$

Commitment: 0x97c9b8d5019e51b227b7a13cd2c753cae2df9d3b435e4122787aff968e666b0b


Reveal

Message with some added randomness:

"I predict Boris Johnson will resign on 7 July 2022. facc8d3303c61ec1808f00ba612c680f"


Data Identifiers

Sometimes people want to store information in one place and reference it in another. For reference, they need some "fingerprint" or digest.

As an example, they may vote on executing some privileged instructions within the system.

The hash of the information can succinctly represent the information and commit its creator to not altering it.


Data Structures (in Brief)

This is the focus of a later lesson.

Notes: For now, just a brief introduction.


Pointer-Based Linked Lists

Pointer-based linked lists are a foundation of programming.

But pointers are independent from the data they reference,
so the data can be modified while maintaining the list.

That is, pointer-based linked lists are not tamper evident.


Hash-Based Linked Lists

Hash-based lists make the reference related to the data they are referencing.
The properties of hash functions make them a good choice for this application.

Any change at any point in the list would create downstream changes to all hashes.


Merkle Trees

Notes:

Each leaf is the hash of some data object and each node is the hash of its children.


Proofs

Merkle trees allow many proofs relevant to the rest of this course,
e.g. that some data object is a member of the tree
without passing the entire tree.

More info in the next lesson.


Questions


Hash Examples in Substrate

Sr25519 Signatures

Sr25519 hashes the message as part of its signing process.

Transactions

In transactions in Substrate, key holders sign a
hash of the instructions when the instructions
are longer than 256 bytes.


Database Keys

TwoX64 is safe to use when users (read: attackers)
cannot control the input, e.g. when a
database key is a system-assigned index.

Blake2 should be used for everything else.

Again, there is a whole lesson on hash-based data structures.


Other Uses of Hashes in Substrate

Hashes are also used for:

  • Generating multisig accounts
  • Generating system-controlled accounts
  • Generating proxy-controlled accounts
  • Representing proposals
  • Representing claims (e.g. the asset trap)

Activity: Crack the Many Time Pad

Instructors: there is a private guide associated with this activity to assist with hints and further details, be sure to review this before starting.

Introduction

The symmetric one-time pad is known to be secure when the key is only used once. In practice key distribution is not always practical, and users sometimes make the critical mistake of reusing a pre-shared key.

In this activity, you will experience first hand why reusing the key is detrimental to security.

The Challenge

The following several ciphertexts were intercepted on a peer-to-peer communication channel:

  • Messages definitively originate in the USA, destined for the UK.
  • Each line contains one hex encoded message, in it's entirety.
  • We believe all messages were encrypted with the same key.

Your task is to use cryptanalysis to recover the plaintexts of all messages, as well as the encryption key used for them.

160111433b00035f536110435a380402561240555c526e1c0e431300091e4f04451d1d490d1c49010d000a0a4510111100000d434202081f0755034f13031600030d0204040e
050602061d07035f4e3553501400004c1e4f1f01451359540c5804110c1c47560a1415491b06454f0e45040816431b144f0f4900450d1501094c1b16550f0b4e151e03031b450b4e020c1a124f020a0a4d09071f16003a0e5011114501494e16551049021011114c291236520108541801174b03411e1d124554284e141a0a1804045241190d543c00075453020a044e134f540a174f1d080444084e01491a090b0a1b4103570740
000000000000001a49320017071704185941034504524b1b1d40500a0352441f021b0708034e4d0008451c40450101064f071d1000100201015003061b0b444c00020b1a16470a4e051a4e114f1f410e08040554154f064f410c1c00180c0010000b0f5216060605165515520e09560e00064514411304094c1d0c411507001a1b45064f570b11480d001d4c134f060047541b185c
0b07540c1d0d0b4800354f501d131309594150010011481a1b5f11090c0845124516121d0e0c411c030c45150a16541c0a0b0d43540c411b0956124f0609075513051816590026004c061c014502410d024506150545541c450110521a111758001d0607450d11091d00121d4f0541190b45491e02171a0d49020a534f
031a5410000a075f5438001210110a011c5350080a0048540e431445081d521345111c041f0245174a0006040002001b01094914490f0d53014e570214021d00160d151c57420a0d03040b4550020e1e1f001d071a56110359420041000c0b06000507164506151f104514521b02000b0145411e05521c1852100a52411a0054180a1e49140c54071d5511560201491b0944111a011b14090c0e41
0b4916060808001a542e0002101309050345500b00050d04005e030c071b4c1f111b161a4f01500a08490b0b451604520d0b1d1445060f531c48124f1305014c051f4c001100262d38490f0b4450061800004e001b451b1d594e45411d014e004801491b0b0602050d41041e0a4d53000d0c411c41111c184e130a0015014f03000c1148571d1c011c55034f12030d4e0b45150c5c
011b0d131b060d4f5233451e161b001f59411c090a0548104f431f0b48115505111d17000e02000a1e430d0d0b04115e4f190017480c14074855040a071f4448001a050110001b014c1a07024e5014094d0a1c541052110e54074541100601014e101a5c
0c06004316061b48002a4509065e45221654501c0a075f540c42190b165c
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

Instructions

  1. Team up with 2-4 students to complete this activity.

  2. Briefly inspect the ciphertext to see if you can identify patterns that may hint some things about their origin.

  3. Research based on what we know bout the messages to find clues to help come up with a theory and game plan to complete your task.

  4. Write a program in Rust that finds the key to generate the plaintext from the provided cipher texts.
    The general steps are:

    1. Find the length of the longest input cipher text.
    2. Generate a key of that length.
    3. Find what the correct key is...

Note that this task is intended to be a bit vague, give it your best effort. We will be sharing hints as time progresses for everyone. Don't hesitate to ask for support if you're feeling stuck, or just ask your peers!

Finished?

Once complete, let a faculty know!

One last ciphertext using the same key that should prove tricky:

1f3cb1f3e01f3fd1f3ea1f3e61f3e01f3e71f3b31f3a91f3c81f3a91f3f91f3fc1f3fb1f3ec1f3e51f3f01f3a91f3f91f3ec1f3ec526e1b014a020411074c17111b1c071c4e4f0146430d0d08131d1d010707040017091648461e1d0618444f074c010e19594f0f1f1a07024e1d041719164e1c1652114f411645541b004e244f080213010c004c3b4c0911040e480e070b00310213101c4d0d4e00360b4f151a005253184913040e115454084f010f114554111d1a550f0d520401461f3e01f3e71f3e81f3e71f3ea1f3e01f3e81f3e51f3a91f3e01f3e71f3fa1f3fd1f3e01f3fd1f3fc1f3fd1f3e01f3e61f3e71f3a7

Notice a pattern? why might that be... ๐Ÿค”

If you want more to do, find ways to improve your solution, perhaps:

  • Create a tool that automates the cipher key generation.
  • Add a way to generate new cipher texts.
  • Create your own cipher texts using other cipher methods.
  • Provide a new set of ciphertexts that were intentionally constructed not to use the most common English words.

Citation

This activity is cribbed from Dan Boneh's Coursera Cryptography I course.

Encryption


How to use the slides - Full screen (new tab)
Slides Content
--- title: Encryption description: A lesson on symmetric and asymmetric encryption. duration: 1 hour ---

Encryption


Goals for this lesson

  • Learn about the differences between symmetric and asymmetric encryption.

Symmetric Cryptography

Symmetric encryption assumes all parties begin with some shared secret information, a potentially very difficult requirement.
The shared secret can then be used to protect further communications from others who do not know this secret.

In essence, it gives a way of extending a shared secret over time.


Symmetric Encryption

Examples: ChaCha20, Twofish, Serpent, Blowfish, XOR, DES, AES


Symmetric Encryption API

Symmetric encryption libraries should generally all expose some basic functions:

  • fn generate_key(r) -> k;
    Generate a k (secret key) from some input r.
  • fn encrypt(k, msg) -> ciphertext;
    Takes k and a message; returns the ciphertext.
  • fn decrypt(k, ciphertext) -> msg;
    Takes k and a ciphertext; returns the original message.

It always holds that decrypt(k, encrypt(k, msg)) == msg.

Notes:

The input r is typically a source of randomness, for example the movement pattern of a mouse.


Symmetric Encryption Guarantees

Provides:

  • Confidentiality
  • Authenticity*

Does not provide:

  • Integrity*
  • Non-Repudiation

Notes:

  • Authenticity: The message could only be sent by someone who knows the shared secret key. In most cases, this is functionally authentication to the receiving party.
  • Integrity: There is no proper integrity check, however the changed section of the message will be gibberish if it has been changed. Detection of gibberish could function as a form of integrity-checking.

Non-repudiation for Symmetric Encryption

There is cryptographic proof that the secret was known to the producer of the encrypted message.


However, knowledge of the secret is not restricted to one party: Both (or all) parties in a symmetrically encrypted communication know the secret. Additionally, in order to prove this to anyone, they must also gain knowledge of the secret.

Notes:

The degree of non-repudiation given by pure symmetric crytography is not very useful.


Symmetric Encryption

Example: XOR Cipher

The encryption and decryption functions are identical: applying a bitwise XOR operation with a key.

Plain: 1010  -->Cipher: 0110
Key:   1100  |          1100
       ----  |          ----
       0110--^          1010

Notes:

A plaintext can be converted to ciphertext, and vice versa, by applying a bitwise XOR operation with a key known to both parties.


Symmetric Encryption

โš  Warning โš 

We typically expect symmetric encryption to preserve little about the original plaintext. We caution however that constructing these protocols remains delicate, even given secure primitives, with two classical examples being unsalted passwords and the ECB penguin.


ECB penguin

Original image

Encrypted image

(by blocks)

Encrypted image

(all at once)

Notes:

The ECB penguin shows what can go wrong when you encrypt a small piece of data, and do this many times with the same key, instead of encrypting data all at once.

Image sources: https://github.com/robertdavidgraham/ecb-penguin/blob/master/Tux.png and https://github.com/robertdavidgraham/ecb-penguin/blob/master/Tux.ecb.png and https://upload.wikimedia.org/wikipedia/commons/5/58/Tux_secure.png


Asymmetric Encryption

  • Assumes the sender does not know the recipient's secret "key" ๐ŸŽ‰๐Ÿ˜Ž
  • Sender only knows a special identifier of this secret
  • Messages encrypted with the special identifier can only be decrypted with knowledge of the secret.
  • Knowledge of this identifier does not imply knowledge of the secret, and thus cannot be used to decrypt messages encrypted with it.
  • For this reason, the identifier may be shared publicly and is known as the public key.

Asymmetric Encryption


Why "Asymmetric"?

Using only the public key, information can be transformed ("encrypted") such that only those with knowledge of the secret are able to inverse and regain the original information.

i.e. Public key is used to encrypt but a different, secret, key must be used to decrypt.


Asymmetric Encryption API

Asymmetric encryption libraries should generally all expose some basic functions:

  • fn generate_key(r) -> sk;
    Generate a sk (secret key) from some input r.
  • fn public_key(sk) -> pk;
    Generate a pk (public key) from the private key sk.
  • fn encrypt(pk, msg) -> ciphertext;
    Takes the public key and a message; returns the ciphertext.
  • fn decrypt(sk, ciphertext) -> msg;
    For the inputs sk and a ciphertext; returns the original message.

It always holds that decrypt(sk, encrypt(public_key(sk), msg)) == msg.

Notes:

The input r is typically a source of randomness, for example the movement pattern of a mouse.


Asymmetric Encryption Guarantees

Provides:

  • Confidentiality

Does not provide:

  • Integrity*
  • Authenticity
  • Non-Repudiation

Notes:

  • Authenticity: The message could only be sent by someone who knows the shared secret key. In most cases, this is functionally authentication to the receiving party.
  • Integrity: There is no proper integrity check, however the changed section of the message will be gibberish if it has been changed. Detection of gibberish could function as a form of integrity-checking.

Diffie-Hellman Key Exchange

Mixing Paint Visualization

Notes:

Mixing paint example. Image Source: https://upload.wikimedia.org/wikipedia/commons/4/46/Diffie-Hellman_Key_Exchange.svg


Authenticated Encryption

Authenticated encryption adds a Message Authentication Code to additionally provide an authenticity and integrity guarantee to encrypted data.

A reader can check the MAC to ensure the message was constructed by someone knowing the secret.

Notes:

Specifically, this authenticity says that anyone who does not know the sender's secret could not construct the message.

Generally, this adds ~16-32 bytes of overhead per encrypted message.


AEAD (Authenticated Encryption Additional Data)

AEAD is authenticated with some extra data which is unencrypted, but does have integrity and authenticity guarantees.

Notes:

Authenticated encryption and AEAD can work with both symmetric and asymmetric cryptography.


AEAD Example

Imagine a table with encrypted medical records stored in a table, where the data is stored using AEAD. What are the advantages of such a scheme?

UserID -> Data (encrypted), UserID (additional data)

Notes: By using this scheme, the data is always associated with the userID. An attacker could not put that entry into another user's entry.


Hybrid Encryption

Hybrid encryption combines the best of all worlds in encryption. Asymmetric encryption establishes a shared secret between the sender and a specific public key, and then uses symmetric encryption to encrypt the actual message. It can also be authenticated.

Notes:

In practice, asymmetric encryption is almost always hybrid encryption.


Cryptographic Properties

PropertySymmetricAsymmetricAuthenticatedHybrid + Authenticated
ConfidentialityYesYesYesYes
AuthenticityYes*NoYes*Yes
IntegrityNo*No*YesYes
Non-repudiationNoNo*NoNo*

Notes:

  • Symmetric-Authentication and Authenticated-Authenticity: The message could only be sent by someone who knows the shared secret key. In most cases, this is functionally authentication to the receiving party.
  • Symmetric-Integrity and Asymmetric-Integrity: There is no proper integrity check, however the message will be gibberish if it has been changed. Detection of gibberish could function as a form of integrity-checking.
  • Non-Repudation: Even though none of these primitives provide non-repudiation on their own, it's very possible to add non-repudation to asymmetric and hybrid schemes via signatures.
  • Note that encryption also, most importantly, makes the data available to everyone who should have access.

Questions

Digital Signature Basics


How to use the slides - Full screen (new tab)
Slides Content
--- title: Digital Signature Basics description: Hands-on application of digital signature duration: 1 hour ---

Digital Signatures Basics


Signature API

Signature libraries should generally all expose some basic functions:

  • fn generate_key(r) -> sk;
    Generate a sk (secret key) from some input r.
  • fn public_key(sk) -> pk;
    Return the pk (public key) from a sk.
  • fn sign(sk, msg) -> signature;
    Takes sk and a message; returns a digital signature.
  • fn verify(pk, msg, signature) -> bool;
    For the inputs pk, a message, and a signature; returns whether the signature is valid.

Notes:

The input r could be anything, for example the movement pattern of a mouse.

For some cryptographies (ECDSA), the verify might not take in the public key as an input. It takes in the message and signature, and returns the public key if it is valid.


Subkey Demo

Key Generation and Signing

Notes:

See the Jupyter notebook and/or HackMD cheat sheet for this lesson.

  1. Generate a secret key
  2. Sign a message
  3. Verify the signature
  4. Attempt to alter the message

Hash Functions

There are two lessons dedicated to hash functions.
But they are used as part of all signing processes.

For now, we only concern ourselves with using Blake2.


Hashed Messages

As mentioned in the introduction,
it's often more practical to sign the hash of a message.

Therefore, the sign/verify API may be used like:

  • fn sign(sk, H(msg)) -> signature;
  • fn verify(pk, H(msg), signature) -> bool;

Where H is a hash function (for our purposes, Blake2).
This means the verifier will need to run the correct hash function on the message.


Cryptographic Guarantees

Signatures provide many useful properties:

  • Confidentiality: Weak, the same as a hash
  • Authenticity: Yes
  • Integrity: Yes
  • Non-repudiation: Yes

Notes:

If a hash is signed, you can prove a signature is valid without telling anyone the actual message that was signed, just the hash.


Signing Payloads

Signing payloads are an important part of system design.
Users should have credible expectations about how their messages are used.

For example, when a user authorizes a transfer,
they almost always mean just one time.

Notes:

There need to be explicit rules about how a message is interpreted. If the same signature can be used in multiple contexts, there is the possibility that it will be maliciously resubmitted.

In an application, this typically looks like namespacing in the signature payload.


Signing and Verifying

Notes:

Note that signing and encryption are not inverses.


Replay Attacks

Replay attacks occur when someone intercepts and resends a valid message.
The receiver will carry out the instructions since the message contains a valid signature.

  • Since we assume that channels are insecure, all messages should be considered intercepted.
  • The "receiver", for blockchain purposes, is actually an automated system.

Notes:

Lack of context is the problem. Solve by embedding the context and intent _within the message being signed. Tell the story of Ethereum Classic replays.


Replay Attack Prevention

Signing payloads should be designed so that they can
only be used one time and in one context.
Examples:

  • Monotonically increasing account nonces
  • Timestamps (or previous blocks)
  • Context identifiers like genesis hash and spec versions

Signature Schemes


ECDSA

  • Uses Secp256k1 elliptic curve.
  • ECDSA (used initially in Bitcoin/Ethereum) was developed to work around the patent on Schnorr signatures.
  • ECDSA complicates more advanced cryptographic techniques, like threshold signatures.
  • Nondeterministic

Ed25519

  • Schnorr signature designed to reduce mistakes in implementation and usage in classical applications, like TLS certificates.
  • Signing is 20-30x faster than ECDSA signatures.
  • Deterministic

Sr25519

Sr25519 addresses several small risk factors that emerged
from Ed25519 usage by blockchains.


Use in Substrate

  • Sr25519 is the default key type in most Substrate-based applications.
  • Its public key is 32 bytes and generally used to identify key holders (likewise for ed25519).
  • Secp256k1 public keys are 33 bytes, so their hash is used to represent their holders.

Questions

Advanced Digital Signatures


How to use the slides - Full screen (new tab)
Slides Content
--- title: Advanced Digital Signatures description: More exotic digital signature methods duration: 1 hour ---

Advanced Digital Signatures


Certificates

A certificate is essentially a witness statement concerning one or more public keys. It is a common usage of digital signatures, but it is not a cryptographic primitive!

Notes:

A certificate is one issuing key signing a message containing another certified key, which attests to some properties or relationship about the certified key.

We must already trust the issuing key to give this attestation any significance, traditionally provided under "Certificate Authority" or "Web of Trust" schemes.


Certificates

A certification system specified conventions on who is allowed to issue certificates, the rules over their issuance (e.g. time limits and revocation) as well as their format and semantics.

For example, the certificate transparency protocol for TLS certificates helps protect against compromised Certificate Authorities.

Notes:

Certificate transparency: explanation and dashboard

  • Maybe mention PGP web-of-trust style schemes

Certificates in Web3

We are building systems that don't have a "Certificate Authority".
But we can still use certificates in some niche instances.

Notes:

Potential example to give verbally:

  • Session keys are a set of keys that generally run in online infrastructure. An account, whose keys are protected, can sign a transaction to certify all the keys in the set.
  • Session keys are used to sign operational messages, but also in challenge-response type games to prove availability by signing a message.

Multi-Signatures

We often want signatures that must be signed
by multiple parties to become valid.

  • Require some threshold of members to
    agree to a message
  • Protect against key loss

Types of Multi-Signature

  • Verifier enforced
  • Cryptographic threshold
  • Cryptographic non-threshold
    (a.k.a. signature aggregation)

Verifier Enforced Multiple Signatures

We assume that there is some verifier, who can check that some threshold of individual keys have provided valid signatures.

This could be a trusted company or third party. For our purposes, it's a blockchain.


Verifier Enforced Multiple Signatures

Multiple signatures enforced by a verifier generally provide a good user experience, as no interaction is required from the participants.

Notes:

This good experience comes at the cost of using state and more user interactions with the system, but is generally low.

Even in a web3 system, the verifier can be distinct from the blockchain. 5 people can entrust a verifier with the identity of "all 5 signed this" associated to a verifier-owned private key.


Cryptographic Multi-Sigs

We want a succinct way to demonstrate that everyone from some set of parties have signed a message. This is achieved purely on the signer side (without support from the verifier).

Example: "The five key holders have signed this message."


Key Generation for Multi-Sigs

In regular multi-signatures,
signatures from individual public keys are aggregated.

Each participant can choose their own key to use for the multi-signature.

Notes:

In some cases, a security requirement of these systems is that every participant demonstrates ownership of the public key submitted for the multi-signature, otherwise security can be compromised.


Cryptographic Threshold Multi-Sigs

This makes more compact signatures compatible with legacy systems. Unlike a regular multi-sig, the public key is associated with a threshold number of signing parties, so not all parties are needed to take part in the signing process to create a valid signature.

This requires MPC protocols and may need multiple rounds of interaction to generate the final signature. They may be vulnerable to DoS from a malfunctioning (or malicious) key-holder.

Example: "5 of 7 key holders have signed this message."

Notes:

These require multi-party computation (MPC) protocols, which add some complexity for the signing users.


Key Generation - Threshold

Threshold multi-signature schemes require that all signers run a distributed key generation (DKG) protocol that constructs key shares.

The secret encodes the threshold behavior, and signing demands some threshold of signature fragments.

This DKG protocol breaks other useful things, like hard key derivation.


Schnorr Multi-Sigs

Schnorr signatures are primarily used for threshold multi-sig.

  • Fit legacy systems nicely, and can reduce fees on blockchains.
  • Reduce verifier costs in bandwidth & CPU time, so great for certificates.
  • Could support soft key derivations.

Schnorr Multi-Sigs

However, automation becomes tricky.

We need agreement upon the final signer list and two random nonce contributions from each prospective signer, before constructing the signature fragments.


BLS Signatures

BLS signatures are especially useful for aggregated (non-threshold) multi-signatures (but can be used for threshold as well).

Signatures can be aggregated without advance agreement upon the signer list, which simplifies automation and makes them useful in consensus.

Verifying individual signatures is slow, but verifying aggregated ones is relatively fast.

(Coming to Substrate soonish.)


BLS Signatures

Allows multiple signatures generated under multiple public keys for multiple messages to be aggregated into a single signature.

  • Uses heavier pairing friendly elliptic curves than ECDSA/Schnorr.
  • Very popular for consensus.

BLS Signatures

However...

  • DKGs remain tricky (for threshold).
  • Soft key derivations are typically insecure for BLS.
  • Verifiers are hundreds of times slower than Schnorr, due to using pairings, for a single signature.
  • But for hundreds or thousands of signatures on the same message, aggregated signature verification can be much faster than Schnorr.

Schnorr and BLS Summary

Schnorr & BLS multi-signatures avoid complicating verifier logic,
but introduce user experience costs such as:

  • DKG protocols
  • Reduced key derivation ability
  • Verification speed

Ring Signatures

  • Ring signatures prove the signer lies within some "anonymity set" of signing keys, but hide which key actually signed.
  • Ring signatures come in many sizes, with many ways of presenting their anonymity sets.
  • Anonymous blockchain transactions typically employ ring signatures (Monero, ZCash).

Notes:

  • ZCash uses a ring signature based upon Groth16 zkSNARKs which makes the entire chain history be the anonymity set.
  • Monero uses ring signatures with smaller signer sets.
  • Ring signatures trade some non-repudation for privacy.

Questions

Hash Based Data Structures


How to use the slides - Full screen (new tab)
Slides Content
--- title: Hash Based Data Structures description: Hash functions in Substrate duration: 1 hour ---

Hash Based Data Structures


Comparison to
Pointer Based Data Structures

  • A hash references the content of some data;
  • A pointer tells you where to find it;
  • We can not have cycles of hashes.

Hash Chains

A hash chain is a linked list using hashes to connect nodes.

Notes:

Each block has the hash of the previous one.


Merkle Trees

A binary Merkle tree is a binary tree using hashes to connect nodes.

Notes:

Ralph Merkle is a Berkeley alum!


Proofs

  • The root or head hash is a commitment to the entire data structure.
  • Generate a proof by expanding some but not all hashes.

Crucial for the trustless nature of decentralised cryptographic data systems!


Proofs: Merkle Copaths

Notes:

Given the children of a node, we can compute a node Given the purple nodes and the white leaf, we can compute the white nodes bottom to top. If we compute the correct root, this proves that the leaf was in the tree


Security

Collision resistance: we reasonably assume only one preimage for each hash,
therefore making the data structure's linkage persistent and enduring (until the cryptography becomes compromised ๐Ÿ˜ฅ).

Notes:

Explain what could happen when this fails.


Proof Sizes

Proof of a leaf has size $O(\log n)$
and so do proofs of updates of a leaf


Key-Value Databases and Tries


Key-value database

The data structure stores a map key -> value.
We should be able to:

  • put(key, value)
  • get(key)
  • delete(key)

Provability in key-value databases

We should also be able to perform the following operations for a provable key-value database:

  1. For any key, if <key,value> is in the database, we can prove it.
  2. If no value is associated to a key, we need to be able to prove that as well.

Types of Data Structures

  • Trees are rooted, directed acyclic graphs where each child has only one parent.
  • Merkle Trees are trees which use hashes as links.
  • Tries are a particular class of trees where:
    • Given a particular piece of data, it will always be on a particular path.
  • Radix Tries are a particular class of a trie where:
    • The location of a value is determined the path constructed one digit at a time.
  • Patricia Tries are radix tries which are optimized to ensure lonely node-paths are consolidated into a single node.

Notes:

Just a selection we'll cover in this course.


Radix Trie

Words: to, tea, ted, ten, inn, A.

Each node splits on the next digit in base $r$

Notes:

In this image, $r$ is 52 (26 lowercase + 26 uppercase).


Patricia Trie

Words: to, tea, ted, ten, inn, A.

If only one option for a sequence we merge them.


Patricia Trie Structures

#![allow(unused)]
fn main() {
pub enum Node {
  Leaf {
    partial_path: Slice<RADIX>,
    value: Value
  },
  Branch {
    partial_path: Slice<RADIX>,
    children: [Option<Hash>; RADIX],
    value: Option<Value>,
  },
}
}

Notes:

The current implementation actually makes use of dedicated "extension" nodes instead of branch nodes that hold a partial path. There's a good explanation of them here.

Additionally, if the size of a value is particularly large, it is replaced with the hash of its value.


Hash Trie

  • Inserting arbitrary (or worse, user-determined) keys into the Patricia tree can lead to highly unbalanced branches, enlarging proof-sizes and lookup times.
  • Solution: pre-hash the data before inserting it to make keys random.
  • Resistance against partial collision is important.
  • Could be a Merkle trie or regular.

Computational and Storage
Trade-offs

What radix $r$ is best?

  • Proof size of a leaf is $r \log_r n$
    • $r=2$ gives the smallest proof for one leaf

...but:

  • Higher branching at high levels of the tree can give smaller batch proofs.
  • For storage, it is best to read consecutive data so high $r$ is better.

Merkle Mountain Ranges

  • Efficient proofs and updates for a hash chain
  • Append only data structure
  • Lookup elements by number

Merkle Mountain Ranges

Notes:

we have several Merkle trees of sizes that are powers of two. The trees that are here correspond to the binary digits of 13 that are 1.


Merkle Mountain Ranges


Merkle Mountain Ranges

Notes:

  • Not as balanced as a binary tree but close
  • Can update the peak nodes alone on-chain

Questions

Exotic Primitives


How to use the slides - Full screen (new tab)
Slides Content
--- title: Exotic Primitives description: More cutting edge cryptography used in blockchain duration: 1 hour ---

Exotic Primitives


Outline

  1. Verifiable Random Functions (VRFs)
  2. Erasure coding
  3. Shamir Secret Sharing
  4. Proxy Re-Encryption
  5. ZK Proofs

Verifiable Random Functions
(VRFs)

  • Used to obtain private randomness, that is publicly verifiable

  • A variation on a signature scheme:

    • still have private/public key pairs, input as message
    • in addition to signature, we get an output

VRF Interface

  • sign(sk, input) -> signature

  • verify(pk, signature) -> option output

  • eval(sk,input) -> output

Notes:

The output of verification being an option represents the possibility of an invalid signature


VRF Output properties

  • Output is a deterministic function of key and input
    • i.e. eval should be deterministic
  • It should be pseudo-random
  • But until the VRF is revealed, only the holder
    of the secret key knows the output
  • Revealing output does not leak secret key

VRF Usage

  • Choose input after key, then the key holder cannot influence the output
  • The output then is effectively a random number known only to the key holder
  • But they can later reveal it, by publishing the VRF proof (signature)

Notes:

The signature proves that this is the output associated to their input and public key.


VRF Example

  • Playing a card game in a distributed and trustless way
  • For player A to draw a card, the players agree on a new random number x
  • A's card is determined by eval(sk_A,x) mod 52
  • To play the card, A publishes the signature

VRF Extensions

  • Threshold VRFs / Common coin

    • Generate the same random number if $t$ out of $n$ people participate
  • RingVRFs

    • The VRF output could be from any one of a group of public keys.

Notes:

Common coins were used in consensus before blockchains were a thing. Dfinity based their consensus on this. But this needs a DKG, and it's unclear if a decentralized protocol can do those easily.

A participant in a RingVRF could still only reveal one random number.


Erasure Coding

Magical data expansion

  • Turn data into pieces (with some redundancy) so it can be reconstructed even if some pieces are missing.

  • A message of $k$ symbols is turned into a coded message of $n$ symbols and can be recovered from any $k$ of these $n$ symbols


Erasure Coding Intuition

Erasure coding relies on both parties sharing an understanding of what possible messages are valid. This lets mistakes be noticed and corrected.

Imagine you are receiving a message, and you know ahead of time that the only two possible messages you would receive are file and ruin.

Notes:

This concept of a subset of messages being valid is super common in real life, and occurs all over the place. At a restaurant, when they ask you if you want soup or salad, even if you mumble they will probably understand you.

---v

Erasure Coding Intuition

How would you classify each of the following words?

file pile pale tale tall rule tail rail rain ruin

---v

Erasure Coding Intuition

How would you classify each of the following words?

file pile pale tale tall rule tail rail rain ruin

You can classify them based on how close they are to a valid input. This also means we can find the errors in these messages.

Notes:

There is no perfect way to separate these, but one very reasonable one is to do it based on the edit distance of the received word with any valid messsage you could receive.

---v

Erasure Coding Intuition

Now, you are receiving messages that could be msg1 or msg2. Can you apply the same technique? Is it as easy to separate received messages?

What if you receive msg3?

Notes:

If the messages are not far apart, it is impossible to distinguish in many cases. There is not enough "distance" between the two possibilities.

---v

Erasure Coding Intuition

With erasure coding, we extend each message magically so they are different enough. The sender and receiver know the same encoding procedure. These extensions will be very different, even if the messages are similar.

msg1 jdf and msg2 ajk

Notes:

It is actually always possible to make the extra magic only appended to the message. This is called a systematic encoding.

For those curious about how the "magic" works:

The magic here is polynomials, and the fact that a polynomial of degree $n$ is completely determined by $n+1$ points. There are many good explanations online.


Erasure Coding


Erasure Coding Classical use

  • Used for noisy channels
  • If a few bits of the coded data are randomly flipped,
    we can still recover the original data
  • Typically $n$ is not much bigger than $k$

Use in Decentralized Systems

  • We have data we want to keep publicly available

    • but not have everyone store
    • but we don't trust everyone who is storing pieces
  • Typically we use $n$ much bigger than $k$


Shamir Secret Sharing

Redundancy for your secrets

  • Turn data (typically a secret) into pieces so it can be reconstructed from some subset of the pieces.

  • A secret is turned into $n$ shares, and be recovered by any $k$ of the shares. $k-1$ shares together reveals nothing about the secret.


Shamir Secret Sharing

Notes:

Image source: https://medium.com/clavestone/bitcoin-multisig-vs-shamirs-secret-sharing-scheme-ea83a888f033


Pros and Cons

  • Can reconstruct a secret if you lose it.
  • So can other people who collect enough shares.

Proxy Reencryption

Generate keys to allow a third party to transform encrypted data so someone else can read it, without revealing the data to the third party.


Proxy Reencryption

Notes:

img source


Proxy Reencryption API

  • fn encrypt(pk, msg) -> ciphertext;
    Takes your public key and a message; returns ciphertext.
  • fn decrypt(sk, ciphertext) -> msg;
    Takes your private key and a ciphertext; returns the message.
  • fn get_reencryption_key(sk, pk) -> rk;
    Takes your private key, and the recipient's public key; returns a reencryption key.
  • fn reencrypt(rk, old_ciphertext) -> new_ciphertext;
    Take a reencryption key, and transform ciphertext to be decrypted by new party.

ZK Proofs

How do we do private operations on a public blockchain
and have everyone know that they were done correctly?

Notes:

(we are working on substrate support for these and will use them for protocols)


What is a ZK Proof?

  • A prover wants to convince a verifier that something is true without revealing why it is true.

  • They can be interactive protocols, but mostly we'll be dealing with the non-interactive variety.


What can we show?

  • NP relation: function(statement, witness) -> bool

  • Prover knows a witness for a statement:

    • They want to show that they know it (a proof of knowledge)

    • ... Without revealing anything about the witness (ZK)


ZK Proof Interface

  • NP relation: function(statement, witness) -> bool

  • prove(statement, witness) -> proof

  • verify(statement, proof) -> bool


ZK Proof Example

Example: Schnorr signatures are ZK Proofs

  • They show that the prover knows the private key (the discrete log of the public key) without revealing anything about it.
  • The statement is the public key and the witness the private key.

zk-SNARK

Zero-Knowledge Succinct Non-interactive Argument of Knowledge

  • Zero knowledge - the proof reveals nothing about the witness that was not revealed by the statement itself.
  • Succinct - the proof is small
  • Proof of knowledge - if you can compute correct proofs of a statement, you should be able to compute a witness for it.

What can we show?

  • NP relation: function(statement, witness) -> bool

    • They want to show that they know it (a proof of knowledge)

    • ... Without revealing anything about the witness (ZK)

  • With a small proof even if the witness is large (succinctness)


What can we show?

  • There are many schemes to produce succinct ZK proofs of knowledge (ZK-SNARKs) for every NP relation.

ZK Proof Scaling

A small amount of data, a ZK proof, and execution time can be used to show properties of a much larger dataset which the verifier doesn't need to know.


Scaling via ZK Proofs in Blockchain

  • Large amount of data - a blockchain
  • Verifier is e.g. an app on a mobile phone

Notes:

e.g. Mina do a blockchain with a constant size proof (of correctness of execution and consensus) using recursive SNARKs.


Scaling via ZK Proofs in Blockchain

  • The verifier is a blockchain: very expensive data and computation costs.

  • Layer 2s using ZK rollups

Notes:

Of which Ethereum has many, ZKsync, ZKEVM etc. Polkadot already scales better!


Privacy

A user has private data, but we can show
publicly that this private data is correctly used.
An example would a private cryptocurrency:

  • Keep who pays who secret
  • Keep amounts secret,
    But show they are positive!

Notes:

You can do some of keeping amounts secret without ZK-SNARKs, but the positive part is difficult. To do everything well, ZK-SNARKs are needed in e.g. ZCash and its many derivatives e.g. Manta.


Practical Considerations

  • Very powerful primitive

  • Useful for both scaling and privacy

  • One can design many protocols with ZK Proofs that wouldn't otherwise be possible


Downside

  • Slow prover time for general computation
  • To be fast, need to hand optimize
  • Very weird computation model:
    Non-deterministic arithmetic circuits

Downsides Conclusion?

  • So if you want to use this for a component,
    expect a team of skilled people to work for at least a year on it...
  • But if you are watching this 5 years later,
    people have built tools to make it less painful.

Succinct Proving
with Cryptography?

  • ZK friendly hashes
  • Non-hashed based data structures
    • RSA accumulators
    • Polynomial commitment based
      (Verkle trees)

Summary

  • VRF: Private randomness that is later publicly verifiable
  • Erasure Coding: Making data robust against losses with redundancy
  • Shamir Secret Sharing: Redundancy for your secrets.
  • Proxy Re-encryption: Allow access to your data with cryptography.
  • ZK Proofs: Just magic, but expensive magic

Questions

Cryptography In Context


How to use the slides - Full screen (new tab)
Slides Content
--- title: Cryptography In Context description: Real-world considerations around cryptography duration: 1 hour ---

Cryptography in Context


Outline

  1. Keeping Secrets Secret
  2. Security and Usability

Secrets

What is a secret in cryptography?

Data that you know, that nobody else knows.


How Secrets Stay Secret

In order for a cryptographic secret to stay secret, the only thing about it that can be revealed is the output of known, secured cryptographic operations.

  • A (sufficiently random) secret can be hashed, and the hash revealed.
  • A private key can be used to generate a public key, and the public key revealed.
  • A private key can be used to generate a signature, and the signature revealed.

How Secrets Get Leaked

  1. Inadvertently leaking information about the secret during normal operation.
  2. Compromised digital or physical security leading to private key loss.

Notes:

Let's go over each of these in order.


Bad Randomness

Some algorithms require randomness. If the randomness is compromised, the private key or encrypted message can possibly be exposed.

Notes:

one source


Side Channel Attacks

A side channel attack is when a cryptographic system is attacked, and the attacker has another source of information outputted by the system.


Timing Attacks

A timing attack can be possible if any of the following
depend on the contents of a secret:

  • Which instructions execute
  • Branching (if statements)
  • Memory access patterns

Notes:

There are many crazy forms of side channel attack, but the primary one is timing. Timing is also the only one that gets reliably sent back over a long distance.


An Example

Imagine this is the source code for a password checker:

#![allow(unused)]
fn main() {
fn verify_password(actual: &[u8], entered: &[u8]) -> bool {
 if actual.len() != entered.len() {
    return false;
 }

 for i in 0..actual.len() {
  if entered.get(i) != actual.get(i) {
   return false;
  }
 }
 true
}
}

What's the problem?

Notes:

Imagine you compile this into a little binary, and you are able to hit it repeatedly. When sending a guess into this, what information do you get back?

A boolean, and the amount of time from sending the password to getting back a response.

The problem is that the amount of time for a response reveals information about the password. An attacker can send in guesses repeatedly, and if it takes a longer amount of time to respond, that means more of the guess is correct.


Example (Cont)

What if we changed the code to look like this?

#![allow(unused)]
fn main() {
fn verify_password(actual: &[u8], entered: &[u8]) -> bool {
 actual == entered
}
}

Is this safe?

Notes: Now, we don't see any difference in the amount of lines of code or loops, right?


Example (Cont)

What does the source code look like?

#![allow(unused)]
fn main() {
// Use memcmp for bytewise equality when the types allow
impl<A, B> SlicePartialEq<B> for [A]
where
    A: BytewiseEq<B>,
{
    fn equal(&self, other: &[B]) -> bool {
        if self.len() != other.len() {
            return false;
        }

        // SAFETY: `self` and `other` are references and are thus guaranteed to be valid.
        // The two slices have been checked to have the same size above.
        unsafe {
            let size = mem::size_of_val(self);
            memcmp(self.as_ptr() as *const u8, other.as_ptr() as *const u8, size) == 0
        }
    }
}
}

Is this safe?

Notes:

Ok, still no. It looks like now the attacker can still figure out if the length of the password based on an early return. But what if we make sure all passwords are 16 bytes long. Now we are just using a single syscall. Is is safe then?


Example (Cont)

Let's check on memcmp.

memcmp(3) โ€” Linux manual page

/* snip */

NOTES
       Do not use memcmp() to compare security critical data, such as
       cryptographic secrets, because the required CPU time depends on
       the number of equal bytes.  Instead, a function that performs
       comparisons in constant time is required.  Some operating systems
       provide such a function (e.g., NetBSD's consttime_memequal()),
       but no such function is specified in POSIX.  On Linux, it may be
       necessary to implement such a function oneself.

So how could we do it?

This is from the subtle crate, which provides constant time equality.

#![allow(unused)]
fn main() {
impl<T: ConstantTimeEq> ConstantTimeEq for [T] {
    /// Check whether two slices of `ConstantTimeEq` types are equal.
    ///
    /// # Note
    ///
    /// This function short-circuits if the lengths of the input slices
    /// are different.  Otherwise, it should execute in time independent
    /// of the slice contents.
    /* snip */
    #[inline]
    fn ct_eq(&self, _rhs: &[T]) -> Choice {
        let len = self.len();

        // Short-circuit on the *lengths* of the slices, not their
        // contents.
        if len != _rhs.len() {
            return Choice::from(0);
        }

        // This loop shouldn't be shortcircuitable, since the compiler
        // shouldn't be able to reason about the value of the `u8`
        // unwrapped from the `ct_eq` result.
        let mut x = 1u8;
        for (ai, bi) in self.iter().zip(_rhs.iter()) {
            x &= ai.ct_eq(bi).unwrap_u8();
        }

        x.into()
    }
}
}

Notes:

Now we've seen how hard it can be just to stop a very simple leak of timing information. Let's see what an actual cryptographic library concerns itself with.


Ed25519's Guarantees

This is an excerpt from the ed25519 description.

  • Foolproof session keys. Signatures are generated deterministically; key generation consumes new randomness but new signatures do not. This is not only a speed feature but also a security feature.
  • Collision resilience. Hash-function collisions do not break this system. This adds a layer of defense against the possibility of weakness in the selected hash function.

Ed25519's Guarantees (Cont.)

  • No secret array indices. The software never reads or writes data from secret addresses in RAM; the pattern of addresses is completely predictable. The software is therefore immune to cache-timing attacks, hyperthreading attacks, and other side-channel attacks that rely on leakage of addresses through the CPU cache.
  • No secret branch conditions. The software never performs conditional branches based on secret data; the pattern of jumps is completely predictable. The software is therefore immune to side-channel attacks that rely on leakage of information through the branch-prediction unit.

Takeway

Preventing side channel attacks is hard! Noticing sidechannel attacks is even harder!

DO NOT ROLL YOUR OWN CRYPTO

Notes:

Be very, very careful whenever you do anything that touches a secret. That includes any operation involving the secret, or reading/writing it somewhere.

When necessary, talk to a security expert or cryptographer.


Using Cryptographic Libraries Safely

  • Stay above the abstraction barrier
  • Validate each primitive's assumptions when combining primitives
  • Use the most reputable library you can
  • Realize when things need serious consideration
    • Some potentially scary terms: Curve point, padding schemes, IV, twisted curve, pairings, ElGamal

Notes:

Reputableness of a library is some combination of:

  • used by many people
  • audited for security
  • reliable cryptographic literature
  • minimal external dependencies
  • recommended by cryptographers

If you get low-level enough in cryptography libraries to see these terms referenced in more than just a description, you're probably too low level.


Horror Stories

Notes:

Only go through a few of these based on interest and remaining time.

---v

PS3 Secret Key Leak

Problem: Bad randomness

Description: The ps3 developers didn't use randomness when signing with an algorithm that required randomness.

Consequence: Every PS3 was hardcoded to trust that key. When hackers got the key, they were then able to pretend to be Sony and write any software that ran on the PS3. In practice, it made running pirated games trivial.

Notes:

source

---v

IOTA's Novel Hash Function

Problem: Rolling your own crypto

Description: IOTA was a cryptocurrency with a value of 1.9B at the time. They wrote their own hash function, and researchers found severe vulnerabilities.

Consequence: Kind security researchers reported the flaw directly to devs. They had to pause the blockchain for 3 days, generate new address for all accounts, and swap to KECCAK.

Notes:

IOTA originally rolled their own hash function in an effort to be quantum-proof.

Some hash function weaknesses are weak. This was not. The proof of concept exploit literally found two hashes that correspond to a message for the blockchain sending a small amount of currency, and another that corresponded to a message sending a huge amount of money.

exploit POC shutdown source

---v

How the NSA wiretapped all cellphone calls for years

Problem: Small key space / secret technique

Description: The standard for cellphone calls up until the mid-late 2000s (GSM A5/1) used 54-bit keys, and the method was meant to be secret. It did not stay secret, and became extremely easily crackable.

Consequence: Intelligence agencies could and did easily wiretap calls. There were many brute-force attacks against the key space.

Notes:

When the standardization process started, professors proposed 128-bit keys. Western european (british especially) intelligence agencies wanted weaker security. Snowden eventually came out and said the NSA could easily read A5/1 calls.

article source source on weakening

---v

Why HTTPS isn't as secure as you'd hope

Problem: Cryptographic primitive assumptions not upheld

Description: Encryption does not generally hide the length of the underlying message. HTTPS often uses compression before encryption. The compression makes duplicated strings smaller.

Consequence: An exploit called BREACH can reveal a secret from an HTTPS-protected website in under 30 seconds. All websites have had to add mitigation to offset this attack.

Notes:

Mitigation looks like:

  • randomizing size of response content after compression
  • separating secrest from user input
  • disabling HTTP compression (this is expensive though)
  • randomizing secrets per request

source


Physical Security

Notes:

Source is a classic XKCD comic.


Physical Security

Full physical access to a running computer can usually let an attacker have full access to your secrets with enough effort.

Some possible means:

  • Scanning all disk storage
  • Take out the RAM and swap it into a different computer to read (cold boot attack)
  • Proximate side-channel attacks
    • RF emissions
    • Power consumption
    • Sound of a computer running

Notes:

Sources for exotic attacks:


HSMs

An HSM is a hardware security module. HSMs can make it much harder to impossible to steal cryptographic keys. An HSM will hold cryptographic keys, and perform operations on them.

Notes:

We don't go into this much, as there are many available resources around physical security and HSMs. This is just bringing up the ideas, in the context of what makes a cryptographic secret actually secret.


Security and Usability

The accessibility of a secret is typically inversely proportional to the security.

Making a secret more secure is often impractical, depending on the usage.

Notes:

This is not explicitly true in all cases, but it is a good rule of thumb. Additionally, note that impractical != impossible.


Thought Experiment

Suppose I give you a secret that's too long to memorize.

At the end of a year, if nobody else knows the secret, I'll give you a million dollars.

What do you do?


Thought Experiment

Suppose I give you a secret that's too long to memorize.

At the end of a year, if nobody else knows the secret, I'll give you a million dollars.

What do you do?

Destroy it


Thought Experiment

Suppose I give you a secret that's too long to memorize.

At the end of a year, if nobody else knows the secret and you present me the secret, I'll give you a million dollars.

What do you do?


Thought Experiment

Suppose I give you a secret that's too long to memorize.

At the end of a year, if nobody else knows the secret and you present me the secret, I'll give you a million dollars.

What do you do?

Hide it somewhere secure

Notes: Like a bank vault, box buried in the woods, etc


Thought Experiment

Suppose I give you a secret that's too long to memorize.

At the end of a year, if nobody else knows the secret and you present me the secret once per month, I'll give you a million dollars.

What do you do?


Thought Experiment

Suppose I give you a secret that's too long to memorize.

At the end of a year, if nobody else knows the secret and you present me the secret every day, I'll give you a million dollars.

What do you do?


Application to Cryptographic Secrets

Cryptographic secrets are easy to have multiple of.

So don't make users use the same one for everything!

As much as possible, one root secret shouldn't be both used regularly, and extremely valuable.


Questions

๐Ÿช™ Economics and Game Theory

Applied economics and game theory fundamental to the architecture and operation of web3 applications.

Learning Outcomes

Completing this module will enable you to:

  • Understand and discuss the basic principles of economics, including the influence of incentives on human behavior, supply and demand, the value of time, and the role of prices in allocating resources.
  • Analyze and comprehend the principles of game theory, including strategic games, Nash equilibrium, and the role of incentives in shaping outcomes.
  • Understand and apply the concepts of price discovery mechanisms and their implications in various market settings.
  • Understand the principles of collective decision-making, including voting systems, group decision-making processes, and the role of consensus in collective decisions.

Economics Basics


How to use the slides - Full screen (new tab)
Slides Content
--- title: Economics Basics description: Economics Basics for web3 builders duration: 2 hour ---

Economics Module


Overview

  • Economics Basics
  • Game Theory Basics
  • Price Finding Mechanisms
  • Collective Decision Making

Notes:

  • Focus on the pure economic concepts and not diving into blockchain yet.
  • That is covered by another lecture doing so.
  • Economics of polkadot for the founders track later.
  • Interactive: Ask questions!

Economics Basics


Cryptography

  • Provides assurances about the conduct of machines
  • Objective in nature
  • Promotes deterministic actions
  • Serves as a toolbox to secure certain actions in the digital world.

Economics

  • Offers predictions about the actions of people
  • Intrinsically subjective
  • Driven by individual preferences
  • Serves as a toolbox to understand which actions humans are likely to undertake.

๐Ÿค–$~~~$ ๐Ÿค $~~~$๐Ÿ‘ช

Notes:

  • Cryptography would be meaningless without humans using it.
  • Crypto & Econ pair well! Together they are the toolbox to design unstoppable systems.
  • They have a great symphony: where cryptography's has boundaries, economics offers solutions and vice versa.

Why are economic concepts
important?

  • help make better decisions in daily- and professional life.
  • model & reason about how others make decisions.
  • better understand (crypto-)economic protocols.
  • understand the broader context of web3/blockchain.

Notes:

  • It is also important to understand the limits of these economic concepts and when they might not work as intended.

Outline

  • Lesson
    • Incentives: What motivates others to do things?
    • Demand & Supply: Market Equilibrium, Influences, and Elasticity.
    • Market Forms: Monopoly, Oligopoly, and Perfect Competition.
    • Economics of Specialization: Comparative Advantage, Opportunity Costs, Gains from Trade.
    • Time as Ultimate Resource: The interaction between Time and Value.
    • Behavioral Biases: Where human decision making might be different than traditional economic theories predict.
  • Discussion & Workshop
    • Discussions to foster and apply concepts we learned.
    • Hands-on applications.

Guiding Human Behavior

An incentive is a bullet, a key:
an often tiny object with astonishing power to change a situation.
-- Steven Levitt --

Notes:

  • Human behavior is driven by incentives.
  • The aggregation of these behaviors lead to societies.
  • Carefully designing incentives lays the ground for functioning systems.
  • You might be asked to implement certain protocols, which you have to judge if they make sense / can be improved.

Understanding Incentives

  • Motivate individuals or groups to perform certain actions.
  • Historically, incentives were designed to influence human behavior both on a macroeconomic and microeconomic level.
  • Most commonly, people refer to money, that fall under the umbrella of extrinsic incentives.
  • But not all incentives are monetary!

Different Types of Economic Incentives

  • In a workplace setting (microeconomic level), employees are often motivated by various forms of economic incentives:
    • Flat Rate: A fixed amount of salary irrespective of performance.
    • Piece Rate: Pay based on the quantity of output produced.
    • Bonus: Additional rewards for excellent performance.
    • Relative Incentives: Rewards relative to the performance of others (competitive incentives).

Notes:

  • Question: What are the advantages / disadvantages of those incentives?

Rationality in Economics

  • Economic theories often assume rationality, implying that actors:
    • Are primarily driven by monetary gains (payoff maximization).
    • Act in self-interest.
    • Have unlimited cognitive resources.
  • Remark: In this, all factors can be translated into monetary terms.

Incentives often work well

  • Interest rates set by central banks.
  • Lower rates: Encourage borrowing and investing.
  • Higher rates: Slow down inflation and stabilize the economy.

Incentives can Backfire

  • In India during British rule, the British Government was concerned about the large number of venomous Cobras.
    • They offered a bounty for every dead cobra.
    • People started to breed cobras at home to sell them.
    • Once the government stopped the program, breeders released the cobras causing larger numbers than before.

Goodhart's Law

When a measure becomes a target, it ceases to be a good measure


Crowding Out

  • In Haifa (Israel) parents often got their kids late from day care.
    • This caused childcare workers frustration.
    • Idea: fine parents for getting their kids late.
    • The program was rolled out in some day cares and not others.
  • Traditional economic theory:
    • Less or at least equally many late parents.
    • BUT: The opposite happened, the number of late parents even increased.
    • After the fine was abolished, parents still were more late.
  • Extrinsic motivation can crowd out intrinsic motivation.

Notes:

  • Question: Why did that happen?
  • Gneezy & Rustichini (2000): A fine is a price.

The Power of Intrinsic Incentives

  • Beyond monetary gain, people are also motivated by intrinsic factors.
    • Social preferences (Reciprocity, Fairness, Social Norms, Inequality Aversion, Altruism).
    • Personal morals, and deeply-held beliefs.
  • Economic models often fail to account for these motivations, leading to unexpected outcomes in real-world applications.

From Humans to Automated Agents

  • The digital age emphasizes the importance of automated actors in our economic systems.
  • Algorithms and bots, respond to incentives, often financial ones, automatically.
  • This shifts the application of incentives beyond a human context and into the realm of digital systems and automated processes.

Notes:

  • Still, not always financial incentives - reputation scores
  • While automated agents existed for a while, they become much more accessible for everybody due to blockchain technology
  • Your electric car might soon have a wallet that automatically pays for charging at a red light.
  • Or you might be able to pay for priority green lights.

Conclusion

  • Incentives continue to be fundamental to interactions in society, whether among humans or between automated systems.
  • As we understand more about incentives in different contexts, we can better design them to influence behavior effectively.
  • Intrinsic motivations remain important, but in certain contexts, like automated systems, monetary incentives also play a significant role.
  • The challenge is to balance these factors to achieve the desired outcomes.

Demand & Supply: The Pillars of Market Equilibrium

  • A common model to help predict and rationalize the price
    which emerges from a market.
  • Can be used to (gu)estimate the quantity
    of produced goods vs. their market prices.

Notes:

  • A famous artist dies and her art increases in price.
  • An unusually nice summer reduces hotel prices in popular vacations destinations.

Law of Demand: A Basic Principle

  • When prices increase, quantity demanded of a good (generally) decreases.

Notes:

  • Question: Anybody know exceptions?
  • Exception are "Veblen goods" where demand increases when price increases. These are typically luxury goods that signal something (e.g. status, reputation) in addition to being useful to consumption. Examples are jewelry, luxury cars, etc.
  • This is not a contradiction to the law of demand because the good's social value depends on the price; i.e., the good "changes" and becomes more desirable as the price increases.

Visualizing Demand: The Demand Curve

Notes:

  • Question: What influences the aggregate market demand?
  • The market demand is the sum of all individual curves.

Influences on demand?

  • Price (Demand moves along the curve)
  • Size of consumer base
  • Average disposable income of consumer base
  • Price of substitution goods
  • Preferences prevalent within consumer base
  • Expectations prevalent within consumer base

Notes:

  • In the ice cream example, a substitution good could be frozen yoghurt: if its price goes down, the demand for ice cream will decrease.
  • If people's appetite for ice cream increases (perhaps via advertisement), the demand will increase.
  • In people expect a shortage of ice cream in the near future, the demand will increase immediately.

Moving the demand curve


Law of Supply: The Other Side of the Coin

  • If the price of a good increases, the quantity supplied also increases

(because suppliers can make more profit!).

Notes:

  • That is, assuming that suppliers manage to sell all produced goods at said price.
  • For example, if the ice cream price increases the store will hire a second employee to sell more scoops per hour.
  • As another example, some petroleum deposits are easier and cheaper to extract than others, so some deposits will be extracted only if the oil price is above a certain threshold.

Demand and Supply Interplay: Market Equilibrium

  • What happens when Price is higher than the Eq. Price?
  • What if the Quantity is lower than the Eq. Quantity?
  • Moving either supply or demand leads to new Eq.

Notes:

  • Where the curves meet we obtain a point of equilibrium, which establishes a market quantity and a market price.
  • If price is higher than equilibrium price, there is more supply than demand, so some goods remain unsold, and sellers will be forced to reduce the price to get rid of their stock. Similarly if the price is lower than equilibrium price, there is more demand than supply, which leads to shortages of the good; sellers will notice this and increase the price. Hence, we always converge to the equilibrium price.
  • Similarly, if the quantity supplied is too low, there is a business opportunity to supply more unit of the good and make a profit, so eventually someone will increase the supply quantity.

Elasticity

  • Measures how much buyers and sellers respond to changes in market conditions.
  • Sensitivity of quantity demanded or supplied to changes in price.
    • High elasticity: small price changes have large impact on quantity demanded or supplied.
    • Low elasticity (== inelasticity): price changes do not affect quantity demanded or supplied.

Notes:

  • Draw a diagram with price on y and quantity on x and draw an almost vertical line tilted towards the left.
    • Demand for Gasoline. Low elasticity: If price goes up people buy only a little less, if price goes up they only buy a little more.
  • High Elasticity: luxury goods, where buyers can easily adjust their consumption.
  • Low Elasticity: essential goods such as energy / medicine.

Conclusion: The Dynamics of Demand and Supply

  • Demand/supply model forces in markets and can be used to estimate equilibrium quantity/price.
  • All other things being equal:
    • demand decreases with increasing price.
    • supply increases with increasing price.
  • There are different factors influencing supply & demand curves.
    • Elasticity.
    • Some shift the equilibrium on the curve.
    • Others move the entire curves.

Market forms

(Or: why monopoly bad?)

  • Typically, economists distinguish between three market forms.
    • Perfect competition: Many firms, many consumers.
    • Oligopoly: Few firms, many consumers.
    • Monopoly: One firm, many consumers.
  • The more competition a firm faces, the less market power it has.
  • Market power: ability to affect the price (through choosing a certain production level).

Notes:

  • Special form of Oligopoly is the Duopoly with two firms, something that we see later.

Market Form Examples

  • Monopoly:
    • Postal services in many European countries.
    • Railway services in many European countries.
  • Oligopoly:
    • Cloud provider (AWS, Google, ...)
    • Mobile phone service.
  • Perfect Competition:
    • Consumer products with low barrier to entry (e.g., kebab stand in Berlin, espresso bar in Italy)

Why discuss this?

  • Current web2 landscape consists of several firms with a lot of market power in their respective market segment.
  • We want to understand, from an economic perspective, why too much market power is not a good thing.

Perfect Competition

  • Under perfect competition ...
    • there are so many firms that a single firm's production quantity does not affect market price.
    • the marginal costs (cost of one additional unit) of production equals the market price.
    • firms have 0 profit.

Notes:

  • This makes sense: The firm produces up to the point where an additional unit costs exactly what it gets for it on the market.
  • Notice that having the market price be equal to the marginal production costs is pretty surprising, as it is ideal from a point of view of society welfare, even though every firm is acting selfishly.
  • In this perfect world, firms will have zero economic profit, meaning they make just enough money to cover production costs and keep the company running.

Monopoly

  • Monopolist: single seller of the good or service, serves the entire market.
  • Market characterized by a high barrier of entry (e.g. big investment, legal restrictions).
  • Monopolist is therefore sole price-setter and serves only to maximize their own profits.
  • This leads to sub-optimal pricing and sub-optimal society welfare: consumers either pay more than they should ("monopolist tax") or are excluded from market.

Oligopolies

  • Oligopolies are somewhere between monopolies and perfect competition.
  • Because of the competition between firms
    • prices are lower, and
    • overall market participation is higher.

What did we learn?

  • Markets differ in how much market power the firms possess.
  • Economics typically distinguishes between three forms (ordered by increasing market power):
    • Perfect competition: many firms in the market.
    • Oligopoly: few firms in the market.
    • Monopoly: one firm in the market.
  • The problem of monopoly is the result of an inefficiently high price.

Economics of Specialization

(or: why bother trading at all?)

It is the maxim of every prudent master of a family, never to attempt to make at home what it will cost him more to make than to buy.
-- Adam Smith, 1776 --

Notes:

  • In this part, we take a closer look at the following concepts:
    • Comparative advantage.
    • Opportunity costs.
    • Gains from trade.

The economic question

  • Empirical puzzle: many countries trade even though one has an absolute advantage in producing many of the traded goods.
  • E.g., the US is well-equipped to cheaply produce both
    • complex computer chips, and
    • agricultural goods (e.g., potatoes).
  • However, the US specializes in producing the former and importing many agricultural goods.
  • Question: Why do countries specialize; why do they trade?

Comparative advantage

  • Consider the following example:
    • Suppose Alice can produce 10kg of meat and 20kg of potatoes per hour.
    • Suppose Bob can produce 2kg of meat and 10kg of potatoes per hour.
  • Alice has an absolute advantage over Bob at producing both goods.
  • BUT, Alice has only a comparative advantage over Bob at producing meat.
    • Alice has lower opportunity costs of producing meat (in terms of foregone potatoes).
    • That is, to produce one kg of meat Alice gives up much less produced potatoes than Bob would.
  • By the same observation, Bob has a comparative advantage at producing potatoes.

Notes:

  • Comparative advantage of meat:
    • Opp. cost Alice: 1kg meat = 2kg potatoes
    • Opp. cost Bob: 1kg meat = 5kg potatoes
  • Comparative advantage of potatoes:
    • Opp. cost Alice: 1kg potatoes = 1/2kg meat
    • Opp. cost Bob: 1kg potatoes = 1/5kg meat

Comparative advantage II

  • If both focus on their comparative advantage, then total welfare is higher than in a situation of autarky!
  • Why?
    • Suppose both require both meat and potatoes.
    • Situation A - Autarky: both divide their time equally between producing meat and producing potatoes.
    • Situation B - Trade: Alice focuses most of her time to produce meat and some to produce potatoes, while Bob focuses only on producing potatoes.
    • Total production of meat and potatoes is higher in Situation B!
  • Specialization result: With trade, both are both better off than in Situation A.

Notes:

  • Assume one working day with one worker (8 hrs).

  • Autarky:

    • Alice: 4hrs _ 10 meat, 4hrs _ 20 potatoes = 40 meat, 80 potatoes
    • Bob: 4hrs _ 2 meat, 4hrs _ 10 potatoes = 8 meat, 40 potatoes
  • Trade:

    • Alice: 5hrs _ 10 meat, 3hrs _ 20 potatoes = 50 meat, 60 potatoes
    • Bob: 0hrs _ 2 meat, 8hrs _ 10 potatoes = 0 meat, 80 potatoes
    • Alice could trade 9 meat for 30 potatoes:
      • Alice: 41 meat, 90 potatoes
      • Bob: 9 meat, 50 potatoes

Absolute vs Comparative Advantage

  • Alice can produce both meat and potatoes more efficiently than Bob - this is an absolute advantage.
  • However, the choice to specialize isn't just about absolute production efficiency.
  • What matters for trade is which good Alice can produce at a lower opportunity cost - this is comparative advantage.

Notes:

  • What does that tell us about daily life?
    • Stick with what you are good at: It's often good to stick things where you have a comparative advantage: For example, assume you are a great software engineer with a job that has a high salary, but you also want to help clean up the ocean. Instead of giving up your job and go clean the ocean, you might want to allocate some of your salary to ocean cleanup, having a larger effect.
    • Realize that absolute advantage is not allOnly because one co-worker has a total advantage in two dimensions of the job, another co worker can still have the comparative advantage in one of the tasks.
    • Question: What are your opportunity costs of being here?

Time: The Ultimate Resource

  • Time is the one resource that is uniformly distributed and yet cannot be accumulated or saved for later.
  • It's always moving, never at rest.
  • This unique property of time lends it incredible implications.
  • This concept has big impact in the context of money and finance.

The Relationship of Time and Money

  • Money now is worth more than the same amount in the future.
  • Why?
    • Again, opportunity costs: The money at hand now could be invested, potentially earning more money.
    • This potential earning capacity bestows more value to present money over future money.

Present Value: The Value of Now

  • Present Value brings some future value back to the present moment.
  • It estimates what value would make you indifferent between getting some value now vs. later.
  • It reflects the worth of a future promise in today's terms, based on the potential earnings we could achieve now.

Notes:

  • There are formulas to calculate how much a future payout in the future is worth today.
  • You need to make assumptions about your rate of return in the meantime.
  • Often risk free rate is used.

Time Value of Money: A Guiding Principle

  • It underlines the importance of wise decisions today for a brighter tomorrow.
  • It shows that nothing is for free. You always give up something.
  • It's a fundamental cornerstone guiding personal savings, investments, and wide-ranging financial decisions.

Behavioral Biases

  • Behavioral biases are systematic errors in decision-making when people use mental shortcuts (heuristics) to process information and make judgments.
  • For a long time economic theory was built on the assumption that all actors are rational.
    • No regard for fairness, altruism, equality.
    • All available information is considered.
    • No influence of emotions.
  • This works well in many circumstances (e.g., on an aggregate level or assuming highly rational actors like firms executives) but in many it doesn't.
  • Most humans are prone to behavioral biases and are influenced by social preferences.

Behavioral Economics

  • Behavioral economics is mainly concerned with the bounds of rationality of economic agents.
  • Influenced by psychological aspects such as emotions, cognition as well as cultural and societal factors.
  • Psychologist Daniel Kahnemann received the 2002 Nobel Prize for Prospect Theory, a fundamental theory in Behavioral Economics.
  • PT models the fact that individuals assess their losses differently than their gains.

Behavioral Economics

  • Behavior is not random!
  • Key behavioral biases include:
    • Confirmation Bias - Decision makers focus on information that cater to their prior belief / opinion and neglect other information.
    • Sunk Cost Fallacy - Instead of cutting losses, decision makers continue to pour resources into a failed endeavour.
    • Herding Effect - People follow the crowd without making their own informed decision.
    • Gambler's Fallacy - In many situations probabilities of repeated events are independent from each other. Often decision makers fail to understand this.
    • Default Effect - People tend to stick with pre-selected choices.
    • Overconfidence - People overestimate their abilities.
  • Any other biases you know?

Notes:

  • Confirmation bias especially pronounced in todays time where people get trapped in echo chambers fostered by algorithms that keep feeding you information that you want to hear.
  • Overconfidence: If asked, more than 50% of people think they drive above average. BUT has also positive effects. Sometimes overconfident people are better than others and statistically perform better in practice.
  • Anchoring Effect
  • Endowment Effect: This bias occurs when we overvalue something simply because we own it.
  • Hindsight Bias - This is the tendency to believe, after an outcome is already known, that you would have predicted or could have foreseen the outcome. It is often referred to as the "I knew it all along" effect.
  • Availability Bias - People make decisions based on the information that is readily available to them, rather than the complete data set. This bias can often lead to inaccurate conclusions.

Behavioral Economics

Simply put, the stock market is a creation of man and therefore reflects human idiosyncrasy.
-- Ralph Nelson Elliott --


Break (10 minutes)


Classroom Discussions

Notes:

  • Roughly 20 minutes

Behavioral Biases

  • Which biases might be responsible for the following situations?
  • How can we mitigate the impact of those biases?

Situation 1:

Warren Buffett announces that he increased his exposure to Apple by buying more stocks. The price of AAPL goes up.

Notes:

  • Herding effect
  • Cannot be in the hope of a price response by Buffett's trade because it already happened.
  • Self-fulfilling prophecy
  • Maybe still rational?

Situation 2:

A stock investor bought a big position in Disney at around 100 USD per share. Then, the scandal unfolds and the price plummets. The investor keeps buying in at 80 USD, 50 USD and finally at 5 USD, 3 USD and 1 USD. When a friend asks him to stop he keeps arguing to recover the losses by "buying the dip" and to "further reducing the average buying price".

Notes:

  • Sunk Cost Fallacy
  • Learn to accept losses

Situation 3:

Peter wants to buy a new smartphone and fancies the new iPhone. Mentally, he is already looking forward to finally buy it, but a friend urges him to do some more research. He agrees and is reading through articles that pop up after googling "Why you should buy the new iPhone".

Notes:

  • Confirmation Bias
  • Mitigating by actively looking outside the echo-chamber that is just catering to your opinion.

Situation 4:

A Swiss energy company could increase the number of new renewable electricity contracts from 3% to 80-90% by pre-selecting the green option when customers were renewing their energy contracts.

Notes:


Titan Submersible Incident

  • On 18 June 2023, Titan, a submersible operated by American tourism company OceanGate, imploded during an expedition to the Titanic wreck.

Titan Submersible Incident

  • On board were Stockton Rush (OceanGate CEO), Paul-Henri Nargeolet (deep sea explorer), Hamish Harding and Shahzada Dawood (billionaire businessmen), and Suleman Dawood (Shahzada's son).
  • Communication was lost 1 hour and 45 minutes into the dive and the vessel failed to resurface later that day.
  • Four days later, a debris field containing parts of Titan was discovered near the Titanic.

Prior Safety Concerns

  • Many industry experts had expressed safety concerns about Titan.
  • OceanGate, including CEO Rush, had not sought certification for Titan.
  • They published the following blog post to give reasoning:

Notes:

  • What biases could be happening here?
  • Survivor bias: is the logical error of concentrating on entities that passed a selection process while overlooking those that did not. This can lead to incorrect conclusions because of incomplete data.
  • What bias are we facing maybe? Hindsight Bias: is the common tendency for people to perceive past events as having been more predictable than they were.

Where do you see yourself or others suffer from behavioral biases?

Notes:

Open class discussion.


Monopoly

  • We have seen: Monopolists' market power leads to inefficient outcomes.
  • Web 2.0 is full of firms that have a monopoly-like position in their respective domain:
    • Google
    • Uber
    • AirBnb
    • ...
  • How do these firms exert their market power, if not over price-setting?

Notes:

  • transparency? (stealing tips!)
  • fees (uber, airbnb)
  • surge pricing (uber)
  • opaque auction mechanism (google in the case of online ads)

Activities


Academy Games


Activities Today & Tomorrow

  • You can earn points by applying the concepts presented in this course to similar situations when playing with/against your colleagues.
  • In session 2.1, 2.2 and 2.3, you can increase your budget (starting at 0) during several economic games.
  • We have a total budget of $2250 for this.
  • The budget is split between all the games that we play and total points earned by all of you are dynamically adjusted to correspond to the budget.
  • In 2.3, there is a big final: You can use your earnings to bid for amazing NFTs by a professional artist in an experimental auction setting.

Sign-in

  • You will sign in to each of the games with your Polkadot wallet.
  • You should always sign in with the same wallet to keep earning points to that wallet.
  • Make sure not to forget your wallet credentials, because your final payout / NFT will be tied to that wallet.

Disclaimer

  1. If a bug occurs and the reward points are messed up or something else goes wrong, we may intervene.
  2. We cannot help you recovering your Polkadot wallet. If you lose access, you need to create a new account in the games but you will lose previous earnings.

Tracking your payoff

  • Your balance is tracked in a wallet app.
  • The points are translated to USD after each game.
  • You can constantly check your balance by opening the wallet app link and sign in with your wallet.

Notes:

  • Completely trusted and centralized xD
  • If you leave your cookies you can simply log back in otherwise you need to log in again.

Everybody got a wallet?


Game 1: Ultimatum game

  • You are randomly matched with another student in the classroom.
  • The interaction is anonymous.
  • Player 1 gets an endowment of 100 points.
  • Player 1 can offer Player 2 between 0 and 100 points.
  • Player 2 sees the offer and can decide to accept or reject the offer.
  • If the offer is accepted, the endowment is split according to the offer.
  • If the offer is rejected, both get 0 points.
  • Budget for the game $500.
  • There is no right or wrong here. It is your choice how to behave in this context!

Game 1: Questions?


Let's start!

Link will be distributed!


Game 1: Discussion

  • What were your strategies here?
  • What do you think we would expect from rational players?
  • Why did people offer something?
  • What did we measure here?

Notes:

  • What would we expect from rational players (they do not know the concept of Nash Equilibrium yet but intuitively they can argue).
  • What do you expect the data to look like?
  • Did we measure fairness? No.
  • How could we measure fairness?
    • Dictator game
  • Talk about results in the literature:
    • Many people offer 40-50%
    • Almost nobody offers more than 50%
    • Many people reject below 30%
    • This is not only because of the stakes. There are experiments in Thailand and Indonesia where they play with half year salary.

Game 1: Results!


Game 2: Cournot Competition

  • Economic model
  • Firms compete by setting quantities
  • Identical product
  • Demand always meets supply
  • Named after French mathematician Augustin Cournot.

Notes:

  • Game 1: Cournot Competition with 2 firms (players from the audience) for 10 rounds
  • THe basic outline was given by Antoine Augustin Cournot in 1844, we will hear it again in GT.

Game 2: Overview

  • Cournot competition (Duopoly!)
  • 2 firms (you play 1 firm!)
  • Each of you individually set a quantity to produce.
  • All units are sold, but the price decreases in the quantity.
  • Repeated over 10 rounds
  • Points accumulate and are added to your budget.
  • Budget for the game: $500

Game 2: Rules

Notes:

  • 10 rounds not 2.

Game 2: Questions?


Let's start!

Link will be distributed!


Game 2: Discussions

  • What strategy did you employ?
  • Did your behavior change over the rounds?
  • What was the best choice for each firm?

Notes:

  • Calculate equilibrium:
  • $\pi = P(Q) * q$
  • For Firm 1: $\pi = (60 - q_1 - q_2) * q_1$
  • solve for optimal $q_1$ -> delta-pi / delta-q1 = -2q_1 - q_2 + 60 =! 0
  • q_1* = (60 - q_2) / 2
  • q_2* = (60 - q_1) / 2
  • q1* = (60 - (60-q1*)/2) / 2
  • = 30 - 15 + (q_1*/4) = q_1*
  • = q_1* = q_2* = 20
  • Show with example. If firm 1 plays 20 then 19 leads to smaller payoff and 21 for other firm.

Game 2: Results!


Conclusion

We learned:

  • fundamental economic concepts that are important to navigate in private- and professional life.
  • the notion of rationality and what to expect from actors that behave this way.
  • that especially human individuals deviate from rationality and behave differently.
  • the importance of time on value.
  • to engage in economic situations with other human actors.

Game Theory Basics


How to use the slides - Full screen (new tab)
Slides Content
--- title: Game Theory Basics description: Game Theory Basics for web3 builders duration: 1 hour lesson + 2 hours activities ---

Game Theory Basics

Notes:

Game theory is a field of study at the intersection of mathematics and economics. It consider an economic system to be a game and people to be players. According to the rules of the game, it analyzes the players' best strategies and then uses this analysis to explain the observed behavior of the economic system.

Game theory is an interesting an powerful tool, because it's fairly simple and intuitive, yet extremely powerful to make predictions. Hence, it's a good idea to learn game theoretical principles and keep them in the back of your mind when designing economic systems.


Outline

  • Lesson
    • What is Game Theory?
    • What is a Game?
    • Types of Games
    • Common Games
    • Nash Equilibrium
    • Equilibrium Selection
  • Workshop & Activities
    • Discussions & more games

What is Game Theory?

Game theory studies strategic situations where the outcome for each participant or 'player' depends on the actions of all others. It formulates models to represent these scenarios and predicts the most likely or optimal outcomes.

Notes:

  • Game theory is all about the power of incentives.
  • Helps you understand and design systems and guide behavior of participants.

Game Theory in Web3

In the context of blockchains, game theoretic reasoning is used for modelling and understanding.

Notes:

  • The term is heavily over-used.
  • 9/10 times people using the term they simply mean there is some economic structure behind their protocols.
  • Rarely real game theoretic analysis is done.

Modelling

  • Tokenomics: Macroeconomic design of a token (inflation, utility, etc.).
  • Business Logic: Interaction of the token with different modules of a protocol.
  • Consensus: Providing sufficient incentives to guarantee that participating nodes agree on a distributed state of the network.
  • Collaboration: Nudging (aggregated) human behavior and their interaction with the protocol.

Understanding

  • Economics: Interaction between different protocols and how finite resources are allocated among all of them.
  • Security: Testing economic security of protocols against various types of attacks.

History of Game Theory

  • Early โ€œgame theoreticโ€ considerations going back to ancient times.
  • โ€œGame theoreticโ€ research early 19th century, still relevant.
  • The systematic study of games with mathematically and logically sound frameworks started in the 20th century.
  • Modern game theory is used in economics, biology, sociology, political science, psychology, among others.
  • In economics, game theory is used to analyze many different strategic situations like auctions, industrial economics, and business administration.

Notes:

  • In Plato's texts, Socrates recalls the following considerations of a commentator of the Battle of Delium:
    • An example is a soldier considering his options in battle: if his side is likely to win, his personal contribution might not be essential, but he risks injury or death. If his side is likely to lose, his risk of injury or death increases, and his contribution becomes pointless.
    • This reasoning might suggest the soldier is better off fleeing, regardless of the likely outcome of the battle.
    • If all soldiers think this way, the battle is certain to be lost.
    • The soldiers' anticipation of each other's reasoning can lead to a self-fulfilling prophecy: they might panic and retreat before the enemy even engages, resulting in a defeat.
  • Spanish conqueror Cortez, when landing in Mexico with a small force who had good reason to fear their capacity to repel attack from the far more numerous Aztecs, removed the risk that his troops might think their way into a retreat by burning the ships on which they had landed
  • Antoine Augustin Cournot, a french mathematician, already described a duopoly game with respective solution in 1844.
    • We will see this later.
  • Examples:
    • Biology: Animals fight for resources or are peaceful, why cooperation evolved
    • Political science: Art of conflict, escalation and de-escalation between nations.

Game theory is abstract

  • Game theoretic models aim to get at the essence of a given strategic problem.
  • This often requires many simplifying assumptions.
  • Pro: Abstraction makes the problem amenable to analysis and helps to identify the key incentives at work.
  • Con: A certain lack of realism.
  • In any case: Modeling a strategic situation always entails a tradeoff between tractability and realism.

Notes:

  • Need to explain what we mean by lack of realism:
  • Often people have more choices than we model.
  • Often people take other things into consideration when making choices than the model allows.
  • Often people know more/less than we assume.
  • How to resolve the tradeoff between tractability and realism is often subjective and depends on the taste of the modeler.

What is a Game?


Definition: (Economic) Game

  • A game is a strategic interaction among several players, that defines common knowledge about the following properties:
  • all the possible actions of the players
  • all the possible outcomes
  • how each combination of actions affects the outcome
  • how the players value the different outcomes

Definition: Common Knowledge

  • An event $X$ is common knowledge if:
    • everyone knows $X$,
    • everyone knows that everyone knows $X$,
    • everyone knows that everyone knows that everyone knows $X$,
    • ... and so on ad infinitum.

Examples: Common Knowledge

Auctions
  • Actions: Bids.
  • Outcome: Winner and Payment.
Price-competition
between firms
  • Actions: Price charged.
  • Outcome: Demand for each firm, profit of each firm.

Notes:

Crucial feature of a game: outcome not only depends on own actions but also on the actions of the other players.


Types of games

Game theory distinguishes between:

  • static & dynamic games
  • complete & incomplete information games

Static and Dynamic Games

Static GameDynamic Game
DefinitionAll players take their actions at the same timePlayers move sequentially and possibly multiple times, (at least partially) observing previous actions
Simple ExampleRock-Paper-ScissorsTic-Tac-Toe
Economic ExampleSealed-bid auction.All bidders submit their bids simultaneously (in a sealed envelope).
English auction.Auctioneer publicly raises price if at least one bidder accepts the price.
RepresentationPayoff MatrixDecision Tree

Notes:

  • Also referred to as simultaneous or sequential games

Completeness of Information in Games

Game of Complete InformationGame of Incomplete Information
Information availableAll information relevant to decision-making is known.Not all information relevant to decision-making is known.
Simple ExampleChessPoker
Economic ExampleSealed auction for seized Bitcoin.Used-car market: the resale value of a used car is opaque.

Notes:


Quiz


Three firms want to hire an engineer...

  • The engineer brings added value to each firm of 300,000 USD per year.
  • The payoff of the firm is known by everyone to be 300,000 USD minus the salary.
  • The payoff to the engineer is salary minus cost of working, which is known to everyone.
  • All firms make a salary offer at the same time.

Quiz Questions:

  1. Is this game static or dynamic? What would need to change in the description of the game such that it would fall in the other category?
  2. Is this game of complete or incomplete information? What would need to change in the description of the game such that it would fall in the other category?

Notes:

  1. The game is static. For it to be dynamic, firms would need to make offers sequentially, knowing what the firms before had offered.

  2. The game is of complete information. To make information incomplete, we would need to have that the value of hiring the engineer differs between firms and is unknown between firms. Or that the cost of working for the engineer is not known to the firms. The point is that we need to have uncertainty over payoffs.

  • This lesson focuses on static games of complete information.
  • When we look at auctions in lesson Price finding mechanisms, we will also consider games of incomplete information, both dynamic and static.

Utility

  • Core concept of Game Theory.
  • Can transform any outcome to some value that is comparable.
    • For example: What is better? Scenario A: Going for a vacation to France and drink Wine, or Scenario B: going to Germany and drink Beer?
    • Those dimensions are only comparable if we give both some value like U(A) = 5, U(B) = 3.
  • Essential assumption: Agents are utility maximizers.
  • Monetary payouts can also be transformed to utility.
    • Simplest assumption U(x) = x.
    • But that is likely not true in reality.
  • Most things have diminishing rates of returns.

Notes:

  • Expected Utility is the average utility we get from comparing several outcomes and weigh them with the probability they occur.
  • In the following, we won't need that, because either we deal with money or other dimensions that are comparable.

Common Games


Prisoners' Dilemma

A fundamental problem:

Even though everyone knows there is a socially optimal course of actions, no one will take it because they are rational utility maximizers.

It's a static game of complete information.

Notes:

One of the most famous games studied in game theory.

  • Static because both players take their action at the same time.
  • Complete because everybody is aware of all the payouts.

Bonnie and Clyde

Bonnie and Clyde are accused of robbing two banks:

  • The evidence for the first robbery is overwhelming and will certainly lead to a conviction with two years of jail.
  • The evidence for the second robbery is not sufficient and they will be convicted only if at least one of them confesses to it.

Bonnie and Clyde

In the interrogation they both are offered the following:

  • If you both confess you both go to jail for 4 years.
  • If you do not confess but your partner does, you go to jail for 5 years: 1 extra year for obstruction of justice.
  • However, if you confess but your partner does not, we reduce your jail time to one year.

Notes:

They are interrogated in different rooms, apart from each other.


Bonnie and Clyde

  • Cooperate ($C$) with each other and not say anything
  • Defect ($D$) and confess their crime

Notes:

  • They can either cooperate or defect
  • First payoff is Clyde, second is Bonnie

Bonnie and Clyde

Choosing D is a dominant strategy: a strategy that is always optimal for a player, regardless of what the other players do.

Notes:

No matter what Clyde does, D is always the best choice. So, they end up both defecting, resulting in 4 years each. It would be in their best interest to cooperate and not to say anything. This would minimize the total jail time for the two. However, both Bonnie and Clyde are rational utility maximizers. So, they end up in a situation where they not only fare worse individually (4 instead of 2) but also jointly (the total jail time is 8 years rather than 4 years).


Nash-Equilibrium

  • Fundamental concept in Game Theory
  • A NE is a set of strategies, one for each player, such that no player can unilaterally improve their outcome by changing their strategy, assuming that the other player's strategy remains the same.
  • In the Prisoner's Dilemma, D/D is the only NE.

Prisoners' Dilemma IRL

  • Nuclear Arms Race: NATO and Russia prefer no arms race to an arms race. Yet, having some arms is preferable to having no arms, irrespective of whether the other one is armed.
  • OPEC: Limiting oil supply is in the best interest of all. However, given the high price that thus results, everyone has an incentive to increase individual oil supply to maximize profits.

Notes:

OPEC: Organization of the Petroleum Exporting Countries. It is something like a cartel that agree on limiting the global oil production and keep the oil price artificially high.

OPEC and Cartels generally seems to overcome the Prisoners' Dilemma... More on that later.


Ultimatum Game

  • We played it before.
  • Sequential game.
  • The Nash Equilibrium can be reasoned by backwards induction.
  • The proposer has the following considerations:
    • What would the recipient accept?
    • Answer: every payoff (larger than 0).
    • Therefore, I should offer, since I want to maximize my payout, something equal or slightly higher than 0.
  • That means, the proposer offering something small and the recipient always accepting is the only NE.

Notes:

  • We saw that empirically that is not the case.
  • Fairness concerns are too strong in that setting.

Coordination Game

  • The prediction of play in the Prisoner's Dilemma was easy: both will defect.
    • This is the optimal thing to do no matter what the other player does.
  • In other games, predictions of play are not so clear.
    • One instance is the coordination game.

Coordination Game

A coordination game is a type of static game in which a player will earn a higher payoff when they select the same course of action as another player.


Coordination Game Example

  • Choose $L$ or $R$.
  • The numbers represent the payoffs a player receives.
  • The players only obtain utility if they coordinate their actions.

Notes:

Examples:

  • Driving on the right/left side of the road.
  • Money adoption.
  • Standard adoption.

Coordination Game Example

  • The coordination game has two outcomes $(L,L)$ and $(R,R)$ that stand out.
  • Clearly, if the other player chooses $L$ ($R$), then it is optimal for the other to do so also.
  • So, in the outcomes $(L,L)$ and $(R,R)$ the players choose mutually optimal actions.

Notes:

  • That is, for both players it holds:
    • Playing $L$ is a best response to the other player playing $L$.
    • Playing $R$ is a best response to the other player playing $R$.

Coordination Game Example

  • Both $(L,L)$ and $(R,R)$ are instances of Nash equilibrium.
  • By their very nature, coordination games always have multiple equilibria.
  • The outcome $(D,D)$ in the Prisoner's dilemma is the unique Nash equilibrium.

Notes:

Nash equilibrium: If other players follows the recommended strategy, then the best response for you is to do the same. As the same logic is true for other players, it's reasonable to assume that everybody will indeed follow the recommended strategy.

However, a Nash equilibrium is a weaker notion than a dominant strategy, because if the other players don't follow the recommended strategy, it is not clear what your best response should be.


Equilibrium selection

  • So, which outcome does the theory of Nash equilibrium predict in the coordination game?
    • None? Both?
  • Sometimes people switch between equilibria (if they are made to)...

Sweden, 1967.

Notes:

  • The NE does not predict any outcome.
  • Sweden switched from left-side driving to right-side.

Schelling Points

  • Nash equilibrium does not predict which strategies the players actually take.
  • This is especially pronounced in games with multiple equilibria (e.g., coordination games).
  • There are theories that offer insights into which strategies players actually take.

Notes:

  • In the 1950s American economist Thomas Schelling ran a couple of informal experiments in which he asked his students (quote on slide)

Schelling Points

If you are to meet a stranger in New York City, but you cannot communicate with the person, then when and where will you choose to meet?

  • Literally any point and time is a Nash equilibrium...
    • However, most people responded: noon at (the information booth at) Grand Central Terminal.
  • Basic idea: in case of multiple equilibria, social norms may help to choose one.

Notes:

  • Imagine you are held in prison.
  • You and your significant other is asked to guess a number.
  • If you both guess the same number, you are set free.
  • You have the following options: 0.231, 1, or 0.823
  • Guessing both the same number is a NE every time.
  • It's highly likely you will walk free.

Summary (so far...)

  • Typology of games: static/dynamic, complete/incomplete information.
  • Three canonical games: Prisoner's Dilemma, Ultimatum-, and Coordination Game.
  • The Prisoner's Dilemma has a unique Nash equilibrium, which is dominant, whereas the Coordination game has two Nash equilibria.
  • To select among multiple equilibria, the concept of a Schelling Point is sometimes used.

Why are theories of equilibrium important?

  • Nash Equilibria are used to predict the behavior of others in a closed system.
  • If you can identify a unique Nash Equilibrium or the Schelling point in a system, you have a strong prediction of user behavior.
  • So, you can begin to drive user behavior by designing incentives accordingly.

Public Goods

  • Non-excludable No-one can be excluded from consumption
  • Non-rivalrous My consumption does not affect yours
  • e.g., fireworks, street-lighting.

Notes:

  • We will now talk about public goods and common goods, which are goods enjoyed by everyone.
  • This is, of course, a very important and very tricky class of goods in a collective.

Common Goods

  • Non-excludable No-one can be excluded from consumption
  • Rivalrous My consumption reduces your possibility to consume
  • i.e., a public park, an office coffee machine.

Notes:

  • Recall: Public good was non-rivalrous.

Examples:

  • Public park: anyone can go; too many people spoil the experience or kills the grass.
  • Coffee machine in the office: anyone can use it; too many users may cause congestion or the amount of coffee may be limited.

Public vs.

  • Main difference is that in a common good your consumption reduces the value of the good to others.
  • This is called a consumption externality that you impose on others (and others impose on you.)
  • The tragedy of the commons is that, because you do not take this externality into account, consumption is higher than would be socially optimal.

Stylized Public Good Game:

  • $N$ players have 10 US dollars each, say, $N=4$.
  • Each player can choose how much to place into a project.
  • Funds in the project are magically multiplied by a factor $\alpha$, say, $\alpha=2$.
  • Finally, the funds in the project are split equally among all players.
    • What would be best for the individual?
    • What would be best for the collective?

      Notes:

      • As long as $\alpha>1$, it's best for the collective to contribute as much money as possible, because the money in the project increases magically, so we end up with more money that we started with.
      • However, the problem is that everyone benefits from the project funds regardless of their individual contribution (it is a common good). If a player decreases their initial contribution by one dollar, their individual payoff decreases by $\alpha/N$ dollars, so as long as $\alpha<N$, it is best for each individual to contribute zero.
      • As a result, we can expect that no one will contribute anything, and the money-multiplying powers of the project will be unused. This opportunity cost is a tragedy of the commons.
      • Finally, if $\alpha\geq N$ then it would be individually better to contribute everything, and we would not have a tragedy of the commons.

      Overfishing

      • Fishing gives private benefit but might destroy the broader ecosystem, which has its own value for everyone (e.g., due to tourism).
      • Because individual fishermen do not pay for the damage they cause to the broader ecosystem, they will fish too much.

      Air pollution

      • Producing a good yields private profit but reduces air quality for everyone.
      • Because there is no price attached to air quality, the firms do not have to pay for its reduction and, hence, will produce too much.

      But...

      • There should be fishing/production/mining! After all, there are always benefits to these activities.
      • The tragedy of the commons is that the externality is not priced into these activities, driving them to inefficiently high levels.

      Notes:

      To be precise, in the last example the so-called "tragedy" is not that producing a good leads to air pollution; after all, this may be unavoidable if we want to consume the good. The tragedy is that even if we agree on the level of production and air pollution that is economically ideal for the collective, we will end up with more pollution.


      Break (10 minutes)

      Notes:

      • Next up class activities.

      Open Source

      Providing open-source software is like contributing to a public good and the community will therefore sooner or later collapse!

      Notes:

      • Why it might not fail:
        • Other incentives:
          • Intrinsic motivation
          • Reputation concerns (your github history is part of your CV)
          • Reciprocity
          • Direct benefit: Some contributors also use the software and benefit from improvements.

      Design a 2x2 game

      Jack and Christine are rivals and keep taunting each other in front of others. At one time, Jack challenges Christine to a game of chicken. He proposes that they both get in their cars and drive towards each other on a road. In the middle of the distance between each other, there is a small bridge with a single lane. Whoever swerves away before the bridge chickened out. If both keep straight, there is no way to avoid a strong collision between the two cars. All friends will be present to see the result.

      Design this game in a 2x2 matrix and assign payoffs to the different outcomes.

      Notes:

      • Need a white board!
      • Give Class about 5 minuets to discuss in small groups on this
      • Then have 10 minutes to ask the class you solve the 2x2 matrix and discuss (on next slide).

      Design a 2x2 game

      • What is/are the Nash Equilibrium/Equilibria here?
      • Which type of games does this remind you of?
      • How would you translate this game to real scenarios?

      Notes:

      • Game of chicken or Hawk-Dove Game
      • "Anti-Coordination Game" with the tension between competition and mutual benefit of compromise.
      • Real-world situations of conflict, where both would prefer not to fight but would actually like to intimidate, leading to a real conflict.
      • Two businesses would be better off not to engage in price war, but it would be good to be the only one to reduce the price to grab some market share.

      Workshop: Games

      Notes:

      • roughly 70 minutes

      Game 1: Guessing Game

      • We divide the classroom into three groups and play a guessing game.
      • The budget for this game is: $250.
      • The game is simple: each player enters a number from 1 to 100.
      • The player who guessed closest to 2/3 of the average number wins.
      • If multiple people win, the payoff is split equally.
      • The game is repeated for ten rounds.

      Game 1: Questions?

      Don't ask about strategies!


      Game 1: Guessing Game

      Link will be distributed!


      Game 1: Discussion

      • What number did you choose / what was your strategy? (which group were you in?)
      • Did your strategy change over time?

      Notes:

      • A number above 2/3*100 does not make sense
      • If everybody believes that, choosing a number above 2/3*2/3*100 does not make sense
      • ... it goes to 0
      • But does 0 Win? No!
      • Question: Who made these considerations?
      • Empirical results:
        • Financial Times asked their readers to submit their solution: Winning number was 13 (~1500 participants)
        • Other news magazine: ~3700 subjects, winning number 16.99, ~2800 subjects, winning number 14.7
        • There were spikes at 33 (response to randomness), 22 (response to that), and 0 (rationality)
      • Level-k-thinking: 1 or 2 steps most prevalent, seldom more than that.
      • Question: What would be the NE for multiplication of 1 of the mean?
        • It becomes a coordination game where all players choose the same value.

      Game 1: Results!


      Game 2: Prisoner's Dilemma

      • You play a Prisoner's Dilemma (groups of 2) over 10 rounds.
      • You will be randomly matched to another student in the class.
      • Budget for this game: $500
      • You have the option to chat between rounds.
      • Important: Keep the chat civil and do not reveal any identifying information about yourself.
      • We will read the chat.

      Game 2: Payoffs

      The other participant
      CooperateDefect
      YouCooperate200 points, 200 points0 points, 300 points
      Defect300 points, 0 points100 points, 100 points

      Game 2: Questions?


      Game 2: Let's go!

      Link will be distributed!


      Game 2: Results!


      Game 3: Public Good Game

      • We will play a public good game as presented in the lesson.
      • Budget for this game: $500
      • Groups of 4 over 10 periods.
      • Money in the project is multiplied by factor $1.6$.
      • With one additional mechanism: After each round each player sees the contributions of the other players and can decide to deduct points from them (at own costs).

      Game 3: Instructions


      Game 3: Contribution


      Game 3: Punishment


      Game 3: Payout


      Game 3: Questions?


      Game 3: Let's go!

      Link will be distributed!


      Game 3: Discussion

      • What was your strategy?
      • Were your groups able to sustain cooperation?
      • Did you cooperate?
      • Did you punish?

      Notes:

      • Additional free rider problem: Punishment was fixed to 100% of the other's points. That means, it was better to hope for other players to punish a player.
      • They cooperate maybe because they did not understand the game.

      Game 3: Results!


      Game 3: Discussion

      • How could we characterize players to types?
        • Freerider
        • Cooperators
        • Altruists
      • What do you think happens when playing this ...
        • ... for one round?
        • ... for many rounds?
        • ... when allowing for communication?
        • ... with different group sizes?

      Notes:

      What about empirical evidence?

      • One Round: Little contribution.
      • Many Rounds: Some little contribution but quickly to 0.
      • Some longer and stronger contribution but eventually going to 0 quickly
      • Different group sizes: Larger groups are more prone to freeriding, i.e., cooperation collapses more quickly.
      • Question: How can we distinguish freerider from those that only freeride because they expect others to freeride?
        • Answer: Ask them to provide a "conditional cooperation table" - i.e., they should state how much they contribute given other's /(average) contributions.
        • Real freeriders have 0 even if others contribute.

      Summary

      • the basics of game theoretic concepts.
      • different types of games.
      • how games can be modeled.
      • how to apply game theoretic thinking in our decision making in certain games.

      Questions


      Further Reading

      Price Finding Mechanisms


      How to use the slides - Full screen (new tab)
      Slides Content
      --- title: Price Finding Mechanisms description: Auction Basics for web3 builders duration: 1 hour ---

      Price Finding Mechanisms


      Introduction

      • A central topic of economics is price finding in markets.
      • How does a buyer and a seller agree on the price of a good?

      Supply / demand for BIG markets

      Notes:

      • As we saw yesterday, a market price must balance supply and demand.
      • Demand: number of items bought by consumers at a given price.
      • Supply: number of items produced and sold at a given price.

      Supply-demand for BIG markets

      • Supply-demand model works well for big markets.
      • We have lots of buyers, sellers, lots of info.
      • Predictable market price, balancing supply and demand.
      • Only sensible price at which a buyer and seller can trade.

      Notes:

      • When a particular good has a large number of buyers and sellers, and we have a well established market with lots of real-life trading data, we can develop a supply-demand model.
      • It allows us to accurately predict the price at which the market will converge to for a good, which is the point at which supply and demand balance out.
      • It only makes sense to trade at market price: if lower, the seller can easily find another buyer willing to pay more, if higher, the buyer can easily find another seller willing to charge less.

      What about difficult markets?

      Notes:

      Image source: https://funkypriceguide.com/collectible/512-gold-hopper-2018-sdcc-fundays/


      What about difficult markets?

      Today we focus on markets with few buyers or sellers, few trades, or non-standardized goods.

      • Gov't sells radio frequencies to broadcasters.
      • A painting by a famous artist trades hands.
      • An oil well goes on sale with unknown amount of oil.

      Notes:

      • The supply-demand model does not work in these examples
      • First example: there is a single buyer
      • Second example: very few buyers and sellers, trades are very infrequent, and goods are not standardized
      • Third example: not enough public data about the good.
      • In second and third cases, buyers may have wildly different personal valuations of the good.
      • A standardized good could be a liter of milk. A less standardized good is a wedding ring, or a painting

      A simple model

      • Selena wants to sell one item of a good.
      • Two buyers, Alice and Bob.
      • Each buyer has a secret valuation $v_A$ and $v_B$:
        how much they are willing to pay for the item.

      How does Selena optimally sell the item?

      Notes:

      • We consider the limit case of one single seller and two buyers, to keep things simple.
      • Throughout the talk we assume that a buyer will purchase the item if the price is below or equal to their valuation.

      Let's assume...

      • All of Selena's income is profit.
      • Alice and Bob each have random valuations between 0 and 1.
      • The valuations are secret but their distribution is well-known.

      Notes:

      • We saw the notion of an abstraction already in game theory class: we lose a bit in realism, but gain in structure which allows us to make precise statements, and develop a richer theory. The intuition and conclusions still apply in real life.
      • In particular, we are assuming that each of the three characters has a certain amount of information about the valuations of the other players, namely the distribution they are sampled from, and they can use this info to strategize their actions.
      • Assuming that each valuation $v_i$ follows a uniform distribution between 0 and 1 simplifies Selena's life, because she knows that for any value $x\in [0,1]$, the probability that a bidder's valuation is below $x$ is exactly $x$, i.e., $P[v_i\leq x]=x$.
      • However, assuming a uniform distribution is just for pedagogical reasons: most results in auction theory can be extended to more general distributions.

      Price posting

      • Simplest solution: Selena posts a price $p$, sells item to first taker.
      • What is her optimal price $p$? One maximizing her expected revenue.
      • Expected revenue: price $\times$ prob. of making a sale.
      • For price $p$, prob. of making a sale is $(1-p^2)$.
      • Hence her expected revenue is $p \cdot ( 1-p^2)$.

      Notes:

      • If both buyers accept the posted price, we assume the firm selects a random buyer to trade with.

      • The proof of this and all other math formulas can be found in the slides, for anyone interested.

      • Assuming that 100% of the sale price is profit for the seller, this profit is

        • $p$ in case of selling, $0$ otherwise, where
        • $(1-p^2)$ is the probability of selling the item.
        • To see this probability, consider conversely that the probability that none of the buyers accepts the price must be $p^2$, because each valuation must be below $p$, and these are two individual events each happening with probability $p$ (recall: valuations are uniformly distributed between 0 and 1).
      • $Pr{v_i \leq x}$ is the probability that the randomly drawn value of valuation $v_i$ is below $x$.

      • Independent values will be important: when is this (not) a sensible assumption? (Example: oil drilling rights auctions, ...)

      • Uniform distribution is just for pedagogical reasons; standard model allows for more general distributions.


      Price posting

      • At the optimum point, expected revenue is $\approx 0.38$.
      • We will see that an auction can do better.

      Notes:

      • Prices, valuations, and revenue are given in the same currency unit, say, USD.
      • How do we maximize the expected revenue function $f(p)=p ( 1-p^2)$? The function has a unique peak, so at the maximum, the first derivative must be zero.
      • The function is maximized for a $p$ satisfying $$ f'(p)=1-p^2 - 2p^2 = 0 \implies p = \sqrt{1/3}.$$
      • Expected revenue is then $f(\sqrt{1/3})=\frac{2}{3}\sqrt{1/3}\approx 0.38.$

      Auctions


      Lesson summary

      • Auctions can be superior to posting a price.
      • We discuss four important auction formats.
      • How to make buyers reveal their secret valuations.
      • Bid shilling.
      • The revenue-equivalence theorem.
      • Guides on what auction format to use in practice.

      Notes:

      • Auctions can provide more revenue to the seller, as we will see.
      • In a market with a posted price, buyers never need to reveal their valuation (i.e., how much they would actually be willing to pay for an item). In auctions, the same is generally true, as they may bid values much lower than their valuations, but there is an auction format where they tend to bid their actual valuations (called truthful bidding).
      • There is a specific sense in which, in theory, all four auction formats provide the same revenue for Selena.
      • However, in practice, some formats may be better suited than others, and we will see some such examples.

      Auctions

      • Alice and Bob (bidders) submit bids based on own valuations.
      • Selena (auctioneer) selects winner to allocate the item to.
      • The winner's payment is a function of the bids.

      An auction is a competitive game for buyers, where the seller makes the rules.

      Notes:

      • Bidders submit bids to auctioneer, which are a declaration of how much they are willing to pay for the item. In an ideal world, a bid equals the bidder's valuation (this is called a truthful bid), but of course depending on the bidder's strategy they may choose to underbid or overbid.
      • In this lesson we will have some fun analyzing the bidders' best strategies as well as the auctioneer's best strategies.
      • Both the winner and the winner's payment will dependent on the bids and the auction rules. The payment cannot be higher than the winner's bid, but it could be lower.
      • Auctions yield a higher revenue because they create competition among buyers.

      Auction formats

      1. There are two broad classes:
        1. Static Auctions: bidders submit their bids at the same time.
        2. Dynamic Auctions: bidders submit bids over time.
      • The difference is whether or not bidders can react to the bids of others and adjust their own.
      • Static auctions are also called sealed-bid auctions.

      Notes:

      • In static auctions, bidders give their bids simultaneously and cannot change it later.
      • Static auctions are also called sealed-bid auctions, as many times in practice the bids are submitted in sealed envelopes to the auctioneer.

      Auction formats

      Four auction formats we discuss today:

      • Static auctions
        • with a first-price payment rule
        • with a second-price payment rule
      • Dynamic Auctions
        • with ascending price (a.k.a. English auctions)
        • with descending price (a.k.a. Dutch auction)

      Notes:

      • Example, Selena auctions off a painting. Suppose in a static auction, Alice bids 1000 and Bob bids 2000. In a first-price format, Bob wins and pays 2000. In a second-price format, Bob still wins but only pays 1000.
      • The English format became popular in England. Selena continuously raises the price as long as at least 2 bidders accept it. As soon as only one bidder remains, they win and pay the current price.
      • The Dutch format became popular in the Netherlands, particularly in the tulip market. Selena starts at a high price and continuously lowers it as long as there are no takers. As soon as a bidder accepts the price, they win and pay the current price.
      • We are gonna analyze each of these four formats, and take the opportunity to discuss relevant notions in auction theory along the way.
      • For me, the second-price auction is the least intuitive format. But it turns out to be one of the nicest and easiest to analyze, so we will start there.

      Second-price auction

      • Why make the winner pay the second highest bid?
      • Least intuitive format, but strategically simplest for bidders.
      • Simply bid your valuation truthfully!
      • A bidder wants to maximize their expected profit:
        • own valuation - price paid, in case they win
        • zero, otherwise

      Notes:

      • Out of the four formats, we start our analysis with second-price static auctions, i.e., Alice and Bob give their bids in sealed envelopes to Selena, and she declares the winner to be the one with the highest bid, but the price to pay is only second highest bid.
      • To analyze the bidders' best strategy, we assume that each bidder wants to maximize their expected profit. Again, this is a sensible but strong assumption, as they could have some other equally sensible objectives. By the end of the lesson we will highlight how this assumption can be a bit unrealistic in practice.

      Second-price auction

      • Truthful equilibrium: a dominant strategy to bid truthfully.
      • Dominant strategy: outcome is at least as good as the outcome of any other strategy, no matter what the other bidder bids.

      Notes:

      • We can immediately say that you should never overbid (i.e., bid above your valuation), because in that case your profit can only be zero (if you lose) or negative (if you win). So the key question is whether you should ever underbid.
      • In the case that Bob's bid is higher than your valuation, bidding truthfully or underbidding both lead to losing and having a revenue of zero.
      • But if Bob's bid is lower than your valuation, any bid between these two values leads to you winning and having the same profit (namely the difference of these values), hence there is no gain in underbidding. However if you underbid too much, you run the risk of losing and having zero profit.
      • Hence in both cases we see that underbidding does not increase your profit, and can only decrease your chance of winning. So it is better to bid exactly your valuation.
      • The proof can be easily generalized to any number of bidders (to obtain that in a second-price auction, it is a dominant strategy for each bidder to bid truthfully.

      TIP: use arrow chars in unicode for style: https://www.htmlsymbols.xyz/arrow-symbols


      Second-price auction

      Expected revenue

      • Selena's expected revenue is expected value of 2nd highest valuation.
      • For two independent variables uniformly sampled from $[0,1]$, the expected value of the minimum is $1/3\approx 0.33$.
      • This is not quite as good as posting a price (which provided expected revenue $0.38$). Why not?

      Notes:

      • Let $F(x)$ be the cumulative density function (CDF) of the lowest of Alice and Bob's valuations, $\min{v_A, v_B}$, where $v_A$ and $v_B$ are independent variables sampled uniformly from $[0,1]$.
      • By definition, for any $x$ in $[0,1]$, $F(x)$ is the probability that the lower valuation is below $x$, or equivalently, $g(x):=1-F(x)$ is the probability that both valuations are above $x$; clearly $g(x)=(1-x)^2$, and hence $F(x)=1-(1-x)^2$.
      • From the CDF, we can compute the probability density function (PDF) by derivation: $f(x)=\frac{d}{dx}F(x)=2(1-x)$.
      • Finally, the expected value is computed from the PDF with the integral formula $$\int_0^1 x\cdot f(x) dx = 2\int_0^1 (x-x^2)dx=2\cdot \left(\frac{1}{2}-\frac{1}{3}\right)=\frac{1}{3}.$$

      Reserve price

      • Because the format we considered is not optimal for the auctioneer!
      • The optimal auction involves a reserve price $r>0$:
        • If no bid is above $r$, nobody wins.
        • If one bid is above $r$, the payment is $r$.
        • If both bids are above $r$, the payment is the second-highest bid.

      Reserve price

      Fact: Under any reserve price $r$, it is still optimal to bid truthfully, and if Selena sets $r=0.5$, her expected revenue is $\approx 0.42$, so it is better than posted price (where expected revenue was $\approx 0.38$).


      Reserve price

      Notes:

      • The proof idea is that if $r=0.5$, half of the time one valuation is above it and the other is below it, and the reserve price increases the paying price. On the other hand, if both valuations are below $r$ then there is no sale, which decreases Selena's revenue, but this case only happens one quarter of the time. Finally, if both valuations are above $r$, the presence of a reserve price does no affect the result. Overall, there is a positive net gain in Selena's expected revenue. Below we present the formal proof.
      • The probability that both valuations are below $r$ is $r^2$, the prob. that $r$ is in between the valuations is $2r(1-r)$, and the prob.that both valuations are above $r$ is $(1-r)^2$. You can check that these probabilities sum up to one.
      • In the first case Selena's revenue is zero, and in the second case it is $r$. Hence the expected revenue for the first two cases is $2r^2(1-r)$.
      • To compute the expected revenue of the third case, we simply copy the integral formula for the expected value (from a couple of slides ago), but we restrict the integration limits to $[r, 1\]$: \begin{align} \int_r^1 x\cdot f(x) dx &= 2\int_r^1 (x-x^2)dx\ &=2\cdot \left[\frac{x^2}{2}-\frac{x^3} {3}\right\]_r^1 \&=\frac{1}{3} - r^2 + \frac{2}{3}r^3. \end{align}
      • Putting all terms together, we obtain that the expected revenue is $$R(r)=0+2r^2(1-r)+\left(\frac{1}{3} - r^2 + \frac{2}{3}r^3\right)=\frac{1}{3}+r^2-\frac{4}{3}r^3.$$
      • To maximize the auctioneer's expected revenue function $R(r)$ above:
        • We evaluate the function at all the local extrema (minima and maxima).
        • Since the function is continuous and differentiable, the local extrema are found at the borders $r=0$ and $r=1$, and at the roots of the derivative $R'(r)=2r-4r^2=0$. These roots are $r=0$ and $r=1/2$.
        • By inspecting all these points, we find that the global maximum is found at $r=1/2$, with a value of $R(1/2)=5/12\approx 0.42$.
      • Literature: Myerson, Roger B. "Optimal auction design." Mathematics of Operations Research 6, No. 1 (1981): 58-73.

      English auction

      Recall the rules:

      • Selena continually raises the price.
      • At any price, you decide whether to stay or leave.
      • If you leave, you may not return.
      • If you are the last one in the auction you win
        and pay the price at which the second-to-last
        bidder left.

      Notes:

      • Next we move to English auctions, which have a very similar analysis.
      • Notice it has a flavor of a second-price auction: the price you pay is decided by another bidder, not you.
      • In a sense, these two auction formats are strategically equivalent!

      English auction

      • English auction is strategically equivalent to static second-price auction.
      • It is a dominant strategy to stay until the price reaches one's valuation.
      • The expected revenue for Selena is the also the same!
      • Consequently, these two formats are also revenue equivalent.

      Notes:

      • Why is staying until the price reaches your valuation a dominant strategy?
        • If you leave you get zero utility.
        • If you stay while the price is below your valuation, your eventual utility may be zero or positive, so there is no incentive to leave.
        • If you stay while the price is above your valuation, your utility is zero or negative, so you should leave immediately.

      Shill bidding

      • Second-price and English auctions popular among theorists, not so popular among practitioners.
      • One reason is that they are prone to shill-bidding: bidder that acts on behalf of the auctioneer to drive up the price.

      Notes:

      • Both second-price auctions and English auctions have the truthful bidding property, which is very satisfactory for theorists. However, as you can probably tell, they are not so popular in practice.
      • A shill bidder has no intention of winning, but just increasing the price that the winner must pay. They do it to benefit the auctioneer.
      • Shill bidding is frowned upon, sometimes illegal, but it is usually hard or impossible to prove and to prevent.

      First-price auction

      • The winning bidder pays her bid.
      • Other rules same as in the second-price auction; i.e.,
        • all bidders submit their bids simultaneously,
        • the highest bid wins.

      Notes:

      • New topic, time to wake up!
      • We move on to first-price auction, which as you recall, is a type of sealed auction.
      • Its analysis is more involved, and the resulting optimal bidding strategy is quite different from before.

      First-price auction

      Bidding in the first-price auction is not truthful.

      • Bidding truthfully can never be optimal: if you win, you earn nothing.
      • Underbidding is strictly better, you win sometimes and when you do you have a positive utility.

      First-price auction

      Equilibrium strategy: It is a Nash equilibrium for each bidder to bid half their own valuation.

      Nash equilibrium: A set of strategies, one per player, where no one has an incentive to change their strategy.

      Notes:

      • Recall Nash equilibrium we already covered.
      • A Nash equilibrium is a "recommended strategy per player", such that if you assume that the other bidder will follow their recommendation, then the best you can do is to follow yours.
      • As the same is true for the other players, it is indeed a reasonable assumption to think the other players will follow their recommendation.
      • However, if the other players picks an unexpected strategy, your recommended strategy may be suboptimal. In this sense, a Nash equilibrium is a weaker concept than a dominant strategy, in which the strategy of the other players is irrelevant.

      First-price auction

      Intuition: suppose you are Alice

      • If you bid 0, winning prob. is zero.
      • If you bid your valuation, profit is zero.
      • Hence, there is a sweet spot between 0 and your valuation where your expected profit is maximal.
      • It turns out this is bidding half your valuation, at which point you and Bob each wins half of the time.

      Notes:

      • Unfortunately, the full proof is outside the scope of the presentation, but we provide it here in the lesson notes.
      • Assume you are Alice, with valuation $v_A$ and bid $b_A$.
      • Assuming that Bob's bid $b_B$ equals half his valuation $v_B$, that $v_B$ is sampled uniformly at random between 0 and 1, and that your bid $b_A$ is at most $1/2$, your winning probability is: \begin{align} Pr[winning] &= Pr[b_B\leq b_A] \ &= Pr[v_B/2 \leq b_A] \ &= Pr[v_B \leq 2b_A] \ &= 2b_A. \end{align}
      • Your profit in case of winning is $(v_A-b_A)$, hence your expected profit is $2b_A(v_A-b_A)$.
      • It can be checked that this expression is maximized for $b_A=v_A/2$.
      • In summary, if Bob bids half his valuation, it is optimal for Alice to bid has her valuation, and vice versa. Together we have a Nash equilibrium.

      First-price auction

      Expected revenue

      • Reasonable to assume each bidder bids half their valuation.
      • Hence, Selena's revenue is $\frac{1}{2}\max{v_A, v_B}$.
      • The expected value of $\max{v_A, v_B}$ is $2/3$.
      • Hence, her expected revenue is $1/3$.

      The same as in second-price auction!

      Notes:

      • I find this result fascinating. The optimal strategies are so different, but somehow the fact that you underbid is exactly compensated by the fact that you pay more when you win.
      • Is there a deeper connection going on here? Or is it just a coincidence that the expected revenues are all the same so far?
      • Formal proof of the expected revenue: recall that the valuations $v_A$ and $v_B$ are assumed to be independent variables uniformly drawn from $[0,1]$.
      • The cumulative density function (CDF) of the higher valuation, $\max{v_A, v_B}$, is $F(x)=x^2$. This is because $F(x)$ is lower than $x$ only if both valuations are below $x$, and these are two independent events each happening with probability $x$.
      • Then, the probability density function (PDF) is $f(x)=F'(x)=2x$.
      • And the formula for the expected value of $\max{v_A, v_B}$ is $$R = \int\limits_0^1 x\cdot f(x)\mathrm d x = \int\limits_0^1 2x^2 \mathrm d v = \frac{2}{3}.$$

      Revenue Equivalence

      Fact: When valuations are secret and independent, there is no reserve price, and item goes to highest bidder, then all auction mechanisms are revenue equivalent.

      Notes:

      • This is a theorem established by Myerson in 1981.
      • Notice all auction formats are not strategically equivalent (namely, you should bid truthfully in some formats but not in others), yet they are still revenue equivalent for the auctioneer.
      • This result is very surprising to me, and one of the coolest results in game theory.

      Dutch auctions

      • Selena continually lowers the price.
      • As soon as a bidder accepts the price,
        they are declared winners and auction is over.
      • Winner pays the price they accepted.

      Notes:

      • We finally say a few words about the fourth and last auction format.
      • Notice it has a flavor of a first-price auction: you pick the price you pay. So you definitely do not want to bid truthfully, but rather underbid.

      Dutch Auction

      Recall the rules:

      • The auctioneer continually lowers the price.
      • At any price, you can decide whether or not to accept the price.
      • If you are the first to accept the price, you win and pay the price you just accepted.

      Dutch Auction

      • It turns out that the Dutch auction is strategically equivalent and revenue equivalent to the static first-price auction.
      • The price that you accept in the Dutch auction corresponds to the price that you'd bid in a static first-price auction.
      • The tradeoffs that the bidders face are very similar: take the current price or wait a bit at the risk of another bidder accepting first. It is an equilibrium to wait till the price is half your valuation.

      Recap

      Analyzed important auction formats:

      • Static first-price auction.
      • Static second-price auction.
      • English auction.
      • Dutch auction.

      Learned under standard assumptions:

      • First-price and Dutch auctions are strategy equivalent.
      • Second-price and English auctions are strategy equivalent.
      • All four actions are revenue equivalent.
      • Having a reserve price increases the expected revenue, and it beats posting a price.

      Break (10 minutes)


      Discussion


      Independence of valuations

      In our analysis, it was important to assume that bidders' valuations are independent from one another.

      Can you think of examples where this assumption isn't sensible?


      Independence of valuations

      Answer:

      Sensible: - a piece of art, where the bidders are final clients.

      Not sensible: drilling rights to an oil well. Bidders will have similar estimates of amount of oil, hence valuations are highly correlated.


      Common value auctions

      Special scenario: there is a unique valuation of item, but each bidder only has a private estimate of it.

      In these cases, it is observed that sealed-bid auctions tend to give higher revenue than dynamic auctions.

      Why do you think this is the case?


      Common value auctions

      The auction may be used as a means of gathering information from other participants to triangulate a price

      Answer: In a dynamic auction, a bidder can use the bids of others as additional signals of the correct valuation. If bids so far seem high, my initial estimate must be low, and vice versa, so I can adjust my personal estimate. Hence estimates converge.

      In a static auction, there is no convergence of estimates, so it is more likely that some bidders keep unusually high estimates. As a result, there is a higher chance that the winner ends up paying more than the correct valuation. This is known as the winner's curse.


      Equivalence of revenues

      It is observed in practice that first-price auctions lead to higher revenue than second-price auctions.

      This violates the equivalence of revenues, so an assumption in our analysis fails consistently.

      What do you think it is?


      Equivalence of revenues

      Answer: Risk aversion. People prefer lower uncertainty games, even if this means lower expected profits.

      Would you rather win a million dollars with a 50% chance, or 300 thousand with a 90% chance?

      In Nash equilibrium analysis for first-price auctions, we claimed that if Bob bids half his valuation, then Alice should bid half hers, so each wins 50% of time. But we implicitly assumed that Alice is risk neutral. Yet she might prefer to bid more and win more often.

      Notes:

      • Of course, Bob may react the same way and also bid more in an attempt to bid more often. As a result, they may still win about 50% of the time each, but both end up bidding too much due to their risk aversion.
      • In contrast, in our analysis for second-price auctions, the bidders' aversion to risk is not a factor.
      • Important: If you want to maximize the auctioneer's revenue and don't mind playing psychological games with the bidders, you should choose first price auctions over second-price auctions.

      Front Running

      Computer systems may have front runners: special nodes can see an incoming bid, react by creating their own bid, and make it look like their bid was created first.

      If you run an auction on a system with front runners, which of the four auctions would you use? Why?

      Notes:

      • Front runners are a known problem in eBay, trading platforms, and blockchain networks, etc.
      • Front runners have a special, faster setup that allows them to see an incoming message from a bidder, react to it by creating their own message, and make their message get registered first in the system.

      Front Running

      Answer: Meet front runner Fred.

      In a Dutch auction, if Fred is a bidder he waits for first bidder to signal accepting the price, and Fred makes the signal first. He's guaranteed to win with least possible price.

      In second-price auction, if Fred is auctioneer he can shill bid successfully: when a highest bid arrives, he creates bid slightly under it and makes it appear as if it was created first.


      Front Running

      Answer: Meet front runner Fred.

      In a first-price auction, if Fred is bidder and if he can "open the seals" he can win by slightly outbidding highest bid. (Poor data privacy, a common issue in computer systems)

      Hence, it might be best to run an English auction.


      Sniping

      In a dynamic auction with a fixed bidding time window, sniping is placing a highest bid as late as possible, so other bidders can't react and overbid you. The practice leads to low revenue. Particularly bad when done by a front runner (microseconds).

      How to protect an auction from snipers?

      Notes:

      • eBay uses such types of auctions, and is infamous for having snipers.

      Sniping

      Answer: candle auctions.
      Dynamic first-price auction with random ending time.


      Sniping

      Answer: candle auctions.
      Dynamic first-price auction with random ending time.

      • Similar to first-price auction
      • except that ending time is unpredictable.
      • At ending time, current highest bidder wins, pays own bid.
      • Sniping protection: the longer you wait to bid, the higher the chances the auction ends.

      Notes:

      • Fun fact: candle auctions were popular in England in the 17th and 18th centuries.
      • They would use a candle (hence the name). When the flame extinguishes, the auction is over.
      • Fun fact: Polkadot implemented a candle auction for assigning block space to applications.

      Workshop: Auction Games


      NFT Auction

      • You have the chance to bid on one of 25 unique NFTs that are created by a professional artist.
      • Use your budget that you accumulated during the last Academy Games.
      • Everything that you will not use for bidding (or if your bid was lower than your budget), you will receive in cash at the end of the Academy.
      • 100% of the revenue of the auctions goes to the artist.
      • You are randomly assigned to one of three auction formats

      The Artist & NFTs!


      Jeremy Gluck (Nonceptualism)

      • Canadian Artist living in Great Britain.
      • Diverse Practice: Post-digital fine art in NFT, digital art, film, photography, sound and video art, installation, and performance.
      • Exhibitions: Works showcased offline in cities like London, Sydney, Bath, Anglesey, and Swansea, and online in various publications.
      • Art is exhibited in Tides Fine Art Gallery, Mumbles, Wales.


      Format 1: Activity Rule Auction

      • The initial bidding phase lasts 30 seconds.
      • Every valid bid resets the timer.
      • You need to bid at least 30 cents more than the previous highest bid.
      • Whoever has the highest bid at the end, wins. Winners pay their bids.

      Format 2: Candle Auction

      • Auction Format from the 16th Century.
      • The auction lasts for exactly 4 minutes.
      • A โ€œcandle mechanismโ€ randomly determines, after the auction, when the auction ended
      • Grace-period of 1 minute.
      • Candle Phase of 3 minutes.
      • Whoever had the highest bid when the auction actually ended, wins.

      Format 3: Hard Close Auction

      • Similar to eBay auctions.
      • Auction lasts for 4 minutes.
      • Whoever has the highest bid at the end of the auction, wins.
      • Winners pay their bids.

      Auction 2: Questions?


      Auction 2: NFT Auction

      Link will be distributed!


      Auction 2: Discussion


      Auction 2: Results!


      Further Reading

      Polkadot & Kusama Auctions
      Data & Background information:


      Questions


      Bonus Game: Dollar Auction

      • Each group of five players is bidding on a one dollar note.
      • Minimum bid is 5 cents, bid increments of 5 cents.
      • Auction lasts 90 seconds.
      • Highest bidder gets the prize.
      • Both highest and second highest bidders pay respective bids.

      Notes:

      • Thought experiment designed by economist Martin Shubik.
      • Serves as analogy for war: it may be apparent for both sides that even winning the war won't make up for the losses, yet both sides are motivated to keep spending money on it.
      • You could get 1 dollar for 5 cents if no one else bids.
      • But if you end up second you have a loss.

      Dollar Auction: Questions?


      Dollar Auction


      Discussion

      • Would you consider the auction paradoxical?
      • Did you catch yourself acting irrational?
      • Did you observe any phases in the game?
      • Why do you think the game could be considered an analogy for war?

      Auction 1: Results!

      Collective Decision Making


      How to use the slides - Full screen (new tab)
      Slides Content
      --- title: Collective Decision Making description: Collective Decision Making for web3 builders duration: 1 hour ---

      Collective Decision Making


      Lesson highlights

      Decision
      making systems
      • Common systems
      • Goals and trade-offs
      Voting mechanisms
      • Common mechanisms
      • Desirable criteria

      Notes:

      • This lesson will be markedly less "exact" than the previous ones in the module. It will be less math and more critical thinking.
      • Hence, I encourage class participation.
      • However, many topics may be contentious and we might not be able to agree, especially not within the lesson time. The objective of the lesson is to awaken your curiosity on the topic, and I encourage you to continue the discussions outside the classroom.
      • In the second half of the lesson we'll talk about voting mechanisms, which are widely used within many decision making systems.

      Collective decision making

      An umbrella term that contains voting mechanisms,
      governance processes, policy making, budget allocation.


      Collective decision making

      • A way to gather the opinions of a heterogenous collective.
      • A way to aggregate their preferences in a concise way.
      • A way to reach a decision over the definition, resources, privilege or authority of a collective.

      Notes:

      • In this lesson we will study some of the different ways in which people can make decisions as a unit.
      • When people think of collective decision making, they usually only think of voting for a representative or representatives, and then letting them make all the decisions. But it is important to remember that this is just one of many possible mechanisms to reach a decision as a collective.

      Why?

      • A prerequisite for the formation of countries, companies, dynasties, i.e., strong institutions.
      • Agile decision making helps collective react to external changes: war, trade, migration.
      • Internally, they enable stability and progress: money, laws, private property.

      Why?

      • Even high tech, highly automated projects remain human collectives at their core.
      • A project's decision-making mechanisms ensure its strength and relevance over time.
      • We should consider these mechanisms along with the project's economics and security.

      Notes:

      • We use the words "collectives" in a very broad sense, to include not only nations and dynasties, but also modern collectives like tech companies, social networks, and online communities. It is important to remember that these are human collectives, no matter how technical or automated they appear.
      • Only well designed decision-making mechanisms can ensure that a collective stays strong and relevant over time. Otherwise, it will disintegrate due to internal discontent or external changes.
      • Hence, analyzing a project's decision making mechanisms is just as important as analyzing the soundness of its cryptography, economics and security.

      A more profound interpretation

      Long you live and high you fly
      Smiles you'll give and tears you'll cry
      And all you touch and all you see
      Is all your life will ever be

      -- Pink Floyd (Breathe)

      Any single entity may be identified as the aggregate of its (inter-)actions with its external environment. If a system's governance wholly defines these, it may properly be considered to identify the entity.


      Common systems

      • Dictator, unelected or elected.
      • Unelected committee (Zurich guilds).
      • Representative democracy -- elected committee.
      • Political parties
      • Direct democracy -- all participate in all decisions.

      Notes:

      • Can you think of other examples?
      • In the middle ages, the city of Zurich in Switzerland was run by guilds (merchants, farmers, tailors). Women, and anyone outside these guilds, had no say in the decision-making process.

      Goals

      Utility- decision maximizes society's welfare
      - it reflects people's preferences
      Legitimacy- decision is considered fair
      - people trust the process
      Practicality- process is fast
      - it is simple to understand

      Notes:

      • The three key goals of any decision making process are utility, legitimacy and practicality.
      • While utility is about the objective quality of the final result, legitimacy is about the perceived quality of the process. A mechanism is considered legitimate if most people trust the process, agree with it and feel empowered by it, even if they may not personally like the outcome.
      • For instance, if Fifa had decided unilaterally that Argentina wins the world cup, this decision making mechanism would have the same utility but not the same legitimacy as playing the games: everyone would be angry at it.
      • Finally, the practicality goal plays an important role especially if we have a very large population, or a short time window to make a decision, e.g., in an emergency.

      Goals

      • A dictator is practical. Is it legitimate?
      • Direct democracy is legitimate. Why not always use this?
      • Which mechanism maximizes utility?
      • Which is best in case of war?

      Notes:

      • This is the type of questions we are gonna explore in this lesson.
      • Direct democracy is considered legitimate but is unfortunately slow. If would not be the right mechanism for an emergency, such as war.
      • In the next slides, we look more closely at some of the most common systems.

      Dictator

      • Legitimate?
      • It is very practical.
      • Good in case of an emergency.
      • Only stable as long as the individual is stable (unlikely to be forever!).

      Notes:

      • What's wrong with a dictator? We typically think of it as illegitimate, but keep in mind that kings used to be considered legitimate in the past. And even today, the owner of a large tech company may be able to make unilateral decisions that impact millions of lives, and we consider this to be legitimate.
      • It is practical and allows for fast decision making in case of an emergency. For example, in the next pandemic, would you like to have every city vote on whether and when face masks should be mandatory in public spaces? Or would you prefer to have an authority impose it?
      • You could also argue that it leads to stability, as people can expect things to stay the way they are for the foreseeable future, and can plan accordingly. However, we also have several real-life examples where a dictator behaves erratically.

      Democracy

      Notes:

      • Lots of different things to lots of different people.
      • Democracy doesn't really exist, only systems which are democratic to a some degree.

      Simple Direct Democracy

      • Every person is consulted on every decision.
      • Considered highly legitimate.
      • Good at revealing the collective's opinions.
      • Leads to progress as anyone can propose ideas.

      Notes:

      • Direct democracy represents an extreme where every person is consulted on every decision, for instance via referendums.
      • It is great at revealing the people's opinions, and adapting at evolving opinions.
      • Similarly, if we assume that anyone is free to raise a proposal in direct democracy, then there is a sort of "free market of ideas". This leads to quick progress, because innovative ideas that challenge the status quo are not silenced.

      Simple Direct democracy

      Compare to a two-party system

      Notes:

      • For instance, imagine a two-party system like the US where people are forced to pick one of two parties, instead of directly casting their vote on every decision. Imagine the left party traditionally supports legalizing abortion and banning firearms, while the right party traditionally supports banning abortion and legalizing firearms. Such idiosyncrasy could have made sense in the past, but if there is a shift in preferences whereby more and more people want to legalize both abortions and guns, the system will not have the means to detect these preferences and adapt properly.
      • Similarly, if neither the left party nor the right party supports universal healthcare, then people who support healthcare do not have a voice.
      • Of course, similar issues remain for a multi-party system (more than two political parties), and the issues are even worse in a dictatorship.

      Simple Direct democracy

      • Not very practical:
        asking everyone to vote.
      • One decision at a time.
      • Not great for emergencies.

      Notes:

      • On the down side, direct democracy is unfortunately not very practical for two reasons. First, because asking every person to vote very slow. In particular, having a referendum would probably not be wise in an emergency such as war, or a detected vulnerability in a computer network.

      Voting paradox

      Does direct democracy maximize utility? Probably not.

      Example: Option A: 0 dollars Option B: 100 dollars

      Prob. that my vote flips the result from A to B: <0.1%.

      So, my expected profit for voting is <10 cents.

      Notes:

      • We saw that direct democracy is legitimate but not practical. How does it fare in terms of utility? Relatively well, but we argue here that it is not ideal, due to something called the voting paradox.
      • Suppose the collective must decide between options A and B, and I personally value option A as worth zero and option B as worth $100 to me.
      • Now, if most people vote for A, then A wins even if I vote for B, and if most people vote for B, then B wins even if I don't vote at all. Hence the most likely event is that my vote does not change the outcome.
      • When the number of voters is in the millions, the likelihood that my vote actually changes the outcome is tiny, well below $0.1%$. My expected profit for voting is then (my profit for B - myr profit for A)$\times$prob. that my vote flips the result.
      • In this example, my expected profit is less than 10 cents, which probably does not justify the effort of voting.

      Voting paradox

      Voting paradox: For a rational, self-interested voter, voting costs will normally exceed the expected benefits.

      Minimal chance that single vote changes collective decision, so most of the time the individual voting benefit is zero.

      A common good game: globally optimal if everyone votes, but personally optimal not to vote.

      Notes:

      • Voting paradox is a known concept in voting theory: if voters are rational and self interested, they should not vote. The fact that many people vote means that they are either not rational, or not self interested.

      Voting paradox

      Voting paradox: For a rational, self-interested voter, voting costs will normally exceed the expected benefits.

      It leads to a low turnout, which affects legitimacy.

      Turnout: percentage of people who choose to vote.

      Notes:

      • Indeed, we see for instance that in Switzerland, where direct democracy is used, the turnout is typically around 30%, and it's even lower in other systems.
      • Only a basic interpretation. Better technology and progressive voting systems (OpenGov!) can mitigate this issue.

      Rational ignorance

      Rational ignorance: refraining from learning when the cost of educating oneself on an issue exceeds the expected potential benefit.

      As a result, most people will not vote judiciously.

      Notes:

      • A notion related to the voting paradox is that of rational ignorance: when the expected benefit we obtain from voting is minimal, we might not vote judiciously, even when we vote.
      • We have limited time and limited capacity to learn things, so we will give preference to acquiring the knowledge that personally brings us the most expected potential benefit.
      • In the previous example, if our expected benefit for voting at most 10 cents, and it takes us two hours to understand what the best option is, it may be wiser to vote at random even if we vote, and use that time to learn to code, or learn to play guitar, etc.

      Rational ignorance

      Whether or not someone votes can be biasable.

      40% of the population supports a new proposal, while 60% rejects it. However, a company who will profit from the decision lobbies for the 'aye'. 50% of supporters and 30% of detractors vote. As a result, the proposal passes.

      Notes:

      • Another consequence of rational ignorance is that someone's vote can be biasable. If you don't have a strong opinion about a vote because you haven't spent the time to learn about the issue deeply, you're more likely to be easily influenceable.

      Solutions to the voting paradox

      • Representative democracy: elect a committee, whose members represent well the preferences of the passive people.

      • Delegated voting: give your voting power to a person who understands the issue well, so you don't have to.

      Notes:

      • We saw that a critique of direct democracy is that it's unrealistic to expect everyone in the population to invest their time to understand every issue, and then vote.
      • This leads to low turnout, which raises the question of legitimacy if a majority of the population does not voice their opinion.

      Advanced Forms of Democracy

      • Liquid democracy/delegation
      • OpenGov/multi-track delegation
      • Quadratic "replacement" voting
      • OpenGov/Optimistic approval
      • ...

      Solutions to the voting paradox

      • Both solutions based on desire of representing every clique in the collective.
      • Delegated voting is organic, helps with turnout, but remains slow.
      • A committee ignores preferences of small cliques, but can take complex decisions fast.

      Notes:

      • However, people naturally tend to congregate with other people with similar preferences, to form what we call cliques. So, you tend to be well represented by voters in your clique. The idea of both systems is to "sample" some opinions for every clique.
      • From my point of view, delegated voting remains very close in spirit to direct democracy, as every individual remains free to vote directly if they choose to do so, or they can choose a representative that represents them very well. We gain in terms of practicality as now fewer people need to vote, but the process is still slow and can only take binary decisions.
      • On the other hand, a committee is a bit closer to a dictator. It does a good job at representing the most common preferences of the collective, but not the preferences of small cliques. On the other hand, it can make subtle, non-binary decisions, and can make them fast.
      • Hence, it could be a good idea to use delegated voting for important, binary decisions (such as whether to make abortions legal), and an elected committee to make emergency decisions (such as what to do in case of war). I encourage you to consider how you would design an ideal decision making system.

      System capture

      Does a dictatorship maximize utility? Also probably not.

      System capture: A decision-making system in which a clique manages to impose its opinion consistently.

      Their special interest is prioritized over the interest of the collective, leading to a decrease of general welfare.

      Notes:

      • Before we argued that pure direct democracy might not maximize utility. Now we also argue that a pure dictatorship is also not ideal for utility.

      System capture

      Example: in a movie club, 40% of people like comedy, 30% like drama, 30% like action.

      Should they watch a comedy every time?

      What if it was 60%, 20%, 20%?

      Notes:

      • Imagine that a group of friends meet every week to watch a movie, and 40% of members always pick comedy while 30% pick dramas and 30% pick action movies.
      • Would it be fair if a comedy is chosen every time? Or what would decision-making system would you choose?
      • What if the percentages were 60%, 20%, 20%? Is it now fair to always pick a comedy, since it makes a majority of people happy?

      System capture

      • A dictatorship system is captured by definition.
      • A majority vote can lead to capture
        if the minority's opinion is consistently ignored.
        Known as "tyranny of the majority".
      • It can affect legitimacy.

      Notes:

      • Although a dictatorship is obviously a captured system, other systems also have varying degrees of risk of being captured.
      • For instance, as we saw with the movie club example, a majority vote can also lead to capture.
      • In turn, this can not only negatively affect the utility goal but also the legitimacy goal, if a group of people feel that their voice is always ignored.

      Random dictator

      Whenever there is a decision to be made,
      pick a person at random in the collective
      and let them decide as a dictator.

      Notes:

      • As a curious note, we mention a system called random dictator.
      • It works quite well in the example of the movie club: every week, pick a person at random and let them select the movie to watch.
      • On the other hand, it would be a bad idea for a country to let a random dictator decide on the tax policy every month. This is because in the case stability is more important than avoiding a tyranny of the majority.

      Random dictator

      • Very efficient.
      • Very capture resistant (on average).
      • Terrible for stability.
      • Where is it used?

      Notes:

      • A random dictator system is captured in the short term, but it is highly resistant to capture in the long term. This is because every clique, even very small ones, will have their voice heard with a frequency proportional to the clique size.
      • In fact, a version of random dictator is used in most blockchain consensus protocol, both based in proof-of-work and proof-of-stake. We do this for efficiency, and because stability of block production is not a priority, while the highest priority is to resist capture by an adversarial minority.
      • This is because, if most block producers act honestly, then it is a Nash equilibrium to also act honestly when it is your turn to produce a block. But if a minority manages to capture the system for some amount of time, and arbitrarily decide on the content of a majority of blocks produced during this time, then the minority has an incentive to attack the network for personal gain.

      Capture resistance vs. stability

      • A dictatorship is fully captured but also stable.
      • Direct democracy is at risk of capture, and still stable.
      • A random dictatorship is most resistant to capture, but unstable.

      Notes:

      • In this chart we consider the size of a clique or constituent, and plot it against the influence it will have over the collective decision in expectation.
      • For instance, if there is an unelected dictator then no clique, big or small, has any influence on decisions.
      • In a two-party system, only fairly large constituents will have a voice, while small constituents will have no voice.
      • If we elect a larger committee then smaller constituents are more likely to be represented and hence have an influence on the final decision.
      • Finally, in a random dictator system we have that every constituent, no matter how small, has a chance of influencing the final decision in proportion to the constituent size.
      • In general, we see that there is a trade-off between being resistant to capture and being stable.

      Recap

      • Goals: utility, legitimacy and practicality.
      • In emergencies, practicality may be more relevant.
      • Simple direct democracy is legitimate, not so practical.
      • Dictatorship is practical, not legitimate.
      • Probably neither extreme maximizes utility.
      • With new technology, we have better tools & mechanisms to make direct democracy effective

      Voting mechanisms

      • Widely used within decision-making systems.
      • Candidates to choose from: people, policies.
      • Voters who declare preferences through ballots.
      • A mechanism takes these ballots as input, and outputs a single candidate as the winner.
      • Many criteria to judge the mechanism.

      Notes:

      • New topic, wake up!
      • Voting mechanisms are some of the most popular ways to make collective decisions, and have been for centuries.
      • Why do you think that is? They are used for the election of leaders, for decision making within a governance body, choosing the winners in contests, among friends to decide on a restaurant, or in a family to name a dog.
      • Without a doubt, one of the most desirable criteria for a collective decision making mechanism is simplicity, meaning that it should be easy to understand and participate for the population. This is probably a big pro for voting mechanisms.
      • In the next slides we will explore some of the most popular voting mechanisms, along with some of the most popular criteria used to compare them.
      • What criteria do you think are most intuitive and desirable for a mechanism? Fairness? Proportionality? Utility? How would you define these terms precisely?

      Voting mechanisms

      We will see the following notions:

      • Single-vote mechanisms: plurality, two-round.
      • Ranked-vote mechanisms: Borda count, ranked-pairs.
      • Vote splitting, strategic voting.
      • Monotonicity criterion.

      Notes:

      • Two main types, depending on how the voters declare their preferences: in single-vote mechanisms, each voter selects one candidate, while in ranked-vote mechanisms, each voter ranks all candidates.

      Single-vote mechanisms

      Mechanisms where each voter picks one candidate.

      • Plurality voting: Candidate with most votes wins, even if they do not receive an absolute majority of votes.
      • Two-round voting: Top two candidates are voted on again. The candidate with most votes in second round wins.

      Notes:

      • These are some of the most popular voting mechanisms. They are certainly among the simplest ones.
      • Plurality is also known as "first past the post" in the UK, where it is widely used.
      • Consider also the generalization - multi-round mechanisms

      Single-vote mechanisms

      Plurality: 8 votes for A, 5 votes for B, 7 votes for C.

      Two-round:

      • 1st: 8 votes for A, 5 votes for B, 7 votes for C.
      • 2nd: 8 votes for A, 12 votes for C.
      1st2nd3rd
      8 votersABC
      7 votersCBA
      5 votersBCA

      Notes:

      • Recall that voters only vote for a single voter per round. But for convenience in what follows we assume we know the full candidate ranking of each voter, so we can analyze what would happen if any two out of the three candidates go head to head in the second round.
      • Are they equivalent? No. Look at the example above where there are three candidates A, B and C, and 20 voters with three types of preferences.
      • In plurality voting, we would only get to see the first column, and elect A with 8 votes.
      • In two-round, candidates A and C would go to second round, where the 5 voters in the last row would change their vote from B to C, and candidate C would win with 12 votes.
      • In general, several seemingly reasonable mechanisms result in different winners for the same collection of voters' preferences. This is why we need to look more closely into what properties we want out of a mechanism.

      Vote splitting

      Notes:

      • Which of these mechanisms is better? Here is an argument against plurality voting.
      • In 2014, Candidate Chandu Sahu, from a political party called BJP in India, was a favorite to win in an election against opponent Ajit Jogi. To his surprise, on the day of the elections there were 10 other independent candidates registered with the same exact name. Apparently most of these candidates had no history in politics, but someone found them and offered them money to run on the elections. Of course, the obvious suspect is the opponent, but it was never proven. Luckily, the original Chandu Sahu still managed to win, but by only a razor-thin margin of 1200 votes, as the other 10 fake candidates accumulated around 70 thousand votes.
      • This technique is known as vote splitting, and plurality voting is well known to be susceptible to it.

      Image source: https://indianexpress.com/article/political-pulse/the-10-other-sahus-how-namesakes-almost-cost-bjps-chandu-lal-sahu-a-win/


      Vote splitting

      • Imagine B and C are similar candidates, and most people prefer either over A.
      • Known issue in plurality. Two-round helps alleviate it.
      1st2nd3rd
      8 votersABC
      7 votersCBA
      5 votersBCA

      Notes:

      • Imagine that candidates B and C are very similar, i.e., they have similar points of view (or a similar name!) In the example, we see that a majority of the population prefers either of them to A, but since their votes are split, both candidates would lose out to A in plurality voting.
      • This issue is known as vote splitting. Two-round voting helps reduces its effect.
      • In electoral systems that implement plurality voting, vote splitting is sometimes used as a weapon by the less popular party (in our example, A), who strategically nominates a candidate similar to their rival to split their votes.
      • In turn, a common defense against vote splitting is to form alliances among similar candidates.

      Monotonicity criterion

      • A voter raising their rank for the winning candidate, or lowering their rank for a losing candidate, should not change the winner.
      • Plurality is monotonic, two-round voting is not.

      Notes:

      • Conversely, here is an argument against two-round voting.
      • Monotonicity is one of the most intuitive criteria that we would expect from a voting rule. It says that the winning candidate should be stable relative to monotonic changes of the voter's preferences. In particular, if a voter raises their preference for the current winner, or lowers their preference for other candidates, then the winner should not change.
      • Failure of this criterion means that the mechanism is susceptible to strategic voting, because a rational voter may be better off hiding their real preferences.

      Monotonicity Criterion

      Two-round:

      • 1st: 8 votes for A, 6 votes for B, 7 votes for C.
      • 2nd: 11 votes for A, 10 votes for C.
      1st2nd3rd
      6 votersABC
      2 votersACB
      3 votersBAC
      3 votersBCA
      5 votersCBA
      2 votersCAB

      Notes:

      • We will see now that two-round voting is not monotonic.
      • In this example, there are 21 voters, with 6 types of preferences.
      • In two-round voting, A and C go to the second round, where A wins 11 against 10.

      Monotonicity Criterion

      Bottom-row voters raise their preference for A.

      Two-round:

      • 1st: 10 votes for A, 6 votes for B, 5 votes for C.
      • 2nd: 10 votes for A, 11 votes for B.
      1st2nd3rd
      6 votersABC
      2 votersACB
      3 votersBAC
      3 votersBCA
      5 votersCBA
      2 votersACB

      Notes:

      • But now suppose the 2 voters in the bottom row raise their preference for A. They simply swap the positions of A and C in their ranking.

      • Now C doesn't make it to the second round. In the second round, now B wins 11 to 10 against A.

      • Hence, two-round voting is not monotonic.

      • If the bottom-row voters want candidate A to win, they are better off marking C as their top preference.


      Strategic voting

      A mechanism in which a voter increases the chances of a candidate by not voting for them.

      Notes:

      • A mechanism that is not monotone is susceptible to strategic voting.
      • This is the case with two-round voting.
      • Indeed, in the previous example, the bottom-row voters want candidate A to win, but are better off marking C as their top preference.

      Ranked-vote mechanisms

      Mechanisms where each voter ranks all candidates.

      • In 1770, Jean-Charles de Borda showed 2-round may elect a candidate that would lose out head-to-head to every other candidate.
      1st2nd3rd
      8 votersABC
      7 votersCBA
      5 votersBCA

      Notes:

      • Are there monotonic mechanisms better than plurality voting?Yes!
      • But they require more information from the voters: they need to rank all the candidates.
      • The unfair nature of plurality voting was pointed out in 1770 by French mathematician Jean-Charles de Borda, in a presentation at the French Royal Academy of Sciences.
      • He proved that plurality voting can elect the least favorite candidate: one that would be defeated by every other candidate in a head-to-head election.
      • Going back to our first example, recall that A was the winner in plurality voting, yet if A and B go head-to-head, A loses 8 to 12, and if A and C go head-to-head, again A loses 8 to 12.
      • Finally, if B and C go head-to-head, B would win 15 to 5, so in a sense B is the overall winner of pair-wise elections.

      Borda count

      • Each voter gives a rank of all $k$ candidates
      • Interpreted as if giving $k-1, k-2, \cdots, 1, 0$ points
      • Add up all points, select candidate with most points.
      • Borda count is monotone.

      Notes:

      • Borda proposed a new election mechanism, known today as the Borda count, that can be thought of as a generalization of two-round voting.
      • If there are $k$ candidates, this is interpreted as if the a voter gives $k-1$ points to their favorite candidate, $k-2$ points to their second favorite candidate, and so on until their least favorite candidate gets zero points. Then, we elect the candidate that received the most points overall.

      Borda Count

      2 points1 point0 points
      8 votersABC
      7 votersCBA
      5 votersBCA

      16 points for A, 25 points for B, 19 points for C.

      Notes:

      • In our example, we see that B wins, as expected. We highlight that for the exact same example, the winner was A under plurality and C under two-round!

      Recap

      • Borda count is arguably more fair.
      • But plurality and two-round voting are simpler.
      • Simplicity seems to be important in practice.
      • No voting mechanism is perfect.

      Notes:

      • In the area of voting theory, there are many other voting mechanisms known, and many other possible criteria to classify them. Voting theory is a fascinating topic at the intersection between mathematics and political theory.
      • In general, we see that the rules with better properties tend to be more complicated, which is unfortunately why they are not used as much. Hence, we can say that no mechanism is perfect.
      • Still, some mechanisms will be a better fit than others depending on our priorities for a system.

      Break

      Notes:

      10 minutes


      Discussion


      Goals clash

      We mentioned that the main goals of a decision-making system are utility, legitimacy and practicality.

      Can you think of real-life scenarios where these goals clash?

      Notes:

      • We saw that ranked-based voting, like Borda count, seems to be more fair (utility), but it is less simple to participate (practicality), which is probably why it's not so popular.

      Sampling voters

      In a country that runs referendums periodically, the turnout is only 2%, and older people tend to vote much more often than younger people.

      The president changes the voting mechanism so that in each referendum, only 5% of the population elected uniformly at random is allowed to vote. As a result, every person is allowed to participate in one out of 20 referendums on average.

      It is now observed that 80% of the people allowed to vote actually vote, so the global turnout grows from 2% to 4%. Moreover, the previously correlation between age and willingness to vote decreases considerably.

      Would you support this new mechanism?

      Notes:

      • We saw that representative democracy and delegated voting can be seen as sampling opinions from the collective, so that not everyone has to vote. In the exercise we take the sampling idea literally.
      • Is this new mechanism more or less legitimate? Does it have higher utility? I think it has higher utility but less legitimacy.
      • Is it practical? It could be an issue to have a sampling process that is verifiably random and unbiased. This affects practicality and legitimacy.

      A holy grail

      A "holy grails" of voting theory is finding a voting mechanism that simultaneously offers local privacy and global transparency: no one should be able to see another person's ballot, but enough aggregate information about the ballots should be public so that everyone can verify the correctness of the result.

      Suppose such a mechanism is created using cryptography; however, it requires every voter to use a private key and be trained in the basics of cryptography. A nation's president proposes to use such a mechanism, but the feedback from the population is negative, because people don't trust digital voting or cryptography.

      What goals are at odds here? What would you do as president?

      Notes:

      • While there is a lot of active research in this topic, there is no fully satisfactory solution yet.
      • The goals at odds are legitimacy and utility. If people don't trust the process then it is not legitimate, even if it has many advantages in theory.
      • The president could try to educate people about the new system, but should probably not impose it.

      Proof of Stake

      In a Proof-of-stake (PoS) based blockchain network, validators are the nodes that participate in the consensus protocol and produce blocks. While it is possible to have a "pure" PoS mechanism where every token holder can participate in consensus directly (imitating PoW), most high-profile projects bound the number of validators that get to be active at any given moment.

      Instead, these project opt for "representative democracy" and let token holders express their preferences for the set of active validators. Examples of blockchain projects that do this are: Polkadot, Cardano, Cosmos, EOS, Tezos and Tron.

      What do you think are the main reasons behind this choice?

      Notes:

      • Mostly for operational reasons
      • As the validator set size increases, the communication costs grow quickly (superlinearly), while the increase in security grows slowly past a certain point.
      • It is not sensible to run nodes for only sporadic participation. Validators would naturally tend to form pools anyway, to decrease the variance of their revenue, and profit from economies of scale.

      Secure Validators

      We saw that it's critically important for security to ensure that the set of validators is not captured by an adversary. Suppose we succeed, so that a super majority of validators participate in consensus honestly.

      A priori, the mechanism for electing validators can be completely independent of the mechanism for deciding on upgrades. However, some projects merge them together. In particular, consider a project that on any referendum, delegates the voting power of all the passive stakeholders to the set of validators.

      Does this constitute capture of the governance body?

      Can you think of examples where the interests of validators are markedly different from the interests of the collective?

      Notes:

      • Yes, for instance if we vote to increase the staking rewards, all validators would support it, but this negatively affects stakeholders who are using their tokens in other ways (such as in financial applications like lending).

      The Economics of Polkadot


      How to use the slides - Full screen (new tab)
      Slides Content
      --- title: The Economics of Polkadot description: Tokenomics of Relay Chains and more duration: 2 hour ---

      The Economics of Polkadot


      Overview

      • Which economic pieces build the the Polkadot Network?
      • What are their mechanisms and incentives?
      • How are those pieces interrelated?

      Remark: There are currently many changes planned



      Token Economics


      DOT Token

      • Native token of the Polkadot network.
      • 1 DOT = \(1\mathrm{e}{10}\) Plancks
      • Planck = smallest unit of account.
        • Reference to Planck Length = the smallest possible distance in Physics.
      • Utility Token with several use-cases:
        • Governance (decentralization)
        • Bonding in slot auctions (utility)
        • Staking (security)
        • Message passing (e.g., transfers)

      Inflation Model

      • Expansion in token supply.
      • Token minted from thin air.
        • Used to pay staking rewards for validators and nominators.
        • (Indirectly) fund Treasury.
      • Central economic variables of the model are:
      • Exogenous:
        • Staking rate (Total amount of staked DOT / Total amount of DOT).
      • Endogenous:
        • Optimal staking rate (a sufficient backing for validators to provide reasonable security).
      • Total inflation rate (10%).

      Inflation Model

      • Different states of DOT:
        • Liquid: Used for messaging and liquidity on markets.
        • Bonded (Staking): Economic mass that guarantees the security of the network.
        • Bonded (Parachains): The demand for DOT tokens by parachains.
      • The goal is to obtain (some) sensible ratio between those three token states.

      Inflation Model

      • Central variable: Ideal staking rate (currently ~53.5%).
      • Highest staking rewards at the ideal staking rate.
      • Incentives to (increase) decrease the staking rate it is (below) above the optimal.
      • Staking inefficiencies -> Treasury.
      • Ideal staking rate scales with number of active parachains (0.5% less with each parachain).

      Inflation

      • In the fiat-world, inflation has a negative connotation.
      • This is a general discussion in economics.
      • My take on this:
        • Predictable (maximum) inflation is good.
        • It incentivizes to work with the tokens (i.e., bond for good parachains, use for message passing).
        • Deflation can cause a halt of economic activity, because people start hoarding tokens.

      Notes:

      Question: What do you think about Inflation?


      Potential changes incoming

      • The current system incentivizes to move the staking rate to the ideal rate.
      • Then, Treasury inflow would be 0 DOT.
      • That is not sustainable.
      • Proposed change: Detach inflation to stakers from total inflation and divert the rest to Treasury directly.

      Staking: Concept

      • Nominated Proof-Of-Stake (NPoS).
      • Economic incentives of validators and nominators are aligned with those of the network.
        • Good behavior is rewarded with staking rewards.
        • Malicious / Neglecting behavior is punished (slashed).
      • Currently, minimum total stake is ~1.6M DOTs.
      • The total stake in the system directly translates to the economic security that it provides.
      • Total stake is pooled from validators (self-stake) and their nominators (nominated stake)
        • High degree of inclusion
        • High security
        • The goal is to get as much skin-in-the-game as possible.

      Validators

      • What makes Validators resilient:
        • Self-stake
        • Reputation (identity)
        • High future rewards (self-stake + commission)

      Nominators

      • Bond tokens for up to 16 validators that they deem trustworthy.
      • They have an incentive to find the best ones that match their preferences.
      • They are tasked to collectively curate the set of active validators.

      Rewards

      • What are staking rewards for?
      • Validators: Hardware, networking, and maintenance costs, resilience.
      • Nominators: Curation of the active set of validators, sort out the good from the bad ones (Invisible Hand).

      Validator Selection

      • The job of nominators is to find and select suitable validators.
      • Nominators face several trade-offs when selecting validators:
        • Security, Performance, Decentralization
        • Ideally those variables in their historic time-series.
      • Economic Background:
        • Self-stake as main indicator of skin-in-the-game.
        • Higher commission, ceteris paribus, leaves a validator with more incentives to behave.
      • Various sources of trust
      • Efficient validator recommendation is one of my research topics.

      Parachains


      What are Parachains?

      • Parachains (or cores) are the layer-1 part of the protocol.
      • Blockchains of their own that run in parallel.
        • Highly domain specific and have high degree of flexibility in their architecture.
        • Share same messaging standard to be interoperable and exchange messages through the Relay Chain.
      • Polkadot: 43 Parachains, Kusama: 46 Parachains.
      • Their state transition function (STF) is registered on the Relay Chain.
        • Validators can validate state transitions without knowing all the data on the Parachain.
        • Collators keep the parachain alive (but are not needed for security).
      • Offer their utility to the network.

      Parachain Slots

      • The access to the network is abstracted into the notion of โ€œslotsโ€.
        • Leases for ~2 years on Polkadot (~1 year on Kusama).
        • Only limited amount of slots available (networking).
        • The slots are allocated through a candle auction.
      • Bonded tokens held (trustlessly) in custody on the Relay Chain.
      • The tokens will be refunded after the slot expires.

      Economic Intuition

      • Tokens cannot be used for anything (staking, transacting, liquidity, governance).
        • That means, tokens locked cause opportunity costs.
        • An approximation is the trust-free rate of return from staking.
      • Parachains need to compete with those costs and generate benefits that exceed those opportunity costs.
        • Sufficient crowdloan rewards.
        • Sufficient economic activity on-chain that justifies renewal.
      • Slot mechanism creates constant demand for DOT token.
      • It is costly to be and remain a parachain.
        • Natural selection mechanism to select useful parachains.
        • Continuous pressure to gather funds for extending slots.

      What do Parachains get?

      • Parachains pay for security.
        • Every parachain is as secure as the Relay Chain.
        • Polkadot is a security alliance with network effects.
        • Not only scaling number of transactions, but it also scaling of security.
      • Security is a pie of limited size, because financial resources are limited.
      • Every chain that secures itself need cut a piece of the cake, which leaves less to others (zero-sum).
      • Shared security protocols allow to keep the cake whole and entail it to all participants.
      • Shared security is a scaling device, because the amount of stake you need to pay stakers to secure 100 shards is less than you need to pay stakers to secure 100 individual chains.

      Outlook Polkadot 2.0

      • Based on Gavโ€™s Keynote at Polkadot Decoded 2023.
      • A new narrative of the whole Polkadot system.
      • We move away from regarding Parachains as a distinct entity but rather regard Polkadot as global distributed computer.
      • It's spaces and apps rather than chains.
      • This computer has computation cores that can be allocated flexible to applications that need it.
      • Coretime can be bought, shared, resold.

      Core Attributes of Blockspace

      • Security: The scarcest resource in blockchain, crucial in preventing consensus faults or 51% attacks that could compromise transactions.
      • Availability: Ensuring blockspace is available without long waiting times or uncertain costs for a smooth, seamless interaction within the decentralized ecosystem.
      • Flexibility: The ability of blockspace to be fine-tuned by the consumer for specific use-cases.

      Blockspace Ecosystem

      • A networked collection of individual blockspace producers (blockchains) offering secured, fit-for-purpose, efficiently-allocated, and cost-effective blockspace.
      • A valuable aspect of a blockspace ecosystem is its connective tissue of shared security and composability.
      • Dapp developers or blockspace providers can focus on their unique features, reusing existing capabilities within the ecosystem.
      • For example, a supply chain traceability application could use different types of blockspace for identity verification, asset tokenization, and source traceability.

      Bulk markets

      • It's not yet finalized how they work but likely:
        • Around 75% of cores are allocated to the market.
        • Cores are sold for 4 weeks as NFT by a broker.
        • Unrented cores go to the instantaneous market.
        • Price de-/increases relative to demand.
        • Current tenants have a priority buy right for their core(s).

      Why the change?

      • This allows for low barriers of entry for people to simply deploy their code to a core and test stuff
      • It makes blockspace more efficient, because not all teams can/want to have a full block every 6/12 seconds.

      Treasury

      • The treasury is an on-chain fund that holds DOT token and is governed by all token holders of the network.
      • Those funds come from:
        • Transactions
        • Slashes
        • Staking inefficiencies (deviations from optimal staking rate)
      • Through governance, everybody can submit proposals to initiate treasury spending.
      • It currently holds around 46M DOT.
      • Spending is incentivized by a burn mechanism (1% every 26 days).

      Treasury as DAO

      • A DAO (decentralized autonomous organization) that has access to funds and can make funding decisions directed by the collective (that have vested interest in the network).
      • This has huge potential that might not yet have been fully recognized by the people.
      • This provides the chain the power to fund its own existence and improves the utility in the future. It will payโ€ฆ
        • โ€ฆ core developers to improve the protocol.
        • โ€ฆ researchers to explore new directions, solve problems and conduct studies that are beneficial for the network.
        • โ€ฆ for campaigns educating people about the protocol.
        • โ€ฆ for systems-parachains (development & collators).
      • A truly decentralized and self-sustaining organization.

      How does it all fit together?



      Takeaways

      • Polkadot is a system that offers shared security and cross-chain messaging.
      • Security scales, i.e., it takes less stake to secure 100 parachains than 100 individual chains.
      • The DOT token captures the utility that the parachains provide and converts it to security.
      • The slot mechanics (renewal, auctions) creates a market where parachains need to outcompete opportunity costs to be sustainable (i.e., they need to be useful).
      • Polkadot is a DAO that will be able to fund its own preservation and evolution.
      • There are many changes to come with Polkadot 2.0 creating a much more agile system.

      Further Resources

      โ›“๏ธ Blockchains and Smart Contracts

      Blockchain and applications built on them covered in depth conceptually and hands-on operation and construction.

      Blockchains and Smart Contracts Overview


      How to use the slides - Full screen (new tab)
      Slides Content
      --- title: Blockchains and Smart Contracts Overview description: Initial look at web history and the problems blockchain and web3 are solving duration: 30 - 45 minutes ---

      Blockchains and Smart Contracts Overview


      Upholding Expectations

      What is the core problem we want to solve?

      Trustless provisioning of infrastructure.

      Notes:

      Something kind of like a server, that doesn't rely on a server operator, and has strong guarantees like Cryptography has to offer.

      One framing: Coming to a shared understanding of a common history, and therefore a common state, of a system.


      Comparison with Cryptography

      Cryptography provides strong guarantees about messages without a trusted party, regardless of the conduit over which a message was transported.

      Notes:

      Crypto guarantees:

      • No tampering
      • No eavesdropping
      • Authorship

      Application Disentanglement

      Removing trust allows us to unpackage applications.

      Notes:

      The idea here is to discuss how applications are often seen as an entire bundle: e.g. Instagram is the database, the algorithms, the UX. But when we have credible expectations that we're interacting with the same system, rules, data, it's possible to build lots of ways to access and interact with the system. It also removes the need for a central authority to deal with all appeals/complaints from various users.


      Desired Properties

      ---v

      Permissionless access

      Anyone should be able to access and interact with the system.

      ---v

      Privacy

      Users should have credible expectations about what information they give up about themselves.

      ---v

      Authenticity

      Users should have credible expectations about the messages they see, regardless of the platform the messages are on.

      ---v

      Finality

      Users should be able to form credible expectations about when a state transition is final.

      ---v

      Behavior

      The system should behave as expected, even if system operators do not.

      ---v

      Unstoppability

      No individual actor, company, state, or coalition should be able to degrade any of these properties.


      A Shared History

      Notes:

      So now we understand the goals of web3. How do we achieve them? The key is allowing users to agree on a shared history. The simplest blockchains do nothing more than timestamp and attest to a stream of historical records. In Web 2 users have no visibility into the history of the app. They must trust the provider to accurately represent the current state. By giving the service provider the power to change the story, we give them the power to shape our understanding of reality and consequently our behavior.

      ---v

      A Shared History

      Any large-scale operation - whether a modern state, a medieval church, or an archaic tribe - is rooted in common stories that exist only in people's collective imaginations.

      Telling effective stories is not easy. The difficulty lies ... in convincing everyone else to believe it. Much of history revolves around this question: How does one convince millions of people to believe particular stories about gods, nations, or LLCs?

      -- Yuval Noah Harari, Sapiens --

      ---v

      Shared Story of a State Machine

      If we agree on:

      • The starting state
        (aka genesis state)
      • The history of transitions

      Then we MUST agree on:

      • The current state

      Notes:

      Now that we have a formal math-y model of systems that we care about, we can see that the notion of shared stories being powerful is more than slick language of philosophical mumbo jumbo. Even the term genesis state (or genesis block) is taken straight from mythology. We aren't newly discovering or inventing the idea that having a shared understanding of our past is important. It dates back to pre-history. We are just formalizing it and applying it to digital services.


      Blockchains (Finally)

      A blockchain can be thought of in three parts

      State Machine

      What does the state hold?

      What are the rules to change it?

      Shared History (data structure)

      Which potential histories exist?

      Consensus

      Which history is the real one?

      What part of history is final?

      Notes:

      First, each blockchain tracks some state machine. We've discussed several examples of what that might be already, we'll code some simple examples shortly, and we'll spend all of module 5 digging into how to create a blockchain-friendly production-ready state machine.

      Next is the Blockchain Data structure. This data structure is basically a linked list of state transitions. But unlike the linked lists you studied in your data structures course, it isn't just linked by memory addresses or any other malleable thing. Instead it is cryptographically linked so that if anyone presents a different history, you can tell right away that you don't agree on a shared history. We'll dive into this data structure in the next lesson.

      Finally, is a consensus mechanism. Defining a state machine alone does not uniquely define a history. There are many possible valid histories. Just like the many worlds interpretation of quantum mechanics. To really agree on the current state, we need to agree on which of the possible histories is the real one.

      ---v

      Short History of Blockchains

      ---v

      Bitcoin

      Uses an unspent transaction output (UTXO) model & Proof of Work (PoW) consensus.

      Notes:

      Who knows where the figure is from?

      CLICK

      Bitcoin was first. It was a money app. The first app chain. It hosts a smart contract in the broad definition.

      It was the first time most people considered a digital service that was not run by a particular person.

      Figure source: Bitcoin white paper

      ---v

      Litecoin, Monero, Dogecoin

      Notes:

      Only a few year later people realized they could fork the code and make small changed and improvements. Some changes were trivial. Some were interesting: monero and privacy.

      For me personally, this was a small crisis. I thought bitcoin was the one global one. Aren't these other coins undermining the narrative? NO! The point is that anytime you don't like the system or have an idea for a better one, you can do it! If you don't like bitcoin, build your own coin and make it better. Let the market decide.

      ---v

      Ethereum

      • Generalization of Bitcoin: provides a quasi-Turing-complete VM
      • Uses an account-based system
      • Accounts can store balances, but can also store executable code (smart contracts)
      • Each contract can have its own internal state and API

      ---v

      Smart contracts - Two Definitions



      Broad Definition "Szabo definition"

      A machine program with rules that we could have defined in a contract, but instead a machine performs or verifies performance.

      Narrow Definition "web3 definition"

      A program that specifies how users can interact with a state machine and is deployed permissionlessly to a blockchain network

      Rocket Cash

      Web2 digital cash example.

      ๐Ÿ“ฅ Clone to start: Rocket Cash

      See the README included in the repository for further instructions.

      Digital Services as State Machines


      How to use the slides - Full screen (new tab)
      Slides Content
      --- title: Digital Services as State Machines description: Digital Services approximate smart contracts; both are modeled as state machines. ---

      Digital Services as State Machines


      Web0

      Telegraph, Telephone

      Users transmit information peer-to-peer.

      Crypto not typically used except by military, but upheld guarantees when used.

      Notes:

      I want to go through a brief history of the web. This is not a history course, but it is important to understand the context of the web and how we got to where we are today. I will be using the term "web" to refer to the internet, but I will also be using it to refer to the world wide web. The internet is the physical infrastructure that allows computers to communicate with each other. The world wide web is one of the several utilizes including e-mail, File Transfer Protocol (FTP), Internet Relay Chat (IRC), Telnet, and Usenet that form the Internet. Okay so let's back up before all of that though - The term web0 isn't a common nomenclature - but basically this is old school Telegraph and telephones and in this era users transmitted information peer-to-peer. There was no intermediary. There was no service provider. There was no web server. There was no database. I would call up a friend - be connected via a wire - and we would talk, end to end. sure there are switchboard operators, but you could think of it as one really long wire connecting me to my friend I'm calling. Which is actually a perfect scenario for cryptography to uphold its guarantees. Once I talk to the switchboard operator and get routed to you, then we can just talk over our Cipher. It's really peer to peer, which is great! However, people were not really aware of the threat model and cryptography was not really used except by the military. But when it was used, it was used to uphold its guarantees. So this is like a property that we want to try to restore in web3.


      Web1

      Introduction of always-on servers.

      Still mostly peer-to-peer.

      Cryptography more often, but still not ubiquitous.

      Notes:

      Web 1 brings us to the 70s and 80s (maybe early 90s). This is an introduction to always-on servers. so with web0, with the telephone - I pick up my phone when I'm calling someone or when someone calls me and then I put it down and I'm offline the rest of the time. But in web1, we have these always on servers but it's still mostly peer-to-peer because people, at least two kind of a first order approximation had their own servers so in the clipart, we've got users and their servers - these pink lines are a connection to your own server. So this user has her own server and these two users maybe share a server - I don't know, they work for the same company or go to the same University or something and then this user doesn't have her own, maybe she's just consuming information and she's more of a casual user. So web1 is pretty similar to web0 and cryptography did start to become used a little bit more, but it still was not very ubiquitous and it's basically just because of the kinds of applications that people were using web1 for - it was mostly just for sharing information. Classic example is publishing - you would publish your paper or your data set and the whole idea is if you're publishing something, you really don't need to be keeping it secret, and in fact you're actually trying to NOT have it be secret. You're trying to put it out there for everyone to use and read. Now authenticity guarantees for sure would have still been great here - to know that your data didn't get tampered with along the wire... but again the threat model just wasn't in practice quite so bad and so like people got away without cryptography.


      Web2

      Introduction of Digital Services with Rich State.

      Administered by service providers: "Send us your information."

      However, users must place faith in the service provider.

      Cryptographic guarantees are about interactions with the service provider, not peers.

      Notes:

      So that brings us to web2. 1999 or 2004, wherever you want to put it really. Web2 brings the introduction of Digital Services with Rich state - so we're talking about web apps. Who here remembers Myspace? I was obsessed with curating my top 8. And then of course Facebook or whatever social media platform and beyond. But it's not just media either, there's eBay and Amazon and you can register to vote online and there's all these Services where you just go online you fill some data into a web form - maybe you don't even have to do a page reload - and that basically submits a transaction to some back-end server somewhere! So we begin to get these really really useful Digital Services. You can even think about things like Google search for example, that hardly existed in web1. Notice that there are no more pink lines in the diagram - people don't have their own servers - they're all connected to this one Central server. Obviously in reality it isn't a single Central server, but the the point kind of remains - it's a small number of centralized service providers.

      ---v

      Digital Services

      People rely on digital services every day. They are inescapable and valuable.

      • Game Servers (AOE or Minecraft)
      • Twitter, Instagram, Facebook, etc.
      • Journalism and sources
      • Banks
      • Lawyers, notaries, regulators

      Notes:

      I should note - Digital services are not bad in and of themselves. They are very valuable. We use all of these every day. We are even using some to administer this course. But they are also hard to escape. What are other examples? I have some listed here - online banking, what else?

      And the key thing here is that they're all administered by these service providers and the model is - don't call your friend on the phone and communicate directly with them anymore - instead send us your information -

      ---v

      Trust Example

      Two users on Twitter:

      • Trust that we are seeing information from the same database*
      • Trust that if a tweet is from X, then X wrote that tweet*
      • Trust that others see our messages as from us*
      • Trust that the messages we see are the messages the users wrote*
      • Trust that we're interacting with the application as equals

      Notes:

      Or rather, send your Tweet directly to Twitter server and then we'll publish it and then when your friend comes to us, we'll tell them that this is what you tweeted. trust me bro, this is what lauren tweeted, it's real. So this is asking you to place trust in the service provider. Once web2 became popular, cryptography became a bit more widespread. Look at the icon at the top of your browser - let's use github for example - That is ensuring me that I'm communicating over SSL to GitHub and they're encrypting all the communication and I am too, which is awesome! Because suddenly we're using this really cool cryptography that we've been developing over decades! But here's the thing - say if you go to pull my repo, you're not pulling it from me anymore, you're pulling it from github. So GitHub knows that nothing was corrupted between me and them and you know that nothing was corrupted between GitHub and you. But the thing is, github's right there - they could change my repo if they wanted to or they could maybe even have the power to take my repo offline. But here, the slide reminds us that we all essentially accept a trusting relationship with the service provider. We trust that we're seeing information from the same database, we trust that if a tweet is from Aaron, then Aaron wrote that tweet, we trust that others see our messages as from us, we trust that the messages we see are the messages the users wrote, and we trust that we're interacting with the application as equals. Cryptography actually provides a lot of these guarantees, but not when an intermediary has stepped in between users.

      ---v

      God Mode Enabled

      In web 2, service providers can perform abuses:

      • Censoring user interaction
      • Restricting users interaction
      • Failing to produce requested data
      • Mutating state opaquely

      Notes:

      Okay in summary, these are the problems that trusted service providers can do to us. They can censor our interactions, they can restrict our interactions, they can fail to produce requested data, and they can mutate state opaquely.

      ---v

      Thought experiment: Digital Currency

      Bitcoin's application was digital currency - a trivially simple application.

      Could this be built with Web2 technology?

      Notes:

      Let's do a thought experiment. Bitcoin came out 2009, it's a trivially simple application. n Bitcoin's application was digital currency. When you think about just the money thing - sending money from one person to another. How many of you think that if I give you like a weekend hackathon project, you could develop some kind of web2 style service and deploy it to AWS or digitalocean or wherever - that would basically act like a currency where I could log in and send my peers money. Could you build this with web2 technology? Yep you absolutely could. This is the kind of simple app you might build in a Freshman year course. You just need to maintain a set of bank notes and their owners (or alternatively a set of accounts and their balances.) So why didn't this exist in web 2? I hope this is pretty obvious. Because the provider could print money. Or steal money. Right? The moment somebody pisses me off I can go and freeze their funds or whatever. So even though coding the cryptocurrency application is like super simple it's totally useless because there's no incredible way to build trust in it. And that's what the blockchain is about. And that's what Joshy is going to speak to very soon.

      ---v

      Distributed Applications in Web 2

      Providers run redundant data centers to prevents accidents.

      But it still assumes benevolent participants and some trusted leader.

      Notes:

      Something we should call out - there are some distributed applications in web2, but it's a little bit different than what we're talking about here. It's not like AWS folks are noobs and they have all their stuff in one Data Center and the moment that there's an earthquake or a fire or whatever everything that hosted on AWS goes down. They're better than that, they have data centers all over the world. But the difference there is that all of those data centers are operating as trusted peers. So one of them will lead and those changes will be synced to the others and the moment one goes down it fails back to the next one to keep the services live. Which is for sure a good thing, I'm not hating on that aspect of it, but the point that I'm making is that it doesn't make it trustless just because it's geographically distributed. It just makes it more resistant to accidents. AWS could still cut it down. Ultimately, the point is that in web2, the masses become beholden to the service providers who were free to extract value and manipulate the users.


      Web3

      A provision of digital services without the need to trust a service provider.

      Providers do not need to be trusted; they are economically incentivized to behave honestly.

      Notes:

      Okay so web3 - We want to maintain the value, versatility, and richness of Web2, but remove the trust, and possibility of extractive behavior. Here I have: It's a provision of digital services without the need to trust a service provider. At an abstract level, the way that we make people trustworthy is that we give them economic incentives to behave honestly. So throughout this course we're going to design protocols and we're going to say things like, as long as two-thirds of the operators in this protocol are following the protocol, then all these guarantees I talked about are upheld! That's great. But then the glaring question is: well okay - how do you make sure that two-thirds of the operators are following the protocol? It always come back to economic incentives. Hence why you had econ and tokenomics last week. Because that's how we're going to get all of these mutually distrusting, not otherwise incentive aligned parties to agree to follow the protocol - by creating the incentive such that following the protocol is the best strategy: they can maximize their own utility by following the protocol.

      ---v

      Desired Approach

      Allow users to interact with a common system without trusting any intermediaries.

      Opens the door to new application stacks:

      • Shared state and state change rules
      • Custom rendering, moderation, interfaces

      Notes:

      We want to allow users to interact with the system without having to trust intermediaries who might stand in the middle between the users. If we can do that, it opens the door to this new application stack where we disentangle all the layers.

      OKAY - so we're at the halfway point within this lecture, let's check in. At this point, we've been through the history of the web and hopefully understand the evolution from simple peer to pee communications to straight up digital services.

      Adding to that, from my from first lecture, we ALSO understand that both software and contracts are solutions to expressing agreements, and during Aaron's activity, we practiced building a simple web2 style smart contract.

      Now I want to start formalizing the semantics of how we actually build a smart contract.

      It turns out there is a mathematical (or computer science maybe?) term called a state machine. And it maps really well to what we just built and the kinds of interactions we want to build.


      State Machines

      Notes:

      What's interesting is that most systems, maybe all of them, but definitely most that we care about can be modeled as state machines. So State machine, it's not a real machine that you can build and touch and operate and stuff like that. It's a model comprised of a set of states and a set of rules about how to transition between the states. Looking at the diagram, maybe my state machine starts in state one over here and then somebody pokes it in some particular way, or to use more blockchain-y lingo, they send a transaction and that puts the state machine into into State-two. And so, what we're going to be doing with the blockchain is tracking a complete history of all of these transitions and if we know the history of all the transitions, then we should know what state we're in right now

      ---v

      Labelled Transition Systems

      Sometimes you can map the entire state space as an LTS.

      Other times it is too big.

      Notes:

      There's this notion of a labelled transition system. Sometimes if your state machine is simple enough you can actually map the entire State space as a labeled transition system. Most of the times when we're building real world applications, it's a little too big. But labeled transition systems are a really good model to keep in your mind when you're thinking about designing your state machines. Consider if we tried to map all possible states of a social media app or a digital currency. Sometimes an LTS drawing like this is useful, other times it would be too large or even infinite. Even still, sometimes drawing part of it can help you think about what the states and transitions might be.

      ---v

      Example: Light Switch

      Simple Switch: 2 States, 1 Transition

      Labelled Transition System

      History

      Notes:

      Let's look at a simple example. A light switch. CLICK It has two states: on and off. And one transition: flipping the switch. We can draw this as a labeled transition system. CLICK And we can also draw a history of the transitions that have happened. So if we know the history of all the transitions, then we should know what state we're in right now. So we start in some State down here for example where the light is on and then when a transition comes in it's we already know what kind of transition it's going to be. There's only one - it's the toggle transition that turns the light off and Etc and Etc and this could go on forever. Right now this is such a simple State machine that like this isn't going to be a very interesting history it's just going to be toggle toggle toggle toggle forever. But with that information, if if for some reason knowing whether this light was on or off is like super important to your Society or your community, you've now achieved it - or at least we know a way to start achieving it.

      ---v

      State Machine Example: Digital Cash

      Each state is a set of bank notes. Where a bank note has an amount and an owner. A transition involves a user consuming (spending) some bank notes and creating new ones.

      Notes:

      Let's look at a more interesting example. Digital cash. This is one where I can't draw the entire label transition system because it's way too huge. But I can draw a little bit of it. Each state is a set of bank notes. Where a bank note has an amount and an owner. A transition involves a user consuming (spending) some bank notes and creating new ones. Not all conceivable transitions are valid. Imagine a user consuming a bank note worth 5 coins, and creating two new ones each worth 3 coins. That's not allowed. So we have to be careful about what transitions are allowed and what transitions are not allowed. And again, if we know the history of all the transitions, then we should know what state we're in right now.

      ---v

      Sate Machine Example: Social Media

      Each state is a set of posts and their associated comments and emoji reaction counts. A transition involves, making a new post, or reacting to someone elses, or commenting

      Notes:

      state is a set of posts and their associated comments and emoji reaction counts. A transition involves, making a new post, or reacting to someone elses, or commenting There is not a single model here. Some state machines will allow deleting or editing posts, while others will not. Some will allow disliking posts while others only allow liking. Here the diagram basically we see - if the state starts out simple with this one Post in it and then someone makes a new post (and there would have to be more details in here about who's posting and what they're posting)But then we get to this new state and then maybe the next transition is that somebody likes post number one and so now the state contains a record that this thing is liked. Just to drive the point home the idea is that if you know the starting State and you know the history of the transitions, then you know the the current state and so you can sit and watch as every transition comes in you can validate them yourselves and say this one's good we're adding it to our history or this one's bad we're chucking it's invalid transition. And then therefore you can know the final state

      ---v

      More State Machine Examples:

      • Ride sharing platform
      • Voting system
      • Blackjack table
      • The app you just built

      Notes:

      There are all kinds of these - uber, polkadot's governance, and gambling! Take a moment to brainstorm with your table about each of these three examples as well as the app you built in the previous activity. For each example, write out what are the state transitions and the the states. If possible, draw a complete LTS. If that is not possible, draw a partial LTS.

      ---v

      Garage Door Lock

      Notes:

      One more team exercise. Draw a state transition diagram for a garage door lock keypad.

      If the user enters the correct 4 digits, the door opens (or closes if it was already open). The thing is, you could enter some random digits before you enter the correct code, and it would still open after the correct code. As you sketch your states, be sure to handle that correctly.

      This example is drawn from https://eng.libretexts.org/Under_Construction/Book%3A_Discrete_Structures/09%3A_Finite-State_Automata/9.02%3A_State_Transition_Diagrams which has many other excellent examples


      Build Some State Machines

      Blockchain From Scratch Activity

      Chapter 1 State Machines

      Notes:

      This activity is going to run over the next four days. We will have a good amount of time to work on it in class. We will also set some minimum completion goals each evening.

      Blockchain from Scratch

      Learn the fundamentals of blockchain by building it from scratch. In Rust.

      ๐Ÿ“ฅ Clone to start: Blockchain from Scratch

      See the README included in the repository for further instructions.

      Peer-to-Peer (P2P) Networking


      How to use the slides - Full screen (new tab)
      Slides Content
      --- title: Peer-to-Peer (P2P) Networking description: Peer-to-Peer (P2P) networking for web3 builders ---

      Peer-to-Peer Networking


      Introduction/Agenda

      • History of p2p networks
      • Discuss the network layer and network conditions that blockchains operate on(Mostly)
      • Talk about traditional web2 network overlays pros vs cons with web3 network overlays
      • Discuss attacks and how to address along with the underlying threat model

      ARPANET

      • First operational packet-switching network
      • Developed in the late 1960s by DARPA(The Defense Advanced Research Projects Agency)
      • Laid the foundation for the modern internet

      Notes:

      Total Information Awareness (TIA): In the early 2000s, DARPA initiated the TIA program aimed at developing technologies for mass surveillance and data analysis. The project raised concerns about privacy and civil liberties, eventually leading to its cancellation in 2003 due to public outcry.


      Packet Switching

      • Mode of data transmission in which a message is broken into a number of parts that are sent independently(Packets)
      • Packets are sent over whatever route is optimal
      • Packets are reassembled at the destination

      Packet Switching

      Notes:

      Mention that headers contain some addressing, destination information, and ordering typically depending


      Packet Switching


      Packet Switching


      Packet Switching


      Packet Switching


      Peer-to-Peer (P2P) Networks

      • P2P is a decentralized form of network structure
      • Unlike client-server model, all nodes (peers) are equal participants
      • Data is shared directly between systems without a central server
      • Peers contribute resources, including bandwidth, storage space, and processing power

      Historical P2P applications

      Notes:

      Napster, Limewire


      Napster

      • Launched in 1999, popular P2P platform
      • Central server for indexing, P2P for transfers
      • Shutdown in 2001 due to legal issues

      Notes:

      Napster's story is closely tied with the band Metallica. In 2000, Metallica discovered that a demo of their song "I Disappear" was being circulated via Napster before its official release. This led to Metallica filing a lawsuit against Napster for copyright infringement. Napster had to comply by banning hundreds of thousands of users from their platform who were sharing Metallica's music. This was a turning point in digital copyright law and played a significant role in Napster's eventual shutdown in 2001.


      Napster Setup


      Napster Setup


      Napster Setup


      Napster Setup


      Gnutella(Limewire)

      • Each node serves as both a client and a server no central server
      • Query all connected nodes for files
      • Gain peer connections to the network via Bootnodes
      • Ordered to shutdown in 2010 by United States Court

      Notes:

      • Check local filestore for file and if it is not available, forward the request to all connected peers.
      • Gnutella generates a significant amount of network traffic by flooding the network with requests.

      Client-Server vs Peer-to-Peer (P2P) Networks

      Client-Server Network P2P Network
      Structure Centralized: One or more central servers control the network Decentralized: All nodes (peers) participate equally
      Data Flow Server provides data to clients Peers directly share data with each other
      Resource Management Servers manage resources and control access Peers contribute resources including bandwidth, storage space, and processing power
      Scalability Can be limited by server capacity Highly scalable due to the distribution of resources
      Security Centralized security measures, single point of failure Potential for some security issues, malware(Depending on how it is implemented)

      Centralized vs Decentralized Networks

      Notes:

      Talk about how when a partition happens in P2P vs Centralized. In p2p, only one node needs to have a full copy in order for the file to be able to be distributed across the network.


      Centralized vs Decentralized Networks


      Centralized vs Decentralized Networks


      Centralized vs Decentralized Networks


      Advantages to Decentralized Networks

      • No privileged nodes
      • Less bottlenecks with bandwidth
      • DOS resistant
      • No centralized infrastructure necessary (Except internet for now...)

      Notes:

      1. No single node or nodes (CDN) have access to all of the content or files or is critical for operating the network. Each node has a copy of the data.
      2. No central node carrying all of the load of traffic. Block production and Block peering/importing can be mentioned here.
      3. Difficult to overload the network or DOS (Not a single node is privileged).
      4. Although many nodes are run on Centralized cloud compute platforms, they don't have to be (Typically).

      Difficulties or Disadvantages

      • Since it is permissionless, a node can share malicious resources
      • Latency
      • Difficult to regulate illicit activity
      • The network is limited by nodes with the weakest hardware

      Notes:

      1. Latency may be an issue if we need to wait for many peers to receive the data produced from a single node since everyone may not have a direct connection. Mention finality time!
      2. No central point to go and snoop all users data (for better or for worse).
      3. Why we have hardware requirements for blockchain networks.

      Gossip Protocol

      Notes:

      • Talk about how we have and want block 45 being peered to others

      ---v

      Gossip Protocol

      Notes:

      Talk about advertising vs just blind sending and how that can be inefficient


      Structured vs Unstructured P2P Networks

      Structured P2P Networks Unstructured P2P Networks
      Organization Nodes are organized following specific protocols and structures (like Distributed Hash Tables) Nodes are connected in an ad-hoc manner without any particular organization
      Search Efficiency Efficient search operations due to structured nature Search operations may be less efficient and can involve flooding the network
      Flexibility Less flexible as changes in topology require restructuring Highly flexible as nodes can freely join, leave, and reorganize
      Privacy Data location is predictable due to structured organization Greater potential for anonymity

      Discovery

      1. Connect to a peer
      2. Ask peer for a list of their known nodes
      3. Connect to random subset of peers from the list
      4. Repeat steps 2 and 3

      Applications

      Notes:

      1. What are some of the types of applications that lend themselves to this kind of network topology? Can anyone think of any?
      2. File sharing(Music)?
      3. Messaging and communication?

      Initial Discovery

      • Bootnode/bootnodes (More on this later in Substrate)

      Notes:

      1. Must know someone who is participating in the network initially(Bootnode)

      Attacks

      Notes:

      • Can anyone think of a way to exploit some of these networks?
      • What would be some things to try to take advantage of?

      Attacks

      Notes:

      1. Distorts view of the healthy normal honest state of the network
      2. Transaction confirmations can be fictions

      Attacks


      Eclipse Attack Execution

      1. Flood a target node with a bunch of malicious peer addresses
      2. The targeted node then stores these malicious peers and utilizes them when re-syncing on next bootup
      3. DOS targeted node to take it offline to force a resync with these new malicious peers

      Preventing Attacks

      • Restrict inbound connections in some way
      • Random selection of peers to connect with
      • Deterministic node selection (Bootnodes)
      • Restricting new nodes (Probably not what we want...)

      Notes:

      1. Be wary of new connections with other nodes
      2. Don't just take the most recent request for connections to avoid the flooding
      3. Bootnodes with higher credibility and trust (Can be a bottleneck) - Rotate bootnodes as they are also subject to attacks

      Conclusion

      P2P networks offer us a path forward towards applications which are more decentralized and censorship resilient

      Blockchain from Scratch

      Learn the fundamentals of blockchain by building it from scratch. In Rust.

      ๐Ÿ“ฅ Clone to start: Blockchain from Scratch

      See the README included in the repository for further instructions.

      Platform Agnostic Bytecode


      How to use the slides - Full screen (new tab)
      Slides Content
      --- title: Platform Agnostic Bytecode description: What are PABs and why they exist? duration: 1 hour ---

      Platform Agnostic Bytecode


      Review of Compilers

      ๐Ÿคฏ Fun Side Reading: Reflections on Trusting Trust

      Notes:

      Just a very quick reminder of how compilers work. Humans write programs in some human readable language like Lauren talked about. Then the compiler translates the semantics of that program into an equivalent program in a much lower more machine-readable language called a bytecode.

      CLICK

      Whenever I show this diagram or talk about compilers, I always like to mention one of my favorite essays ever. Ken Thompson's 1984 Turing Award lecture.


      Definition

      A PAB is a bytecode that follows two main principles:

      • Turing Completeness, as a standard bytecode would respect
      • Support for tooling that makes it executable on every machine

      Notes:

      Ideally a bytecode like this is designed to be executed on a virtual machine that follows general known patterns.


      High Level Languages
      PABs
      Architecture's bytecode

      Notes:

      From left to right you can see different levels of abstraction over the program that will ultimately be run on some machine. Generally, from a high level language you need two compilation step if you want to pass through a PAB.

      Other examples of PABs used right now:

      • Inside the Linux Kernel -> eBPF
      • Inside browsers -> Wasm
      • Inside Blockchains -> Wasm
        • Full nodes
        • Light nodes (Wasm inside Wasm)
      • LLVM Toolchain -> LLVM IR

      ---v

      Compiling in a PAB

      Notes:

      So when we are using a PAB, we need to compile twice. This is, of course, the cost to using a PAB. In this lesson we'll also explore the advantages.


      What a PAB allows is:

      • Portability
        - Avoid Hardware Centralization
        
      • Determinism
        - Make consensus possible
        

      Notes:

      The main goal of a PAB is to make the code portable, you should be able to compile it once and then share it around without caring about the architecture on which will be executed. Of course in a decentralized network we want that different nodes, with different architectures came up to the same result if the input are the same, that's called determinism, if a PAB would not have determinism then reaching consensus is impossible.

      ---v

      That's why PABs are so important

      Desireable Features

      • Hardware Independence
      • Efficiency
      • Tool Simplicity
      • Support as Compilation Target
      • Sandboxing

      Notes:

      • Hardware Independence: It should not be tightly related to a specific architecture, otherwise the execution on different machine could be convoluted
      • Efficiency: the execution of a PAB should be efficient, the problem for a PAB is that in the execution time is also considered the "translation" to the machine's bytecode or the interpretation
      • Support as Compilation Target: The PAB should be possible to be compiled by as many as possible High Level languages
      • Tool Simplicity: If the tools that makes the PAB executable are extremely complex then nobody will use it

      ---v

      Sandboxing?

      An environment for running untrusted code without affecting the host.

      A SmartContract is Arbitrary Code that may be executed on other people's infrastructure, we don't want SmartContracts capable of destroying the nodes on which they are executed

      Notes:

      CLICK read definition

      The term sandbox is an analogy to kids playing in a sandbox. The parent puts the kid in the sandbox and tells them they can play in the sandbox and they are safe as long as they stay in. Don't go in the woods and get bitten by a snake or in the road and get hit by a car. Just stay in the sandbox.

      Of course the analogy isn't perfect. The children in the sandbox stay there because the parent asked them to. They could leave anytime they wanted to. For actual untrusted code, a better analogy would be a walled garden or a Jail

      ---v

      Sandboxing?

      A sandboxed environment must be created by the executor of the PAB.

      Notes:

      Of course the security can be seen by various point of view and some examples are:

      • Compilation takes too much time -> compiling bomb
      • Access to the environment -> "buffer overflow" techniques

      Those things can't be addressed by the PAB itself but they can give good guidelines and code design to make an 100% secure implementation of the executor possible.


      PAB's lifecycle example



      WebAssembly


      Wasm's key points

      • Hardware-independent
        • Binary instruction format for a stack-based virtual machine
      • Supported as compilation target by many languages
        • Rust, C, C++ and many others
      • Fast (with near-native performance)
      • Safe (executed in a sandboxed environment)
      • Open (programs can interoperate with their environment)

      Notes:

      Wasm seems to respect every rating points we defined before


      Stack-Based Virtual Machine Example

      Adding two number in wasm text representation (.wat)

      (module
        (import "console" "log" (func $log (param i32)))
        (func $main
          ;; load `10` and `3` onto the stack
          i32.const 10
          i32.const 3
      
          i32.add ;; add up both numbers
          call $log ;; log the result
        )
        (start $main)
      )
      

      Notes:

      Wasm has also a text representation, Wat has some features that allow for better readability:

      • Stack push operations can be grouped to its consuming instruction.
      • Labels can be applied to elements.
      • Blocks can enclosed with parenthesis instead of explicit start/end instructions.

      Instructions push results to the stack and use values on the stack as arguments, the compilation process generally translate this stack-based bytecode to register based, where registers are used to pass values to instructions as a primary mechanism. The compilation will try to elide the wasm stack and work with only the architecture registers.

      There is another type of stack used in wasm and that's called: shadow stack, resource to learn more: https://hackmd.io/RNp7oBzKQmmaGvssJDHxrw


      Wasm seems to be a perfect PAB, but

      • How does communication with the environment work?
      • How the memory is managed?
      • How is it executed?

      Notes:

      Assuming all the things we said before wasm seems to be perfect but how those things really works?


      Communication with the Environment

      Let's call Embedder the program that will take the wasm blob as input and execute it

      • the wasm blob may expect parameters from the embedder
        • embedder -> wasm
      • the embedder may act on a return value from the wasm
        • wasm -> embedder

      ---v

      Problem

      Wasm has no ambient access to the computing environment in which code is executed


      Solution

      Notes:

      • Every interaction with the environment can be done only by a set of functions, called Host Functions, provided by the embedder and imported in wasm
      • The embedder is able to call the functions defined in wasm blob, called Runtime API, and pass arguments through a shared memory

      Memory

      In addition to the stack Wasm has also access to memory provided by the embedder, the Linear Memory.


      • This area will be used also used as a frontier for data sharing
      • To make everything secure the Embedder is doing incredibly convoluted things

      Notes:

      From Wasm the Linear Memory is byte addressable Linear Memory can be manipulated using functions called 'store' and 'load'

      The Rust compiler uses for dynamic/heap memory and to pass non primitives values to functions by emulating an additional stack within the linear memory, this emulated stack (the shadow stack) is what we would understand as stack in other architectures

      ---v

      Example

      Notes:

      Here's an example, wasm sees linear memory like a byte array and if it tries to access the second byte, it would use an index 1. When it's time to execute it the embedder will see this access and translate the linear memory access at index 1 to a standard memory access to base_linear_memory + 1.

      Buffer overflow? Wasm uses 32 bit, this makes impossible to have an offset bigger then 4GiB, this means that the embedder can leave those 4GiB free in its virtual memory to makes impossible to the wasm blob to access any environment information. Even if the offset is only positive there are embedded that are defining as protected the 2GiB before the BLM so that if for some reason the wasm code trick the embedder to treat the offset as a signed number that would cause an Operating System error.


      How Wasm is executed

      There are multiple ways to execute wasm:

      • Ahead Of Time Compilation
      • Just in Time Compilation
      • Single Pass Compilation
      • Interpretation
      • ...

      Notes:

      AOT: Compile all the code at the beginning, this allows to makes a lot of improvement to the final code efficiency JIT: The code is compiled only when needed, examples are functions that are compiled only when called, this leave space only to partials improvements SPC: This is a specific technique of compilation that is made in linear time, the compilation is done only passing once on the code Interpretation: The wasm blob is treated as any other interpreted language and executed in a Virtual Machine

      ---v

      Wasmtime

      • It is a stand alone wasm environment
      • Wasmtime is built on the optimizing Cranelift code generator to quickly generate high-quality machine code either at runtime (JIT) or ahead-of-time (AOT)
      • It executes the compiled wasm blob in sandboxed environment while keeping everything extremely secure

      Notes:

      Cranelift is a fast, secure, relatively simple and innovative compiler backend. It takes an intermediate representation of a program generated by some frontend and compiles it to executable machine code

      ---v

      Wasm lifecycle in Wasmtime

      ---v

      Wasmi

      • It is a wasm environment with support for embedded environment such as WebAssembly itself
      • Focus on simple, correct and deterministic WebAssembly execution
      • The technique of execution is interpretation but:
        • The wasm code is transpiled to WasmI IR, another stack-based bytecode
        • The WasmI IR is then interpreted by a Virtual Machine

      Notes:

      proposal to switch from a stack based ir to registry based ir https://github.com/paritytech/wasmi/issues/361

      paper explaining the efficiency of translating wasm to registry based code https://www.intel.com/content/www/us/en/developer/articles/technical/webassembly-interpreter-design-wasm-micro-runtime.html

      Due to it's characteristics it is mainly used to execute SmartContracts on chain

      ---v

      Wasm lifecycle in Wasmi


      Alternatives

      ---v

      EVM

      • The Ethereum Virtual Machine executes a stack machine
        • Interesting: here the bytecode was create to be executed in a blockchain, so instructions are not hardware-dependent but there are instruction tightly related to Cryptography and others blockchain instructions

      ---v

      CosmWasm

      • Wasm is always used but with different tools
      • They use CosmWasm as Embedder and internally is used Wasmer, a Single Pass Compiler

      ---v

      Solana eBPF

      • eBPF is used as PAB, but intrinsically eBPF has a lot of restrictions
      • Solana forked the eBPF backend of LLVM to makes every program to be compiled in eBPF
      • The Embedder is rBFP, a virtual machine for eBPF programs

      Notes:

      https://forum.polkadot.network/t/ebpf-contracts-hackathon/1084

      ---v

      RISC-V ?!

      • RISC-V is a new instruction-set architecture
      • main goals are:
        • real ISA suitable for direct native hardware implementation
        • avoids โ€œover-architectingโ€

      Being so simple and "Hardware-Independent" there are work in progress experiments to test if it is suitable to become the new polkadot smart contract language

      Notes:

      Discussion about using RISC-V as smart contract language: https://forum.polkadot.network/t/exploring-alternatives-to-wasm-for-smart-contracts/2434

      RISC-V Instruction Set Manual, Unprivileged ISA: https://github.com/riscv/riscv-isa-manual/releases/download/Ratified-IMAFDQC/riscv-spec-20191213.pdf


      Activity: Compiling Rust to Wasm

      • Let's make a simple Rust crate that compiles to Wasm!
      • Clone the repo

      ---v

      Activity: Compiling Rust to Wasm

      • A target triple consists of three strings separated by a hyphen, with a possible fourth string at the end preceded by a hyphen.
      • The first is the architecture, the second is the "vendor", the third is the OS type, and the optional fourth is environment type.
      • wasm32-unknown-emscripten: Legacy, provides some kind of std-like environment
      • wasm32-unknown-unknown โœ“ WebAssembly: Can compile anywhere, can run anywhere, no std
      • wasm32-wasi โœ“ WebAssembly with WASI

      ---v

      Rust -> Wasm Details

      #![allow(unused)]
      fn main() {
      #[no_mangle] // don't re-name symbols while linking
      pub extern "C" fn add_one() { // use C-style ABI
        ...
      }
      }

      and if a library:

      [lib]
      crate-type = ["cdylib"]
      

      ---v

      Activity: Compiling Rust to Wasm

      rustup target add wasm32-unknown-unknown
      
      cargo build --target wasm32-unknown-unknown --release
      
      wasmtime ./target/wasm32-unknown-unknown/release/wasm-crate.wasm --invoke <func_name> <arg1> <arg2> ...
      

      ---v

      Additional Resources! ๐Ÿ˜‹

      Check speaker notes (click "s" ๐Ÿ˜‰)

      Notes:

      Wasm Executor

      In this activity, you will practice compiling Rust code to a few different wasm targets. You will also learn to embed a web assembly executor in your own binary application. We will focus on Rust, but also briefly look at executing wasm from Python to demonstrate the platform agnostic nature of Wasm.

      ๐Ÿ“ฅ Clone to start: Wasm Executor Activity

      See the README included in the repository for further instructions.

      Blockchain Structure


      How to use the slides - Full screen (new tab)
      Slides Content
      --- title: Blockchain Structure description: The Blockchain data structure including hash-linking, forks, header vs body, and extrinsics. duration: 30 min ---

      Blockchain Structure


      Shared Story

      A Blockchain cryptographically guarantees that a history of events has not been tampered with. This allows interested parties to have a shared history.

      Notes:

      And it allows them to know whether they have identical histories in O(1) by just comparing the tip of the chain.


      Hash Linked List

      Notes:

      This is a simplified blockchain. Each block has a pointer to the parent block as well as a payload.

      ---v

      Hash Linked List

      Notes:

      The pointer is a cryptographic hash of the parent block. This ensures data integrity throughout the entire history of the chain. This is the simplest form that a blockchain could take and indeed it allows us to agree on a shared history.

      ---v

      Hash Linked List

      Notes:

      This ensures data integrity throughout the entire history of the chain. This is the simplest form that a blockchain could take and indeed it allows us to agree on a shared history.

      ---v

      Genesis Block

      Notes:

      The first block in the chain is typically called a the "Genesis block" named after the first book in the judaeo-christian mythology - The beginning of our shared story. The parent hash is chosen to be some specific value. Typically we use the all-zero hash, although any fixed widely agreed-upon value would also do.


      State Machines (Again)

      A state machine defines:

      • Set of valid states
      • Rules for transitioning between states

      ---v

      Blockchain meet State Machine

      Notes:

      The simplest way to join a blockchain to a state machine is to to make the blockchain's payload a state machine transition. By doing so, we effectively track the history of a state machine in a cryptographically guaranteed way.

      ---v

      Where do the States Live?

      Somewhere else!

      Notes:

      There is a state associated with each block. But typically the state is NOT stored in the block. This state information is redundant because it can always be obtained by just re-executing the history of the transitions. It is possible to store the state in the blocks, but the redundancy is undesirable. It wastes disk space for anyone who wants to store the history of the chain. Storing the state in the block is not done by any moderately popular blockchain today. If you want to store the states, you are welcome to do so. Software that does this is known as an Archive node or an indexer. But it is stored separately from the block ...Pause... One more time to make sure it sinks in: The state is NOT in the block.

      ---v

      State Roots

      A cryptographic anchor to the state

      Notes:

      Some data redundancy can be good to help avoid corruption etc. It is common for a block to contain a cryptographic fingerprint of the state. This is known as a state root. You think of it as a hash of the state. In practice, the state is typically built into a Merkle tree like structure and the tree root is included. Not all blockchains do this. Notably bitcoin doesn't. But most do. We'll go into details about exactly how this state root is calculated for Substrate in the next two modules, but for now we just consider the state root to be some kind of cryptographic fingerprint.


      Forks

      A state machine can have different possible histories. These are called forks.

      Notes:

      You can think of them like alternate realities. We need to decide which of the many possible forks is ultimately the "real" one. This is the core job of consensus and we will talk about it in two upcoming lessons in this module.

      ---v

      Invalid Transitions

      Notes:

      Before we even get to hardcore consensus, we can rule out some possibilities based on the state machine itself


      Realistic Blockchain Structure

      • Header: Summary of minimal important information about this block
      • Body: A batched list of state transitions

      Notes:

      The header is a minimal amount of information. In some ways it is like metadata. The body contains the real "payload". It is almost always a batch of state transitions. There are many name aliases for what is included in the body:

      • Transitions
      • Transactions
      • Extrinsics

      ---v

      Blocks in Substrate

      #![allow(unused)]
      fn main() {
      /// Abstraction over a Substrate block.
      pub struct Block<Header, Extrinsic: MaybeSerialize> {
      	/// The block header.
      	pub header: Header,
      	/// The accompanying extrinsics.
      	pub extrinsics: Vec<Extrinsic>,
      }
      }

      Notes:

      This example is from Substrate and as such it strives to be a general and flexible format, we will cover Substrate in more depth in the next module. This is representative of nearly all real-world blockchains


      Headers

      Exact content varies per blockchain. Always contains the parent hash. Headers are the actual hash-linked list, not entire blocks.

      Notes:

      The parent hash links blocks together (cryptographically linked list). The other info is handy for other infrastructure and applications (more on that later).

      ---v

      Header Examples

      Bitcoin

      • Version
      • Previous Hash
      • Tx Merkle Root
      • Time
      • N_Bits
      • Nonce

      Ethereum

      • Time
      • Block Number
      • Base Fee
      • Difficulty
      • Mix Hash
      • Parent Hash
      • State Root
      • Nonce

      ---v

      Substrate Header

      • Parent hash
      • Number
      • State root
      • Extrinsics root
      • Consensus Digest

      Notes:

      Extrinsics root is a crypto link to the body of the block. It is very similar to the state root. Consensus Digest is information necessary for the consensus algorithm to determine a block's validity. It varies widely with the consensus algorithm used and we will discuss it in two upcoming lectures.

      ---v

      Substrate Header (Full Picture)


      Extrinsics

      Packets from the outside world with zero or more signatures attached.

      • Function calls to the STF
      • Some functions require signatures (e.g., transfer some tokens)
      • Others don't, but usually have some validation means

      DAGS

      Directed Acyclic Graphs

      Notes:

      In math there is a notion of a Directed Acyclic Graph. Define graph, than directed, than acyclic. Blockchains are examples of DAGs. Actually blockchains are a specific kind of a DAG called a tree. Sometimes you will hear me talk about the "block tree" which really means all the histories of the chain.

      But there are more kinds of DAGs than just trees. Consider if someone authored a block that looks like this.

      CLICK

      ---v

      DAGS

      Directed Acyclic Graphs

      Notes:

      What if a block could have more than one parent!? It could allow parallelization and increased throughput! But it also leads to problems. What if there are conflicting transactions in the two parent histories? How do you even know if there are conflicting histories?


      Blockchain ๐Ÿ’’ P2P Networks

      Notes:

      So hopefully some parts of this figure look familiar. What do you see here?

      • Diverse servers.
      • In a p2p network.
      • Each with their own view of the blockchain.

      ---v

      Nodes

      Software agents that participate in blockchain network.
      May perform these jobs:

      • Gossip blocks
      • Execute and Validate blocks
      • Store blocks
      • Store states
      • Gossip transactions
      • Maintain a transaction pool
      • Author blocks
      • Store block headers
      • Answer user requests for data (RPC)
      Notes:

      Many nodes only perform a subset of these tasks

      ---v

      Types of Nodes

      • Full Nodes
      • Light Nodes (aka Light clients)
      • Authoring nodes
      • Archive nodes
      • RPC nodes

      Blockspace

      A resource created, and often sold, by a decentralized blockchain network.

      Learn more:

      Notes:

      A Blockchain network is a replacement for a centralized server. It sells a product to application deployers. The state machine is the application layer, and the blockchain is the server replacement. In the same way that applications pay data centers for server resources like cpu time, disk space, bandwidth etc. Applications (maybe via their developers or users) pay for the privilege of having their history attested to and their state tracked by a trustless unstoppable consensus layer.

      ---v

      Transaction Pool

      • Contains transactions that are not yet in blocks.
      • Constantly prioritizing and re-prioritizing transactions.
      • Operates as a blockspace market.

      Notes:

      Sometimes known as mempool (thanks bitcoin ๐Ÿ™„) Authoring nodes determine the order of upcoming transactions. In some sense they can see the future.

      Foreshadow forks where players disagree on the rules History: dao fork bch fork foreshadow consensus: arbitrary additional constraints for a block to be valid


      Let's #BUIDL It

      Blockchain from Scratch

      Learn the fundamentals of blockchain by building it from scratch. In Rust.

      ๐Ÿ“ฅ Clone to start: Blockchain from Scratch

      See the README included in the repository for further instructions.

      Consensus Authoring


      How to use the slides - Full screen (new tab)
      Slides Content
      --- title: Consensus Authoring description: Authoring restrictions related to consensus duration: 30min ---

      Consensus: Authoring

      ---v

      Consensus is...

      ...a decision making process that strives to achieve acceptance of a decision by all participants.

      ---v

      Blockchain Consensus is...

      ...a decentralized consensus system to reach agreement over a shared history of a state machine.

      ---v

      Blockspace

      Blockchain consensus systems produce a resource called blockspace.

      Strong incentive alignments and strong guarantees make for high quality blockspace.

      Notes:

      As we discussed blockspace represents the right to contribute to the shared history. This is a valuable resource that is offered to users as a product. We will discuss the selling of this resource in a later lecture on allocation and fees. The consensus system used plays a large role in determining the quality of the blockspace.


      Forks Review

      There are going to be forks. We need to decide which one is the real one.

      We can rule some forks out to reduce the problem space. Then we are left to decide which is canonical.

      Notes:

      Forks represent alternate courses that history could take. They arise every time there is a difference of opinion.

      You can think of them at a social level. Court cases, arguments, wars. Ideally we can resolve them peacefully

      You can think of them at a very low physics-y level. Every time an electron encounters a potential barrier it either reflects of tunnels. When consensus is high-quality, the result is as objective as the outcome of a physical process.


      Five Aspects of Consensus

      • State machine validity
      • Arbitrary / Political validity
      • Authorship throttling
      • Fork choice heuristic
      • Finality

      Notes:

      The first two aspects are relatively simple and I'll discuss them briefly right now. The third and fourth are the main topic of this lecture. The fifth is covered in detail two lectures from now in great detail.

      The first three aspects are about ruling possibilities out. The fourth and fifth are about deciding between any remaining possibilities.

      ---v

      State Machine Validity

      Some forks can be ruled out simply because they contain invalid state transitions.

      Notes:

      Example spending more money than you have. Noting your present location such that you would have traveled faster than speed of light since last record. Calling a smart contract with invalid parameters.

      ---v

      Arbitrary / Political Validity

      Similar to state machine validity.
      Examples:

      • Blocks that are too big
      • Blocks that have "hack" transactions
      • Empty blocks
      • Block with Even state roots

      Notes:

      This concept is similar to the previous slide. In some sense this is even the same. This allows us to rule out some forks just for not having properties that we like. Or for having properties that we dislike.

      Not everyone will agree on these properties ad that leads to long-term network splits.


      Authorship Throttling

      Real-world blockchains impose additional restrictions on who can author blocks. Why?

      Notes:

      These blockchains are supposed to be permissionless right? At least many of them are. Some are even very hardcore about that goal. So why would we want to restrict the authoring. Answer: So the nodes are not overwhelmed. Unthrottled authoring leads to fork chaos. If anyone authored at any moment there would be blocks raining down left and right. It would be impossible to check them all. It would be DOS central. So we need some organization / order to the process.

      ---v

      Leader Election

      We need to elect a small set (or ideally a single) entity who are allowed to author next.

      In pre-blockchain consensus this was called the "leader", and still often is.

      Notes:

      By electing a few leaders, we are able to throttle the authoring.

      ---v

      Liveness

      The ability of the system to keep authoring new blocks

      Notes:

      Before we go on, I want to introduce the concept of liveness. It is a desireable property that consensus systems want to have. Systems that have better liveness properties offer higher quality blockspace. Chains without liveness guarantees become stalled.


      Proof of Work

      Satoshi's Big invention.

      Solve a Pre-image search - earn the right to author.

      ---v

      Proof of Work: Pros

      • Permissionless (or so we thought)
      • Requires an external scarce resource: Energy
      • Blind: Nobody knows the next author until the moment they publish their block
      • Expensive to author competing forks - Clear incentive

      Notes:

      On the surface one big strength of PoW is that anyone can spin up a node and join at any time without anyone's permission. This is clearly how it was described in the whitepaper. In practice, many systems now have such a high hashrate that your home computer is useless. It is now permissioned by who can afford and acquire the right hardware.

      The reliance on an external resource is good in some sense because it is an objective measure of the market's valuation of the consensus system. This helps valuate the blockspace.

      The blindness is a good property because it makes it impossible to perform a targeted attack (DOS or physical) on the next leader to stall the network.

      Some attacks rely on the leader authoring two competing forks and gossiping them to different parts of the network. With PoW, it costs energy for every block you author. This makes it expensive to perform such attacks. This provides an economic incentive for authors to only author blocks on the "correct" fork.

      ---v

      Proof of Work: Cons

      • Energy Intensive
      • Irregular block time
      • Not so permissionless

      Notes:

      Energy consumption is more often considered a negative property. Sometimes called proof of waste. I won't go that far, but in a world where climate change is a reality, it is certainly not ideal to be spending so much energy if we can get away with far less.

      Worth noting that some PoW schemes (eg Monero's) strive to minimize the energy impact by choosing algorithms that are "asic resistant". While these are decidedly better than Bitcoin's, they do not fundamentally solve the problem. Just alleviate it somewhat in practice.

      Secondly, the block time is only probabilistically known. When waiting for block to be authored, there are sometimes spurts of blocks followed by long stretches without any.

      Although it seems permissionless on its face, in practice, to be a bitcoin miner you need to have expensive specialized hardware.

      ---v

      Why Author at All?

      • Altruism - You feel good about making the world a better place
      • Your Transitions - Because you want to get your own transitions in
      • Explicit incentives - Eg block reward

      Notes:

      If it costs energy to author blocks, why would anyone want to author to begin with?

      Mining only when you want to get your transaction in seems like a good idea to me. People who don't want to self author, can pay other a fee to do it for them. This is the purpose of transaction fees. Most chains have transaction fees specified in the transactions themselves which go to the author

      Some networks also add an explicit incentives such as a 50BTC reward per block.


      Proof of Authority

      Traditional class of solutions.

      Divide time into slots.

      Certain identities are allowed to author in each slot.

      Prove your identity with a signature.

      ---v

      Proof of Authority: Pros

      • Low energy consumption
      • Stable block time

      Notes:

      Stable block time is a property of high-quality block space. It allows applications that consume the blockspace to have expectations about throughput. In PoW you will occasionally have long periods without a block which can negatively affect applications.

      ---v

      Proof of Authority: Cons

      • Permissioned
      • No external resource to aid valuation
      • Incentives for honesty are not always clear

      Notes:

      Does anything bad happen if they misbehave? Not inherently. We will need an incentive for that.


      Some PoA Schemes

      Reminder: PoA is a family of leader election schemes

      ---v

      Aura

      The simple one.

      Everyone takes turns in order.

      #![allow(unused)]
      fn main() {
      authority(slot) = authorities[slot % authorities.len()];
      }

      Notes:

      Pros:

      • Simple
      • Single leader elected in each slot

      Cons:

      • Not blind - welcome targeted attacks

      ---v

      Babe

      Blind Assignment for Blockchain Extension

      • In each slot, compute a VRF output.
      • If it is below a threshold, you are eligible to author.
      • If eligible, author a block showing VRF proof
      • If NOT eligible, do nothing

      Notes:

      Pros:

      • No fixed order helps alleviate DOS attacks

      Cons:

      • Some slots have no authors - There is a workaround for this.
      • Other slots have multiple authors which leads to forks - There is no workaround for this.

      ---v

      Sassafras

      Single blind VRF-based leader election

      ๐Ÿ™ˆTBH, IDK how it works internally.

      But Jeff does!

      Notes:

      • Has most of the Pros of PoW (except for the external resource based valuation hint)
      • Has all the Pros of PoA

      ---v

      Sassafras Analogy

      Sassafras is kinda cards against humanity

      ---v

      Sassafras Analogy

      ---v

      Sassafras Analogy

      Ring VRF outputs are "cards". You anonymously "play" the best aka smallest cards in your hand.
      Those cards are sorted, not by funniness since they're just numbers, but by the number.
      The order in which they wind up is the block production order.
      You claim the ones that're yours by doing a non-ring VRF with identical outputs.

      ---v

      Proof of Stake

      It's just PoA in disguise ๐Ÿคฏ

      Uses an economic staking game to select the authorities.

      Restores the permissionlessness to at least PoW levels.

      Restores clear economic incentives

      Notes:

      There is an economic game called staking as part of the state machine that allows selecting the authorities who will participate in the PoA scheme. Actually there isn't just one way to do it, there are many. Kian will talk a lot more about this and about the way it is done in Polkadot later. I'll just give the flavor now.

      The basic idea is that anyone can lock up some tokens on chain (in the state machine). The participants with the most tokens staked are elected as the authorities. There is a state transition that allows reporting authority misbehavior (eg authoring competing blocks at the same height), and the authority loses their tokens. There are often block rewards too like PoW.

      ---v

      ๐Ÿ’’ Consensus ๐Ÿชข State Machine

      • Loose coupling between consensus and state machine is common
      • Eg Block rewards, slashing, authority election
      • In PoW there is a difficulty adjustment algorithm

      In Substrate there is a concept of a Runtime API - Consensus can read information from state machine.

      Notes:

      So far I've presented consensus as orthogonal to the state machine. This is mostly true. But in practice it is extremely common for there to be some loose coupling. We already saw an example when we talked about block rewards. The consensus authors are rewarded with tokens (in the state machine) for authoring blocks. Now we see that they can have tokens slashed (in state machine) for breaking consensus protocol. And we see that even the very authorities can be elected in the state machine.


      Fork Choice Heuristics

      Each node's preference for which fork is best

      • Longest chain rule
      • Most accumulated work
      • Most blocks authored by Alice
      • Most total transactions (or most gas)

      Notes:

      The fork choice allows you, as a network participant, to decide which fork you consider best for now. It is not binding. Your opinion can change as you see more blocks appear on the network

      ---v

      Reorganizations

      Dropped transactions re-enter tx pool and re-appear in new blocks shortly

      Notes:

      Having seen more blocks appear, we now have a different opinion about what chain is best. This is known as a reorg. Re-orgs are nearly inevitable. There area ways to make sure they don't happen at all, but there are significant costs to preventing them entirely. Typically short reorgs are not a big problem, but deep reorgs are.

      You can experience this in a social way too.

      • Imagine that you are waiting for a colleague to submit a paper. You believe they have submitted it yesterday, but it turns out that they didn't submit it until today or won't submit it until tomorrow. Might be annoying, but not world shattering (usually).
      • Imagine that you believe the colleague submitted the paper months ago and the paper has been published. You have applied on a job having listed the publication.

      ---v

      Double Spends

      Notes:

      The name comes from bitcoin, but the attack generalizes. It exploits the existence of forks. Attacker has to get two conflicting transactions into two forks. And convince a counterparty to believe one chain long enough to take an off-chain action before they see the reorg.

      ---v

      Double Spends


      Five Aspects of Consensus

      • State machine validity
      • Arbitrary / Political validity
      • Authorship throttling
      • Fork choice heuristic
      • Finality

      Notes:

      We just discussed the first four aspects. Finality will be discussed in an upcoming lesson

      Manual Consensus (aka BitStory)

      In this activity students will encounter and explore blockchain consensus mechanisms such as PoW, PoA, and PoS by acting them out manually.

      Students will collaborate in telling a shared story. The shared story is represented by a blockchain with one word in each block. To add a word to the story, a student must create a valid block, and draw that block on the community whiteboard*

      • If no whiteboard is available, you may be able to substitute; a paper, online drawing tool, or even a chalkboard if you are desperate.

      Proof of Work

      The PoW phase is the original, and a working hasher tool is available here: https://joshorndorff.github.io/BitStory/bitStoryLive.html

      To begin we'll explore the Nakamoto style Proof of Work. To add a block a student must:

      • Choose a parent block hash to build on
      • Choose a word to add to the story
      • Find a nonce so that the block's hash meets the required difficulty

      The process of finding a nonce is accomplished by entering the parent hash and word into the hasher and repeatedly mashing the increment nonce button until you find a valid one. In this way students are manually performing the preimage search.

      The game ends at a pre-determined time. The winner is the person who authors the most blocks in the longest chain at the end of the game.

      Attacks

      The instructor can encourage student groups to perform various attacks on the PoW network.

      • Censor specific words or plot points by forking around them.
      • Publish invaild blocks to trick other miners into wasting their hash rate on invalid chains.
      • Perform a hard fork where your block start to have two words each. Some students will like the two-words-per-block rule and keep building on them. Other students (ideally ones who don't know the instructor is in on the shenanigans) may think the two-word blocks are not following the rules, and choose to ignore them. Welcome to bitstory cash.

      Formalize the Language

      Our previous story had basically no structure. Let's change it to specifically allow a small set of "transaction types". From here on each block will contain a very short sentence (can be abbreviated on the whiteboard) The following is an example and your group is encouraged to make modifications if you like. The degree of formalization is up to you. The more formal it gets, the more it starts to look like real world blockchains.

      • Choose a fixed cast of max 3 or 4 characters
      • Choose a fixed set of a few locations
      • Allow one simple action in each block (a character moves, or does one simple thing)
      • Consider letting your characters affect your "real world" by letting them change the difficulty necessary for a block to be valid. You now have a difficulty adjustment algorithm.

      Play a few more games. This time each miner write down a few goals that they want to accomplish in-story (eg, bob and alice have a fight, charlie is elected mayor). This time the winners are whoever best accomplishes their in-story goals. Notice that in some cases the goals may al lbe mutually compatible, and in other cases they may not. In subsequent rounds, establish goals together. Try to play at least one round where there are incompatible goals. Miners may choose to form alliances (pools) and censor blocks that don't build on their allies' in-story goals.

      Proof of Authority (Aura)

      In this phase, students will choose some authorities to act as block authors and a slot duration and they will take turns authoring in order. They should observe that this method is much simpler and less chaotic. It does indeed take less work.

      Attacks

      • You can skip your slots to slow the progress of the story once it isn't going your way.
      • Censor plot points.
      • Publish blocks in slots in which they are not valid to try to trick other miners. This may be particularly convincing when the correct author is skipping their slot.
      • Polarize the authorities to work on two different forks

      Discussion

      What attacks worked best in PoW vs PoA. What strengths and weaknesses does each have.

      Proof of Stake

      One weakness of PoA is that only some of the participants are authors. We can improve this situation by allowing the authority set to rotate. Challenge the students to consider ways in which the authority set might change. What if the story itself is about who is elected as authorities?

      Notice that this is the foundation of proof of stake. The on-chain story is about people bonding and unbonding their security deposits. Formalize a new language around this story.

      Mixed PoW PoA

      Some early PoS designs envisioned interleaving PoS and PoW blocks. Invent such a scheme, and try it out.

      Grandpa and finality

      This one is a little different. Here we assume an external block production method. At first this can just be one student who is not playing grandpa, but instead being the sole block producer.

      The other students use a shared whiteboard to post their pre-votes and pre-commits and personal paper to track which blocks they have finalized.

      Hybrid Consensus

      If there is time in the activity slot, we can combine any of the block production methods with grandpa. My personal favorite is PoW + Grandpa.

      Econ & Game Theory in Blockchain


      How to use the slides - Full screen (new tab)
      Slides Content
      --- title: Econ & Game Theory in Blockchain description: Blockchain applications of Econ & Game Theory duration: 60 minutes ---

      Econ & Game Theory in Blockchain

      Notes:

      • Presenter Introduction
      • Timeline clarification (after mod 2 and 3)
      • Today we'll be going through a brand new lecture on Econ & Game Theory in Blockchain

      What's the goal?

      Notes:

      What's the goal of this lecture?

      ---v

      What's the goal?

      Econ Box Open
      • Demand & Supply
      • Markets
      • Nash Equilibrium
      • Schelling Point
      • ...

      Notes:

      By now you all should be familiar with all the terms and techniques from the Economy and Game Theory module like the Demand & Supply or Nash equilibrium.

      ---v

      What's the goal?

      Econ Box Blockchain Box Open
      • Consensus
      • Protocol
      • Tokens
      • State Transition Functions
      • ...

      Notes:

      And you also just finished covering the basics of Blockchains so you all should be familiar with state transition functions as well as have some understanding of consensus protocols.

      ---v

      What's the goal?

      Econ and Blockchain Handshake

      Notes:

      Having all that in mind, the goal of this lecture is to combine the two and see how the economy and game theory can be applied to blockchain. To do that we'll provide some exemplary use-cases and case studies.


      Landscape

      Notes:

      But first let's start with a quick summary of how we'll be approaching to bridge the gap between the two modules.

      ---v

      Landscape

      • Systems as Games
      Game Pad

      Notes:

      Firstly, we'll be looking at various blockchain systems as isolated games.

      ---v

      Landscape

      • Systems as Games
      • Nodes/Miners/Validators as Players
      Laptop user cheering

      Notes:

      Participants in those games will be for instance the miners, like in the Bitcoin protocol, maybe the validators in the Polkadot protocol or just some users of your smart contract.

      ---v

      Landscape

      • Systems as Games
      • Nodes/Miners/Validators as Players
      • Protocols as Game Rules
      Checklist with points

      Notes:

      The protocol is essentially a set of rules that will define the game itself. The one we will be analyzing.

      ---v

      Landscape

      • Systems as Games
      • Nodes/Miners/Validators as Players
      • Protocols as Game Rules
      • Cryptocurrencies as Points
      BTC symbol

      Notes:

      And to properly analyze those games we need to have some value representation, which in our case will be usually cryptocurrencies.

      ---v

      Landscape

      • Systems as Games
      • Nodes/Miners/Validators as Players
      • Protocols as Game Rules
      • Cryptocurrencies as Points
      • Rewards & Punishments as Incentives
      Gavel Hammer

      Notes:

      And finally we'll be having various reward and punishment schemes that will shape the incentives of the players.

      Now that we have all the elements defining our blockchain games we can look for further parallels.

      ---v

      Market Emergence

      Notes:

      And first let's take a look at something that hopefully very familiar: Markets

      ---v

      Market Emergence

      "A market is a composition of systems, institutions, procedures, social relations or infrastructures whereby parties engage in exchange."

      Notes:

      So a market is a composition of systems where ppl exchange stuff. This is the simplest way of putting it.

      And Markets are the cornerstone of our economy and they are everywhere which also means they are thoroughly studied. Which is very fortunate for us.

      Now let's look at something that might also be pretty familiar to many...

      ---v

      Market Emergence

      Fee Market

      Chart of Bitcoin fees over time

      Notes:

      The fee market. That's a chart of the average transaction fees in the Bitcoin network over time. As you can see the price is constantly fluctuating.

      ---v

      Market Emergence

      Fee Market

      Fees/Tips
      • Users bid with the fees
      • Miners select highest fee transactions
      Scales Icon

      Notes:

      Let's dive a bit deeper. So the users are bidding with how much they are willing to pay and miners are choosing the most lucrative transactions. If miners already included all the best transactions and there are not many new ones coming in they need to start accepting lower fees, so the price drops. But wait...?

      ---v

      Market Emergence

      Fee Market

      What is the product? Blockspace in Polkadot

      Notes:

      What is the actual product being exchanged here? When you have the crude oil market the goods being exchanged are clearly defined. It's crude oil barrels. What is actually being traded in here?

      Anyone has an idea what it might be?

      ---v

      Market Emergence

      Fee Market

      What is the product?
      • What is being exchanged is the blockspace
      • Miners produce blockspace and effectively auction it to users
      Blockspace in Polkadot

      Notes:

      It's the BLOCKSPACE. Miners secure and produce blockspace and sell it in an auction to various users. Blockspace as a product is actually a pretty new way of thinking about blockchains so I highly suggest checking out the article linked in the notes.

      ---v

      Market Emergence

      Fee Market

      Market Equilibrium
      • The transactions that manage to get into the blocks have fees representative of the current market equilibrium
      Chart showing a market equilibrium

      Notes:

      The price of blockspace is determined by the market equilibrium of supply and demand. Demand force is the users willing to transact and supply is the miners producing the blockspace. Transactions that are actually included represent the current equilibrium.

      ---v

      Market Emergence

      Fee Market

      Supply & Demand
      • In extreme cases the demand can raise so much that the fees overwhelm the network
      Rising graph icon

      Notes:

      We all know that the transaction fees can sometimes get outrageous. In cases where many users want to transact the demand can skyrocket. And what we see is the price equilibrium following suit.

      ---v

      Market Emergence

      Fee Market

      Supply & Demand
      • In extreme cases the demand can raise so much that the fees overwhelm the network
      • Most economic markets can react to growing demand with increased supply, but that is not directly possible in most blockchains
      Stack of goods on a conveyor belt

      Notes:

      And now let's think of what normal markets do. If our demand and price go extremely high the suppliers would have an incentive to produce more. But in blockchains that's not directly possible. More miners often don't mean more blockspace but MORE SECURE blockspace. That means a better quality blockspace.

      So the supply is often fixed and the demand is fluctuating. This is often the cause for the very volatile fee market.

      ---v

      Market Emergence

      Fee Market

      Supply & Demand
      • In extreme cases the demand can raise so much that the fees overwhelm the network
      • Most economic markets can react to growing demand with increased supply, but that is not directly possible in most blockchains
      CryptoKitties Logo

      Notes:

      And according to that theory some large blockchains actually tried implementing variable size blocks to react to the demand, but it's a pretty complex task and a big tradeoff.

      Some of you maybe recognize this image as the CryptoKitties game on Ethereum. It's a game where you can buy and breed digital cats. It was so popular that it actually clogged the Ethereum network and made normal transactions nearly unfeasible. And maybe now you have some additional insights into what actually happened there.

      =5 sec pause=


      Nash Equilibrium

      Notes:

      Now let's look at something a bit different that allows us to predict what users would do... a Nash Equilibrium.

      ---v

      Nash Equilibrium

      • Strategy from which no player wants to deviate
      • But what's the strategy for BTC mining?

      Notes:

      Nash Equilibrium is strategy from which no player wants to deviate. Pretty simple but what's the strategy for BTC mining? How can we leverage that knowledge?

      ---v

      Nash Equilibrium

      Honest Mining Meme

      Notes:

      And more importantly... being honestly is the answer, right?

      ---v

      Nash Equilibrium

      Bitcoin Mining

      Bitcoin Logo

      Notes:

      Let's dive in and actually see what's the answer. What's the Nash Equilibrium of Bitcoin mining? Is it being honest or dishonest?

      ---v

      Nash Equilibrium

      Assumptions:
      • Only 2 miners
      • Block reward = 2
      • Difficulty scales with number of honest miners
      • Miners are rational actors
      • Dishonest miners do not cooperate
      Payoff Table

      Notes:

      Here we have a few assumptions, but the main ones are:

      • imagine we have only 2 miners
      • the block rewards is 2
      • miners can be honestly following the protocol or try to cheat and push invalid blocks (for instance one where they get 999 bitcoins)

      Let's figure out how many bitcoins each one of them will mine on average.

      ---v

      Nash Equilibrium

      Payoff Table

      Notes:

      If both miners are not mining honestly, none of them produce valid bitcoin blocks... then there are no ACTUAL bitcoins being mined. Remember that they don't cooperate but for instance try and push blocks with 9999 BTC in their account. So the reward is 0 for both.

      ---v

      Nash Equilibrium

      Payoff Table

      Notes:

      If both are working honestly and making valid blocks they have an equal chance to mine a block and reap the rewards. Rest of the nodes will accept those blocks but the two miners compete for the same resource so each one will get a single BTC on average (as the mining reward is 2).

      ---v

      Nash Equilibrium

      Payoff Table

      Notes:

      If one of them is dishonest the other one can still keep working honestly and reap greater rewards as there is no competition. In that case the honest party gains 2 BTC.

      Now that we know who earns how many bitcoins in each scenario we need to shift out attention to another important force. Things like Twitter, Facebook or Bitcoin are often deemed as valuable because of their Network Effect.

      ---v

      Nash Equilibrium


      Network Effect

      "The network effect is a business principle that illustrates the idea that when more people use a product or service, its value increases."

      Notes:

      Generally the more ppl use some platform or believe in something the more valuable it is. It's a pretty simple concept but it's very important in the blockchain space. Bitcoin is precious to many because many people believe in it and mine it and exchange it. In blockchains if more people use your chain it's more valuable.

      ---v

      Nash Equilibrium


      Network Effect


      Assumptions revisited:
      • Only 2 miners
      • Block reward = 2
      • Difficulty scales linearly with number of honest miners
      • Miners are rational actors
      • Dishonest miners do not cooperate
      • Token price scales quadratically with the number of honest miners
        • 1 honest miner -> 1$
        • 2 honest miners -> 4$

        Notes:

        Let's add this extra assumption into our model. If there are more miners securing the network the coins itself are more valuable. We will use a super simple model here where the price of a coin is a square of the number of miners. We will be investigating small systems so it is a good enough approximation.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        We can apply the changes by changing the evaluations of the miners coins.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        If both miners honestly secure the network the coin price is 4 and in case of single honest miner the price is 1.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        Now we can actually focus on finding the Nash Equilibrium so let's used the simplest method.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        Assuming Miner B is honest the Miner A needs to choose between a payoff of 4 or 0.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        He ofc chooses 4 as it's higher. We temporarily mark it with the red circle.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        Same reasoning for B being dishonest and A chooses 2 over 0.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        Now we reverse the assumptions and assume A is honest and then B chooses between 4 and 0.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        Similarly as before he should choose 4 as it's higher.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        And the last remaining options will result in the circled 2.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        Now the square with all the options circled is the Nash Equilibrium. In our case it seems to be being honest.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        Not a big surprise, and being dishonest lands you with zero coins so it was too be expected. But what if we change the assumptions a bit?

        ---v

        Nash Equilibrium


        What does it mean exactly to be dishonest?

        There are some mainstream rules (the protocol) and if an individual miner breaks them it seems like an isolated mistake or an attempt at cheating.


        If multiple miners break the protocol in the same way, it can be seen as a new protocol deviating from the main one.

        Notes:

        We were assuming that miners are either honest or dishonest, but what does being DISHONEST actually mean?

        =fragment=

        There are some mainstream rules (the protocol) and if an individual miner breaks them it seems like an isolated mistake or an attempt at cheating. This is what we were analyzing before.

        =fragment=

        But what if multiple miners break the protocol in the same way? It can be seen as a new protocol deviating from the main one. And that's what we'll be looking at next. The dishonest miners are cooperating.

        ---v

        Nash Equilibrium

        Assumptions:
        • Only 2 miners
        • Block reward = 2
        • Difficulty scales with number of honest miners
        • Token price scales quadratically wih the number of honest miners
        • Miners are rational actors
        • Decision between which protocol to follow
        Payoff Table

        Notes:

        Assumptions are pretty much the same but this time around the dishonest miners will cooperate. Effectively they will be following a different modified protocol.

        So we will no longer consider them dishonest but they simply follow a different set of rules. Now miners choose to mine for the bitcoin BTC protocol or the bitcoin cash BCH protocol.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        Let's quickly fill up this table with the same rules as before. Only this time of both miners follow BCH they ofc get the rewards in the BCH token.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        Just as before we take the extra step and apply the network effect. If both miners secure the network price is 4 and if only 1 the price is 1.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        Here we have all the prices adjusted.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        Now let's take a look at what happens if Miner B miners BTC. Miner A would prefer to also mine bitcoin. So they are both mining the same thing.

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        On the other hand if miner B mines BCH then it seems that Miner A prefers to mine BCH...

        ---v

        Nash Equilibrium

        Payoff Table

        Notes:

        And in fact what we have in here is two distinct Nash Equilibria!

        ---v

        Nash Equilibrium

        So is being honest the best strategy?



        Not always. If the majority of people are honest then honesty pays off. If the majority of people are dishonest in the same way then be dishonest with them.



        In fact it was proven that in PoW following the majority is the true Nash Equilibrium no matter what strategy/protocol they are using as long as it's consistent.

        Notes:

        So is being honest the best strategy?

        =fragment=

        Not always. If the majority of people are honest then honesty pays off. If the majority of people are dishonest in the same way then be dishonest with them.

        =fragment=

        In fact it was proven that in PoW following the majority is the true Nash Equilibrium no matter what strategy/protocol they are using as long as it's consistent. So Bitcoin mining is in fact a huge coordination game and this is why honesty AKA following the BTC protocol usually pays off.


        Schelling Point

        Notes:

        On that topic... What if we have multiple Nash Equilibria? Schelling point often comes to the rescue.

        ---v

        Schelling Point

        • A solution that people tend to choose (easiest to coordinate on)
        • Generally it is also a Nash Equilibrium
        • How in the world can it be useful in Blockchain?

        Notes:

        So first a Schelling point is a strategy people tend to choose. It is generally a Nash Equilibrium

        =fragment=

        But how in the world is it useful to us in Blockchain?

        ---v

        Schelling Point

        Detective game

        • Two partners in crime
        • Detective interrogates them individually
        Detective and 2 robbers

        Notes:

        But before we can fully answer that let's explore a different game that can show you how Schelling points work. And I promise it will be applicable to Blockchain.

        So the story goes like that. We have to bank robbers and a detective. The detective interrogates he robbers and tries to make them confess or catch them lying.

        ---v

        Schelling Point

        Detective game

        Interrogation Game Payoff Table

        Notes:

        If any of the robbers spill the beans and tell the truth they both loose the game and go to prison. If both of them lie they get away with it. Seems super simple and it should be pretty straightforward that they both simply need to lie.

        Seems like a safe strategy, right? But remember than both of them witnessed the same real events of the robbery...

        ---v

        Schelling Point

        Detective game

        Interrogation Game Large Payoff Table

        Notes:

        But there are multiple lies they can construct. If they have inconsistent stories the detective will catch them. So now they need to coordinate on a specific lie and have all the details exactly the same or they are screwed.

        Contrast it with how easy it would be to just tell the truth. They both witnessed the truth so talking about it in a consistent way is trivial.

        ---v

        Schelling Point

        Detective game

        Truthful answers are one of the easiest strategies coordinate on, so they are the Schelling Points.

        Notes:

        That's why we can say that Truthful answers are one of the easiest strategies coordinate on, so they are the Schelling Points. The truth itself is a Schelling point.

        And this concept is vital to something in blockchain called...

        ---v

        Schelling Point

        Oracles

        Oracles are blockchain entities that provide information from the outside world to the blockchain.

        Oracle getting information from real world

        Notes:

        Oracles. Firstly who in the room heard of oracles? Raise your hands. Oracles are super interesting because what they are trying to achieve is to provide information from the outside world to the blockchain. And that's a very very hard task.

        ---v

        Schelling Point

        Oracles

        Oracles are blockchain entities that provide information from the outside world to the blockchain.

        External Information Examples:
        • What's the temperature in Berkeley?
        • Who won the election?
        • What's the exchange rate of USD to BTC?

        Notes:

        Some examples of the information they can provide are for instance what's the current temperature in Berkeley? Who won the election? What's the exchange rate of USD to BTC? and may others.

        Let's actually see how they might work is a slightly simplified version with the temperate example.

        ---v

        Schelling Point

        Oracles



        Temperature Line

        Notes:

        So what's the temperature in Berkeley? Imagine you have a garden in Berkeley and you have on chain insurance that if the temperature is too high you get a payout. So you want to know the temperature in Berkeley.

        We know the answer lies somewhere on this axis. But what it is exactly?

        ---v

        Schelling Point

        Oracles



        Temperature Line with Some Measurements

        Notes:

        We can ask some users to submit what they think the temperature is. Some of them will check it themselves, some will use weather apps and some will just guess.

        ---v

        Schelling Point

        Oracles

        Temperature Line with a Cluster of Measurements

        Notes:

        What we are hoping for is that the cluster of votes you can see here will be around the actual temperature. The best approach would be to check the median. And why is that?

        ---v

        Schelling Point

        Oracles

        • Honest participants are naturally coordinated
        • Attackers could try to coordinate and lie
        Temperature Line with a Cluster of Measurements

        Notes:

        Honest voters are naturally coordinated. They will check the temperature and vote honestly all within some small margin of error.

        People that would lie to skew the results and would submit random values generally wouldn't cluster like the honest voters. To make a more dangerous attack they would need to strategically coordinate on a specific value and all lie about it. It's much harder to pull of than simply checking the temperature outside.

        Submitting the truth is the Schelling Point in here and it makes it easy to be honest.

        ---v

        Schelling Point

        Oracles

        What to do with attackers? Temperature Line with a Cluster of Measurements

        Notes:

        But what if there are some attackers? What can we do about them?

        ---v

        Schelling Point

        Oracles

        What to do with attackers?

        If they go unpunished they can repeat the attack until successful
        Temperature Line with a Cluster of Measurements

        Notes:

        If we never punish them they can repeat the attack until they are successful. And that's not good.

        ---v

        Schelling Point

        Oracles

        What to do with attackers?

        If they go unpunished they can repeat the attack until successful

        Or even worse, they can make a million fake identities and spam incorrect votes
        Temperature Line with a Cluster of Measurements

        Notes:

        Or even worse, they can make a million fake identities and spam incorrect votes. So we need to punish them.

        But this is no longer a problem of a Schelling Point. The Schelling point did its job already. What we are talking about right now are...


        Incentives

        Notes:

        Incentives. Incentives are the next big topic we will be exploring. And they are vital to the blockchain space.

        ---v

        Incentives

        "Something that encourages a person to do something."

        In our case we want to shape incentives that motivate the users to submit honest values.

        Notes:

        Incentives are things that encourage people to do something.

        =fragment=

        In our case we want to shape incentives that motivate the users to submit honest values. We need to build out incentives in a way that shapes the expected behavior of the users to honesty.

        ---v

        Incentives

        Oracles

        What to do with attackers?

        If they go unpunished they can repeat the attack until successful

        Or even worse, they can make a million fake identities and spam incorrect votes
        Temperature Line with a Cluster of Measurements

        Notes:

        So going back to our oracle problem how can we deal with attackers?

        ---v

        Incentives

        Oracles

        What to do with attackers?

        If they go unpunished they can repeat the attack until successful

        Or even worse, they can make a million fake identities and spam incorrect votes
        Mask

        Notes:

        Let's focus on the second issue of fake identities. How can we prevent that?

        ---v

        Incentives

        Oracles

        Sybil Attacks

        Common problem in blockchain.

        If deploying on a chain, an easy solution is to make users lock some funds.
        Mask

        Notes:

        An attack where a single entity creates multiple fake identities is called a sybil attack and is super common in blockchain. This is one of the things you always will need to ask yourself when you deploy things in blockchain. Is whatever I built safe from Sybil Attacks?

        =fragment=

        One easy ready solution that is often used is making users lock some funds. Only a user with some funds locked can vote. The strength of the vote is proportional to the stake so making a million accounts makes no sense. This is a very simple solution but it's not always applicable.

        ---v

        Incentives

        Oracles

        What to do with attackers?

        If they go unpunished they can repeat the attack until successful

        Or even worse, they can make a million fake identities and spam incorrect votes
        Gavel Hammer

        Notes:

        Now let's go back to the first issue of unpunished attackers. How can we deal with them so they don't continue attacking us?

        ---v

        Incentives

        Oracles

        Punishments

        We already laid out the foundation for punishments.

        Our solution for de-sybiling users makes them lock funds.

        If such a user would vote incorrectly, we can slash their funds.
        Gavel Hammer

        Notes:

        Interestingly we already laid out the foundation for the defense. Voters have some funds locked in the system so they have Skin in the game.

        =fragment=

        If they vote incorrectly we can slash their funds. This is a very common solution in blockchain. Incorrect in that case would be very far from the median.

        We have designed some protective incentives and now the system seems safe.

        ---v

        Incentives

        Oracles


        Did we forget about something?



        Why would anyone participate in this system?

        Notes:

        But did we forget about something? Anyone has any idea what it might be? What we are missing?

        =fragment=

        Why would anyone participate in this system? Why would anyone vote? Why would anyone lock their funds and take the risk?

        ---v

        Incentives

        Oracles

        Ghost Town

        No user wants to participate

        Getting information from the real world is an effort and they voters are doing the protocol a service
        Ghost

        Notes:

        Getting information from the real world is an effort and they voters are doing the protocol a service. So we need to incentivize them to participate. We need a rewards scheme or otherwise the network will be a ghost town.

        ---v

        Incentives

        Oracles

        Reward Scheme

        If users are doing the protocol a service they need to be rewarded

        One way to do it is to mint some token rewards for well-behaved voters

        Or distribute them from a previously acquired pool of rewards
        Hand receiving some money

        Notes:

        If users are doing the protocol a service they need to be rewarded.

        =fragment=

        One way to do it is to mint some token rewards for well-behaved voters.

        =fragment=

        Or distribute them from a previously acquired pool of rewards.

        But what is crucial here the protocol is safe and dependable only if there is enough voters so the incentives need to be designed in a way that encourages participation.

        More precisely, incentives need to be roughly proportional to the value an attacker could gain by compromising the system. Low-stakes oracles don't need to have super aggressive incentives.

        ---v

        Incentives

        Oracles

        Reward Scheme Question

        Can we distribute fixed value rewards for correct votes?
        Correct vote = 10$


        No. We should base rewards on the size of the voter's stake.
        Raised Hands

        Notes:

        Let's do a quick question. Can we distribute fixed value rewards for correct votes? As an example each Correct vote = 10$

        =question time=

        =fragment=

        No. We should base rewards on the size of the voter's stake. Otherwise the system is vulnerable to sybil attacks. If you have a million fake identities you can vote a million times and get a million times the reward. So the reward should be proportional to the stake.

        ---v

        Incentives

        In summary:

        • Make it easy to honest users and hard for attackers
        • Service to the protocol needs to be rewarded
        • Destructive or interfering actions need to be punished
        • De-Sybiling the users can help defend against spam

        Notes:

        Now let's summarize the main points.

        We need to make it easy for honest nodes and hard for attackers. The Schelling Point as the foundational part of the design handles that for us.

        =fragment=

        Service to the protocol needs to be rewarded. We need to incentivize participation to guarantee reliable results.

        =fragment=

        Destructive or interfering actions need to be punished. We need to disincentivize bad behavior. In our case we did the slashes.

        =fragment=

        De-Sybiling the users can help defend against spam.

        We we have all of that our systems should be properly incentivized and safe... and on that note what happens when the incentives are...


        Misaligned Incentives

        Notes:

        Misaligned. When they promote some behavior that is not good for the network.

        ---v

        Misaligned Incentives

        Ethereum State Storage Issue

        Ethereum Logo

        Notes:

        Let's take a look at Ethereum. Ethereum is a blockchain that has a lot of smart contracts. And smart contracts are basically programs that run on the blockchain. They are stored on chain and they can be executed by anyone. For them to work a bunch of code needs to be deployed on the chain.

        ---v

        Misaligned Incentives

        Ethereum State Storage Issue


        State Storage Replication

        Whenever we store something on chain (like a smart contract) it needs to be at least partially replicated among the nodes.

        Multiple nodes store the same data.
        Data Replication Diagram

        Notes:

        And moreover whenever we store something on chain (like a smart contract) it needs to be at least partially replicated among the nodes. Thousands of nodes store the same data which is not super efficient.

        ---v

        Misaligned Incentives

        Ethereum State Storage Issue


        State Storage Replication Costs

        Ethereum deals with the burden of replication by charging more gas for submitting bulky data.

        All of that is ON TOP OF any computation gas costs.
        Stacks of coins next to data blobs

        Notes:

        Ethereum attempts to deal with it by introducing scaling fees. The more data you put in the state the more you need to pay. And that's on top of any computation costs.

        ---v

        Misaligned Incentives

        Ethereum State Storage Issue


        State Storage Duration

        This particular part of the state might be relevant for future state transitions so nodes cannot simply discard it.

        Full nodes need to keep ALL the data.
        Data same over time

        Notes:

        Note that once we put something in state it has to stay there pretty much indefinitely until we use it again. Because who knows, it might be relevant to some future state transitions. So nodes cannot simply discard it.

        Now let's explore an example.

        ---v

        Misaligned Incentives

        Ethereum State Storage Issue


        Meet Bob

        Bob happily deploys his awesome smart contract in Ethereum. He paid a hefty gas fee but so be it.

        Deploying a smart contract on chain

        Notes:

        So let's meet Bob. Bob is a developer and he happily deploys his awesome smart contract in Ethereum. He paid a hefty gas fee but so be it. His code was added to the state and now many nodes hold a copy of it.

        ---v

        Misaligned Incentives

        Ethereum State Storage Issue


        The Problem

        Bob decided to become a musician or just no longer likes programming.

        He doesn't care about his smart contract anymore.
        Smart contract deployed on chain

        Notes:

        But imagine that one day Bob decides to become a musician or he just no longer likes programming. He doesn't care about his smart contract anymore. But the chain doesn't know about it. His code still lives in the state and has to be continuously replicated and maintained.

        ---v

        Misaligned Incentives

        Ethereum State Storage Issue


        The Problem Made Worse

        Many others like Bob follow suit.

        Some of them continue developing but, why bother removing old data? They already paid for it.
        Many smart contracts deployed on chain

        Notes:

        Now imagine there are hundreds of people like Bob. Some of them even continue developing but, WHY BOTHER removing old data? They already paid for it. And some of them just don't care anymore.

        ---v

        Misaligned Incentives

        Ethereum State Storage Issue


        "Why Bother?"

        Getting data on chain is expensive, but there is no incentive to clean it up.

        This is a core misalignment of incentives that lead to Ethereum state size growing out of control.
        Many smart contracts deployed on chain

        Notes:

        We need to focus on this "why bother" part. This is a core example of a misalignment of incentives that lead to Ethereum state size growing out of control.

        Getting the data to state is indeed expensive but once we do... why clean it? There is no incentive to do so. So the chain was getting overwhelmed in junk.

        ---v

        Misaligned Incentives

        Ethereum State Storage Issue


        The Goal

        Design new protocol rules that shape the behavior of the users in a way that they start cleaning up the state.
        Cleaned smart contracts on chain

        Notes:

        So what can we do about it? What's the goal? We need to design new protocol rules that shape the behavior of the users in a way that they start cleaning up the state. Hopefully without any side effects.

        ---v

        Misaligned Incentives

        Ethereum State Storage Issue


        The Solution

        State Storage Gas Refunds

        Pay a hefty fee when deploying data to state, but get some of it refunded when removing it.
        Burning a smart contract

        Notes:

        One of the proposed solutions was the introduction of a Gas Refund. You pay a hefty fee when deploying data to state, but get some of it refunded when removing it. So now there is an incentive to clean up the state.

        ---v

        Misaligned Incentives

        Ethereum State Storage Issue



        behavior Before

        Developer becoming a musician

        Notes:

        So what we had originally is Bob paid for his smart contract and then simply went away to play a guitar.

        ---v

        Misaligned Incentives

        Ethereum State Storage Issue



        behavior After

        Developer removing his contract and becoming a musician

        Notes:

        Afterwards Bob deploys his contract in the same way but before running of to play a guitar he removes it from the state and gets some of the gas back. He likes the extra money so he has an incentive to clean. In here we're presenting the version where he recovers the full value for educational purposes but Ethereum only refunds a portion of the gas.

        But wait... So what is the actual cost if he paid 10 and got 10 back? Anyone has any idea? Cost might not be obvious but it is an...


        Opportunity Cost

        Notes:

        Opportunity Cost. It's a very important concept in economics and it's also crucial in blockchain.

        ---v

        Opportunity Cost

        "The loss of other alternatives when one option is chosen."

        Multiple alternatives with opportunity cost

        Notes:

        d Generally opportunity cost is the loss of other alternatives when making a choice. When choosing between 10 and 30$ the opportunity cost of picking 30 is 10, the other option you are forgoing.

        ---v

        Opportunity Cost

        Ethereum State Storage


        The Real Cost

        Instead of having the funds locked in the storage deposit/refund scheme, Bob could have invested them somewhere else and gain some profit.

        Just locking your funds is sort of a punishment by itself.
        Locking Funds vs Investing

        Notes:

        Going back to the topic of Ethereum the real cost for Bob is not the 10$ he paid for storage as he regains it later. The cost is in losing the opportunity of investing the money elsewhere.

        =fragment=

        Just locking your funds is sort of a punishment by itself. Even if you regain them later. This is especially true in inflationary systems, and spoiler alert that most of them.

        The opportunity cost is a clever mechanism that allows us to include costs without directly charging them, and we also need to be super aware so we don't accidentally punish the users by not thinking of some external opportunity costs.

        ---v

        Opportunity Cost

        Extra Examples

        • Creating invalid blocks in Bitcoin never gets directly punished even if the block is rejected by the network. The real cost is the opportunity cost, as the miner could have mined a valid block instead.
        • Polkadot native token DOT is inflationary (~7.5% per year) but it can be staked to earn rewards (~15% per year). Not staking DOT has an opportunity cost which incentives staking to secure the network.

        Notes:

        There is a lot of awesome examples of opportunity costs in blockchain. For instance Creating invalid blocks in Bitcoin never gets directly punished even if the block is rejected by the network. The real cost is the opportunity cost, as the miner could have mined a valid block instead.

        =fragment=

        and the Polkadot native token DOT is inflationary (~7.5% per year) but it can be staked to earn rewards (~15% per year). Not staking DOT has an opportunity cost which incentives staking to secure the network.

        And there are also many other staking and DeFi examples out there.


        Externalities

        Notes:

        Now to actually appreciate what the we did in the previous section we need to talk about externalities.

        ---v

        Externalities

        "A consequence of an economic activity that is experienced by unrelated third parties."

        Notes:

        An externality is a consequence of an economic activity that is experienced by some third parties.

        As an example, think of the pollution you emit when driving a car. It's a negative externality that affects all the people around you. Alternatively imagine planting a tree in your garden simply because you like how it looks and gives you some shade. The tree improves the quality of air in the neighborhood and that's a positive externality for the people around you.

        ---v

        Externalities

        Ethereum State Storage

        The clogging of the chain with useless data is a negative externality that affects all the users of the chain.

        As protocol designers we need to be aware of such externalities and we can try and limit their effects by pricing them in.

        Notes:

        In the Ethereum example you could argue that the network getting clogged is the externality of the single developer not cleaning after himself. And it affects all the users of the chain. The chain is an example of a common good.

        As protocol engineers or system designers you need to identify those externality costs and price them in.

        ---v

        Externalities

        Ethereum State Storage


        Negative Externality Cost

        In the Ethereum State Storage problem we priced in the negative externality as the opportunity cost of locking your funds.
        Lucking Funds vs Investing

        Notes:

        That's what we did with the opportunity cost in Ethereum. We made it so burdening the chain is actually expensive for the perpetrator. We aligned his incentives with the incentives of the chain.

        ---v

        Externalities

        Oracles


        Positive Externality

        Providing the voting services in the Oracle scheme can be seen as a positive externality for the network that can further use this extra information.

        The voters are providing a valuable service to the protocol.
        Information from the external world entering Blockchain

        Notes:

        But not all externalities are negative.

        For example the whole oracle scheme makes it so the chain can get information from the real world. This is a positive externality for the network that can further use this extra information.

        ---v

        Externalities

        Oracles

        The voters are providing a valuable service to the protocol.

        So if they submit the vote on chain through a transaction, should they pay any fees?

        Notes:

        The honest voters are providing a valuable service to the protocol.

        =fragment=

        So having that mind should they pay any transaction fees when submitting votes?

        ---v

        Externalities

        Oracles


        Beneficial Transactions

        Such a transaction can be totally free.

        But make sure it cannot be spammed!
        Sticker with Free on it

        Notes:

        And contrary to the common belief such a transaction can be totally free.

        =fragment=

        But make sure it cannot be spammed! And it's super important, because if it is free it can be trivial. In our oracle system we can make sure there is only ONE vote per stake. This way we remain safe and can make the transaction free to further incentivize participation.

        ---v

        Free Transactions


        There are other free transactions that are not necessarily positive externalities.

        Inherent Transactions
        • Block rewards in BTC
        • Any logic that needs to be executed for every block (is inherent to the block)
        Sticker with Free on it

        Notes:

        There are other free transactions that are not necessarily positive externalities. For instance in Bitcoin after mining a block you get a reward. This is a free transaction that is not a positive externality. It's just inherent to the block. Usually any logic that needs to be executed for every block (is inherent to the block) is free.


        Complete vs Partial Information Games

        Notes:

        Now let's look at something totally different. We will talk about a concept crucial to game theory and that is information.

        ---v

        Complete vs Incomplete Information Games

        Do players know everything about the game state?

        Do players NEED to know everything about the game state?

        Notes:

        We'll be looking at questions like do players know everything about the game state? Do they NEED to know everything about the game state? And how does it affect the game?

        ---v

        Complete vs Incomplete Information Games

        Polkadot Approval Voting (Simplified)

        Five validators and three approval checkers among them

        Notes:

        To investigate this topic we'll dive deeper into Polkadot and particularly the Approval Voting subsystem. This is something I personally work on at Parity and you will learn a lot more about it in the later modules.

        What you need to understand now is that there are some special nodes called validators. They as they name suggests validate if the new blocks in the network are valid and correct.

        ---v

        Complete vs Incomplete Information Games

        Polkadot Approval Voting (Simplified)


        Approval Checkers

        In Polkadot when new blocks are validated, not everyone does the work. Only some randomly chosen validators - called Approval Checkers - are selected to validate candidate blocks.
        Five validators and three approval checkers among them

        Notes:

        But in Polkadot not every validator does all the work. They share the work and each block is checked only by a subset of validators. They are called Approval Checkers.

        ---v

        Complete vs Incomplete Information Games

        Polkadot Approval Voting (Simplified)


        Attackers

        We assume that attackers can DDoS some but not ALL Validators.

        Being DDoS'ed makes them unable to vote in time.
        Attackers eliminated some validators

        Notes:

        We also need to make some assumptions about the attackers willing to disrupt the network. We assume that attackers can DDoS some but not ALL Validators. Being DDoS'ed makes them unable to vote in time.

        ---v

        Complete vs Incomplete Information Games

        Polkadot Approval Voting

        Default Scenario

        • Randomly select 3 Approval Checkers and announce them
        • Approval Checkers publish votes if they are selected
        • If all the votes received confirm the block is fine it passes
        Five validators and three approval checkers among them

        Notes:

        A default validation scenario would go like that. We randomly select 3 Approval Checkers and announce them. Approval Checkers publish votes if they are selected. If all the votes received confirm the block is fine it passes.

        But there is a problem with it? Does anyone know what it is?

        ---v

        Complete vs Incomplete Information Games

        Polkadot Approval Voting

        Default Scenario

        • Randomly select 3 Approval Checkers and announce them
        • Approval Checkers publish votes if they are selected
        • Attackers use the information and DDoS the selected before they publish the vote (except their insider)
        • If all the votes received confirm the block is fine it passes
        Eliminated approval checkers

        Notes:

        If the attackers learn who the designated approval checkers are they can focus their attack. And only eliminate the relevant targets thus compromising the network.

        ---v

        Complete vs Incomplete Information Games

        Polkadot Approval Voting

        What was the Problem?

        • Attackers learned everything about the game
        • Attackers could use this information before the validators could respond

        How do we fix it?

        • Limit the information the attackers have access to so they cannot plan ahead

        Notes:

        So let's ask ourselves what was the problem? The attackers learned everything about the game and could use this information before the validators could respond. The information is the weapon.

        =fragment= If we could somehow limit the information the game would change in our favour. We need to make this a game of incomplete information, where some things are hidden.

        ---v

        Complete vs Incomplete Information Games

        Polkadot Approval Voting

        Improved Scenario

        • Each validator uses a VRF to generate a random number
        • Validators with a sufficiently small number will have the right to be Approval Checkers
        • Approval Checkers reveal themselves by showing their low numbers and publish a vote at the same time
        • If all the votes received confirm the block is fine it passes

        Notes:

        • Let's imagine this new improved scenario. Each validator uses a VRF to generate a random number.
        • There is some threshold and validators with a sufficiently small number will have the right to be Approval Checkers.

        =fragment=

        • Approval Checkers reveal themselves by showing their low numbers (with a vrf proof attached) and publish a vote at the same time. If all the votes received confirm the block is fine it passes.

        This method has this nice property that attackers don't learn who the approval checkers are until they reveal themselves. So they cannot plan ahead.

        ---v

        Complete vs Incomplete Information Games

        Polkadot Approval Voting

        What can the Attackers do?

        • They no longer know who the Approval Checkers are so they have to guess
        • If they don't guess correctly they get heavily slashed
        Some validators are eliminated

        Notes:

        So what can the attackers do? If they can no longer target DDoS they only can attack at random. This vastly reduces their chances of success and makes thm vulnerable to punishment if they fail.

        This is pretty much the method used in Polkadot and it's called a VRF based random assignment. It's a game of incomplete information.

        ---v

        Complete vs Incomplete Information Games

        Extra Examples

        • BABE is a Polkadot mechanism for selecting new block producers (further covered in Polkadot module). It also uses a similar VRF scheme to generate random assignments

        Notes:

        Some other notable examples you will learn about in later modules are connected to BABE the Polkadot mechanism for selecting new block producers. It also uses a similar VRF scheme to generate random assignments.


        Shifting Assumptions

        Notes:

        By now you might have noticed a pattern that usually we have a list of assumptions.

        ---v

        Shifting Assumptions

        Every game has some assumptions that need to be made before reasoning about them.
        • Number of players
        • Available actions
        • Access to information
        • etc
        Board with assumptions

        Notes:

        Some of the assumptions are connected to the number of players or maybe the access to information. And they are usually pretty stable.

        ---v

        Shifting Assumptions

        If any of the assumptions change the game changes as well. Old incentives that were previously sensible may now motivate a vastly different behavior. Some of the assumptions on the board change

        Notes:

        What happens when for some reason the assumptions evolve and change? The game changes as well. Old incentives that were previously sensible may now motivate a vastly different behavior.

        ---v

        Shifting Assumptions

        Restaking

        Restaking through EigenLayer incentives Ethereum stakers to stake the same funds for multiple apps at the same time.

        The incentive game will be vastly different and the capital will effectively be leveraged (double risk and double rewards).
        EigenLayer Logo

        Notes:

        A super good example of that is what is very recently happening in Ethereum. And I mean literally this month. There is a new layer 2 protocol called EigenLayer that allows stakers to stake the same funds for multiple apps at the same time. This is called restaking.

        This is not something natively available in Ethereum and it wasn't taken into consideration when slashes/rewards were designed.

        ---v

        Shifting Assumptions

        Restaking

        Consequences

        Restaking consequences are still not fully understood and the research is ongoing.

        Speaker Notes ("S") for further reading.
        EigenLayer Logo

        Notes:

        The consequences of restaking are still not fully understood and the research is ongoing. I encourage you to read the speaker notes for further reading. The whole field of blockchain incentivization and protocol design is still developing so there are many unknowns but overall I hope all the methods shown today help you in making more informed decisions later down the line. Thats it...


        Summary

        • Markets - Fee Markets
        • Nash Equilibrium - BTC Mining
        • Schelling Point - Oracles
        • Incentivization - Oracles
        • Opportunity cost - Ethereum State Storage Refunds
        • Externalities - Ethereum State Storage and Oracles
        • Complete vs Incomplete Information Games - Polkadot Approval Voting
        • Assumptions - Restaking

        Notes:

        So to summarize we talked about:


        Thanks everyone!

        Blockchain from Scratch

        Learn the fundamentals of blockchain by building it from scratch. In Rust.

        ๐Ÿ“ฅ Clone to start: Blockchain from Scratch

        See the README included in the repository for further instructions.

        Unstoppable Applications


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: Unstoppable Applications description: Unstoppable Applications in web3 duration: 1 hour ---

        Unstoppable Applications

        Notes:

        Much like tokenomic design, that is a large component in unstoppable apps that incorporate cryptocurrency or other motivating factors, this lesson is far far too short to give you all the tools and techniques to make a robust DApp design.

        Instead we strive to highlight the problem space we face and some classes of solutions to them.


        Motivation

        So far, we have discussed state machines and consensus... in isolation.

        Does the contexts in which they operate within matter?

        Notes:

        • So far mostly on simplified, idealized systems.
          • "Black boxes" of cryptography
          • Rational actors and assumed complete models of behavior in economics
          • Blockchains as an "isolated system" of sorts - external systems cannot be reasoned about in the same way... We will talk about the Oracle Problem.
        • In practice there are far more "unknown unknowns" and "black swan" behavior. More to come on that in this lesson.

        Discussion

        What properties of a system make it "stoppable"?

        Notes:

        • Web2 context: central providers & authorities, ...
        • Web3 context: decentralized, ...
        • What properties of a system make it "stoppable"?

        Unstoppable Apps Properties

        • Anitfragile
        • Trustless*
        • Censorship-less*
        • Accessible*
        • ...perhaps more?

        Notes:

        The "*" indicates the web3 context for defining properties, not generally. Not all of these can apply, nor is is possible all simultaneously apply. We need to craft the system properties based on what we must achieve. In reality we strive for Absolute Unstoppability, but likely cannot grantee it in every possible scenario.


        Anitfragile

        Some things benefit from shocks; they thrive and grow when exposed to volatility, randomness, disorder, and stressors and love adventure, risk, and uncertainty. Yet, in spite of the ubiquity of the phenomenon, there is no word for the exact opposite of fragile. Let us call it antifragile. Antifragility is beyond resilience or robustness. The resilient resists shocks and stays the same; the antifragile gets better.

        -- Antifragile --

        Notes:

        • Read Antifragile quote, recommend book recommended, see the links throughout slides for more after class.
        • Hydra fable & lore: https://en.wikipedia.org/wiki/Lernaean_Hydra - even though can be almost completely destroyed, it is resilient and recovers. Absolutely Unstoppable doesn't mean it cannot be damaged or even paused temporarily, it means it cannot cease to exist and will eventually recover, and ideally come back stronger in doing so.

        An N-lemma

        Hypothesis: a absolutely Unstoppable App cannot exist.

        We must make trade-offs out of all N properties
        that a absolutely Unstoppable App would possess.

        Notes:

        As with crypto, we can have astronomically good odds... But they are not perfect. We want the most robust system possible, given the environment and context the consensus system lives in.

        More relevant trilemma:


        Web3 Tech Stack

        Notes:

        This diagram is a bit dated with advancements in the field, but a good approx. representation.

        Observation and clarification: DApps canonically refer to smart contract applications. These exist within the context of consensus systems that themselves inherit properties of unstoppability from. The academy is more focused on consensus system engineering - we reason about blockchains themselves - rather than "DApp"s that use those as platforms to operate in or on. The Smart contracts lessons may include detains on unstoppable dapps design considerations.


        Much More Than Blockchain Architecture...

        Blockchains only form one part of the stack.

        Web3 applications must prevent attacks at all layers.

        • Networking
        • Consensus
        • Node access
        • Validator power
        • Inter-consensus trust
        • Human factors
        • Extrinsic factors

        Notes:

        These are for discussion today, but there are many more thank those listed here!


        Human Level


        Attacking Web3

        Notes:

        Key point: your "perfect" system in is likely weak to things outside of the "rules"! especially

        Image Source: https://xkcd.com/538/

        ---v

        Web3 Criticisms

        There are valid criticisms of how many Web3 apps operate today.

        • Humans are cheap & lazy...
          No individuals run servers.
        • RPC node providers
        • A protocol improves
          slowly vs. a platform.
        • False marketing,
          frauds, & scams!

        Notes:

        https://moxie.org/2022/01/07/web3-first-impressions.html great critique on the state of the space, but founder of Signal messenger.

        Not all hope is lost! This is valid mostly in the present, we will discuss these and what we're building to realize a better stack.


        Systems Level

        ---v

        Prove it!

        We use the word "proof" a lot...
        it means many things in different contexts:

        • Math โ†’ Provable Correct (algo)
        • Consensus โ†’ Proof of X (security)
        • Crypto โ†’ [ZK | VRF | Validity | ... ] Proofs

        Notes:

        The one so far not covered is Provable Correctness - where we can use maths to prove that our logic cannot do unexpected behavior. An interesting example is Cardano's design value proposition using haskell and provably correct most parts of their platform.

        We have a lesson and exercise on formal verification methods latter on - this is how we can approach Provable Correctness in the context of Rust and thus Substrate.

        BUT this property assumes a complete system model! Nuke proposes that when considering factors outside the consensus system, there cannot be a rigorous proof of correctness as we cannot model the universe.

        ---v

        ๐Ÿ”ฎ Oracle Problem

        An oracle provides eternal data to a consensus system.
        (i.e. a partial state of an external chain)

        The oracle problem relates to the trust in the oracle.

        Notes:

        • Example: Random Oracle, NOT like VRF we saw in the crypto module that can be in the consensus system.
        • Oracle needed for input from anything that lives outside of the boundary of the consensus system.
          • Everything in a chain is self-referential. Applications in a consensus system may want to try and reason about something outside itself.
        • Inclusive of bridges

        ---v

        ๐Ÿฆข Black Swans

        • known bounds of operation
          assumed impossible
        • death spirals

        Notes:

        Explain example of luna or other system collapse.

        ---v

        ๐Ÿคฏ Complexity

        • Illustrating how to map the intricacies of coupled, complicated, interactions of systems.
        • * You are not expected to understand this plot ๐Ÿ˜…

        Notes:

        Example: irrational actors can be represented in a very simple model as a completely random act, or opposite act of what a rational actor would do. If you "fuzz" you system you may discover fragility to irrational actions that could undermine your system. Perhaps it's far easier and more likely than it at first appears to experience a black swan event.

        • Image source - Describes the various categories of uncertainty, epistemology limits and statistical subjects touching on Taleb's Black swan / antifragility etc. ideas

        ---v

        ๐Ÿ‘ช Dependency

        Notes:

        • yes in software and hardware, you are at risk of attack from poisoned deps through non-maintenance, up to targeted exploitation. One mitigation is vendoring these, need systems inn place to monitor. Dependabot is not sufficient.
        • Also in dependance on specific operational contexts. For example that it is legal to operate the software for nodes.

        Image source: https://xkcd.com/2347/

        ---v

        ๐Ÿฆธ Dependency in Polkadot

        Foundational to Polkadot ecosystem!

        Notes:

        • Jaco is effectively the only maintainer of how just about everything communicates with Substrate nodes!
        • Capi is on the way, but just getting started.

        ---v

        ๐Ÿ™ˆ Unknown unknowns

        Notes:

        Outside of the system itself, we cannot guarantee/prove that every possible condition is accounted for in our models & system design. We must expect forces outside our system & it's model may interact in unexpected ways. Assumptions about context must be rigorously evaluated (i.e. - what does finality mean in the chain this pallet or contract lives in?) (Formal mathematical proofs reason only about the things we can and do account for.)


        Network Level

        ---v

        ๐Ÿ•ธ๏ธ Peer-to-Peer Networks

        ---v

        Network Attacks

        • Entry/Boot nodes and peer discovery
        • Data center faults
        • Traffic analysis and targeted takedowns
        • Eclipse attacks

        Notes:

        The network lesson covers these, just a reminder that the network is not in the direct command of the consensus system, so it's a threat!

        • security & stability
        • privacy! On chain might be ZK, but how about the gossip and RPCs?

        Boot nodes typically hard coded to "bootstrap" and start peer discovery. Boot nodes can decide what peers to advertize, or can be inaccessible. Common data centers (AWS, GCP, ...) could fail or censor, potentially large number of peers go dark. Hard to hide! Most p2p traffic is easy to identify vs. web2 traffic.

        ---v

        Node Queries

        Running a node is hard, most people outsource.

        These service have power to deceive, censor, and surveil.

        ---v

        Multi-Chain Applications

        If running one node is burdensome, try multiple.

        ---v

        Trustless Messaging

        In order to handle messages without trust,
        systems must share common finality guarantees.

        A should never process a message from B,
        where B is reverted and A is not.

        ---v

        A Note on Synchronicity

        Smart contracts on a single chain (e.g. Ethereum)
        can interact trustlessly because of their shared view of finality.

        Asynchronous systems can also share finality.
        i.e., be members of the same consensus system.

        ---v

        Discussion

        Minimum viable decentralization.

        What key aspects should be considered?

        Notes:

        • Quantitative: nodes needed (for what), incentives, ... FIXME TODO
        • Qualitative: social norms, ... FIXME TODO

        Consensus

        ---v

        Mining Pools

        Proof of Work authority sets have no finite bound.
        But people like to organize.

        [Collaborating | Colluding] authority sets creates risk.

        Notes:

        Call out that Nomination pools exist and are discussed in the NPoS lesson latter. Similar issues, but in a more bounded set.

        ---v

        Mining Pools

        Notes:

        Source: Buy Bitcoin Worldwide

        ---v

        Security Dilution

        Security is always a finite resource:

        • Centralized: Cost of corruption/influence
        • Proof of Work: Number of CPUs in the world
        • Proof of Stake: Value (by definition, finite)

        ---v

        Security Dilution

        Consensus systems compete for security,
        and they have reason to attack each other.

        Emergence of obscure/niche "Proof of X" algorithms
        to shelter from attack only goes so far.

        ---v

        โš” Blockchain Wars

        Systems with high security have the
        incentive to attack systems with low security
        whom they perceive as competitors.

        For fun and profit.

        Notes:

        "In a galaxy the universal consensus far far away not so far away..."

        ---v

        โš” Proof of Work Battles

        What might it cost to successfully attack?

        Notes:

        • For PoW, hashing power for the same algo can be attacked! Buying hash power is possible:
        • Most GPU miners switch tasks to the mine the highest paying (relative to some base currency) chain using software like https://www.nicehash.com/.
        • ASICs are less flexible, but also can to the highest paying coin.
        • Example: ETH classic deep re-ogs

        ---v

        Proof of...
        Nothing at Stake

        Forks are "free" to vote in favor of...
        vote on them all!

        (If you are not eventually slashed!)

        What might it cost to successfully attack?

        Notes:

        • Unlike PoW where voting on a chain costs something extrinsic to the system, PoS has only intrinsic measures to do accounting of consensus rules.
        • Critical: This was a problem with early naive implementations of PoS. Modern PoS schemes avoid this specific problem by having the security deposit and slashing for equivocation (in a few slides)
        • Good explainer, source of image: https://golden.com/wiki/Nothing-at-stake_problem-639PVZA

        ---v

        Proof of...
        Relatively Nothing at Stake

        Risk-to-reward ratio of attacks is
        relative to the valuation of the staked assets.

        Rational actors take into account
        extrinsic motivators in calculating the highest reward.

        What might it cost to successfully attack?

        Notes:

        • Again PoS ha only intrinsic measures to do accounting of consensus rules, but the system doesn't exist in a vacuum: the relative valuation of what is at stake needs to be accounted for.

        ---v

        Validator Consolidation

        How many validators does a system need?

        Higher numbers should lead to a decrease in the ability for entities to collude.

        But validators are expensive, both economically and computationally.

        Notes:

        Yet another N-lemma to consider.

        ---v

        PoS Economic Security

        Proposition: The upper bound of economic security in PoS is relative valuation can secure, that is correlated with the market capitalization of the network.

        Market capitalization refers to the total market value of all assets inherent to a single company/chain/token.

        Notes:

        • This market capitalization could be company shares, or total ETH in existence, or total X token associated with a specific smart contract or parachain.

        ---v

        โš” PoS Economic Security Battles

        Notes:

        Here like in PoW we have relative safety in networks, but there is no way to "hop" from one chain to another, so the war is still in the relative security, but one stake cannot directly attach another stake in a separate consensus system...

        What about an system of value within consensus?

        ---v

        DApp PoS Economic Security

        Notes:

        Consideration: these notes are an oversimplification! We may talk more about this kind of problem in NPoS lesson (Nuke thinks at least). The details of a formal analysis are out of scope for this Academy.

        Proposition: Total applications valuation of their assets (tokens on smart contracts, or parachains) is limited and that limit is correlated with the total economic security of the consensus system they reside in.

        In Polkadot's relay chain model, Nuke would argue it's feasible that an attack to extract value from a very highly valued asset could outweighs the cost of obtaining a byzantine level of stake to execute. Therefore the sum of all parachains market cap is also limited as that same level of stake control would enable take over of all chains on it.

        Nuke argue this is the same for the sum of all contracts valuations on something like Ethereum.

        ---v

        Authority Misbehavior

        • Equivocation
          • Authorship: Proposing mutually
            exclusive chains
          • Finality: Voting for mutually
            exclusive chains to be final
        • Invalidity
        • Lack of availability
        • Intentional protocol abuse (selfish mining)

        Notes:

        We already talked consensus faults, but abuse is newer. Nuke argues "abuse" as a term here isn't the intended mechanism design, and is adverse to the health of the system. Selfish mining where it's impossible to prove an author is withholding valid blocks to "cheat" by mining ahead of the rest of th network is a good example in the class of attacks that consensus authorities and others may have.

        ...Could other actors abuse the protocols?

        ---v

        Accountability of Authority

        Authority should imply accountability.

        No matter how you design an authority selection mechanism, some people will have a privileged position within it.

        Those who choose to become authorities should be liable for their actions.

        ---v

        Provability and Equivocation

        Some types of misbehavior are harder to prove than others.

        Equivocation is simple:
        Someone can just produce two signed messages as cryptographic proof.

        Others rely on challenge-response games and dispute resolution.

        Notes:

        Nothing at stake solution, with the possible caveat of long range attacks Weak subjectivity can still potentially cause the same behavior in a much harder to orchestra way, with bad actors having already have their stake released to produce a valid, finalized fork.

        ---v

        Design Considerations
        in Polkadot

        • More validators increases the state transition throughput of the network: parachains.
        • Individual shards have full economic freedom by being members of a larger consensus system.
        • Superlinear slashing puts colluding validators at existential risk,
          while well-meaning ones should have little to worry about).

        Notes:

        A few interesting design decisions in Polkadot's architecture.

        We will cover polkadot much more in depth latter!

        ---v

        Transaction Censorship and Ordering

        Block authors choose the transactions they include and in what order.

        • Censorship attacks
        • "Maximal extractable value" (MEV)

        ---v

        Web3 Goal: Non-Censorship

        There are a lot more system users than system authorities.

        However, every transaction must be included by an authority.

        If no authority will include a user's transaction, they do not have permissionless access.

        If any authority (author) decides not to censor, it may be included.

        Notes:

        Most present systems have no mechanism to penalize censorship, and a much harder problem can be the ability to discover this is happening on the network at all, depending on the actors involved.

        ---v

        Maximal Extractable Value (MEV)

        A measure of the value that block authors can extract based on their knowledge of pending transactions and ability to order them.

        • Frontrunning
        • Backrunning
        • Sandwiching

        https://www.mev.wiki/

        Notes:

        Emergent behavior. Not realized as possible by many until it quietly became the norm.

        ---v

        Maximal Extractable Value

        An environment in which detection means certain death...
        ...identifying someoneโ€™s location is as good as directly destroying them.

        -- Ethereum is a Dark Forest --

        Notes:

        Tell the story of this article, basically a white hat engineered obfuscation to try and remove funds in a bugged contract -> someone decoded, realized extractable valued, and front-ran them.

        This is now the norm on Ethereum at least, and further it's becoming institutionalized.

        ---v

        ๐Ÿ‘ผ Flashbots

        Flashbots is a research and development organization formed to mitigate the negative externalities posed by Maximal Extractable Value (MEV) to stateful blockchains, starting with Ethereum.

        -- Flashbots --

        Notes:

        This might be misleading, in that they are profiting in making MeV more effective and institutionalized!

        ---v

        Flashbots ๐Ÿ˜ˆ

        • Flashbots Auction: a marketplace for transaction ordering including the Flashbots Relay and MEV-Geth.
        • MEV-Boost: an out-of-protocol implementation of proposer-builder separation (PBS) for proof-of-stake Ethereum.
        • Flashbots Protect: an rpc endpoint that anyone can use for protection from frontrunning and failed transactions.
        • Flashbots Data: tools and dashboards to improve the transparency of MEV activity on Ethereum and the Flashbots Auction.

        Notes:

        Centralizing force, as information asymmetry generally drives towards a monopoly on MeV. Competitive landscape for this exists, and to Flashbots' credit, they seem genuine in trying to help the health of Ethereum by decentralizing...

        (BUT FIRST a discussion!) Especially in light of recent OFAC pressures revealing fragility in the system...

        ---v

        Discussion

        Front-running as a Service (FaaS) & MEV Auctions (MEVA)

        A solution or crutch?

        Notes:

        • Flashbots & Friends

        ---v

        Compliance

        Notes:

        https://cryptoslate.com/op-ed-is-ethereum-now-under-u-s-control-99-of-latest-relay-blocks-are-censoring-the-network/

        • code is unstoppable, but platform can sensor. Ability -> responsibility (we may talk more on that latter)

        ---v

        Social Context

        Social systems and norms can help cover up weaknesses in protocols.

        Public monitor to shame OFAC censors:

        https://www.mevwatch.info/

        Notes:

        • Pressure from peers through breaking norms, perhaps even losing of authority in consensus due to this. Peer reputation in computer networks, and here also in human ones.
        • Sometimes social pressures are healthy for the system, sometimes toxic depending on point of view and who benefits!
        • In monero "run your own node" culture helps keep it decentralized.
          Bitcoin big block wars show social pressures help decide the canonical forks.
        • Normalizing MEV for the profit of middlemen, providing extraction services in the worst case.

        ---v

        Unbundling

        Notes:

        From before, but here point out how this is getting more fine grained as well, and where a single actor would do it all (early bitcoin for example) we are moving more and more to appear.

        • Especially if more things like MeV can be enhanced by doing so.
        • This introduces more complexity and interfaces that can provide weakness (especially when a network is required!)

        ---v

        Unbundling

        The vision of "blockspace" leads
        more and more to this end

        ---v

        Diversity

        Notes:


        Final Thoughts

        • Complexity generally increases the risks of failures, but we should not fear it.
          $~~~$Hypothesis: this _usually makes systems more brittle._
        • Observable behavior trumps models and theory.
          $~~~$Complex systems are not intuitive and may show your assumptions and models are wrong!
        • This lesson was a start down holes... $~~~$We encourage you to keep diving deeper!

        Notes:

        • Risks and unknown unknowns increase exponentially so in many cases.
        • Examples of observables in things like MEV OFAC dominance and Babe fallback dominance etc.
        • Looking forward to explore the great unknown horizons in web3 together!

        ๐Ÿค Together, into the Deep


        Questions


        Additional Slides

        Notes:

        For reference mostly, outside of formal class time ๐Ÿ˜€


        Governance... Unstoppable?

        Unstoppable Code: The Difference Between Can't and Won't

        Notes:

        Watch after class! Perhaps assigned informally to everyone to watch in the next few days.

        ---v

        Unstoppable Code

        It seizes power from dominate forms of power: governments, corporations, states, associations, cultures, religions. It seizes power, from these big things and gives it to little people. And sooner or later, the people who are losing their undeserved, abusively applied power will start to fight back. And at that point, we will start to find out how unstoppable our code is.

        -- Andreas Antonopoulos --

        ---v

        Can't vs. Won't

        The moment you go over the the line from "can't to won't, what started as an ability becomes a responsibility. And then if you claim that you don't have the ability anymore, that responsibility just became negligence, criminal negligence.

        -- Andreas Antonopoulos --

        Notes:

        • The difference?
        • Silk road founder getting 2 life sentences + 40 years.
        • moral relativism -> "who's law?"
        • Don't make your "oops clause" -> not too narrow.

        DAOs

        Decentralized Autonomous Organizations (DAOs).

        A coordination mechanism.

        ---v

        Democratic Systems

        Democratic Mediums is a directory of patterns
        for decision, deliberation, and noise.

        Notes:

        Very much encouraged to explore after class! Many novel and niece definitions in this wiki.


        Modeling Behavior

        Token Engineering
        {especially the Academy & cadCAD Edu}

        Notes:

        Mostly free education and tools to dive deeper on tokenomics. Remember, these are models of idealized systems in general, real world conditions will differ!

        Consensus Finality


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: Consensus Finality description: Finality providing methods and mechanisms for blockchains duration: 45min ---

        Consensus: Finality


        Consensus is...

        ...a decision making process that strives to achieve acceptance of a decision by all participants.

        Notes:

        If we have external factors like trust relationships or Social hierarchy it can be quite easy.

        Trust example: If everyone trusts each other, then any one person can propose an outcome, and all the others will trust that it is in the community's best interest.

        Social Hierarchy example: If a community has an executive (President, Queen, CEO), the executive can propose her preference and everyone will accept it based on the external mechanism that keeps her as the executive.

        But these conditions are not interesting, and are often not realistic. It gets interesting when we don't make these simplifying assumption.

        ---v

        Five Aspects of Blockchain Consensus

        • State machine validity
        • Arbitrary / Political validity
        • Authorship throttling
        • Fork choice heuristic
        • Finality

        ---v

        Adversarial Conditions

        • The participants want to agree on something
        • They don't trust each other
        • Some could be faulty, or make mistakes
        • Some could be straight up malicious (aka Byzantine)

        One Shot Consensus

        A single decision to make.

        Once it is made, we are done.

        ---v

        Desired Properties

        All honest participants...

        • Termination - Eventually reach some decision
        • Agreement - Reach the same decision
        • Integrity - Reach at most one decision, which was proposed by at least one participant.

        ---v

        Also Validity

        If all participants unanimously propose the same decision, all honest participants reach that decision.

        Notes:

        Another desireable property that is sometimes too obvious to say.


        Example: Where to go for dinner

        • Termination - Stan, Kyle, and Kenny will each eventually decide where to eat.
        • Agreement - Stan, Kyle, and Kenny will each decide to eat at the same place.
        • Integrity - Stan, Kyle, and Kenny will eat at either Whistlin' Willies or Casa Bonita.

        Notes:

        Stan, Kyle, Kenny, and Cartman are trying to decide where to have dinner. Stan, Kyle, and Kenny want to have something yummy and spend time together. Cartman is secretly pursuing his own goals and doesn't care whether dinner is yummy. In fact he secretly hopes they won't decide at all so he can go home and eat KFC with his mom.

        Stan and Kyle both propose Whistlin' Willie's for dinner. Kenny and Cartman propose Casa Bonita.

        In the end Kenny may not agree that Whistlin' Willie's is the best option for dinner, but he will agree that it is the legitimate group decision.

        Image source: https://southparkstudios.mtvnimages.com/images/shows/south-park/clip-thumbnails/season-14/1402/south-park-s14e02c03-ill-show-them-f-obscene-16x9.jpg

        ---v

        Mistakes vs Malice

        Crash Tolerance - A system can keep operating normally when some actors crash or go offline.

        Byzantine Fault Tolerance - A system can keep operating normally when some actors are intentionally malicious. Byzantine actors may:

        • Crash - Or pretend to have crashed - Byzantine faults are a superset of crash faults
        • Lie - Sending false information to peers
        • Equivocate - Send inconsistent information to different peers

        Notes:

        These examples and many others are instances of the Byzantine Generals Problem

        ---v

        Mistake or Malice?

        • Consider an Airplane flight computer.
        • The pilot must know the airspeed at all times.
        • Airspeed sensors can fail.
        • Sensors can be buggy.
        • Are these malicious?

        Notes:

        Consider an Airplane flight computer. It is critical that the pilot (human or automated) knows the aircraft's airspeed at all times. Airspeed sensors can fail due to extreme temperatures, icing, solar radiation, and other reasons. For this reason, there are redundant sensors, and they run a consensus protocol.

        Imagine that one of the sensors has an overflow bug such that when the airspeed exceeds a certain threshold, maybe u8::max_value(), it actually reports a much lower speed.

        Are these crash faults or byzantine?

        In a blockchain system, bugs in the code may cause nodes whose operators are intending to be honest, to deviate from the protocol anyway. This is why client diversity is important.

        Image source: https://thepointsguy.global.ssl.fastly.net/uk/originals/2020/12/pitot-ice-scaled.jpg


        Byzantine Generals Problem

        • Divisions plan to attack
        • Must make coordinated attack to succeed
        • Some generals might be traitors and behave maliciously

        Cornell Lecture Slides

        Notes:

        Several divisions of the Byzantine army are camped around an enemy city. They plan to attack the city, but have not yet decided when to attack. If the attack is coordinated, they will probably be successful, but if it is uncoordinated, they will probably be defeated. What do our three properties mean in this scenario?

        • Termination - every honest general will eventually decide when to attack
        • Agreement - honest general will all agree to attack ta the same time
        • Integrity - the attack time will have been proposed by at least one honest general

        Leslie Lamport proposed this problem in the form of a story as a typical representative of the distributed consensus problem.

        This is a big moment for us, so let's stop and savor it. Two days ago Lauren kicked us off by talking about human coordination. How it can achieve long railroads and the great pyramids and sports teams and dancing, and even this academic program. Since then we've looked through so many layers of abstraction (contracts, smart contracts, state machines, dags) and so many technical implementation details (P2P networking, platform agnostic bytecodes, blockchain headers, merkle proofs). We've taken a huge class of problems - human coordination problems - and distilled them down to their absolute essence. Human coordinate at global scale reduced to this one cute, carefully stated problem.

        ---v

        Byzantine Generals: Solved

        Miguel Castro and Barbara Liskov, 1999

        Notes:

        Image = Cover page: Practical Byzantine Fault Tolerance

        And great news. The problem is solved! At least under some conditions. And also great news. We have a pretty good understanding of under what circumstances it is solvable.

        Before I can tell you exactly how and where it can be solved, we have to understand networks a little bit.


        History Lesson


        Synchrony

        A network is one of:

        • Synchronous - When sent, a message is received immediately by all.
        • Asynchronous - When a message is sent it may be received after some delay, or not at all. The sender doe not know whether it is received. Messages may be received in different orders by different parties.
        • Partially Synchronous - When a message is sent, it may be received after some delay up to a maximum delay, $T$. It may not be dropped entirely. Messages may be received in different orders by different parties.

        Roughly analogous to real-time (async) vs. turn-based (sync) games.

        ---v

        Sync or Async?

        Classify each of these:

        • Telephone call
        • Mail communication
        • Text message
        • Jitsi video call
        • Element chat

        Notes:

        • Telephone call is sync. You say something and wait for the other party to reply
        • Mail is async. You send a letter, then go do other stuff for several days
        • Jitsi video call is basically sync just like the phone call.
        • The others can be either. Sometimes you text back and forth quickly and wait for replies. Other times you send and come back later like mail.

        Determinism

        A system is one of:

        • Deterministic - The same inputs give the same outputs every time.
        • Probabilistic - The same inputs may not give the same outputs.
          $~~~~~~~~~~~~~~~~~~~~~~~~~$ Requires a source of entropy. eg coin flipping.

        FLP Impossibility

        Excellent Podcast with Ethan Buchman

        Notes:

        Once it was shown that deterministic consensus is impossible in an async network, the field split into two major parts. Either you:

        • Assume the network is (at least partially) synchronous
        • Introduce non-determinism

        One interesting thing about Nakamoto pow consensus is that it does both.

        • Nondeterminism because you don't know who will solve the pow first
        • Partial synchrony because it only works if the time to download and execute the block is much less than the time to solve the PoW

        Ongoing Consensus

        We want to continue agreeing on
        an every-growing history of events

        Notes:

        Blockchains are intended to keep processing and finalizing transactions forever. This is more like deciding where to go to dinner every night over and over.

        ---v

        Desireable Properties

        • Safety aka Finality - Nothing bad will happen
        • Liveness - Something good will eventually happen

        Notes:

        Traditional literature typically uses the term safety. Blockchain folks typically talk about finality. They basically mean the same thing.

        Finality can be proved to entities that are not involved in consensus.

        We spoke about liveness previously in the authoring system. There it means that more blocks will be authored, more blockspace will be created and put up for sale. Here, in finality, it means that more blocks will be finalized. Of course liveness in the finality depends on liveness in the authoring.

        These two decisions can be entirely orthogonal to one another, or wrapped up together.

        ---v

        Liveness vs Termination

        Earlier I described Termination as desireable,
        now I'm saying Liveness is desireable.

        Are these at odds with each other?

        Notes:

        Termination guarantees that, for a given decision, each honest participant will eventually decide something. This concept is relevant when there is a single decision to be made. In the blockchain context, it guarantees that we will eventually know which block is final at height n.

        Liveness guarantees that a system that is intended to continue making progress, will indeed eventually make progress. In the context of the blockchain, liveness means that once we've decided what block is final at height n, we will eventually go on to decide what block is final at height n + 1.


        Finality in Nakamoto Consensus

        • Longest chain rule
        • Longest chain is "best"... for now
        • Someone could always start mining a chain
          and,with low but non-zero probability,
          end up with it longer.
        • There could already be a longer chain
          that you just haven't heard of.

        The finality is only probabilistic.
        Nakamoto consensus in only safe in a synchronous network.

        Notes:

        This is to say that Nakamoto consensus is NOT safe on the real asynchronous internet. In practice, as long as blocks can be downloaded and executed much more quickly than the target block time, it is usually good enough.

        • Longest could also mean most accumulated work

        Deterministic Finality

        • Based on traditional methods (BFT)
        • Requires an honest-majority finite authority set
        • Consensus protocol that assumes honest majority
        • Economic game that keeps them honest

        Notes:

        If you want deterministic finality, it basically means employing BFT agreement protocols that we talked about in the history lesson. This means we need a finite authority set with an honest majority. And that means we need incentives to keep them honest.

        ---v

        Incentives: Game Theory!

        Abstractly: You behave honestly
        when the utility of doing so exceeds the cost.

        Incentive designers may potentially:

        • Reward honest behavior
        • Punish (aka slash) dishonest behavior

        Notes:

        Many systems use both of these, but doing so is not strictly necessary. Even without slashes, the opportunity cost of staking and the missed rewards from authoring invalid blocks may be sufficient.

        It is often the case that blockchain systems give rewards in the authorship and punishments in the finality. There is no fundamental reason for this; it is just a little more straightforward to implement.

        ---v

        What is potentially punishable?

        • Authoring when you aren't supposed to
        • Failing to author when you are supposed to
        • Casting finality votes for conflicting blocks
        • Casting a finality vote for a block (or chain)
          that includes an invalid state transition.

        How severe are each of these offenses?
        Do they all warrant a slash?
        A full slash?

        Notes:

        PoW has inherent punishment through wasted energy. BFT based system's don't.

        Instead, aspiring participants will typically lock up a security deposit which can be slashed in part or in whole.

        ---v

        Concrete Punishment Example

        Let's say a slash is 100 units, and the reporter gets 10%.
        I plan to attack.

        If my attack is successful,
        I expect to gain roughly 200 units worth of utility.

        I ask another authority to cooperate with me:
        "I'll pay you 20 units to not rat me out for my attack".

        How would you respond?

        Notes:

        "I don't believe you can carry out the attack because someone else will report you and take the 10 units, leaving me with 0."


        Case Study: Tendermint

        • Authorship is like Aura - simple round robin
        • Naive but simple BFT implementation
        • If the block has enough votes
          by the end of the slot, it is finalized.
          Otherwise, it is rejected via timeout.
        • "Instant finality"
        • Forkless - Forks are disallowed
          because blocks can only be authored
          on finalized parents.

        Notes:

        Tendermint assumes a partially synchronous network, like all the BFT based systems - That is to say that messages may not arrive immediately, but will arrive within a finite time bound. In practice this means it is slot-based just like so many of the authoring schemes.

        Tendermint is often touted as "instant finality". It is instant in the sense that finality is tied to authorship. In practice this means that authorship, which is inherently O(n), is slowed down to stay in sync with finality which is O(n^2). They sacrifice liveness to guarantee absolute safety at all times.

        ---v

        Tendermint Deep Dive

        1. Wait for a block (or author one if it is your turn)
        2. Prevote
          • If the block is valid, Prevote for it.
          • If the block is invalid, Prevote `Nil`
        3. Precommit
          • Wait for 2/3 prevotes then Precommit
          • If you don't get 2/3 prevotes, Precommit `Nil`
        4. Complete
          • Wait for 2/3 Precommits them finalize
          • If you don't get it, throw the block away

        Very useful blog post


        Hybrid Consensus

        • Separates block production from finality.
        • Block production stays live even if finality lags.
        • Allows lower overhead in the finality layer.
        • Used in Substrate.

        ---v

        What About Re-Orgs


        Notes:

        Previously we talked about how a node's view of the best block can change, and that is called a re-org.

        ---v

        Modified Fork Choice Rule


        Only extend best finalized chain Notes:

        Once you have a finality gadget installed, you have to make sure you only ever author on top of finalized blocks. Even if another chain is longer.


        Case Study: Grandpa

        • Deterministic finality only
        • Requires an external block authoring scheme
          with its own liveness proof.
        • Kind of like Tendermint but better.
        • Finalizes chains, not blocks.

        ---v

        Vote on Chains, not Blocks

        Notes:

        BFT finality with $n$ authorities is in $O(n^2)$. Tendermint does this at every block. This bounds the size of the authority set.

        With separated, we treat each vote as a vote not only for one block,but also for each ancestor block. This significantly reduces the number of total messages sent. Allows the chain to stay live even when many validators are offline


        A GRANDPA Round

        Each validator broadcasts a prevote for the highest block that it thinks should be finalized

        • For honest validators, this block must include the chain that was previously finalized
        • This new chain could be several blocks longer than the last finalized chain

        A validator that is designated as the primary broadcasts the highest block that it thinks could be final from the previous round


        A GRANDPA Round

        Notes:

        • 7 = # Validators
        • 5 = # Threshold

        A GRANDPA Round

        Notes:

        • 7 = # Validators
        • 5 = # Threshold

        A GRANDPA Round

        Each validator computes the highest block that can be finalized based on the set of prevotes

        • i.e. we find the common ancestor of all votes that has support from more than $\frac{2}{3}N + 1$ validators
        • We refer to this block as the prevote GHOST

        A GRANDPA Round

        Notes:

        • 7 = # Validators
        • 5 = # Threshold

        A GRANDPA Round

        We define the round estimate as the highest ancestor of the prevote GHOST for which it is still possible to precommit

        • i.e. when no precommit votes have been sent out yet, then:

        estimate == prevote GHOST


        A GRANDPA Round

        Notes:

        • 7 = # Validators
        • 5 = # Threshold

        A GRANDPA Round

        If the estimate extends the last finalized chain, then each validator will cast a precommit for that chain.

        Each validator waits to receive enough precommits to be able to finalize

        • We again find the common ancestor of the estimate which has threshold precommits
        • We declare that block as finalized

        A GRANDPA Round

        Notes:

        • 7 = # Validators
        • 5 = # Threshold

        A GRANDPA Round

        Notes:

        • 7 = # Validators
        • 5 = # Threshold

        A GRANDPA Round

        The round is deemed completable:

        • if the estimate is lower than the prevote GHOST
        • or if it's impossible to have a supermajority on any block higher than the current estimate

        Validators start a new round after it becomes completable.


        A GRANDPA Round

        Notes:

        • 7 = # Validators
        • 5 = # Threshold

        A GRANDPA Round

        Notes:

        • 7 = # Validators
        • 5 = # Threshold

        A GRANDPA Round

        Notes:

        • 7 = # Validators
        • 5 = # Threshold

        A GRANDPA Round (Alt)

        Notes:

        • 7 = # Validators
        • 5 = # Threshold

        A GRANDPA Round (Alt)

        Notes:

        • 7 = # Validators
        • 5 = # Threshold

        A GRANDPA Round (Alt)

        Notes:

        • 7 = # Validators
        • 5 = # Threshold

        A GRANDPA Round (Alt)

        Notes:

        • 7 = # Validators
        • 5 = # Threshold

        Summary

        • Networks can be {Synchronous, Asynchronous}
        • Consensus systems can be {Deterministic, Probabilistic}
        • Consensus systems can be {Open participation, Finite participation}
        • There is always an assumption that at least {1/2, 2/3} participants are honest
        • In decentralized systems, we use Economics and Game Theory
          to incentivize honest execution of the consensus protocol

        Game Time

        I want to play a game...
        a board game!

        Grandpa - The Board Game

        • Players: 5+ (4 actual players, 1 author)
        • Play time: 15 - 60 min
        • Materials: A large whiteboard and many colored markers

        Overview

        Grandpa is a Byzantine fault tolerant blockchain finality gadget (formal spec). This collaborative board game allows players to learn and practice the Grandpa protocol while also having fun together.

        Your goal in the game is to finalize blocks in an ever-growing blockchain data structure. You will work together to share information with other players and reach consensus. But watch out; some players may be Byzantine!

        Pictured: Byzantine Generals solving distributed consensus via a board game.

        Some less important details of the grandpa protocol (such as primaries, and timeout conditions) are omitted from the board-game for the sake of playability and clarity.

        Setup

        Select one participant to act as the "author" who is responsible for creating the blockchain structure, but will not actually play the grandpa protocol. The remaining participants are all players in the grandpa protocol.

        Give one marker to each participant including the author. Each player should have their own marker color. Avoid colors that are hard to distinguish such as light red and pink. If you have colorblind players, take special care when choosing marker colors.

        Choose a goal number of blocks that you wish to finalize together as a team. The game will end when you reach this block number.

        Views and Gossip

        The grandpa protocol operates in a decentralized asynchronous blockchain network. As such, there is no universal view of the blockchain data structure or the messages that are being passed between players in the protocol. Some players may see more information than others and information may arrive to the various players in different orders.

        Divide the whiteboard into a dedicated space for each player in the protocol. Each player should have roughly 50cm X 50cm. The author does not need their own dedicated space.

        Throughout the game all participants including the author are responsible for communicating with other players by adding information to other players dedicated spaces. In fact, most of the marks that you make during the game will be on someone else's space rather than your own. For a more realistic game, take care to share information with other players in a different order each time.

        Genesis Block

        Before game play begins, the author draws a single genesis block labeled G on each player's view. Each player marks the genesis block as final by shading it with their color in their own view.

        Authoring

        The author is responsible for creating the blockchain data structure and gossiping it to the players. As the game progresses the author will grow the blockchain by creating descendant blocks of this genesis block. The author may create blocks anywhere in the chain they see fit. They may create forks, or linear runs without forks. They may create a long chain and then go back and create shorter forks from earlier in the chain.

        When the author creates a block they should gossip it to all players by drawing it on each player's view. A block is drawn with a pointer to its parent block and a short unique block identifier like a few characters or digits. The author should take care to vary to order in which they place new blocks on various players' views. In fact, the author may even gossip multiple blocks to a single player before going back and gossiping any of them to other players. However the author should ensure that all blocks are eventually gossiped to all players.

        Two blocks have been authored since genesis. Not all players have heard about block DEF yet.

        In some ways the author acts as a "party host" or "dungeon master" for the game. They should observe the players progress, struggles, and enthusiasm, and author accordingly. If players are struggling to keep up or getting frustrated or overwhelmed the author should slow down the authoring rate or build a simpler chain with fewer forks. If players are easily finalizing blocks or getting bored the author should speed up, create a more complex tree with many forks, or decrease the synchrony.

        Game Play

        The Grandpa protocols proceeds in rounds. Each player has their own view of what round they are on, and not all players will be on the same round at the same time. In each round, each player casts two votes known as the "prevote" and "precommit" in that order. Each player begins in round 1.

        Like many other BFT protocols, Grandpa requires strictly greater than 2/3 of players (not counting the author) to be properly following the protocol. For the remainder of this section this will be referred to as a "threshold".

        Prevoting

        Each player begins a round by casting their prevote. A prevote can be thought of as a non-binding signal for what the player hopes to finalize in this round. Generally the prevote should be for the longest chain that extends the best finalized block. A player casts their prevote by writing the current round number off to the right of the block they are prevoting for first on their own view, and then on other players' views. Remember you should send your prevotes out to other players in a different order each time, and it is okay to allow some latency between sending it to each player.

        Players 1 and 4 have cast prevotes for block ABC in round 2. Their prevote has not yet been gossiped to all players. Players 2 and 4 have not yet cast prevotes for round 2.

        If a player hasn't yet seen the block you're prevoting for, you may add the block and its parents to their view.

        The Prevote Ghost

        When a player has seen a threshold of prevotes in the current round, they can may mark the round's "Prevote Ghost" on their own view. They may also choose to wait a short time to see if any new prevotes are coming in. The prevote ghost is defined as the highest block that has a threshold of prevotes, and it is marked by drawing the letters PG and a round number off to the left of the block. For example PG2 for the round two prevote ghost. (Or optionally ๐Ÿ‘ป2 if you are feeling artistic).

        Now that player 1 has seen a threshold of prevotes (3 in this case of 4 players), they can mark the prevote ghost. No other players have seen enough prevotes to mark a prevote ghost yet.

        The Estimate

        As you mark your prevote ghost, also mark your estimate on your own view to the left of the same block that is the prevote ghost with the letter E and a round number. For example, E4 for round four's estimate.

        In round 4, Player 1 has marked their prevote ghost previously, and is now setting the estimate to the same slide.

        A round's estimate is defined as the highest block that is in the chain of the prevote ghost that could possibly achieve a threshold of precommits. So while the estimate begins at the same block as the prevote ghost, it may move up the chain as more precommits come in.

        Precommitting

        Once you have marked a prevote ghost, you may, again, wait a short time for any more prevotes to come in. Once you get tired of waiting (or when you have seen all the prevotes), you may cast your precommit for the block that you see as the prevote ghost. Mark your precommit first on your own view and then on other players' views by writing the round number off to the right of the block and circling it. Precommits are distinguished from prevotes by the circle. Remember not all players will agree on which block is the prevote ghost, so others may precommit for blocks different than you have.

        Player 1 casts their round 4 precommit for the block that they see as the prevote ghost. They have also observed a Player 2's precommit.

        As you observe more precommits appearing on your view, your estimate may change. Specifically it may move up the chain to ancestor blocks.

        As Player 1 observes more precommits arriving, it becomes impossible for blockGHI to achieve a threshold of precommits, and thus the estimate moves up the chain.

        Completing a Round

        We will decide that some block is finalized in each round, although it may be a block that was already finalized in a previous round. We will only ever finalize an ancestor of the estimate. Once some ancestor of the estimate has achieved a threshold of precommits, you can declare that block finalized by shading it with your color on your view.

        After a round has completed, you may choose to erase the votes for that round from your view to keep the board tidy. But you are not required to do so. Be careful not to erase votes for future rounds by accident as some players may have advanced to the next round before you.

        As Player 1 has completed round 4 and has erased markings related to round 4. They have NOT erased marking related to round 5.

        Proceed to the next round.

        Ending the Game

        Grandpa is intended to continue finalizing blocks forever. Since you likely don't want to play this board game forever, the board game does have an end.

        The honest players win when they all finalize the goal number of blocks chosen at the beginning without a safety violation.

        The Byzantine players (if any; see next section) win when two honest players finalize conflicting blocks or the honest players get fed up and flip the whiteboard over.

        Byzantine Actors

        Once you have played a few rounds of the game and are able reliably finalize new blocks, you can spice things up by assigning one or more players to be Byzantine. Byzantine players are not required to follow the protocol rules. For example they may:

        • Prevote for chains that do not extend the latest finalized chain
        • Precommit for blocks other than the ones indicated by the prevote
        • Go back and cast votes in previous rounds
        • Fail to participate at all.

        When first adding Byzantine players, you may assign the Byzantine roles such that everyone knows who is Byzantine. Or, for a more realistic experience, you may assign it blindly by eg drawing straws. Remember that in order for Grandpa to work you must have strictly less than one third of grandpa players Byzantine.

        For the most realistic experience, allow players to self select whether they are Byzantine. By doing this there is no guarantee that the honest super majority criteria is met and you experience safety faults where different players finalize conflicting chains.

        Designing DAG-based consensus


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: Designing DAG-based consensus description: A formal, yet friendly consensus framework ---

        Designing DAG-based consensus


        Goals of this lecture


        1. formalize the consensus problem and related concepts
        2. provide a framework for designing DAG-based consensus protocols

        What is consensus?


        • a process of agreeing on the same result among a group of participants
        • a fundamental problem in distributed computing
        • a key component of blockchain technology stack

        Consensus features


        liveness, safety, integrity


        We have already seen some


        Nakamoto

        Babe

        Grandpa

        Sassafras

        Tendermint

        ...


        Who is running the protocol?


        Participants, called nodes


        Nodes


        • nodes can be either honest or malicious
        • honest nodes follow the protocol
        • malicious nodes can deviate from the protocol in any way they want
        • malicious nodes can collude with each other
        • malicious nodes can be controlled by an adversary

        Public key infrastructure


        • every node has its own private and public key
        • every node signs messages with its private key
        • every node verifies messages with other nodes' public keys

        Public key infrastructure


        authenticated point-to-point communication


        Adversary


        Adversary can control the network delays, but is computationally bounded, i.e. it cannot break the cryptography (like forging the signatures).


        Network


        Communication via network... but what kind of network?


        Network models


        synchronous

        partially synchronous

        asynchronous


        Network models: synchronous


        There exists a known upper bound \(\Delta\) on message delivery time.



        Intuition: there's a well-defined notion of a protocol round


        Network models: asynchronous


        There is no upper bound on message delay, though delivery is guaranteed.



        Intuition: you can't tell whether a node has crashed or has a long delay


        Network models: asynchronous


        There is no upper bound on message delay, though delivery is guaranteed.



        • We assume that the adversary has full control over the message delays.
        • The concept of a timeout is basically useless.

        Network models: partially synchronous


        There exists a known bound \(\Delta\), and an unknown point in time GST after which the communication becomes synchronous with a delay \(\Delta\).



        Intuition: protocol will eventually work synchronously, but it needs to be safe before


        Crucial theoretical results


        [FLP theorem] It is impossible to have a deterministic protocol that solves consensus in an asynchronous system in which at least one process may fail by crashing.



        [Castro-Liskov theorem] It is impossible to have a protocol that solves consensus in a partially synchronous system with \(3f+1\) nodes in which more than \(f\) processes are byzantine.


        Crucial theoretical results


        [FLP theorem] It is impossible to have a deterministic protocol that solves consensus in an asynchronous system in which at least one process may fail by crashing.



        [Castro-Liskov theorem] It is impossible to have a protocol that solves consensus in a partially synchronous system with \(3f+1\) nodes in which more than \(f\) processes are byzantine.


        Consequence


        The best one can hope for in asynchronous scenario is probabilistic protocol tolerating up to \(f\) faults for \(3f+1\) participants.


        โœ… Doable!


        Note on randomness


        Real probability is actually needed in the extremely hostile environment. In case where the adversary is not legendarily vicious, even a dumb (but non-trivial) randomness source will do.


        Responsiveness


        Responsiveness


        Protocols that are not responsive have to wait for \(\Delta\) time to proceed to the next round.


        Responsiveness


        Protocols that are not responsive have to wait for \(\Delta\) time to proceed to the next round.

        • \(\Delta\) must be long enough to allow all honest nodes to send their messages.
        • \(\Delta\) must be short enough to allow the protocol to make progress.
        • In case of failure, they have to perform a pretty expensive recovery procedure (like the leader change).

        Responsiveness


        Protocols that are responsive wait for \(2f+1\) messages to proceed to the next round.



        Why \(2f+1\)?


        Responsiveness


        Protocols that are responsive wait for \(2f+1\) messages to proceed to the next round.



        Among \(2f+1\) nodes, there are at least \(f+1\) honest ones, i.e. honest majority.


        Responsiveness


        Protocols that are responsive wait for \(2f+1\) messages to proceed to the next round.

        • Asynchronous protocols must be responsive.
        • In good network conditions, they significantly much faster.

        Checkpoint


        Up to this point, we covered:

        • consensus problem
        • node types and adversary
        • inter-node communication
        • network models (synchronicity)
        • protocol limitations in asynchronous network (honesty fraction and the need for randomness)
        • responsiveness

        Warmup exercise: broadcast


        (In an asynchronous network) reliably send a single message to all other nodes.



        • (validity) If the sender is honest and broadcasts a message \(m\), then every honest node outputs \(m\).
        • (integrity) If an honest node outputs a message \(m\), then it must have been broadcast by the sender.
        • (agreement) If an honest node outputs a message \(m\), every other honest node outputs \(m\).

        Reliable broadcast protocol (RBC)



        Reliable broadcast in practice


        Due to the very high communication complexity we use heuristics or cryptography-based tricks.


        Blockchain protocol vs Atomic broadcast


        Atomic broadcast


        Randomness formalized


        Randomness beacon


        Atomic broadcast: timeline



        Atomic broadcast: timeline



        Fun fact


        Aleph paper, as the first, also achieved fully asynchronous randomness beacon:

        • with efficient setup (\(O(1)\) rounds, \(O(N^2)\) communication)
        • with \(O(1)\) expected rounds to output a random value with \(O(N)\) communication per round

        Consensus protocols (selection)


        Classical protocols:

        • [DLSโ€™88], [CRโ€™92],
        • PBFT [CLโ€™99]
        • Random Oracles โ€ฆ [CKSโ€™05]
        • Honey Badger BFT [MXCSSโ€™16]
        • Tendermint [BKMโ€™18]
        • VABA [AMSโ€™19]
        • Flexible BFT [MNRโ€™19]
        • HotStuff [YMRGAโ€™19]
        • Streamlet [CSโ€™20]
        • Grandpa [SKK'20]

        DAG-based protocols:

        • [L. Moser, P. Meliar-Smith โ€˜99]
        • Hashgraph [Bโ€™16]
        • Aleph [GLSSโ€™18]
        • DAG-Rider [KKNSโ€™21]
        • Highway [KFGSโ€™21]
        • Narwhal&Tusk [DKSSโ€™22]
        • Bullshark [SGSKโ€™22]

        DAG-based protocols


        DAG


        Directed Acyclic Graph


        How does it relate to consensus?


        Intuition: graph represents the dependencies between messages (units).


        Framework core


        1. We maintain a local DAG representing our knowledge of the units.
        2. We perform a local, offline consensus on our DAG.

        Framework core


        1. We maintain a local DAG representing our knowledge of the units.
        2. We perform a local, offline consensus on our DAG.

        Framework core (in other words)


        1. (online): sending and receiving units that contribute to the local DAG
        2. (offline): everybody performs a local consensus on the DAG, just by looking at it

        Clue observations


        • local DAGs might differ...
        • but they are guaranteed to converge to the same DAG
        • the offline consensus is guaranteed to produce the same result

        Adversary control



        Randomness? Where is randomness?


        It is put into the local consensus protocol.


        Relation to the atomic consensus problem


        • nodes receive transactions and put them into units
        • nodes send each other their new units
        • (locally) nodes come up with a linear ordering of the units and make blocks from chunks

        Digression: block production, information dissemination and finalization


        The common approach (e.g. in Substrate):

        • production and dissemination is done in the same layer
        • afterwards, nodes perform consensus on finalizing disseminated blocks

        Natural approach for DAG-based protocols:

        • information dissemination happens as 'the first phase'
        • block building and (instant) finalization happens locally

        Main consequences of the different separation


        • block signatures
        • speed

        Local consensus: goal


        Local copies might differ significantly, blocks might have not come to all nodes yet, etc... but we have to make common decision about unit ordering!


        Key concept: availability


        Intuitively, a unit is available if:


        • most of the nodes have it
        • it was distributed pretty promptly (we won't call a unit available, if it finally arrived everywhere after a month)
        • most of the nodes know that most of the nodes know that most of the nodes know... that it is available (mutual awareness)

        Availability


        If a unit is available, it is a good candidate for being chosen as an 'anchor' in extending current ordering.


        Lightweight case study


        Aleph Zero BFT protocol




        Building blocks



        Choosing head



        Availability determination


        Units vote for each other's availability.


        (Part of) availability determination


        VoteU(V) =

        • [[U is parent of V]] if V is from the round just after the round of U
        • 0/1 if all children of U voted 0/1
        • CommonVote(round(U ), round(V )) otherwise

        (U comes from the earlier round than V)


        Bonus: generating randomness


        Sig sk(nonce)


        • randomness must be unpredictable
        • delayed reveal
        • must depend on \(f+1\) nodes
        • cannot be disturbed by the adversary

        Standard way



        Standard way


        Problem: need for trusted dealer!


        One simple trick


        Everybody is dealing secrets


        Combining randomness


        Accounting Models & User Abstractions in Blockchains


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: Accounting Models & User Abstractions in Blockchains ---

        Accounting Models & User Abstractions in Blockchains


        Overview

        • Cryptography, Signatures, Hash functions, Hash based Data Structures
        • Economics/Game Theory
        • Blockchain structure

        Where do we go from here?

        • We have some base elements, ideas, and concepts
        • Now, let's put them together into something cool..

        What are we talking about?

        • Now that we have this structured decentralized tamper proof state machine..
        • Let's think of ways we can formulate a state and a state transition in terms of representing users

        State User Model


        State User Model


        How to represent Joshy and Andrew?


        User Representation


        How to send from Joshy to Andrew? What do you need?

        Notes:

        What would be catastrophic if we got wrong??


        What if we want to spend this?

        Notes:

        Why do we say spend here and not modify?


        Input


        Transaction

        Notes:

        Why do we not send all of the 70 to Andrew?


        How to verify this state change is valid?

        • We can actually spend this thing signature verification!
        • Sum of the inputs is >= sum of the outputs
        • No coins are worth 0
        • Has this already been spent before?

        Notes:

        Which did I forget??


        Our new state


        How do we generalize beyond money?


        How do we generalize beyond money?

        Notes:

        How are we going to verify now that the state transition is valid?


        Transaction


        Transaction


        Is this a good model? Why or why not? Let's discuss

        • Scalability
        • Privacy
        • General Computation

        Is there a different way?

        Notes:

        Now ease them to the solution of Accounts


        Accounts

        Notes:

        Now ease them to the solution of Accounts


        State Transition Accounts


        State Transition Accounts


        How do we verify and handle this transaction?

        • Verify enough funds are in Joshy's account
        • Verify this amount + Andrews amount don't exceed the max value
        • Check the nonce of the transaction
        • Do the actual computation of output values

        Notes:

        Did I forget any?


        State Transition Accounts


        What did we do differently in Accounts vs UTXO model?

        Notes:" />

        Verify as opposed to determining the outcome. Not submitting output state in transaction


        Account Arbitrary Data


        Is this a good model? Why or why not? Lets Discuss

        • Scalability
        • Privacy
        • General Computation

        Notes:

        Parallelization? Storage space, privacy solutions?


        Small shill... Tuxedo ๐Ÿ‘”

        https://github.com/Off-Narrative-Labs/Tuxedo


        Questions

        Adding Privacy to the UTXO model


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: Adding Privacy to the UTXO model description: Explore some techniques for adding privacy to the UTXO model duration: 30min ---

        Adding Privacy to the UTXO model


        Input/Output-based Cryptocurrencies

        • Transactions have a list of unspent transaction outputs (UTXOs) as its inputs
        • Each input is signed
        • The transaction is allowed to spend as much funds as the sum of its inputs
        • The transaction spends funds by creating outputs and by paying a fee

        Input/Output-based Cryptocurrencies

        • Inputs must only refer to actually existing outputs (membership)
        • The output spent must not be prior spent (linkability)
        • The output's owner must consent to this transaction (ownership)
        • The transaction's inputs and outputs must be balanced (sum check)

        Bitcoin

        • Bitcoin specifies the spent output. This satisfies membership and linkability
        • Each Bitcoin output has a small, non-Turing complete program (Script) specifying how it can be spent
        • Each input has a scriptSig which proves the script is satisfied and this is an authorized spend (ownership)
        • The outputs cannot exceed the inputs, and the remainder becomes the fee (sum check)

        ZK Proofs

        • ZK-SNARKs - A small proof that's fast to verify (<= $O(\sqrt{n})$)
        • ZK-sNARKs - A small proof that's not fast to verify (>= $O(n)$, frequently $O(n log n)$)
        • ZK-STARKs - A small proof that's fast to verify, based on hash functions
        • All of these can prove the execution of an arbitrary program (via an arithmetic circuit)
        • None of these reveal anything about the arguments to the program

        Merkle Proofs

        • Merkle proofs support logarithmically proving an item exists in a tree
        • For 2**20 items, the proof only requires 20 steps
        • Even if a ZK proof is superlinear, it's a superlinear encoding of a logarithmic solution

        Private Membership

        • When an output is created on-chain, add it to the Merkle tree
        • When an input specifies an output, it directly includes the output it's spending
        • It also includes a Merkle proof the output exists somewhere on chain, embedded in a ZK proof

        Pedersen Commitments

        • A Pedersen commitment has a value (some number) and a mask (also some number)
        • There's as many masks as there private keys, hiding the contained value
        • Pedersen commitments can be extended to support multiple values

        A New Output Definition

        • ID
        • Amount
        • Owner
        • Mask
        • All in a single
          Pedersen commitment

        Private Membership

        • We don't prove the included output exists on chain
        • We prove an output with identical fields exists on chain yet with a different mask
        • This allows spending a specific output without revealing which it is

        Ownership and linkability

        • A ZK proof can take the output ID and apply some transformation
        • For every output ID, the transformation should output a single, unique ID
        • Not just anyone should be able to perform this transformation
        • This provides linkability, and if only the owner can perform the transformation, ownership

        Sum check

        • One final ZK proof can demonstrate the sum of inputs - the sum of outputs = fee
        • There are much more efficient ways to prove this

        Summary

        • This hides the output being spent and the amounts transacted
        • Combined with a stealth address protocol, which replaces addresses with one time keys, it hides who you're sending to as well
        • This builds a currency which is private w.r.t. its on-chain transactions

        Light Clients and Bridges


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: Light Clients and Bridges description: Light clients principles and application in bridges ---

        Light Clients

        What can I say?
        It's a client but light.

        ๐Ÿ˜ข Running a Node is Hard ๐Ÿ˜ญ

        Ideally:

        • Everyone runs their own node.
        Reality:
        • It takes a lot of disk, memory, etc
        • It takes some know-how
        • I don't need it all the time

        Notes:

        The bitcoin whitepaper clearly assumes that users will run their own nodes. This is the most trustless and decentralized way to operate, and you should do it whenever you can. If you think you can't you're probably wrong. Just ask the Monero community.

        There are some reasons not to run a full node and the reality is that not everyone will. So even though we should always run our own nodes, let's look at some alternatives and ways we can make node running more accessible.

        ---v

        RPC Nodes

        AKA, trust somebody else's node.

        • ๐Ÿ•ต๏ธ Spy on you (like infura).
        • ๐Ÿ”ž Censor you
        • ๐Ÿคฅ Lie to you
        • ๐Ÿ’” Steal your boyfriend

        Notes:

        The easiest thing to do is just trust some expert to run a node for you. Very web2. Lot's of things can go wrong.

        So this is definitely not the best option. Let's see if we can do better.

        ---v

        Lighten the Load

        For resource constrained systems and people in a hurry

        • Phone
        • Raspberry pi
        • Microcontroller
        • Inside Web Browser

        Notes:

        One of the complaints was that the node takes too much resources. This is especially true if we want people to be able to run the node in all kinds of exotic environments. And we do want that because we want people to run their own node even when they're just paying the bill at dinner from their phone or liking social posts while scrolling on the bus. Let's make the client lighter so it doesn't require as much resources.

        ---v

        Light Client Duties

        • โŒ Sync blocks
        • โŒ Execute blocks
        • โœ… Sync headers
        • โ” Maintain Transaction Pool
        • โœ… Checks consensus
        • โŒ Maintains state

        Notes:

        This is what a typical light client does. There is not a single definition of light client. There are varying degrees of lightness to suit your needs.

        ---v

        Trustless

        • Relies on full node for data
        • Does not have to trust data
        • State root helps a lot

        Notes:

        The figure is from the Bitcoin whitepaper. The concept of light clients has been around since bitcoin. At that time it was known as Simplified Payment Verification. You could confirm that a payment was sent or received. But you couldn't confirm that the tokens in question still existed or anything else about the state.

        Chains with state roots can have much more powerful light clients

        ---v

        Syncing Strategies

        • Full header sync
        • Checkpoints in code
        • Warp sync

        Notes:

        We also need to address the use case of clients that are not always on. For example if you only need your node on your phone, or when using a specific web page, that means it will have some syncing to do.

        Doing a full sync is already a lot faster than on a full client because you aren't downloading or executing the blocks. But by the time you have a few million headers, it does still take some time.

        The naive solution is to just have relatively recent headers hard-coded in the client. This works pretty well. You already have to trust the client developers for the entire implementation so you aren't trusting a new party at least.

        Warp sync is possible when you have deterministic finality. In dead simple PoA you just check that the authorities have signed the latest block and you are good. If you have authority hand-offs, there is more work to be done. You have to check that each authority set signs the transition to the next authority set. But this is still only even N blocks instead of every block.

        ---v

        Self Defense

        Stay in the gossip protocol or you might get got.

        Notes:

        In the main gossip protocol, if authorities finalize two conflicting blocks, then we can prove that they have broken the rules and slash them. If we don't watch the gossip and only peer with a single full node, then our view is entirely defined by that node. They may gossip us an attack chain and we won't know. So it is important to communicate with many different full nodes.


        Bridges

        Transport layers between independent consensus systems

        Notes:

        Generally speaking bridges move arbitrary data between unrelated consensus systems. Basically between different blockchains, and those messages can evoke arbitrary side effects on the target chain. To keep it concrete, we'll mostly talk about moving tokens.

        ---v

        Source and Target Chain

        Notes:

        By convention we speak of bridges as being one-directional. When we talk about trustless bridge design this is a core concept in the design. It is less critical but still useful for trusted bridges.

        A two-way bridge is really just two one-way bridge. Think of a two-way street. There is a dedicated lane for each direction.

        ---v

        Source Chain Re-Orgs

        Notes:

        On PoW chains this is truly just a judgement call and a prayer. If the source chain has deterministic finality we can do better. We need to wait for finality. But even this isn't foolproof. More on this after we cover the basic design.

        ---v

        Bridge Models

        Trust-based

        Trusted intermediary makes a transfer manually.
        Eg. Wrapped bitcoin on ethereum (WBTC)

        Trustless

        Trustless is the goal,
        like almost everything in web3.

        Notes:

        The trust based bridges are not at all ideal. You have to entirely trust an intermediary. You send the intermediary tokens on the source chain. Once the intermediary is satisfied that they really own the source tokens, they send you some target tokens on the target chain. Or they don't whatever, not their problem.

        You can make they trust properties slightly better by using a multisig or a group of people so you only have to trust some subset of them. But this does not fundamentally eliminate the trust agreement. One classic example is the WBTC foundation. You send them bitcoin, they wait for "enough" block confirmations, and then they mint you an ERC20 token on Ethereum. And they provide the same service in reverse too.

        A lot of the trusted bridge design can be improved and we'll talk about that in detail in the next few slides. But it's worth observing here that we will never be able to eliminate the part about "Once the intermediary is satisfied that they really own the source tokens". The bridge can never be stronger than the consensus on the source chain


        Trustless bridge design

        • Most trustless way to interact with blockchain
          is to run a node
        • This is true for individuals and other blockchains
        • A blockchain is extremely resource constrained.
        • Run a source chain light client on the target chain

        Notes:

        ---v

        BTC Relay


        Bridge Design Challenges

        Notes:

        Bridges present their own set of design challenges beyond what we encounter in regular stand-alone light clients.

        ---v

        Peers?

        • How can we peer without networking?
        • Enter the Relayer - a permissionless and trustless role
        • Need at least one honest relayer

        Notes:

        On-chain logic doesn't have network IO, so how do we peer? There is a role known as a relayer. It is an off-chain agent who watches the source chain, and submits headers and finality proofs from the source chain to the target chain through transactions. Anyone can start a relayer. It is typically a little piece of software that you run. But there is nothing magic about it. You could perform the relayer task manually by copying header data from an explorer into metamask for example.

        You do need at least one honest relayer for the chain to get the correct header info. For this reason a large decentralized relayer group is nice. But even if you don't trust any relayer out there, you can always run your own.

        ---v

        Finality and Equivocation

        Notes:

        It is not safe to accept headers as finalized immediately even if there is a deterministic finality proof. Let that sink in. Even if there is a valid finality proof, it is not safe to accept them as finalized. Why not?

        Because the validators may be equivocating. They don't send equivocations to real nodes on the network because those equivocations will be gossiped around and reported on the source chain and the validators will be slashed accordingly. But remember a light client on the target chain has no way to report such equivocations back to the source chain.

        ---v

        Equivocation Incentives

        • Add a Challenge Period and
        • Add Fishermen - reverse of relayers

        OR

        • Stake Relayers so they can be slashed

        Notes:

        There are basically two classes of solutions. Both of them require a waiting period aka challenge period before accepting a header with a finality proof as final.

        One is to add a role of fishermen. They are responsible for noticing when the header candidate on the target chain is different from the one in the main source chain protocol and reporting this behavior back to the source chain so the validators can be slashed there. Two problems:

        1. Fishermen have weak incentives. If they do a good job there will be no equivocations and they will not get paid.
        2. Target chain is relying on the foreign source chain to keep the bridge secure instead of owning that security itself.

        The other is to have the relayer role require a security deposit. If it turns out that a relayer relays an attack header, that relayer is slashed and the relayer who reports it gets a reward. Relayers will expect to earn some reward for the opportunity cost of their stake which makes the bridge operation more expensive.


        Multichain Apps

        ---v

        We have a header, now what?

        • App users submit proofs
        • Need a source chain transaction?
          Submit an spv-style transaction proof
        • Need some source chain state?
          Submit a state proof

        Notes:

        The header sync is just the foundation. Now Applications can build on top of it with the best possible trust guarantees.

        If you need some source chain transaction, your app needs to require an spv-style transaction proof to check against the header's extrinsics root.

        If you need some source chain state, your app needs to require a state proof to check against the header's state root.

        ---v

        Multichain Security

        Notes:

        This kind of trustless bridge with proper incentives gets us information about the source chain to the target chain with security about as high as it was on the source chain. If you are building an app that spans multiple chains consider the security guarantees on both chains. The weaker security of the two is the security your app has. More abstractly, your app consumes two different kinds of blockspace that may be of different qualities. Your app is only as quality as the lower of the blockspaces.

        ---v

        Example: Depository Mint Model

        • Send tokens to a contract on source chain
        • Message is relayed to destination chain
          • Offchain relay and transaction
          • XCM
          • "Somehow"
        • New "wrapped" tokens are minted on the destination chain

        Notes:

        The same process works in reverse to get the original tokens back. This can get get complex when there are multiple bridges. Are their wrapped tokens interchangeable? What if one of the bridges gets hacked?

        EVM, Solidity, and Vyper


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: EVM, Solidity, and Vyper description: Overview and history of the EVM and languages that compile to it. Focus on architecting real-world smart contracts. ---

        EVM, Solidity, and Vyper


        EVM

        Ethereum Virtual Machine

        A VM designed specifically for the constraints and features of Ethereum


        EVM Properties

        • Deterministic: execution outcome easy to agree upon
        • Spam-resistant: CPU and other resources are metered at a very granular level
        • Turing-complete (with a caveat)
        • Stack-based design
        • Ethereum-specific (EVM can query block hash, accounts and balances, etc.)

        Notes:

        It is critical that the EVM be 100% deterministic and that each implementation produce the same outcome. Even the smallest discrepancy between two running nodes would lead to different block hashes, violating consensus about the results.


        History of Ethereum

        • Nov 2013: Vitalik released WP
        • Apr 2014: Gav released YP
        • July 2014: $18M raised (ICO lasted 42 days)
        • July 2015: Frontier released -- bare bones, Proof of Work
        • Sept 2015: Frontier "thawed", difficulty bomb introduced
        • July 2016: DAO Fork
        • 2016-2019: Optimizations, opcode cost tuning, difficulty bomb delays
        • 2020: Staking contract deployed, Beacon Chain launched
        • 2021: EIP-1559, prep for The Merge
        • 2022: The Merge (Proof of Stake)
        • 2023: Staking withdraw support

        ---v

        DAO Hack

        1. 2016: raised $150M worth of ETH
        2. Later that year: 3.6M ETH drained
        3. Reentrancy attack
        4. "Mainnet" Ethereum forks to retroactively undo hack
        5. Ethereum Classic: code is law

        Notes:

        A DAO ("Decentralized Autonomous Organization") is much like a business entity run by code rather than by humans. Like any other business entity, it has assets and can carry out operations, but its legal status is unclear.

        The earliest DAO ("The DAO") on Ethereum suffered a catastrophic hack due to a bug in its code. The DAO community disagreed on whether or not to hard-fork and revert the hack, resulting in Ethereum splitting into two different chains.

        ---v

        The first smart contracting platform

        Ethereum has faced many challenges as the pioneer of smart contracts.

        • Performance: Underpriced opcodes have been attacked as a spam or DoS vector
        • High gas fees: Overwhelming demand low block-space supply
        • Frontrunning: Inserting transactions before, after, or in place of others in order to economically benefit
        • Hacks: Many hacks have exploited smart contract vulnerabilities
        • Problems with aggregating smart contract protocols together
        • Storage bloat: Misaligned incentives with burden of storage

        ---v

        Idiosyncrasies

        • Everything is 256bits
        • No floating point arithmetic
        • Revert
        • Reentrancy
        • Exponential memory expansion cost

        Gas

        Turing completeness and the Halting Problem

        • EVM: Turing-complete instruction set

        • But what about the Halting Problem?

        • Obviously cannot allow infinite loops

        • Solution: Gasometer, a way to pre-pay for each opcode execution

        Notes:

        The Halting Problem tells us that it's not possible to know that an arbitrary program will properly stop. To prevent such abuse, we check that there is gas remaining before every single opcode execution. Since gas is limited, this ensures that no EVM execution will run forever and that all work performed is properly paid for.

        ---v

        Gasometer

        • Checks before each opcode to make sure gas can be paid
        • Safe: prevents unpaid work from being done
        • Deterministic: results are unambiguous
        • Very inefficient: lots of branching and extra work

        Notes:

        This not only makes it possible to prevent abuse, but crucially allows nodes to agree on doing so. A centralized service could easily impose a time limit, but decentralized nodes wouldn't be able to agree on the outcome of such a limit (or trust each other).

        ---v

        Gas limits and prices

        Gas: unit of account for EVM execution resources.

        • gas_limit: specifies the maximum amount of gas a txn can pay
        • gas_price: specifies the exact price a txn will pay per gas

        A txn must be able to pay gas_limit * gas_price in order to be valid.

        Notes:

        This amount is initially deducted from the txn's sender account and any remaining gas is refunded after the txn has executed.

        ---v

        EIP-1559

        An improvement to gas pricing mechanism.

        gas_price --> max_base_fee_per_gas
                  \-> max_priority_fee_per_gas
        
        • Separates tip from gas price
        • base_fee is an algorithmic gas price, this is exactly what is paid and is burned
        • ...plus maybe tip if (base_fee < max_base_fee + max_priority_fee)
        • Algorithmic, congestion-based multiplier controls base_fee

        Notes:

        https://eips.ethereum.org/EIPS/eip-1559 Introduced in London hard-fork.

        ---v

        OOG and Gas Estimation

        If a txn exhausts its gas_limit without finishing, it will produce an OOG (out-of-gas) error and all changes made in the EVM are reverted (except for fee payment).

        In order to estimate the amount of gas a txn will need, an RPC method (eth_estimateGas) can perform a dry-run of the txn and record the amount used.

        However, there are a few caveats:

        • Run against current state (state may change)
        • The RPC node could lie to you
        • This is expensive infrastructure overhead and can be a spam vector

        Account Types

        There are two types of Ethereum accounts. Both use 160-bit account IDs and can hold and send Ether, but they are controlled very differently.

        ---v

        Account Types

        Externally-owned Account (EOA)

        • Traditional user-controlled account
        • Controlled via private keys
        • Account ID generated by hashing public key
        • Uses an incrementing nonce to prevent replay attacks

        Contract Account

        • Controlled by immutable bytecode
        • May only ever do precisely what the code specifies
        • Account ID generated deterministically when bytecode is deployed

        Transactions

        A transaction is a signed payload from an EoA which contains details about what the transaction should do and how it will pay for itself.

        ---v

        Transactions fields:

        • value: 0 or more Ether to send with the txn
        • to: the target of this transaction
        • input: Optional input data for creating or calling a contract
        • gas_limit: Max gas the txn will pay
        • gas_price: (or EIP-1559 equivalent)
        • nonce: prevents replay attacks and forces ordering
        • signature: proves ownership of private keys, allows receiving Account ID
        • there can be more

        ---v

        Possible use-cases:

        • Call a contract's external function (input specifies function and arguments)
        • Create a contract (input specifies contract's bytecode)
        • Neither (input empty)

        In all cases, Ether can be sent (Neither being a normal Ether send).

        ---v

        Transaction Validity

        Before executing (or gossiping) txns, some validity checks should be run:

        • Is the gas_limit sufficient? (21_000 minimum at least pays for processing)
        • Is the signature valid? (Side effect: public key recovered)
        • Can the account pay for gas_limit * gas_price?
        • Is this a valid (and reasonable) nonce for the account?

        Notes:

        These checks come with overhead, so it's important to discard invalid txns as quickly as possible if it is invalid. This includes not gossiping it to peers, who have to also verify it.


        Opcodes and Bytecode

        An opcode is a single byte which represents an instruction for the VM to execute.

        The EVM executes bytecode one opcode at a time until it is done, explicitly halts, or the gasometer runs out of gas.

        Notes:

        Functions compile down into a sequence of opcodes, which we call bytecode. This bytecode is bundled together and becomes the on-chain contract code.


        ABI

        ABI ("Application Binary Interface") describes the bytecode for a contract by annotating where functions and other objects exist and how they are formatted.

        ---v

        Exercise

        Review this Contract Code on Etherscan

        ---v

        Sandboxed Contract State

        Contract Accounts contain a sandboxed state, which stores everything that the contract writes to storage. Contracts may not write to storage outside of their own sandbox, but they can call other contracts whose bytecode might write to their respective storage.

        ---v

        Calling Contracts

        Contract functions can be invoked in two ways different ways:

        • EoAs can call a contract functions directly
        • Contracts can call other contracts (called "messaging")

        ---v

        Types of contract messaging

        • Normal call: Another contract is called and can change its own state
        • staticcall: A "safe" way to call another contract with no state changes
        • delegatecall: A way to call another contract but modify our state instead

        Notes:

        Transactions are the only means through which state changes happen.

        ---v

        Message Object

        Within the context of a contract call, we always have the msg object, which lets us know how we were called.

        msg.data (bytes): complete calldata (input data) of call
        msg.gas (uint256): available gas
        msg.sender (address): sender of the current message
        msg.sig (bytes4) first 4 bytes of calldata (function signature)
        msg.value (uint256) amount of Ether sent with this call
        

        ---v

        Ether Denominations

        Ether is stored and operated on through integer math. In order to avoid the complication of decimal math, it's stored as a very small integer: Wei.

        1 Ether = 1_000_000_000_000_000_000 Wei (10^18)
        

        Notes:

        Integer math with such insignificant units mostly avoids truncation issues and makes it easy to agree on outcomes.

        ---v

        Named Denominations

        Other denominations have been officially named, but aren't as often used:

        wei                 = 1 wei
        kwei (babbage)      = 1_000 wei
        mwei (lovelace)     = 1_000_000 wei
        gwei (shannon)      = 1_000_000_000 wei
        microether (szabo)  = 1_000_000_000_000 wei
        milliether (finney) = 1_000_000_000_000_000 wei
        ether               = 1_000_000_000_000_000_000 wei
        

        gwei is often used when talking about gas.


        Programming the EVM

        The EVM is ultimately programmed by creating bytecode. While it is possible to write bytecode by hand or through low-level assembly language, it is much more practical to use a higher-level language. We will look at two in particular:

        • Solidity
        • Vyper

        Solidity

        • Designed for EVM
        • Similar to C++, Java, etc.
        • Includes inheritance (including MI)
        • Criticized for being difficult to reason about security

        ---v

        Basics

        // 'contract' is analogous to 'class' in other OO languages
        contract Foo {
            // the member variables of a contract are stored on-chain
            public uint bar;
        
            constructor(uint value) {
                bar = value;
            }
        }
        

        ---v

        Functions

        contract Foo {
            function doSomething() public returns (bool) {
                return true;
            }
        }
        

        ---v

        Modifiers

        A special function that can be run as a precondition for other functions

        contract Foo {
            address deployer;
        
            constructor() {
                deployer = msg.sender;
            }
        
            // ensures that only the contract deployer can call a given function
            modifier onlyDeployer {
                require(msg.sender == deployer);
                _; // the original function is inserted here
            }
        
            function doSomeAdminThing() public onlyDeployer() {
                // this function can only be called if onlyDeployer() passes
            }
        }
        

        Notes:

        Although Modifiers can be an elegant way to require preconditions, they can do entirely arbitrary things, and auditing code requires carefully reading them.

        ---v

        Payable

        contract Foo {
            uint256 received;
            // this function can be called with value (Ether) given to it.
            // in this simple example, the contract would never do anything with
            // the Ether (effectively meaning it would be lost), but it will faithfully
            // track the amount paid to it
            function deposit() public payable {
                received += msg.value;
            }
        }
        

        Notes:

        The actual payment accounting is handled by the EVM automatically, we don't need to update our own account balance.

        ---v

        Types: "Value" Types

        contract Foo {
            // Value types are stored in-place in memory and require
            // a full copy during assignment.
            function valueTypes() public {
                bool b = false;
        
                // signed and unsigned ints
                int32 i = -1;
                int256 i2 = -10000;
                uint8 u1 = 255;
                uint16 u2 = 10000;
                uint256 u3 = 99999999999999;
        
                // fixed length byte sequence (from 1 to 32 bytes)
                // many bitwise operators can be performed on these
                bytes1 oneByte = 0x01;
        
                // address represents a 20-byte Ethereum address
                address a = 0x1010101010101010101010101010101010101010;
                uint256 balance = a.balance;
        
                // also: Enums
        
                // each variable is an independent copy
                int x = 1;
                int y = x;
                y = 2;
                require(x == 1);
                require(y == 2);
            }
        
        }
        

        ---v

        Types: "Reference" Types

        contract Foo {
            mapping(uint => uint) forLater;
        
            // Reference types are stored as a reference to some other location.
            // Only their reference must be copied during assignment.
            function referenceTypes() public {
                // arrays
                uint8[3] memory arr = [1, 2, 3];
        
                // mapping: can only be initialized from state variables
                mapping(uint => uint) storage balances = forLater;
                balances[0] = 500;
        
                // dynamic length strings
                string memory foo = "<3 Solidity";
        
                // also: Structs
        
                // Two or more variables can share a reference, so be careful!
                uint8[3] memory arr2 = arr;
                arr2[0] = 42;
                require(arr2[0] == 42);
                require(arr[0] == 42); // arr and arr2 are the same thing, so mod to one affects the other
            }
        }
        

        ---v

        Data Location

        Data Location refers to the storage of Reference Types. As these are passed by reference, it effectively dictates where this reference points to. It can be one of 3 places:

        • memory: Stored only in memory; cannot outlive a given external function call
        • storage: Stored in the contract's permanent on-chain storage
        • calldata: read-only data, using this can avoid copies

        ---v

        Data Location Sample

        contract DataLocationSample {
            struct Foo {
                int i;
            }
        
            Foo storedFoo;
        
            // Data Location specifiers affect function arguments and return values...
            function test(Foo memory val) public returns (Foo memory) {
                // ...and also variables within a function
                Foo memory copy = val;
        
                // storage variables must be assigned before use.
                Foo storage fooFromStorage = storedFoo;
                fooFromStorage.i = 1;
                require(storedFoo.i == 1, "writes to storage variables affect storage");
        
                // memory variables can be initialized from storage variables
                // (but not the other way around)
                copy = fooFromStorage;
        
                // but they are an independent copy
                copy.i = 2;
                require(copy.i == 2);
                require(storedFoo.i == 1, "writes to memory variables cannot affect storage");
        
                return fooFromStorage;
            }
        }
        

        ---v

        Enums

        contract Foo {
            enum Suite {
                Hearts,
                Diamonds,
                Clubs,
                Spades
            }
        
            function getHeartsSuite() public returns (Suite) {
                Suite hearts = Suite.Hearts;
                return hearts;
            }
        }
        

        ---v

        Structs

        contract Foo {
            struct Ballot {
                uint32 index;
                string name;
            }
        
            function makeSomeBallot() public returns (Ballot memory) {
                Ballot memory ballot;
                ballot.index = 1;
                ballot.name = "John Doe";
                return ballot;
            }
        }
        

        Solidity Hands-On

        ---v

        Dev Environment

        We will use the online Remix IDE for our sample coding. It provides an editor, compiler, EVM, and debugger all within the browser, making it trivial to get started.

        https://remix.ethereum.org

        ---v

        Flipper Example

        Code along and explain as you go

        ---v

        Exercise: Multiplier

        • Write a contract which has a uint256 storage value
        • Write function(s) to multiply it with a user-specified value
        • Interact with it: can you force an overflow?

        Overflow checks have been added to Solidity. You can disable these by specifying an older compiler. Add this to the very top of your .sol file:

        pragma solidity ^0.7.0;
        

        ---v

        Bonus:

        • Prevent your multiplier function from overflowing
        • Rewrite this prevention as a modifier noOverflow()

        The constant constant public MAX_INT_HEX = 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff; may be helpful.


        Vyper

        • Also designed for the EVM
        • Similar to Python
        • Intentionally lacks some features such as inheritance
        • Auditable: "Simplicity for the reader is more important than simplicity for the writer"

        ---v

        Compared to Solidity

        Vyper mostly lacks features found in Solidity, all in the spirit of improving readability. Some examples:

        • No Inheritance
        • No modifiers
        • No function overloading
        • No recursive calling (!)
        • No infinite-loops

        ---v

        Basics

        # There is no `contract` keyword.
        # Like Python Modules, a contract is implicitly
        # scoped by the file in which it is found.
        
        # storage variables are declared outside of any function
        bar: uint
        
        # init is used to deploy a contract and initialize its state
        @external
        def __init__(val):
            self.bar = val
        

        ---v

        Functions

        @external
        def doSomething() -> bool:
            return True
        

        ---v

        Decorators and Payable

        # Vyper contains decorators for restricting functions:
        
        @external # function can only be called externally
        @internal # function can only be called within current context
        @pure # cannot read state or environment vars
        @view # cannot alter contract state
        @payable # function may receive Ether
        
        # also, to cover the most common use case for Solidity's modifiers:
        @nonreentrant(<unique_key>) # prevents reentrancy for given id
        

        Notes:

        source: https://docs.vyperlang.org/en/stable/control-structures.html#decorators-reference

        ---v

        Types

        # value types are small and/or fixed size and are copied
        @external
        def valueTypes():
            b: bool = False
        
            # signed and unsigned ints
            i: int128 = -1
            i2: int256 = -10000
            u: uint128 = 42
            u2: uint256 = 42
        
            # fixed-point (base-10) decimal values with 10 decimal points of precision
            # this has the advantage that literals can be precisely expressed
            f: decimal = 0.1 + 0.3 + 0.6
            assert f == 1.0, "decimal literals are precise!"
        
            # address type for 20-byte Ethereum addresses
            a: address = 0x1010101010101010101010101010101010101010
            b = a.balance
        
            # fixed size byte arrays
            selector: bytes4 = 0x12345678
        
            # bounded byte arrays
            bytes: Bytes[123] = b"\x01"
        
            # dynamic-length, fixed-bounds strings
            name: String[16] = "Vyper"
        
        # reference types are potentially large and/or dynamically sized.
        # they are copied-by-reference
        @external
        def referenceTypes():
            # fixed size list.
            # It can also be multidimensional.
            # all elements must be initialized
            list: int128[4] = [1, 2, 3, -4]
        
            # bounded, dynamic-size array.
            # these have a max size but initialize to empty
            dynArray: DynArray[int128, 3]
            dynArray.append(1)
            dynArray.append(5)
            val: int128 = dynArray.pop() # == 5
        
            map: HashMap[int128, int128]
            map[0] = 0
            map[1] = 10
            map[2] = 20
        

        ---v

        Enums

        enum Suite {
            Hearts,
            Diamonds,
            Clubs,
            Spades
        }
        
        # "hearts" would be considered a value type
        hearts: Suite = Suite.Hearts
        

        ---v

        Structs

        struct Ballot:
            index: uint256
            name: string
        
        # "someBallot" would be considered a reference type
        someBallot: Ballot = Ballot({index: 1, name: "John Doe"})
        name: string = someBallot.name
        

        Vyper Hands-On

        ---v

        Remix Plugin

        Remix supports Vyper through a plugin, which can be easily enabled from within the IDE. First, search for "Vyper" in the plugins tab:

        ---v

        Remix Plugin

        Use Vyper through the new Vyper tab and use "Remote Compiler"


        Reentrancy

        The DAO Vulnerability

            function withdraw() public {
                // Check user's balance
                require(
                    balances[msg.sender] >= 1 ether,
                    "Insufficient funds.
         Cannot withdraw"
                );
                uint256 bal = balances[msg.sender];
        
                // Withdraw user's balance
                (bool sent, ) = msg.sender.call{value: bal}("");
                require(sent, "Failed to withdraw sender's balance");
        
                // Update user's balance.
                balances[msg.sender] = 0;
            }
        

        We make a call to withdraw user's balance before updating our internal state.

        ---v

        How can this be avoided?

        • Commit state BEFORE contract call
        • Modifier that prevents reentrancy (Solidity)
        • @nonreentrant decorator (Vyper)

        Storing Secrets On-Chain

        Can we store secrets on-chain?

        What if we want to password-protect a particular contract call?

        Obviously we can't store any plaintext secrets on-chain, as doing so reveals them.

        ---v

        Storing Hashed Secrets On-Chain

        What about storing the hash of a password on chain and using this to verify some user-input?

        Accepting a pre-hash also reveals the secret. This reveal may occur in a txn before it is executed and settled, allowing someone to frontrun it.

        ---v

        Verifying with commit-reveal

        One potential solution is a commit-reveal scheme where we hash our reveal with some salt, then later reveal it.

        // stored on-chain:
        secret_hash = hash(secret)
        
        // first txn, this must execute and settle on chain before the final reveal.
        // this associates a user with the soon-to-be-revealed secret
        commitment = hash(salt, alleged_secret)
        
        // final reveal, this must not be made public until commitment is recorded
        reveal = alleged_secret, salt
        verify(alleged_secret == secret)
        verify(commitment == hash(salt, alleged_secret))
        

        ---v

        Alternative: Signature

        Another approach is to use public-key cryptography. We can store the public key of some key pair and then demand a signature from the corresponding private-key.

        This can be expanded with multisig schemes and similar.

        How does this differ from the commit-reveal scheme?

        Notes:

        Commit-reveal requires that a specific secret be revealed at some point for verification. A signature scheme provides a lot more flexibility for keeping the secret(s) secure.

        Contract Workshop

        Web2 digital cash example.

        ๐Ÿ“ฅ Clone to start: Contract Workshop

        See the README included in the repository for further instructions.

        Start Your Own Blockchain

        In this activity, you will get hands on experience operating a blockchain node and seeing it interact with other blockchain nodes.

        ๐Ÿ“ฅ Clone to start: Academy PoW Node

        See the README included in the repository for further instructions.

        Learning objectives:

        1. How to run a node
        2. What the log messages mean
        3. Preview: How to configure a network with a chain spec
        4. Experience peer connections coming and going
        5. Practice checking for common not-peering issues (different genesis, firewall)

        Outline

        Clone & Compile

        โฐ Because the compile time is quite long, this should be done in advance of class.

        Ideally, compilation will be given to you as as homework on the day before to run (at worst) over night. The faculty may also have a docker image ready to go, please request this if needed.

        Generate User Keys

        With Polkadot JS

        Get Tokens

        You request tokens from the teacher by sharing your address with the teachers in a public channel. After the first five you have tokens, we pass them on to your peers. This manual approach allows you to get familiar with sending and receiving tokens in your wallet.

        Optionally, we may install a faucet or UBI pallet in the chain to allow you to self service after the initial manual distribution.

        Optional Treasure Hunt

        You could have tokens stored at eg the Alice key or other leaked keys and challenge you to find and recover the tokens. Make up a story/myth about some mysterious figure who lost your key and left clues, etc.

        Run Nodes

        You start your own nodes and join the network

        Start Mining

        Node runners can contribute PoW hashrate to the network in exchange for tokens.

        Fork the Network

        In BA we forked the difficulty, but maybe there is something more compelling. Maybe fork to adjust max block size. Nice because it happened for real in bitcoin. Maybe hard because I believe max block size is controlled in the runtime.

        Another idea: When we launch the chain, we allow PoW block to have either of two valid kinds of seals. For example, one based on sha3 and another based on keccak. Then we pretend there is a cultural war and some people like one function or the other hand some people don't care. When the time comes to fork, we have three different versions of the node available: 1 only accepts sha3, 2 only accepts keccak, 3 is the original that accepts either. In this way we can see that there will be at least two viable chains (the sha 3 chain and the keccak chain). If there are a lot of nodes that still run the old version and accept either hash, we will form a third chain. But if there are only a few such nodes, we will re-org into one of the other two chains, and potentially reorg back and forth between them.

        Smart Contracts

        The Smart Contracts module has an activity about launching smart contracts on a running chain. When these modules are run back-to-back, it makes an excellent learning experience to use this same chain we just launched to also launch the smart contracts.

        Blockchain Forks


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: Blockchain Forks description: Detailed Classification for Blockchain Forks Types duration: 60 minutes ---

        Blockchain Forks


        Landscape

        ---v

        Landscape

        Ideal World

        In an ideal world blockchains would look like this:

        ---v

        Landscape

        Real World

        Things don't always go according to plan:


        ---v

        Landscape

        Chaotic Real World

        And sometimes they get extra messy:



        What's the goal?

        ---v

        What's the goal?

        Fork Identification

        Notes:

        There are different forks, they can have different shapes and reasons. We'll try to identify some examples.

        ---v

        What's the goal?

        Fork Categorization

        Notes:

        Why? Forks in the same category will exhibit similar behavior and will require similar handling. Then when making changes we can easily figure out to which box the change belongs and react accordingly.

        It's also worth pointing out that the whole web3 space is still very young and we are still figuring out how to name things appropriately. There is definitely still a lot of confusion about the fork types and the convention I will be using here today is based on the naming proposed by the MIT Digital Currencies Initiative. It will cover most of the common terms and hopefully will not be as self-contradictory as some of the terms used within the community.

        ---v

        What's the goal?

        Fork Confusion


        BABE (sometimes):

        Notes:

        To demonstrate the confusion think of BABE. BABE can have multiple block authors create blocks at the same time and then the chain forks. What type of a fork is it? Some of you might have heard about soft forks and hard forks, any idea which one it is?


        Fork Categorization

        ---v

        Fork Categorization

        Forks Family Tree


        Notes:

        This is the core categorization of forks we'll be waking you through today. You don't need to understand the whole tree as we'll be going step by step. For now let's actually go back to the example from BABE and let's place it on the map.


        Transitory Forks

        Notes:

        For that we'll talk about transitory forks.

        ---v

        Fork Categorization

        Transitory Forks


        Notes:

        One of the simplest forks that is rarely talked about so the name might not sound familiar. They are often also called short-lived forks or temporary forks.

        ---v

        Transitory Forks


        Notes:

        They usually arise from the fundamental protocol uncertainty or networking lag, but luckily they are usually resolved quickly by the network itself. For instance in BABE even if all the nodes are running the same software those forks can still happen when two nodes roll low enough numbers. In Bitcoin two miners might mine a block at roughly the same time. Over time one of the blocks wins due to some heuristics like the longest chain. Those forks are generally not a problem and only live on short timescales.


        Consensus Forks

        Notes:

        So now let's move to something more interesting. Consensus forks.

        ---v

        Fork Categorization

        Consensus Forks


        Notes:

        Usually when you hear about forks you hear about those guys. They are the other branch and they also come in many flavours we'll discuss in a minute.

        ---v

        Consensus Forks

        Validity Set

        Notes:

        But first before we understand the intricacies of consensus forks we need to understand the concept of the validity set and how it connects to the protocol.

        ---v

        Consensus Forks

        Validity Set


        Notes:

        It's best to see it through an example so let's look at the BTC block. You don't need to understand all the fields that are in it but for now take a look at the blocksize field as well as the block header itself.

        Validity Set


        ---v

        Consensus Forks

        Validity Set


        ---v

        Consensus Forks

        Validity Set


        Notes:

        So the validity set is a set of all hypothetical blocks that could be produced by the protocol. It's a set of all valid blocks under those rules.

        So if there is a block D that for instance is too big, it's blocksize is bigger than the allowed one...

        ---v

        Consensus Forks

        Validity Set


        Notes:

        Then it falls out of the validity set into the universal set of all possible data blobs. Only some of those data blobs are valid blocks.

        ---v

        Consensus Forks

        Validity Set


        Notes:

        Let's actually look at an example. Imagine this is Bitcoins validity set and here we see a few blocks from it. The numbers at the top are initial few digits from the hashes representing those blocks.

        Imagine all the Bitcoin nodes suddenly decide they really dislike when the first hash digit is odd. They only like when it's even so they band together and change the protocol to only accept hashes with an even first digit.

        ---v

        Consensus Forks

        Validity Set


        Notes:

        This change in the protocol would reduce the validity set. It would be more constrained then before. Some of the previously valid blocks would no longer be valid under the new rules. What happens in that case? Can we predict that?

        ---v

        Consensus Forks

        Validity Set

        N โŠ† O

        Notes:

        To generally represent the same idea we'll be using the simpler representation on the right. Where the new set N is contained within the old set O. The fancy notion at the bottom says the same N is contained in O.


        Soft Forks

        To understand the example from a second ago we'll dive into soft forks.

        ---v

        Fork Categorization

        Soft Forks


        Notes:

        Firstly soft forks are a type of a consensus fork and they are results of a change in the protocol and thus the validity set.

        ---v

        Fork Categorization

        Soft Forks

        N โŠ† O
        • Backwards Compatible
        • By making the consensus rules more restrictive the set of valid blocks gets smaller.
        • Not every (and often none) of the blocks produced under old rules will get accepted by new nodes.

        Notes:

        So based on the venn diagram we have on the side we can see that the new consensus rules are more restrictive as the validity set shrinks.

        New nodes produce blocks that are always accepted by old nodes. Old nodes generally don't produce blocks accepted by the new nodes.

        Before we jump into demonstration is decreasing or increasing blocksize a soft fork?

        ---v

        Fork Categorization

        Soft Forks

        N โŠ† O
        Examples:

        • Decreasing blocksize
        • Accepting only even/odd hashes
        • Disallowing some transaction types

        Notes:

        Decreasing the block size restricts how many different blocks can be constructed so it makes the set smaller. It is a soft fork. The example we had a moment ago with the even hashes is also a soft fork as it adds another constraint on the previous protocol rules restricting them even further. Another good example would be banning some transaction types from use.

        Now let's take a look at how forks work in practice and how they vary based on the hash power or staking power in favour of the protocol change.

        ---v

        Fork Categorization

        Soft Forks

        N โŠ† O

        Notes:

        So in this scenario we'll be looking at what happens if nodes with less than 50% hash power or stake want to soft fork. Remember that soft forking is simply making the consensus stricter.

        In that case the blocks produced by new nodes are marked with N and they are accepted by the old chain but the old chain mines faster so they don't care about the new nodes. Blocks produced by old nodes are NOT accepted by the new nodes so the longest chain for new nodes is the short chain with only the N blocks. This is effectively a permanent fork.

        ---v

        Fork Categorization

        Soft Forks

        N โŠ† O

        Notes:

        In the similar example when the new nodes control more than 50% power the situation changes drastically. The new nodes mine faster and are the longest chain. But remember that old nodes accept the new blocks so if they new nodes mine faster the old nodes blocks get constantly reorged out. They are forced to update the software if they want their blocks to get accepted at all otherwise they loose all the rewards.


        Hidden Forks

        Notes:

        Now let's take a look at something a bit less known. Hidden forks.

        ---v

        Fork Categorization

        Hidden Forks


        Notes:

        An edge case of soft forks.

        ---v

        Fork Categorization

        Hidden Forks

        N โŠ† O
        • Conflict-less
        • The old, now excluded blocks were allowed but never used in practice.
        • New nodes are theoretically stricter but practically accept all old blocks.
        • Old nodes accept new blocks.

        Notes:

        So the venn diagram is exactly the same as in the normal soft fork case. But imagine that the the orange crescent, this is the part we're excluding from the old protocol when shifting to the new one... was never actually used. So for instance the block had an empty field that could have some arbitrary data inside, but everyone left it empty and never checked what was inside. The new protocol puts something meaningful in the empty field but doesn't require it. Because old nodes never used this field pretty much all old blocks will be accepted under the new rules.

        TL;DR the stuff we removed from the validity set wasn't even used despite being technically valid.

        ---v

        Fork Categorization

        Hidden Forks

        N โŠ† O
        Examples:

        • Assigning non-conflicting uses to empty opcodes.
        • BTC Ordinals using empty opcodes to implement BTC NFTs.

        Notes:

        A good example of that assigning new optional use-cases for previously unused opcodes as with the example of the recent Bitcoin Ordinals update.

        ---v

        Fork Categorization

        Hidden Forks

        N โŠ† O

        Notes:

        And the reason why they are hidden... is they are not even manifesting as forks despite the consensus change. All nodes effectively accept each others blocks so there is no conflict.


        Hard Forks

        Notes:

        ---v

        Fork Categorization

        Hard Forks


        ---v

        Fork Categorization

        Hard Forks

        O โŠ† N
        • Forwards Compatible
        • By making the consensus rules less restrictive the set of valid blocks gets bigger.
        • Not every (and often none) of the blocks produced under new rules will be accepted by the old nodes.
        • Every block produced under old rules will get accepted by new nodes.

        ---v

        Fork Categorization

        Hard Forks

        O โŠ† N
        Examples:

        • Increasing blocksize
        • BTC Cash fork at first*
        • Adding new transaction types
        • Increasing max nonce value

        ---v

        Fork Categorization

        Hard Forks

        O โŠ† N

        Notes:

        First let's go through the scenario of a hard fork with less than 50% support. Remember that the rules were loosened this time. If new guys accept old blocks so because they have less than 50% power they get constantly reorged out. There is no permanent fork in this scenario and the change will not go through if if it has this limited support.

        ---v

        Fork Categorization

        Hard Forks

        O โŠ† N

        Notes:

        In the case of more than 50% support the new guys miner faster but they are no accepted by the old nodes so they go ahead. Old nodes maintain the old chain and the community is split. So if there is a major change that most people accept but not everyone it will always fork the chain.


        Small Summary



        Notes:

        Now that we've seen both soft and hard forks... If we'd manually increase the difficulty of mining in the BTC network would that be a soft or hard fork? Hard.

        Also let's reiterate. We only have permanent forks in soft forks with under 50% support and in hard forks with over 50% support.


        Full Forks

        ---v

        Fork Categorization

        Full Forks


        ---v

        Fork Categorization

        Full Forks

        O โˆฉ N = โˆ…
        • Fully Incompatible
        • Soft + Hard
        • By changing the consensus rules the sets can become disjoint or overlapping.
        • Most (and often all) blocks produced under one ruleset are not accepted under the other.

        ---v

        Fork Categorization

        Full Forks

        O โˆฉ N = โˆ…
        Examples:

        • Changing the hashing function
        • Changing the signature scheme
        • Specific combinations of soft and hard forks
        • BTC Cash fork in the end*

        ---v

        Fork Categorization

        Full Forks

        O โˆฉ N = โˆ…

        Summary



        Notes:

        • Bitcoin cash pivot from hard to full because they didn't have enough HP.
        • Soft are often preferred for changes because with >50%HP they do not fracture the community (BTC community logic)
        • Hard can be preferred as they seem to better represent minorities. If some people don't agree with the majority they naturally fork off and are not peer pressured to follow (ETH community logic)

        Thank you!


        ink! Workshop


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: ink! Workshop description: An introduction to the ink! workshop. duration: 20 min ---

        Notes:

        This workshop was an idea that came from first wave of PBA in Cambridge. The first time it was ever played was back in Buenos Aires. It's very much a PBA right of passage, so it's exciting to be able to play today.


        Day 1


        Notes:

        This is us playing in BA earlier this year.


        Components


        • We deploy + run.
        • Runs game loop.
        • Invokes each player.
        • Determines the score.
        • On big screen during game.

        Notes:

        The game has three components:

        • click the first is a game contract, which is something that we have taken care of. I will deploy it and run it on Rococo. The game contract runs the game loop. It invokes each player contract and it determines the score.
        • click And the player.contract, that's your job. so we we have a template for a very basic player and you can basically modify it in any way you like. I will explain the scoring function in a moment.
        • click Lastly, we have a front-end that we will put on the big screen here, but it's online, you can open it up on your laptop and follow the game locally.

        The Process

        1. ๐Ÿง  Create contract that plays on your behalf
        2. ๐Ÿš€ Deploy contract to Rococo testnet
        3. ๐Ÿค Register contract as player with game contract
        4. ๏ธ๐ŸŽฎ We start the game
        5. ๐Ÿ“บ๏ธ๏ธ It runs for some minutes, we watch it on screen

        Notes:

        So the process is as follows:

        • click - you brainstorm and create a contract that plays on your behalf ideally in a better way than the other contracts
        • click - you deploy the contract to the Rococo testnet
        • click - you register your contract as a player with the game contract. We will publish the addresses and it's it's not complicated you can just use contracts UI. I will also show it in a second
        • click - then we start the game. We have a script that calls the game contract regularly and the front end will display the changes
        • click - so the game will run for some minutes. Your contract will already be uploaded at this point - so you can't do anything. The contract will play on your behalf. Meaning it's hands off. I mean you can also change it with upgradable contract patterns and stuff, but in general, you won't have to. So you can just watch the game play out. If you're familiar with AI agents, that's also a similar idea, where you have an agent that plays a game on your behalf.

        How the Game looks

        Notes:

        This is what the game board itself looks like. You can see, it's a it's a coordinate grid.


        How the Game looks

        Notes:

        this is X1 y0. And the idea is for you, as a contract developer, to paint as many fields of this canvas of the script as possible. There's an API function for how to paint. I'll show it in a minute, but the idea is that in the end the contract that painted the most Fields wins.


        How the Game looks

        Notes:

        So yeah, there's bit of scoring to be done. There's also scoreboard here on the side, where all players will be shown as well as their rank. You will be assigned a random color to your player contract.


        How to score?

        • Use as little gas as possible to paint as many fields as possible.
        • Stay within your gas budget.
        • The later you manage to still paint a field the better you score.
        • No overpainting! First player to paint a field owns it.

        Notes:

        This game was designed in a way where all of the best practices for smart contract development will make your player run better. We really tried to gamify some of the concepts that are really best practices of how you should develop a smart contract.

        • click - first one is to use as little gas as possible to paint as many fields as possible. Gas consumption is a super big thing as user fees derived from that and the transaction throughput also relates to how big a contract is so the less complex your contract is the better it will do.
        • click - There's a certain gas budget allocated per player and if you if you're above that then you just won't make a turn. So you have to stay in an optimal gas budget.
        • click - the later in the in the game that you still manage to paint the field the better you score because what will happen is that there will be all kinds of fields painted and it will convert to a place where there's only a few left. so if you have a player that just randomly tries to paint things then at some at some point it just won't no longer work because it doesn't paint the unpainted fields. But in order to find those, you will have to have some more complex logic in your contract.
        • click - and lastly, there's no overpainting. So the first player to paint a field owns it. So if you have a contract that just paints the same field over and over again, it will just not work. So you have to have some logic in there that checks whether a field is already painted or not.

        Basic Player Contract

        #![allow(unused)]
        fn main() {
        #[ink::contract]
        mod player {
            #[ink(storage)]
            pub struct Player {}
        
            impl Player {
                #[ink(constructor)]
                pub fn new() -> Self {
                    Self {}
                }
        
                /// Called during every game round.
                /// Return `(x, y)` coordinate of pixel you want to color.
                #[ink(message, selector = 0)]
                pub fn your_turn(&self) -> Option<(u32, u32)> {
                    Some(1, 2)
                }
            }
        }
        }

        Notes:

        So this is a very basic player contract. We have a repository app set up with the template for this - I will share the link in a second. How it looks is - it's a contract that defines the player module.

        • click - a super simple player doesn't have to contain any storage whatsoever.
        • click- a super simple player also doesn't have to contain any Constructor arguments
        • click - and it can just return a random constant value. So this is the most simple player ever. What your player needs to do is to have one message, that has a defined selector, so you don't have to think more about this. It's just that it needs to have this function and this function is called by the game with every round. And whatever you return from this function. This is your your turn for the game. So if in this case you return an option some(1,2), this would mean you paint the pixel at X1 Y2. In that case, it would be a very boring play. Who can tell me what would happen here? (a player that only makes one turn - it always tries to paint the same field. And if someone else also chose the same magic numbers then it wouldn't even make a turn at all.)
        • click - So there's a couple things of how you can improve on that and I will show some hints later, but for now what we're gonna do is look at the repository and see how we can get started with this.

        How to play

        Notes:

        This is the Squink-Splash-beginner Repository - it contains a couple of things.


        How to play

        Notes:

        It contains the cargo.toml file and the lib.rs, which is the player that I just showed.


        How to play

        Notes:

        It contains the game metadata. You will need this in order to interact with the game itself, to register your player and so on. but yeah we will show that.


        How to play

        Notes:

        And it has 2 todo files. The first is some instructions on General setup and the second one is in order to build your player.


        Now (1)

        github.com/paritytech/squink-splash-beginner โžœ todo-1.md

        Notes:

        So, the first thing that we are going to do is that I would ask all of you to go to this link here and follow the instructions. We will go around and help whoever has an issue. I think you might even have already used some of the requirements in there, so it may not be super complicated.


        Now (2)

        github.com/paritytech/squink-splash-beginner โžœ todo-2.md

        Notes:

        For this stage, you will need the address of the game. We'll post it in the chat. This is a simple example - we're just giving you exposure to the flow of uploading your player contract.


        ๐Ÿ•น๏ธ๐ŸŽฎ Let's play! ๐Ÿ•น๏ธ๐ŸŽฎ

        https://splash.use.ink

        Notes:

        The next slides go over strategy. (maybe wait to to go through until you've played one game)


        The Game Contract

        ink-workshop/game/lib.rs

        • pub fn submit_turn(&mut self)
        • pub fn board(&self) -> Vec<Option<FieldEntry>>
        • pub fn gas_budget(&self) -> u64
        • pub fn dimensions(&self) -> (u32, u32)

        Notes:

        When you deployed it you already saw that there's different functions that you can call. There are a lot of functions that are interesting.

        • click - the game runner calls this - if you're interested, you can take a look how it works and there might be some hints for for the game.
        • click - then there's a function to to query the board in order to find out if certain fields are already occupied or if they are still free.
        • click - there's a function for the gas budget so you can find out how much gas your player is allowed to use per one round. Because worst thing is if you are above this gas budget, then you just won't execute any turn during that round.
        • click - there's also function to query the game dimensions. Same thing there, if you paint outside the bounds, then also you just missed one turn.

        Things to consider ๐Ÿง 

        • Develop a strategy for your player.

        • Use as little gas as possible to paint as many fields as possible.
        • Stay within your gas budget.
        • The later you manage to still paint a field, the better you score.

        • No overpainting! First player to paint a field owns it.
        • paritytech/squink-splash-advanced

        How to test locally?

        paritytech/squink-splash-advanced

        Notes:

        There's details in here. You could deploy locally to test things out. there's also commands you can use to deploy not using the UI.


        Hint: Playfield Dimensions

        • Paint within the bounds of the playfield!
        • Otherwise you wasted a turn.

        Ideas

        • You can call your own contract as often as you want!
        • Random number
        • Query which fields are free
          • Query game state via cross-contract call
          • Off-chain computation

        Notes:

        • click Lastly, some ideas for for your agent in the game you can you yourself can call your contract. As often as you want and this won't contribute to the gas used during the game. You can during the game you could also call a set of function of your contract if you want to adapt something.
        • click You could also Generate random numbers. There are a couple of libraries out there, if you want to go along those lines you have to pay attention that it's a no standard Library. typically a random number generator libraries they typically have a feature that you need to enable in order to make them no standard compatible
        • click A clever strategy would be a query which fields are free. that's a bit more complex with the with the code snippet that we have in this Advanced repository, this would give you a hint.


        Day 2



        Now

        We help you debug!

        Then

        ๐Ÿ•น๏ธ๐ŸŽฎ๐Ÿ•น๏ธ๐ŸŽฎ

        Thereafter

        Solutions Explainer


        Use as little gas as possible to paint as many fields as possible.

        Stay within your gas budget.

        The later you manage to still paint a field, the better you score.

        No overpainting! First player to paint a field owns it.


        paritytech/squink-splash-advanced


        Frontend

        https://splash.use.ink


        Questions

        • What strategy did the winner choose?
        • What strategies did the others choose?
        • What do you think would be the perfect strategy?

        Board Dimensions

        • Worst ๐Ÿ˜ฑ
          • Cross-contract call to game

        #![allow(unused)]
        fn main() {
        #[ink(message)]
        pub fn dimensions(&self) -> (u32, u32)
        }



        • Best ๐Ÿ‘๏ธ
          • const width: u32
          • new(width: u32, height: u32)

        More Pitfalls

        • Forgetting --release
        • Iterating over a datastructure in your contract

        Avoid iteration

        #[ink(message)]
        fn pay_winner()
          let winner = self.players.find(โ€ฆ);
        
          self.transfer(winner, โ€ฆ);
        }
        
        #![allow(unused)]
        fn main() {
        #[ink(message)]
        fn pay_winner(
            winner: AccountId
        ) {
          assert!(is_winner(winner));
        
          self.transfer(winner, โ€ฆ);
        }
        }
        ---

        Strategy 1
        Return Random Numbers


        Strategy 1
        Return Random Number

        • Wasm-compatible RNG
        • Use Storage to hold seed for random number
        • ๐Ÿ“ˆ Uses little Gas
        • ๐Ÿ“‰ Quickly runs into collisions
        • ๐Ÿ“‰ Score function rewards players that late in game still paint fields

        Strategy 2
        Paint only free fields


        Strategy 2
        Paint only free fields

        • Query board for free fields
        • ๐Ÿ“ˆ Succeeds late in game
        • ๐Ÿ“‰ Cross-contract call ๐Ÿ’ฐ๏ธ
        • ๐Ÿ“‰ Need to iterate over Mapping: O(n)

        Strategy 3
        Shift computation off-chain


        Strategy 3
        Shift computation off-chain

        • Off-chain Script
          • Query board โžœ Search free field

        • #![allow(unused)]
          fn main() {
          #[ink(message)]
          fn set_next_turn(turn: โ€ฆ) {}
          
          #[ink(message, selector = 0)]
          pub fn your_turn(&mut self) -> {
            self.next_turn
          }
          }

        Strategy 4
        Exploit player sorting in game loop


        Strategy 4
        Exploit player sorting in game loop

        • On top of Strategy 3 (off-chain computation).
        • Game loop calls players in same order each time.
        #![allow(unused)]
        fn main() {
        #[ink(message)]
        fn submit_turn(&mut self) {
            // -- snip --
        
            for (idx, player) in players.iter_mut().enumerate() {
                โ€ฆ
            }
        
          // -- snip --
        }
        }

        Strategy 4
        Exploit player sorting in game loop

        #![allow(unused)]
        fn main() {
        impl<T: Config> AddressGenerator<T> for DefaultAddressGenerator {
        	fn generate_address(
        		deploying_address: &T::AccountId,
        		code_hash: &CodeHash<T>,
        		input_data: &[u8],
        		salt: &[u8],
        	) -> T::AccountId {
        
            // -- snip --
        
        	}
        }
        }

        โžœ All inputs are known

        โžœ Generate low T::AccountId with known inputs


        Strategy 5
        Checking these slides already yesterday


        Additional Lessons

        Coordination and Trust in Web3


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: Coordination and Trust in Web3 description: Principles of coordination, contracts, smart contracts, and comparison with traditional legal infrastructure adn web3 infrastructure. ---

        Coordination and Trust in Web3

        Notes:

        In this module we're going to talk about Smart Contracts.


        Smart Contracts

        Two Definitions

        Broad Definition

        aka Szabo definition

        A machine program with rules that we could have defined in a contract, but instead a machine performs or verifies performance.

        Notes:

        The first question we should answer is "wtf is a smart contract"? The term has been around for a long time (longer than any blockchain platform) and has evolved over time. It was first introduced by this guy Nick Szabo in the '90s.

        CLICK - read szabo definition

        Szabo's definition is primitive and kind of vague. It might even be argued that it includes all computer programs. But it gets at the essence of his thinking at the time. Szabo studied both law and computer science and his big idea was that agreements that are traditionally legal contracts could be better expressed and enforced as computer programs. Over time, as this vision was gradually realized, the term took on a more specific meaning and was refined quite a bit.


        Smart Contracts

        Two Definitions

        Narrow Definition

        aka web3 definition

        A program that specifies how users can interact with a state machine and is deployed permissionlessly to a blockchain network

        Notes:

        click - read web3 definition. This definition is clearly more technical and precise, which is helpful in a lot of ways. At this point though, we haven't even learned all the terms in this web3 definition. As an educator, that deeply pains me. So - my promise to you - by the end of the week, this sentence will 100% make sense to you. The broad definition (szabo's - the first one we just saw) is useful in understanding the broader space, historical context, and its philosophy. This though, the narrow definition is useful when you're talking about deploying code to actual blockchain platforms. In this first lesson today, we'll focus mostly on Szabo's broader notion and explore the ways in which smart contracts are a modern extension to an ancient technology: contracts. We'll also look at some criticisms of the term "smart contract" and some alternatives that might have been better if it weren't already way too late.


        Coordination

        Examples of coordinated activities:

        • dancing, music band
        • money, business, trade
        • rowing team
        • building homes, buildings, roads

        Doing the above safely through bureaucracies like inspection and certification procedures, automobile registration

        Notes:

        Okay so let's do that! Let's rewind a bit. Because before we can understand the ways in which smart contracts are "smart" versions of contracts, we need to understand what contracts are in the first place.

        After all, we are standing on the shoulders of giants here.

        So, in order to understand what contracts are, we'll start all the way back at human coordination.

        We'll see that contracts are just a pretty good technology to achieve coordination, and that smart contracts are an improvement on that technology.

        So, let's start here. I'm going to assert that coordinated activity is the basis of so much human behavior. Consider the building we're in right now. It has technicians, janitors, grounds people to run it, not to mention the crew it took to build it. Coordination is what allows this team of workers pictured here to complete a railroad that stretches across a continent. Obviously some coordination require more bureaucracy than others. Building a railroad that crosses many state lines? That will require government approval, inspection, and certification procedures - especially as it relates to safety. But two people dancing? That most definitely could be categorized as coordinate - but can be negotiated or navigated between two individuals.


        Coordination

        Notes:

        There are so many examples of things we as humans can achieve only through coordination, not as individuals, it would be impossible to list them all. I hope it's evident that I think coordination is good a good thing! Because of it, we can build big big things and create beautiful music in a symphony together because of it.


        Coordination

        Notes:

        It is how massive organizations like corporations and governments are able to function.


        Voluntary vs Forced Coordination

        Notes:

        A clarifying attribute of coordination is that it can happen voluntarily or by force. Here, we have a picture of folks working together, let's assume it's voluntarily - as it looks like they're volunteering their time at a community garden. This is a wonderful example of human coordination at its best.


        Voluntary vs Forced Coordination

        Notes:

        But like all things, of course coordination can have it's ugly side. Pictured here, a slave boat, is categorically a coordinated effort; people are forced to work together. Most of us consider slavery a bad outcome. It IS a bad outcome. Even those with sketchier moral compasses would agree that they don't want to be the slaves.

        The point I'm making here is this: We want to create conditions (otherwise known as, incentives) where voluntary coordination can happen freely and easily. And thus make slavery as an outcome unlikely. And by building proper tools for voluntary coordination, we can help prevent forced coordination from happening in the first place. And again, voluntary coordination is a good thing between individuals - it's how we together achieve things that are impossible otherwise on our own. Thinking back to our first example of building a railroad - if I were to drive railroad spikes every day for the rest of my life, say I start in Miami where I live now? Maybe I eventually would make it to Georgia? But big collaborative coordination efforts? Railroads made it California! They stretch all over and across a continent!


        Business and Trade

        • primitive trade
        • dead weight loss
        • counterparty risk

        Notes:

        From a historical perspective, one of the first forms of coordination was simple trading. Let's imagine that I grow food and you make houses. So, I'll give you some food, if you build me a house, and I pay your daily wage by giving you food as long as you're building my house. This is a classic example of what is called primitive trade. You the house builder - you need food, so you can only build houses for people who grow food. As in, you can't earn food by building houses for randos who don't have food to pay you with. Obviously there are a whole lot of problems within this situation and it won't scale over time. But it's also a classic example of what is called 'deadweight loss' - The definition of which is: "a cost to society created by market inefficiency, which occurs when supply and demand are out of equilibrium". Another classic example would be if I, for example, grow corn in the summer and you ice fish in the winter. And together, we would love to have some way to feed each other during our off seasons - me to feed you in the summer and you to feed me in the winter - but there's no way to coordinate that because someone has to go first. So let's say I go first and I just give you half of my corn all summer. It's within reason that you could just, you know, piece out and I'mโ€ฆ left with nothing all winter. And that is what we call counterparty risk. And because I don't want to take on that counterparty risk, I don't engage in such a deal, which is then deadweight loss. So, despite the fact that we both kinda want to trade, we have no real way of making it happen. So are we stuck?? Do we just not do the deals because we're afraid of counterparty risk and deadweight loss?


        Trust

        So you want to coordinate with someone.

        But you don't want to get ripped off.

        Notes:

        Letโ€™s then move on to talk about trust - as it makes sense that it's one common way to achieve coordination. Because let's imagine, you want make an agreement, but you don't want to get ripped off. Trust means that we decide to trust whoever we're coordinating with - we're creating an agreement between one another to not to rip each other off. Say for example we're builders - you and I decide to build your house this week. Well then I in turn believe that weโ€™ll build mine next week.


        Personal/Real Trust

        vs

        Rational Expectations

        Personal trust - you can build it over time through a slow give and take like you do with your family or long term business partners. <!!-- .element: class="fragment" -->

        Notes:

        For those houses to be built, for that coordination to happen, we need credible expectations that the promises we make to one another will be upheld. AKA I have to be willing to trust you - that you are going to fulfill your end of the bargain. So here on the slide, I have two concepts: Personal or 'real' trust vs. rational expectation. CLICK - I like to think of personal trust, as a familial thing - or something that's been developed between people over time - I for example trust that my mom, who is watching my son right now, is taking good care of him. But she and I have gradually built up that trust over many years. This sort of trust happens within families, between friends or neighbors, but it can also happen in business relationships - say for example, you have a new business partner and you start that relationship off by doing say one small deal together. And yeah, you may choose to take on a little risk within the deal, but itโ€™s small. And then the next time you interact, the deal evolves and becomes a little bigger. And maybe that time the other party/person takes on that little bit of risk. At no point is there SO much risk that you could get completely wrecked if they rip you off. Again, there's some, and so by accepting that risk slowly more and more over the years, you two are able to actually build up that personal and real trust.


        Trust: A Dirty Word

        Less Trust, More Truth

        Notes:

        Okay pause - I should fully acknowledge that I'm talking a whole lot about this thing trust, and who here has heard or seen ppl in the web3 world wearing hats saying - CLICK - less trust, more truth. Anyone? (check for nods)

        So, some personal historical context on me, when I was a newbie in the web3 space, I hadnโ€™t heard this phrase (or slogan rather?). I saw it on pplโ€™s shirts and felt sort of confused and honestly, kind of excluded from what felt like maybe an inside joke. I remember thinking, wait, I trust people, does that make me wrong... Is that a bad thing? I asked myself the question, why is trust such a dirty world in our world? Well, in time, thanks to some of my fellow instructors in the room, I came to understand that this phrase is referring to the fact that life is not all sunshine and roses! Not all relationships and interactions are filled with trustful actors. In fact, most aren't! Literally every day we're in situations in which we are forced to treat complete strangers as if we've established real, authentic trust between one another. Treat them as if they're a close friend or business partner. And that is definitely bad, and is a dirty word, and is a thing that we really shouldn't do. And yet! This literally happens all the time - every day really -when you give a merchant your credit card number and you trust that they won't go off on an amazon spree with it. Or when you give your social security number to a new employer and you trust that they won't steal your identity. Or when you give your passport to a border agent and you trust that they won't sell it on the black market. Okay, end pause - back to the slides.


        Personal/Real Trust

        vs

        Rational Expectations

        • Personal trust - you can build it over time through a slow give and take like you do with your family or long term business partners.
        • Rational expectations - you can believe that a person will act faithfully because they are incentivized to do so.

        Notes:

        So. We know that we don't have the luxury of only interacting with people we truly personally trust. It's impossible to get personal trust with every person you'd in theory like to because it takes a super long time to bootstrap that real kind of relationship. And frankly, sometimes you want to interact with a stranger more immediately, say today, and not have to court them for years upon years. So, again, this should lead us to the question of - how do you handle these every day situations? How DO I trust that the coffee guy isn't going to run off with my credit card details? CLICK - Well, that's where rational expectations come into play. This is a game theory, incentives-based concept. Where you believe credibly that a person will act faithfully according to an agreement you've made - because the incentives make them do so. So, in a sorta loose way you could say this is another kind of trust? But it's not the genuine, real, authentic kind. It's more like, I trust that you will act in a certain way because you are so incentivized to do so and being truthful and upholding your end of the agreement ultimately is your best strategy and option.


        Personal Trust for Performance

        Personal trust can make things really efficient, and it's reasonable to leverage this trust when it's warranted

        Notes:

        Again, I want to make sure you're all hearing me when I say that personal/familial trust is not wrong, or bad, or dirty. I donโ€™t want us all walking out here calling our moms and dissolving our trust with them. That just isnโ€™t the scenario we speak to when we say 'trust is dirty'. And further, if you have someone who you genuinely trust, you can actually get things done a whole lot faster and more efficiently by leveraging that trust. So, it may be worth it to bootstrap genuine personal trust. You know, start small, accept a small bit of risk and gradually increase that over time. Again, it's only bad when you have to do that with someone that you don't know or you actively distrust AND they aren't incentivized to return that trust.


        Promises and Reputation

        • how enforceable is it?
        • does it work in small setting?
        • does it work in a large setting?

        Notes:

        So these incentives are obviously really important. Let's talk about what they are or could be. Often they are civic duty based - the fear that society or your town will turn on you if you break your promise. OR they're morality or religion based - you know, you keep your promise to someone out of fear of otherwise going to hell. Or they are based on the fact that you have a reputation to uphold. You don't want to be known as a liar or a cheat. I think you get what I'm saying here, yeah? It's social norms and expectations that keep people. Reputation is on the line. And this can get society pretty far. But, it's not perfect. It's not always enforceable - or rather - a threat of a bad reputation isn't always enough to keep people in line. Or a town grows enough that you can't keep track of everyone's reputation. Or, you know, you're in a big city and you don't care about your reputation because you're never going to see these people again. F trust. F reputation. Outlaw life is for me!

        From a sociological perspective, we know that if people within a society tend to keep promises, the society tends to prosper. But! Eventually, it will be sufficiently profitable to defect and to break promises. womp womp!

        And this is why we started the lecture off with discussing coordination. So let's quickly merge some of the concepts weโ€™ve discussed thus far. Coordination and trust. Because letโ€™s be real, to achieve things on this planet, it requires some coordination between the 8 billion people here. And sure, genuine personal trust can help in a small village. And promises based on societal reputation and civic incentives can help getting things done across a town. But promises don't get us all the way up to scalable, global trust. When talking with Joshy, about this problem his example, that I canโ€™t seem to get out of my head was that: โ€œIt's the modern equivalent of a primitive village deciding "okay everyone, let's all agree we're going to poop in the same corner of our village, not just wherever we feel like it.โ€ So as he says: Promises can make a village of 100 shit in the same corner, but not 8 billion. I think this is his way of saying that we need some bigger and more resilient coordination tools.


        Contracts

        • Contracts are Promises
        • Some involve money, but they're much more general than just money.
        • Traditionally, contracts are special promises that the government will enforce.

        Notes:

        So how do we incentivize people to keep their promises beyond a civic duty? Ding ding ding - Finally! This brings us to the idea of contracts. They're basically, pretty much, promises. They could involve money, a lot of them do, though they don't have to. Ultimately, think of them as abstractions over promises. And someone is enforcing the contract.

        On a very small scale, think of siblings fighting over a toy, a parent might encourage them to agree to take turns. Later when it is time to tarde, the parent might remind the child, "you said you would trade in 5 minutes, so now you have to." The parent is big enough to physically overpower the child if it were to come to that, and the child knows it, so therefore it rarely comes to that. The parent in this situation is the enforcer.

        Same thing in sports. We all agree to the rules, and agree to let a referee make the judgement calls. The ref is the enforcer. In big leagues, where there can suspend players and affect their career, this works. In the public park, it doesn't always, because the ref is just a peer and there is no real power differential.

        Traditionally what you've probably learned is that "enforcer" of these contracts (or promises) is the government. The government, by being the enforcer, can help people engage in so much voluntary coordination just by making them keep their own promises.

        And sure, that's valuable because governments are big and powerful. You could say that even in some sense, the government enforcing it is a lot like God enforcing it. As in, the government is big enough to have power over individuals interacting, and if anyone tries to rip someone else off, the government will step in and make sure the contract is upheld.

        This essentially allows us to interact as if we trusted each other. And over time, the stickiness sets in and people actually do start to trust each other! Yay, happy days - But wait! What am I talking about? We know the unfortunate truth which is that government having such a power differential over the people is not always a good thing...


        Promises with Force

        Notes:

        This is a picture of the white house being built - CLICK - and of course the pyramids.

        Both are impressive works of human coordination. But to the workers they are much different. The construction crew building the white house was engaged in voluntary coordination through employment and they were probably excited about their new government too. The slaves building the pyramids, are engaged in forced coordination, and when they die, their corpses won't even make it in that pyramid, they'll just get thrown out in the desert. Our job, in this smart contract module, is to design systems of coordination that make it likely and easy for voluntary coordination so that it is not necessary or desireable to resort to forced coordination.


        Promises with Force

        Notes:

        We're doing that because contracts, which often require enforcers, who usually is the government - CLICK - Yeah, which could of course not always be a good thing...


        What is a Contract Again?

        (to summarize)

        A promise, made between parties, backed by the government, to coordinate together to accomplish something together.

        Notes:

        So yeah, that's the goal and what we're trying to solve. But, again, just before we go any further, letโ€™s make sure weโ€™re all on the same page about what a contract itself is. A promise, made between parties, backed by the government, to coordinate together to accomplish something together. The notion of promise is very abstract:

        • It could involve coordinating over time.
        • It could involve money or even credit.
        • It also could have multiple possible valid outcomes.

        Okay. Cool. All on the same page? Nodding heads? We can agree we understand this concept?


        Smart Contracts ๐ŸŽ‰

        Notes:

        Amazingโ€ฆ because, now onto smart contracts! - CLICK - We've finally made it here, folks, horary! Back in the 90s, this CS guy, Nick Szabo, and some other cypherpunks realized that contract law and computer science actually had a lot in common, and thus developed the idea of smart contracts. And together we'll develop and tease out those same comparisons here.


        Smart Contracts - Szabo Definition

        A machine program with rules that we could have defined in a contract, but instead a machine performs or verifies performance.

        Broad definition aka Szabo definition formulated c 1997

        Notes:

        So Szabo has this idea of the smart contract - which is basically to write contractual agreements as computer programs in programming languages. And then to run those programs on a computer that will then execute the instructions faithfully with machine-like precision. He was pretty excited about some of the research in multi-party computation that was happening at the time and had a sense that this vision might be right around the corner. Remember, there is a second smart contract definition that we called the "web3 definition" right at the beginning of this lecture. That definition is heavily dependent on content you'll learn during the blockchain module - so Joshy will be sure to build off of that later this week.

        https://nakamotoinstitute.org/the-god-protocols/ https://nakamotoinstitute.org/formalizing-securing-relationships/


        Smart Contracts ๐Ÿ˜ ๐Ÿ‘Ž

        Notes:

        Of course, I want to caveat that the term isn't perfect and not everybody likes it. As you can see in the tweet, Vitalik has worried that the term is too catchy and gives the wrong impression...

        With these caveats and criticisms in mind, it is still worth exploring the idea of contracts as computer programs.


        Quick review of how we got here

        Notes:

        So remember we started all the way back at simple primitive trade. One solution to those problems were to have personal trust with one another. But that doesn't work for everything. So we evolved to use rational expectations based on incentives and reputation and societal norms. But then we had the problem of scalability. Which brought us to, contracts, aka promises, which allowed us to really scale and achieve some incredible coordination. And to ensure that those contracts were upheld, the government functioned as a central point of enforcement. But, as we've seen, that's not always a good thing. And so now, we're going make the next iteration in this long, rich history of how to coordinate: which is the smart contract.


        Smart Contract Objectives

        • Retain all the value of traditional governmental contracts
          • Can trust strangers promises
          • Can coordinate with little risk
          • etc.
        • Remove the need for a powerful government
        • Remove ambiguity about what contracts are enforceable
        • Make writing and understanding contracts more approachable
        • Make contract execution and enforcement more efficient

        Notes:

        These are the goals of a smart contract. We basically want to keep all the good stuff that contracts brought us, but then of course make them better by not making the government the central point of enforcement. We want to know, without a doubt that the contracts will be enforced. We want them to be approachable, legibility wise, and we want them to be efficient.


        Smart Contracts - Two Parts

        1. Expression
        2. Execution / Enforcement

        Notes:

        So, we'll look at smart contracts in two parts. Expression - this is the part about actually codifying or expressing what you agree to unambiguously. From a programmer's perspective, this is by writing code. Or from a lawyer's perspective, this is writing up a bunch of pages of legalese, which basically is a DSL for being really specific in a legal contract. And then the part about making sure it actually happens? and what to do if something goes wrong? Thatโ€™s the execution or enforcement part. Execution is ultimately where the two worlds contrast - running unstoppable code on a blockchain vs. having courts and judges and police officers and things like that. Essentially the idea of a smart contract is: we're going to take a contract, which we all feel really confident in our definition, and we're going to glue it together with all of Szaboโ€™s incredible Computer Science research that's come out way more recently than contract research, and that all together is the next iteration and improvement on contracts. I should note - of these two parts, Programming languages are good at part 1 and computers are good at part 2.


        Expression -- Pt. 1

        Domain Specific Languages

        Legalese: In witness whereof, the parties hereunto have set their hands to these presents as a deed on the day month and year hereinbefore mentioned.

        ink!: signatures = [alice, bob]

        Notes:

        Okay when we're talking expression we're really talking - Domain Specific Languages (which we say DSL for short). Humans have been inventing DSL forever.


        Domain Specific Languages

        Notes:

        From the ancient greek accounting and administrative language, linear b,


        Domain Specific Languages

        Notes:

        to the shorthand for summarizing chess games,


        Domain Specific Languages

        Notes:

        to the ink! programming language that you will soon learn. Contract law is a DSL and is an important part of society. Itโ€™s hard to read (read the sentence on the slide). Computer Programming languages are also DSLs, which of course had itโ€™s own evolution, independent from contract law. It can be quite hard to read (asm or solidity), whereas good ones can also be easier to read. But really, the point I'm trying to make here is that legalese contract law and computer programming languages are not so different. They are both DSL meant to precisely express the nuances of detailed human agreements. And they're both hard to read for non experts. Szabo recognized this similarity and thus had the idea that programming may be the future of law.


        Coding Style

        • Adding laws vs negative diffs
        • Elegance vs handling each special case
        • Testing and Repeatability

        Notes:

        Back to the thing about both being hard to read for non-experts, a goal of mine is to make smart contracts more accessible to everyday people so that people can read their own contracts!

        HOWEVER since we're talking similarities - we should also speak to the differences. And hence be careful with this legal and coding comparison. The culture of the two can certainly be different:


        Execution / Enforcement -- Pt. 2

        Notes:

        Okay, so now that we understand what it means to express and write a contract, imagine now that the legal agreement has been made, what comes next?


        Execution / Enforcement -- Pt. 2

        Notes:

        Well it must be executed / enforced. Where does that happen? How does that work?


        Traditional Execution

        • There is rarely an official submission moment.
        • Parties Self Execute
        • Sometimes, that's it. Success.
        • Sometimes it goes to court.

        Notes:

        So - letโ€™s talk about execution. In a traditional sense, parties often just sign the contract - maybe there is a witness, sometimes even a notary, rarely does it actually get submitted to a judge. On the happy path, no further action is necessary - this is actually a decided strength of the traditional system - it is maximally lazy. When there is a dispute though, parties can go to court for a human judge to rule. A judge enforces through powerful agencies like police and jails, and also through social consensus. In practice there is little consideration for scaling, queue length, ruling priority, resource exhaustion. But in principle there could be. The existing systems of the nation states could be optimized considerably without having to even adopt any real smart contract.


        Computerized Execution

        • There is a deployment moment
        • Sometimes there are checkpoints or milestones or other official interactions
        • Sometimes there is dispute resolution

        Notes:

        Moving beyond traditional execution to computerized: you start some process on some computer and that computer faithfully executes the program for users to interact with. At first you can think of it like a computer in the basement and we walk up to it when we want to interact. This is how nintendo works. Multiple controllers and multiple physically present users. The internet makes it better in some ways. We can interact with our smart contract remotely instead of being physically present. But now, there is the question of the server operator. Are they really executing the correct contract? A corrupt server operator is essentially the same as a corrupt judge. Hmmm... so I wonder how we might solve this?


        Code is Law

        Notes:

        So, after all of that talk comparing law and code, it's inevitable that we get here: 'Code is law' which is another web3-ism you'll hear quite often. We know tech can be used to enforce rules. So we might wonder, do we actually need lawyers? Can code dictate what we can and cannot do? Can smart contracts be the answer? I'm not coming for lawyers. I'm sure they're nice. But could this be the answer to make life more trustless and secure? But this is where we'll end it today. I hope I've primed you with a few lingering questions. These smart contracts do sound cool, but perhaps the execution, where it actually happens, hopefully is a bit opaque and puzzling still. That's what Joshy will be able to speak to this week.


        Activity

        Notes:

        Any questions?

        Now I'm going to turn it over to Aaron to demonstrate this idea of expressing contracts as programming code.

        Resources, Fees, Ordering


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: Resources, Fees, Ordering description: Fees and ordering in blockchains duration: 1 hour ---

        Resources, Fees, Ordering


        Overview

        1. Fees and ordering
        2. Execution models

        Notes:

        • This lecture is a bit all over the place.
        • A bunch of stuff worth covering, but not all directly related.

        Fees & Ordering


        Fees & Ordering

        Blockchains are open, shared systems. They are unrestricted in access.

        But restricted in resources.


        Permissionless Access

        Free ability to access does not mean free cost to access.


        Threads

        A blockchain runtime is single-threaded.

        (For now)


        Time

        A block must terminate in some amount of time.


        Network

        Besides execution time, blocks need to propagate throughout the network.


        Demand

        Many people may want to use the system concurrently.

        But the system needs to decide:

        • Which state transition calls to include in a block, and
        • How to order them.

        Brief Interruption #1

        The block body contains an ordered set of extrinsics: Packets from the outside world with zero or more signatures attached.

        Notes:

        Recall from Lecture 1.


        Brief Interruption #1 (cont.)

        These packets include:

        • A call to the system's state transition function
        • Some contextual information (e.g. a spec version)
        • Perhaps some additional information that would help block authors prioritise

        No-Longer Brief Interruption #1 (cont.)

        Consider packets with:

        • zero signatures attached as "inherents" or "unsigned extrinsics"
        • one or more signatures attached as "transactions"

        This will be pretty straightforward until it's not.


        Fee Models

        Different blockchains have different fee models.

        For this lecture, we will look at three:

        • Size (Bitcoin)
        • Step Metering (Ethereum)
        • Time (Polkadot)*

        * (and coming soon, space)


        Size

        Bitcoin has a very simple STF: Namely verifying signatures and reassigning UTXOs.

        Its block size is limited, and each transaction has some byte-length (instruction, signature, etc.)

        Block authors will normally choose the set of transactions that would yield the highest gross fee.


        Bitcoin Fee Market

        Notes:

        • Fee market evolves based on demand

        Source: Y Charts (Couldn't find the chart I really wanted (per-byte rate), but can discuss)


        Metering

        Ethereum has a more complex STF than Bitcoin, namely one that is quasi-Turing-complete.

        Users could submit transactions with an unknown number of steps to terminate.

        The system uses "gas metering" to halt execution of a transaction and continue to the next.


        Metering

        Ethereum's STF defines a VM with instruction set, where each instruction costs some "gas".

        Users specify:

        • Max amount of gas to use
        • Cost, in ETH, per unit of gas they are willing to pay

        Metering

        Each time an instruction is executed, the system deducts its cost from the max gas.

        If the program terminates, it only charges for the gas used.

        If it runs out of gas, it terminates the program.


        Gas Rates

        Notes:

        Source: Etherscan


        Weight

        Instead of metering during runtime, meter ahead of time.

        Charge a fixed* fee for dispatching some call.


        *Fixed

        #![allow(unused)]
        fn main() {
        #[pallet::weight(100_000_000)]
        fn my_variable_weight_call(input: u8) -> Refund {
        	let mut refund: Refund = 0;
        	if input > 127 {
        		let _ = do_some_heavy_computation();
        	} else {
        		let _ = do_some_light_computation();
        		refund = 80_000_000;
        	}
        	refund
        }
        }

        Time

        Weight is picoseconds of execution time (10E-12).

        Calls are benchmarked on some "standard hardware".

        (There are some changes in the works about making two-dimensional weight.)


        Weight

        Using weight reduced the overhead of runtime metering, but requires some more care from developers.

        • It must be possible to meter before runtime
        • So, users should not be able to deploy untrusted code
        • There is no safety net on execution
        • Some computation is OK, but should be possible from call inspection

        Notes:

        Examples:

        • User-supplied length of a list that will be iterated over
        • Number of calls within a batch

        Brief Interruption #2

        Some of the gas and weight systems are evolving.

        • Ethereum recently added EIP1559, which uses a fee + tip mechanism
        • Parity and Web3 Foundation are discussing some changes to the weights model

        Fee Strategies

        Block authors can include transactions using several strategies:

        • Just take those with the highest fee
        • Take those with the highest fee to {length, gas, weight} ratio

        Fee Burning

        Not all the fees must go to the block author (depends on system design).

        In fact, this is often a bad design. Block authors will want fees to go up, so may indulge in no-op transactions to boost fees.


        Fee Burning Examples

        • Polkadot only gives 20% of fees to block authors (80% goes to an on-chain Treasury)
        • Since EIP1559, Ethereum burns some of its fees (its "base") from each transaction

        In both systems, users can add "tips" to increase the priority of their transaction with authors.


        Filling a Block

        Depending on the limiting factor, systems can have different limits to call a block full.

        • Bitcoin: Size (in bytes)
        • Ethereum: Gas limit (sum of all gas limits of transactions)
        • Polkadot: Weight (sum of all max expected weights)

        Ordering

        We've selected some transactions, but the runtime is single-threaded.

        Block authors must order them.


        Priority Basis

        The naive solution is to maintain an order of pending transactions by some "priority".

        And just include the top N transactions that fit in a block.


        More Advanced

        But, many small transactions might result in a higher fee for greedy block authors.

        So there could exist a set of transactions that is more profitable than just the top N.

        Even some that could be considered attacks.


        Execution Models


        Transactional Execution

        Most blockchains have a "transactional" execution model.

        That is, they need to be woken up.

        A smart contract, for example, won't execute any code unless someone submits a signed, fee-paying transaction to the system.


        Brief Interruption #3

        All of the "packets from the outside world" in these systems are signed.

        Some key holder signs an instruction that authorises a call and is willing to pay for its execution.

        Now is the time to enter the world of unsigned packets.


        Free Execution

        State machines can have autonomous functions in their state transition function.

        System designers can make these functions execute as part of the STF.

        In this model, block authors must execute some logic.


        Free Execution

        These added function calls are powerful, but some care must be taken:

        • They still consume execution resources (e.g., weight)
        • They need some method of verification (other nodes should be able to accept/reject them)

        Hooks

        The Substrate lectures will get into these, but for now just a look at some APIs:

        #![allow(unused)]
        fn main() {
        pub trait Hooks<BlockNumber> {
        	fn on_initialize(_n: BlockNumber) -> Weight {}
        	fn on_finalize(_n: BlockNumber) {}
        	fn on_idle(_n: BlockNumber, _remaining_weight: Weight) -> Weight {}
        	fn on_runtime_upgrade() -> Weight {}
        	fn offchain_worker(_n: BlockNumber) {}
        }
        }

        Source: /frame/support/src/traits/hooks.rs

        The Quest for Infrastructure


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: The Quest for Infrastructure description: Summary and review Smart Contract principles, and observation that we have not yet thwarted the tyrannical server operator ---

        The Quest for Infrastructure

        OR

        Where can I actually run these things!?


        Quick Review of Smart Contracts

        ---v

        Expression

        Write code instead of long confusing legalese

        ---v

        Execution

        Let the computer run it instead of the judge interpreting it.


        Quick Review of Service Providers

        They are the technicians that keep the infra running.

        They are the technicians that keep the infra running.

        They don't have any particular morals, certainly not the same ones you have.

        Notes:

        We need people to keep the computers online and replace broken hardware and make sure the data is backed up etc. But those people should be just that, technicians. They are not elected officials or Judges or anything like that. So we don't want to accidentally let them fil that role.

        Many, probably even most, data center workers are perfectly good people not trying to scam anyone. The point is that any individual server operator could be evil and it will be easier to coordinate voluntarily if people don't have to trust the server operator to do it.


        Where to Run the Contracts?

        Notes:

        We know we need somewhere to run these things with no back door or trusted party.

        We have seen some hints so far. Let's check them out.

        ---v

        Diversity and Redundancy

        • Geographical - for natural disasters and terrorist attacks
        • Jurisdictional - to resist the incumbent governments - they are the ultimate server operators
        • Moral - so all compasses are represented in the network, and no group can impose hegemony
        • Of compute hardware - incase some is bugged or backdoored or prohibitively expensive

        Notes:

        Web2 gets a lot of this right. At least they are good at the first two and preventing accidental data loss etc. There is a lot to be kept from web 2. Some digital services, or subsystems of digital services may never even need web3 upgrades.

        But indeed there is also some to be thrown out or improved.

        ---v

        P2P Networking

        Replace the operator with a system where peers all have power.

        Notes:

        We saw well how this worked out well in the file sharing and anonymous browsing domains (bit torrent,)

        ---v

        Reproducible Execution

        • Computers are better than courts
        • PABs make distributed code execution practical

        Notes:

        We saw even back in the early 2000s with java web applets that allowing more people to run the same program is hugely useful.

        A few decades later we have much better tech for this, and it is even more valuable.

        PABs make it practical for diverse parties all over the world to run the same software deterministically on their bespoke hardware in their unique environments.


        Blockchain

        Solves ALL Your Problems

        ---v

        Solves Some Specific Problems

        Allows us to replace the central server operator with a P2P network and Consensus System

        Notes:

        Actually it is a fallacy that blockchain solves all our problems. It actually solves some very specific coordination problems. It also brings some efficiency improvements to modern bureaucracies. It does not automatically make everyone believe the same things or magically arrive in happy unicorn bunny land. Tomorrow and Thursday will dig in on how the blockchain and its P2P network work together.

        ---v

        Blockchain Data structure

        Notes:

        We'll discuss two new blockchain related topics. First is the blockchain data structure which you can see here. This one is forked which is when things get really interesting, and when you need to invoke the second part

        ---v

        Blockchain Consensus

        Notes:

        Consensus deals with how we agree which version of the data structure is real. It is an interesting and complex topic, but we first need to learn a bit more about how the data structure allows us to track a shared story.

        ---v

        Reading Assignment

        For tomorrow please read the bitcoin whitepaper

        Wasm Smart Contracts in Ink!


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: Wasm Smart Contracts in Ink! description: A working programmerโ€™s guide to the crypto industry ---

        Wasm Smart Contracts in Ink!

        A working programmerโ€™s guide

        Notes:

        • ask questions during the lecture, don't wait until the end
        • practical, but we go deeper where needed
        • some complexity is omitted in the examples (examples are not a production code)

        Intro: ink! vs. Solidity

        ink!Solidity
        Virtual MachineAny Wasm VMEVM
        EncodingWasmEVM Byte Code
        LanguageRustStandalone
        ConstructorsMultipleSingle
        ToolingAnything that supports RustCustom
        StorageVariable256 bits
        Interfaces?Yes: Rust traitsYes

        Notes:

        • students are freshly of an EVM lecture so might be wondering why another SC language
        • Virtual Machine: any Wasm VM: yes in theory, in practice bound pretty close to the platform it runs on (Substrate & the contracts pallet)
        • Tooling: Solidity has been around for years, enjoys the first-to-market advantage (but ink! is a strong contender)
        • The EVM operates on 256 bit words (meaning anything less than 32 bytes will be treated by the EVM as having leading zeros)

        Intro: ink! overview

        • DSL in Rust
        • Inherits all the benefits of Rust
          • Modern functional language
          • Type & Memory safety
        • Compiled to Wasm
          • Ubiquitous
          • Fast

        Notes:

        • ink! is not a separate language
        • enjoys access to a vast collection of libraries developed for other purposes
        • Wasm is targeting the browsers and quickly becoming the "assembly" od the web in lieu of JS

        Intro: ink! & Substrate

        Notes:

        • Technically you could take a SC written in ink! and deploy it to any Wasm-powered blockchain.
          • in practice not that straight-forward.
        • ink! is closely tied to the larger Substrate framework.
        • Substrate is a framework for developing customized blockchain runtimes from composable pallets.

        Intro: ink! & Substrate

        Notes:

        • contracts written in ink! are compiled to Wasm bytecode
        • pallet contracts provides
          • instrumentation
          • execution engine
          • gas metering

        Notes:

        • pallet contracts is oblivious to the programming language
        • it accepts Wasm bytecode and executes it's instructions

        Notes:

        • contracts itself can be written in ink!

        Notes:

        • But also any other language that compilers to Wasm
          • Solang
          • or ask!

        Development: Prerequisites

        Install the required tooling

        sudo apt install binaryen
        rustup component add rust-src --toolchain nightly
        rustup target add wasm32-unknown-unknown --toolchain nightly
        cargo install dylint-link
        cargo install cargo-contract --force
        

        Notes:

        • Binaryen is a compiler and toolchain infrastructure library for WebAssembly
        • at the moment ink! uses a few unstable Rust features, thus nightly is require
        • rust source code is needed to compile it to wasm
        • wasm target is added
        • cargo-contract is a batteries included CLI tool for compiling, deploying and interacting with the contracts

        Development: cargo-contract

        Create a contract

        cargo contract new flipper
        
        /home/CloudStation/Blockchain-Academy/flipper:
          drwxrwxr-x 2 filip filip 4096 Jul  7 11:11 .
          drwxr-xr-x 5 filip filip 4096 Jul  7 11:11 ..
          -rwxr-xr-x 1 filip filip  573 Jul  7 11:11 Cargo.toml
          -rwxr-xr-x 1 filip filip  285 Jul  7 11:11 .gitignore
          -rwxr-xr-x 1 filip filip 5186 Jul  7 11:11 lib.rs
        

        Notes:

        • ask how many student have written some code in Rust, this should feel familiar to them

        Development: Cargo.toml

        [package]
        version = "0.1.0"
        authors = ["fbielejec"]
        edition = "2021"
        
        [dependencies]
        ink = { version = "=4.2.1", default-features = false }
        scale = { package = "parity-scale-codec", version = "3", default-features = false, features = ["derive"] }
        scale-info = { version = "2.6", default-features = false, features = ["derive"], optional = true }
        
        [lib]
        path = "lib.rs"
        
        [features]
        default = ["std"]
        std = [
          "ink/std",
          "scale/std",
          "scale-info/std",
        ]
        

        Notes:

        • who knows why is the std library not included by default?
        • Answer: contracts are compiled to Wasm (executed ib a sandboxed environment with no system interfaces, no IO, no networking)

        Developing contracts

        contract code

        #![allow(unused)]
        fn main() {
        #[ink::contract]
        pub mod flipper {
        
            #[ink(storage)]
            pub struct Flipper {
                value: bool,
            }
        
            impl Flipper {
                #[ink(constructor)]
                pub fn new(init_value: bool) -> Self {
                    Self { value: init_value }
                }
        
                #[ink(constructor)]
                pub fn default() -> Self {
                    Self::new(Default::default())
                }
        
                #[ink(message)]
                pub fn flip(&mut self) {
                    self.value = !self.value;
                }
        
                #[ink(message)]
                pub fn get(&self) -> bool {
                    self.value
                }
            }
        }
        }

        Notes:

        • basic contract that flips a bit in storage
        • contract will have a storage definition, constructor(s), messages
        • grouped in a module

        Developing contracts: Compilation & artifacts

        Compile:

        cargo +nightly contract build
        

        Artifacts:

         [1/*] Building cargo project
            Finished release [optimized] target(s) in 0.09s
        
        The contract was built in RELEASE mode.
        
        Your contract artifacts are ready.
        You can find them in:
        /home/CloudStation/Blockchain-Academy/flipper/target/ink
        
          - flipper.contract (code + metadata)
          - flipper.wasm (the contract's code)
          - flipper.json (the contract's metadata)
        

        Notes:

        • produces Wasm bytecode and some additional artifacts:
        • .wasm is the contract compiled bytecode
        • .json is contract ABI aka metadata (for use with e.g. dapps)
          • definitions of events, storage, transactions
        • .contracts is both of these together

        Developing contracts: instantiate

        Deploy:

        cargo contract instantiate --constructor default --suri //Alice
          --skip-confirm --execute
        

        Output:

         Dry-running default (skip with --skip-dry-run)
            Success! Gas required estimated at Weight(ref_time: 138893374, proof_size: 16689)
        ...
          Event Contracts โžœ CodeStored
                 code_hash: 0xbf18c768eddde46205f6420cd6098c0c6e8d75b8fb042d635b1ba3d38b3d30ad
               Event Contracts โžœ Instantiated
                 deployer: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY
                 contract: 5EXm8WLAGEXn6zy1ebHZ4MrLmjiNnHarZ1pBBjZ5fcnWF3G8
        ...
               Event System โžœ ExtrinsicSuccess
                 dispatch_info: DispatchInfo { weight: Weight { ref_time: 2142580978, proof_size: 9009 }, class: Normal, pays_fee: Yes }
        
           Code hash 0xbf18c768eddde46205f6420cd6098c0c6e8d75b8fb042d635b1ba3d38b3d30ad
            Contract 5EXm8WLAGEXn6zy1ebHZ4MrLmjiNnHarZ1pBBjZ5fcnWF3G8
        

        Notes:

        • we see a bunch of information on gas usage
        • we see two events one for storing contract code another for instantiating the contract
          • why is that?
          • code & instance are separated, we will come back to that
        • finally we see code hash and the newly created contracts address

        Interacting with the contracts: queries

        cargo contract call --contract 5EXm8WLAGEXn6zy1ebHZ4MrLmjiNnHarZ1pBBjZ5fcnWF3G8
          --message get --suri //Alice --output-json
        
        • contract state?
        • tip: default constructor was called

        Notes:

        • who can tell me what will be the contract state at this point?

        Interacting with the contracts: queries

        "data": {
          "Tuple": {
            "ident": "Ok",
            "values": [
              {
                "Bool": false
              }
            ]
          }
        }
        

        Interacting: transactions

        Sign and execute a transaction:

        cargo contract call --contract 5EXm8WLAGEXn6zy1ebHZ4MrLmjiNnHarZ1pBBjZ5fcnWF3G8
          --message flip --suri //Alice --skip-confirm --execute
        

        Query the state:

        cargo contract call --contract 5EXm8WLAGEXn6zy1ebHZ4MrLmjiNnHarZ1pBBjZ5fcnWF3G8
          --message get --suri //Alice --output-json
        

        Result:

        "data": {
          "Tuple": {
            "ident": "Ok",
            "values": [
              {
                "Bool": true
              }
            ]
          }
        }
        

        Notes:

        • if I query it again the bit is flipped
        • no surprises there

        Dev environment: Contracts UI

        Notes:

        • there is also a graphical env for deploying & interacting with contracts
        • deploy & create an instance of flipper

        Dev environment: Contracts UI

        Notes:

        • call a transaction

        Dev environment: Contracts UI

        Notes:

        • query state

        Developing contracts: Constructors

        #![allow(unused)]
        fn main() {
        #[ink(storage)]
        pub struct Flipper {
            value: bool,
        }
        
        #[ink(constructor)]
        pub fn new(init_value: bool) -> Self {
            Self { value: init_value }
        }
        
        #[ink(constructor)]
        pub fn default() -> Self {
            Self::new(Default::default())
        }
        
        #[ink(constructor)]
        pub fn non_default() -> Self {
            Self::new(false)
        }
        }

        Notes:

        • lets dissect what a contract code is built like
        • no limit of the number of constructors
        • constructors can call other constructors
        • constructors return the initial storage
        • a lot of complexity conveniently hidden behind macros

        Developing contracts: Queries

        #![allow(unused)]
        fn main() {
        #[ink(message)]
        pub fn get(&self) -> bool {
            self.value
        }
        }
        • #[ink(message)] is how we tell ink! this is a function that can be called on the contract
        • &self is a reference to the contract's storage

        Notes:

        • returns information about the contract state stored on chain
        • reaches to the storage, decodes it and returns the value

        Developing contracts: Mutations

        #![allow(unused)]
        fn main() {
        #[ink(message, payable)]
        pub fn place_bet(&mut self, bet_type: BetType) -> Result<()> {
            let player = self.env().caller();
            let amount = self.env().transferred_value();
            ...
            self.data.set(&data);
            ...
        }
        • &mut self is a mutable reference to the object youโ€™re calling this method on
        • payable allows receiving value as part of the call to the ink! message

        Notes:

        • constructors are inherently payable
        • ink! message will reject calls with funds if it's not marked as such
        • mutable references allow me to modify the storage.
        • queries are for free, mutations are metered (you pay gas)
          • you will also pay for queries within such transactions

        Contracts: Error handling

        #![allow(unused)]
        fn main() {
        pub enum MyResult<T, E> {
            Ok(value: T),
            Err(msg: E),
        }
        
        #[derive(Debug, PartialEq, Eq, Encode, Decode)]
        #[cfg_attr(feature = "std", derive(scale_info::TypeInfo))]
        pub enum MyError {
            InkEnvError(String),
            BettingPeriodNotOver,
        }
        
        #[ink(message)]
        pub fn spin(&mut self) -> Result<()> {
            if !self.is_betting_period_over() {
                return Err(MyError::BettingPeriodNotOver);
            ...
        };
        
        pub type Result<T> = core::result::Result<T, MyError>;
        }
        • ink! uses idiomatic Rust error handling: Result<T,E> type
        • Use the Err variant to pass your own semantics
        • Type aliases reduce boilerplate & enhance readability

        Notes:

        • ink! uses idiomatic Rust error handling
        • messages are the system boundary
        • returning error variant or panicing reverts the transaction
          • panicing is the same as returning Err variant (Result is just being nice)

        Error handling: call stack

        #![allow(unused)]
        fn main() {
        #[ink(message)]
        pub fn flip(&mut self) {
            self.value = !self.value;
        
            if self.env().block_number() % 2 != 0 {
              panic!("Oh no!")
            }
        
        }
        }
        • what is the state of this contract if the tx is called in an odd block number?

        Notes:

        • answer: whatever it was prior to the tx:
          • returning error variant reverts the entire tx on the call stack

        Contracts: Events

        #![allow(unused)]
        fn main() {
        #[ink(event)]
        #[derive(Debug)]
        pub struct BetPlaced {
            #[ink(topic)]
            player: AccountId,
            #[ink(topic)]
            bet_type: BetType,
            amount: Balance,
        }
        }
        • Events are a way of letting the outside world know about what's happening inside the contract.
        • #[ink(event)] is a macro that defines events.
        • Topics mark fields for indexing.

        Notes:

        • events are especially important for dapps
        • storage is expensive: reading e.g. aggregate data from chain directly is impossible / impractical
        • dapps the can listen to the event, normalize & store off-chain and answer e.g. complex queries

        Contracts: Events

        #![allow(unused)]
        fn main() {
        #[ink(message)]
        pub fn flip(&mut self) {
        
            Self::emit_event(
                self.env(),
                Event::Flipped(Flipped { }),
            );
        
            self.value = !self.value;
        
            if self.env().block_number() % 2 == 0 {
              panic!("Oh no!")
            }
        
        }
        }
        • What happens to the events from reverted transactions?
        • Will this event be emitted in an odd block?

        Notes:

        • answer: yes, but only because I reverted the condition :)

        Contracts: Defining shared behaviour

        #![allow(unused)]
        fn main() {
        #[ink::trait_definition]
        pub trait PSP22 {
            #[ink(message)]
            fn total_supply(&self) -> Balance;
        
            #[ink(message)]
            fn balance_of(&self, owner: AccountId) -> Balance;
        
            #[ink(message)]
            fn approve(&mut self, spender: AccountId, amount: Balance) -> Result<(), PSP22Error>;
        
            #[ink(message)]
            fn transfer(&mut self, to: AccountId, value: Balance, data: Vec<u8>) -> Result<(), PSP22Error>;
            ...
        
        impl SimpleDex {
            use psp22_trait::{PSP22Error, PSP22};
        
            /// Returns balance of a PSP22 token for an account
            fn balance_of(&self, token: AccountId, account: AccountId) -> Balance {
                let psp22: ink::contract_ref!(PSP22) = token.into();
                psp22.balance_of(account)
            }
            ...
        }
        • Trait Definition: #[ink::trait_definition].
        • Sharing the trait definition to do a cross-contract call.

        Notes:

        • (part of) PSP22 (ERC20 like) contract definition
        • all contracts that respect this definition need to implement it
        • you can now share the trait definition with other contracts
        • while getting a typed reference to an instance

        Deeper dive: Storage

        #![allow(unused)]
        fn main() {
        use ink::storage::Mapping;
        
        #[ink(storage)]
        #[derive(Default)]
        pub struct Token {
            total_supply: Balance,
            balances: Mapping<AccountId, Balance>,
            allowances: Mapping<(AccountId, AccountId), Balance>,
        }
        }

        Notes:

        • now that we dipped our toes lets dissect more
        • starting with the storage
        • what does this code actually put into the chain storage?

        SCALE (Simple Concatenated Aggregate Little Endian)

        Notes:

        • Pallet contracts storage is organized like a key-value database
        • each storage cell has a unique storage key and points to a SCALE encoded value
        • SCALE codec is not self-describing (vide metadata)

        SCALE: examples of different types

        TypeDecodingEncodingRemark
        Booleantrue0x0encoded using least significant bit of a single byte
        false0x1
        Unsigned int420x2a00
        Enumenum IntOrBool { Int(u8), Bool(bool)}0x002a and 0x0101first byte encodes the variant index, remaining bytes encode the data
        Tuple(3, false)0x0c00concatenation of each encoded value
        Vector[4, 8, 15, 16, 23, 42]0x18040008000f00100017002a00encoding of the vector length followed by conatenation of each item's encoding
        Struct{x:30u64, y:true}[0x1e,0x0,0x0,0x0,0x1]names are ignored, Vec structure, only order matters

        Notes:

        • this table is not exhaustive
        • struct example: stored as an vector, names are ignored, only order matters, first four bytes encode the 64-byte integer and then the least significant bit of the last byte encodes the boolean

        Storage: Packed Layout

        #![allow(unused)]
        fn main() {
        use ink::storage::Mapping;
        
        #[ink(storage)]
        #[derive(Default)]
        pub struct Token {
            total_supply: Balance,
            balances: Mapping<AccountId, Balance>,
            allowances: Mapping<(AccountId, AccountId), Balance>,
        }
        }
        • By default ink! stores all storage struct fields under a single storage cell (Packed layout)

        Notes:

        • We talked about the kv database that the storage is, now how is it used precisely
        • Types that can be stored entirely under a single storage cell are called Packed Layout
        • by default ink! stores all storage struct fields under a single storage cell
        • as a consequence message interacting with the contract storage will always need to read and decode the entire contract storage struct
        • .. which may be what you want or not

        Storage: Packed Layout

        #![allow(unused)]
        fn main() {
        use ink::storage::traits::{
            StorageKey,
            ManualKey,
        };
        
        #[ink(storage)]
        pub struct Flipper<KEY: StorageKey = ManualKey<0xcafebabe>> {
            value: bool,
        }
        }
        • The storage key of the contracts root storage struct defaults to 0x00000000
        • However you may store it under any arbitrary 4 bytes key instead

        Storage: Packed Layout

        "storage": {
          "root": {
            "layout": {
              "struct": {
                "fields": [
                  {
                    "layout": {
                      "leaf": {
                        "key": "0xcafebabe",
                        "ty": 0
                      }
                    },
                    "name": "value"
                  }
                ],
                "name": "Flipper"
              }
            },
            "root_key": "0xcafebabe"
          }
        }
        

        Notes:

        • demonstration of the packed layout - value is stored under the root key

        Storage: Un-packed Layout

        #![allow(unused)]
        fn main() {
        use ink::storage::Mapping;
        
        #[ink(storage)]
        #[derive(Default)]
        pub struct Token {
            total_supply: Balance,
            balances: Mapping<AccountId, Balance>,
            allowances: Mapping<(AccountId, AccountId), Balance>,
        }
        }
        • Mapping consists of a key-value pairs stored directly in the contract storage cells.
        • Each Mapping value lives under it's own storage key.
        • Mapping values do not have a contiguous storage layout: it is not possible to iterate over the contents of a map!

        Notes:

        • Use Mapping when you need to store a lot of values of the same type.
        • if your message only accesses a single key of a Mapping, it will not load the whole mapping but only the value being accessed.
        • there are other collection types in ink!: HashMap or BTreeMap (to name a few).
          • these data structures are all Packed, unlike Mapping!

        Storage: working with Mapping

        #![allow(unused)]
        fn main() {
        pub fn transfer(&mut self) {
            let caller = self.env().caller();
        
            let balance = self.balances.get(caller).unwrap_or(0);
            let endowment = self.env().transferred_value();
        
            balance += endowment;
        }
        }
        • what is wrong here?

        Notes:

        • working with mapping:
        • Answer: Mapping::get() method will result in an owned value (a local copy), as opposed to a direct reference into the storage. Changes to this value won't be reflected in the contract's storage "automatically". To avoid this common pitfall, the value must be inserted again at the same key after it was modified. The transfer function from above example illustrates this:

        Storage: working with Mapping

        #![allow(unused)]
        fn main() {
        pub fn transfer(&mut self) {
            let caller = self.env().caller();
        
            let balance = self.balances.get(caller).unwrap_or(0);
            let endowment = self.env().transferred_value();
        
            self.balances.insert(caller, &(balance + endowment));
        }
        }
        • Mapping::get() returns a local copy, not a mutable reference to the storage!

        Notes:

        • working with mapping:
        • Mapping::get() method will result in an owned value (a local copy).
        • Changes to this value won't be reflected in the contract's storage at all!
        • you need to inserted it again at the same key.

        Storage: Lazy

        #![allow(unused)]
        fn main() {
        use ink::storage::{traits::ManualKey, Lazy, Mapping};
        
        #[ink(storage)]
        pub struct Roulette {
            pub data: Lazy<Data, ManualKey<0x44415441>>,
            pub bets: Mapping<u32, Bet, ManualKey<0x42455453>>,
        }
        }
        • Every type wrapped in Lazy has a separate storage cell.
        • ManualKey assignes explicit storage key to it.
        • Why would you want to use a ManualKey instead of a generated one?

        Notes:

        • packed layout can get problematic if we're storing a large collection in the contracts storage that most of the transactions do not need access too
        • there is a 16kb hard limit on a buffer used for decoding, contract trying to decode more will trap / revert
        • lazy provides per-cell access, like a mapping
        • lazy storage cell can be auto-assigned or chosen manually
        • using ManualKey instead of AutoKey might be especially desirable for upgradable contracts, as using AutoKey might result in a different storage key for the same field in a newer version of the contract.
          • This may break your contract after an upgrade!

        Storage: Lazy

        Notes:

        • only the pointer (the key) to the lazy type is stored under the root key.
        • only when there is a read of d will the pointer be de-referenced and it's value decoded.
        • lazy is a bit of a mis-nomer here, because storage is already initialized.

        Contracts upgradeability: set_code_hash

        #![allow(unused)]
        fn main() {
        #[ink(message)]
        pub fn set_code(&mut self, code_hash: [u8; 32]) -> Result<()> {
            ink::env::set_code_hash(&code_hash)?;
            Ok(())
        }
        }
        • Within SC's lifecycle it is often necessary to perform an upgrade or a bugfix.
        • Contract's code and it's instance are separated.
        • Contract's address can be updated to point to a different code stored on-chain.

        Notes:

        • append only != immutable
        • proxy pattern known from e.g. solidity is still possible
        • within the Substrate framework contract's code is stored on-chain and it's instance is a pointer to that code
        • incentivizes cleaning up after oneself
        • big storage optimization

        Contracts upgradeability: access control

        #![allow(unused)]
        fn main() {
        #[ink(message)]
        pub fn set_code(&mut self, code_hash: [u8; 32]) -> Result<()> {
            ensure_owner(self.env().caller())?;
            ink::env::set_code_hash(&code_hash)?;
            Ok(())
        }
        }

        Notes:

        • you DO NOT want to leave this message un-guarded
        • solutions to ensure_owner can range from a very simple ones address checks
        • to a multiple-role database of access controlled accounts stored and maintained in a separate contract

        Upgradeability: storage

        #![allow(unused)]
        fn main() {
        #[ink(message)]
        pub fn get_values(&self) -> (u32, bool) {
            (self.x, self.y)
        }
        
        #[ink(storage)]
        pub struct MyContractOld {
            x: u32,
            y: bool,
        }
        
        #[ink(storage)]
        pub struct MyContractNew {
            y: bool,
            x: u32,
        }
        }
        • Make sure your updated code is compatible with the existing contracts state.
        • Will the getter work with the new definition and the old storage ?

        Notes:

        • Various potential changes that can result in backwards incompatibility:
          • Changing the order of variables
          • Introducing new variable(s) before any of the existing ones
          • Changing variable type(s)
          • Removing variables
        • Answer: no, SCALE encoding is oblivious to names, only order matters

        Upgradeability: storage migrations

        #![allow(unused)]
        fn main() {
        // new contract code
        #[ink(message)]
        pub fn migrate(&mut self) -> Result<()> {
            if let Some(OldContractState { field_1, field_2 }) = get_contract_storage(&123)? {
                self.updated_old_state.set(&UpdatedOldState {
                    field_1: field_2,
                    field_2: field_1,
                });
                return Ok(());
            }
        
            return Err(Error::MigrationFailed);
        }
        
        // old contract code
        #[ink(message)]
        pub fn set_code(&mut self, code_hash: [u8; 32], callback: Option<Selector>)
        }

        Notes:

        • if the new contract code does not match the stored state you can perform a storage migration
        • think of regular relational DB and schema migrations
        • a good pattern to follow is to perform the update and the migration in one atomic transaction:
          • if anything fails whole tx is reverted
          • won't end up in a broken state
          • make sure it can fit into one block!

        Common Vulnerabilities

        #![allow(unused)]
        fn main() {
        impl MyContract {
        
          #[ink(message)]
          pub fn terminate(&mut self) -> Result<()> {
              let caller = self.env().caller();
              self.env().terminate_contract(caller)
          }
        
          ...
        }
        }
        • What is wrong with this contract?
        • How would you fix it?

        Notes:

        • we start easy
        • answer: no AC in place
        • parity wallet 150 million hack

        Common Vulnerabilities: blast from the past

        Notes:


        Common Vulnerabilities

        #![allow(unused)]
        fn main() {
            #[ink(storage)]
            pub struct SubstrateNameSystem {
                registry: Mapping<AccountId, Vec<u8>>,
            }
        
            impl SubstrateNameSystem {
                #[ink(message, payable)]
                pub fn register(&mut self, name: Vec<u8>) {
                    let owner = self.env().caller();
                    let fee = self.env().transferred_value();
        
                    if !self.registry.contains(owner) && fee >= 100 {
                        self.registry.insert(owner, &name);
                    }
                }
        }
        • On-chain domain name registry with a register fee of 100 pico.
        • Why is this a bad idea?

        Notes:

        • everything on-chain is public
        • this will be front-run in no time
        • Can you propose a better design?
        • Answer: commit / reveal or an auction

        Common Vulnerabilities

        #![allow(unused)]
        fn main() {
        #[ink(message)]
        pub fn swap(
            &mut self,
            token_in: AccountId,
            token_out: AccountId,
            amount_token_in: Balance,
        ) -> Result<(), DexError> {
            let this = self.env().account_id();
            let caller = self.env().caller();
        
            let amount_token_out = self.out_given_in(token_in, token_out, amount_token_in)?;
        
            // transfer token_in from user to the contract
            self.transfer_from_tx(token_in, caller, this, amount_token_in)?;
        
            // transfer token_out from contract to user
            self.transfer_tx(token_out, caller, amount_token_out)?;
            ...
        }
        }
        • Contract is a DEX Decentralized EXchange, follows the popular AMM (Automated Market Maker) design.
        • Tx swaps the specified amount of one of the pool's PSP22 tokens to another PSP22 token according to the current price.
        • What can go wrong here?

        Notes:

        Answer:

        • no slippage protection in place.
        • bot will frontrun the victim's tx by purchasing token_out before the trade is executed.
        • this purchase will raise the price of the asset for the victim trader and increases his slippage
        • if the bot sells right after the victims tx (back runs the victim) this is a sandwich attack

        Common Vulnerabilities

        #![allow(unused)]
        fn main() {
        #[ink(message)]
        pub fn swap(
            &mut self,
            token_in: AccountId,
            token_out: AccountId,
            amount_token_in: Balance,
            min_amount_token_out: Balance,
        ) -> Result<(), DexError> {
        
            ...
        
            if amount_token_out < min_amount_token_out {
                return Err(DexError::TooMuchSlippage);
            }
        
        ...
        }
        }

        Notes:

        • slippage protection in place

        Common Vulnerabilities

        • Integer overflows
        • Re-entrancy vulnerabilities
        • Sybil attacks
        • ...
        • Regulatory attacks ๐Ÿ˜…
        • ...

        Notes:

        • long list of possible attacks
        • too long to fit into one lecture
        • baseline: get an audit from a respectable firm
        • publish your source code (security by obscurity is not security)

        Pause

        Optional challenge: github.com/Polkadot-Blockchain-Academy/adder

        Notes:

        Piotr takes over to talk about making runtime calls from contracts and writing automated tests. There is a 15 minute challenge for you in the meantime.


        Interacting with the execution environment

        #![allow(unused)]
        fn main() {
        impl MyContract {
          ...
          #[ink(message)]
          pub fn terminate(&mut self) -> Result<()> {
              let caller = self.env().caller();
              self.env().terminate_contract(caller)
          }
          ...
        }
        }

        Blockchain node onion


        Blockchain node onion



        Blockchain node onion

        • networking
        • block production, dissemination, finalization
        • storage management
        • off-chain maintenance, querying, indexing

        Blockchain node onion

        • computing new state based on the previous one and a single transaction

        Blockchain node onion

        • executing contract calls

        Standard API

        • caller()
        • account_id()
        • balance()
        • block_number()
        • emit_event(event: Event)
        • transfer(dest: AccountId, value: Balance)
        • hash_bytes(input: &[u8], output: &mut [u8])
        • debug_message(msg: &str)
        • and many more

        Standard API

        #![allow(unused)]
        fn main() {
        impl MyContract {
          ...
          #[ink(message)]
          pub fn terminate(&mut self) -> Result<()> {
              let caller = self.env().caller();
              self.env().terminate_contract(caller)
          }
          ...
        }
        }

        Interacting with the state transition function


        • token transfer
        • staking
        • voting
        • contract call
        • ...
        • advanced cryptography
        • bypassing standard restrictions
        • outsourcing computation
        • ...

        Interacting with the state transition function


        • token transfer
        • staking
        • voting
        • contract call
        • ...
        • runtime call

        • advanced cryptography
        • bypassing standard restrictions
        • outsourcing computation
        • ...

        • chain extension


        Runtime


        In Polkadot ecosystem state transition function is called runtime


        Calling runtime


        #![allow(unused)]
        fn main() {
        #[ink(message)]
        pub fn transfer_through_runtime(
            &mut self,
            receiver: AccountId,
            value: Balance,
        ) -> Result<(), RuntimeError> {
            let call_object = RuntimeCall::Balances(BalancesCall::Transfer {
                receiver,
                value,
            });
        
            self.env().call_runtime(&call_object)
        }
        }

        Calling runtime


        #![allow(unused)]
        fn main() {
        #[ink(message)]
        pub fn transfer_through_runtime(
            &mut self,
            receiver: AccountId,
            value: Balance,
        ) -> Result<(), RuntimeError> {
            let call_object = RuntimeCall::Balances(BalancesCall::Transfer {
                receiver,
                value,
            });
        
            self.env().call_runtime(&call_object)
        }
        }

        Chain extensions


        Chain extension is a way to extend the runtime with custom functionalities dedicated to contracts.


        Chain extensions


        ink! side:

        • provide ChainExtension trait
        • include extension in the Environment trait instantiation

        runtime side:

        • handling extension calls
        • extension logic itself

        Provide ChainExtension trait

        #![allow(unused)]
        fn main() {
        #[ink::chain_extension]
        pub trait OutsourceHeavyCrypto {
          type ErrorCode = OutsourcingErr;
        
          #[ink(extension = 41)]
          fn outsource(input: Vec<u8>) -> [u8; 32];
        }
        
        pub enum OutsourcingErr {
          IncorrectData,
        }
        
        impl ink::env::chain_extension::FromStatusCode for OutsourcingErr {
          fn from_status_code(status_code: u32) -> Result<(), Self> {
            match status_code {
              0 => Ok(()),
              1 => Err(Self::IncorrectData),
              _ => panic!("encountered unknown status code"),
            }
          }
        }
        }

        Provide ChainExtension trait

        #![allow(unused)]
        fn main() {
        #[ink::chain_extension]
        pub trait OutsourceHeavyCrypto {
          type ErrorCode = OutsourcingErr;
        
          #[ink(extension = 41)]
          fn outsource(input: Vec<u8>) -> [u8; 32];
        }
        
        pub enum OutsourcingErr {
          IncorrectData,
        }
        
        impl ink::env::chain_extension::FromStatusCode for OutsourcingErr {
          fn from_status_code(status_code: u32) -> Result<(), Self> {
            match status_code {
              0 => Ok(()),
              1 => Err(Self::IncorrectData),
              _ => panic!("encountered unknown status code"),
            }
          }
        }
        }

        Include extension in the Environment trait instantiation


        #![allow(unused)]
        fn main() {
        pub enum EnvironmentWithOutsourcing {}
        impl Environment for EnvironmentWithOutsourcing {
            ... // use defaults from `DefaultEnvironment`
            type ChainExtension = OutsourceHeavyCrypto;
        }
        
        #[ink::contract(env = crate::EnvironmentWithOutsourcing)]
        mod my_contract {
          ...
        }
        }

        Include extension in the Environment trait instantiation


        #![allow(unused)]
        fn main() {
        #[ink::contract(env = crate::EnvironmentWithOutsourcing)]
        mod my_contract {
          fn process_data(&mut self, input: Vec<u8>) -> Result<(), OutsourcingErr> {
            self.env().extension().outsource(subject)
          }
        }
        }

        Handling extension calls


        #![allow(unused)]
        fn main() {
        pub struct HeavyCryptoOutsourcingExtension;
        
        impl ChainExtension<Runtime> for HeavyCryptoOutsourcingExtension {
          fn call<E: Ext>(&mut self, env: Env) -> Result<RetVal, DispatchError> {
            match env.func_id() {
              41 => internal_logic(),
              _ => {
                error!("Called an unregistered `func_id`: {func_id}");
                return Err(DispatchError::Other("Unimplemented func_id"))
              }
            }
            Ok(RetVal::Converging(0))
        }
        }

        Chain extension: reaching even further


        Testing contracts


        Testing contracts



        Testing contracts


        Unit tests


        #![allow(unused)]
        fn main() {
        #[ink::test]
        fn erc20_transfer_works() {
          let mut erc20 = Erc20::new(100);
        
          assert_eq!(erc20.balance_of(BOB), 0);
          // Alice transfers 10 tokens to Bob.
          assert_eq!(erc20.transfer(BOB, 10), Ok(()));
          // Bob owns 10 tokens.
          assert_eq!(erc20.balance_of(BOB), 10);
        
          let emitted_events = ink::env::test::recorded_events().collect::<Vec<_>>();
          assert_eq!(emitted_events.len(), 2);
        
          // Check first transfer event related to ERC-20 instantiation.
          assert_transfer_event(
            &emitted_events[0], None, Some(ALICE), 100,
          );
          // Check the second transfer event relating to the actual transfer.
          assert_transfer_event(
            &emitted_events[1], Some(ALICE), Some(BOB), 10,
          );
        }
        }

        Unit tests


        #![allow(unused)]
        fn main() {
        #[ink::test]
        fn erc20_transfer_works() {
          let mut erc20 = Erc20::new(100);
        
          assert_eq!(erc20.balance_of(BOB), 0);
          // Alice transfers 10 tokens to Bob.
          assert_eq!(erc20.transfer(BOB, 10), Ok(()));
          // Bob owns 10 tokens.
          assert_eq!(erc20.balance_of(BOB), 10);
        
          let emitted_events = ink::env::test::recorded_events().collect::<Vec<_>>();
          assert_eq!(emitted_events.len(), 2);
        
          // Check first transfer event related to ERC-20 instantiation.
          assert_transfer_event(
            &emitted_events[0], None, Some(ALICE), 100,
          );
          // Check the second transfer event relating to the actual transfer.
          assert_transfer_event(
            &emitted_events[1], Some(ALICE), Some(BOB), 10,
          );
        }
        }

        Unit tests


        #![allow(unused)]
        fn main() {
        #[ink::test]
        fn erc20_transfer_works() {
          let mut erc20 = Erc20::new(100);
        
          assert_eq!(erc20.balance_of(BOB), 0);
          // Alice transfers 10 tokens to Bob.
          assert_eq!(erc20.transfer(BOB, 10), Ok(()));
          // Bob owns 10 tokens.
          assert_eq!(erc20.balance_of(BOB), 10);
        
          let emitted_events = ink::env::test::recorded_events().collect::<Vec<_>>();
          assert_eq!(emitted_events.len(), 2);
        
          // Check first transfer event related to ERC-20 instantiation.
          assert_transfer_event(
            &emitted_events[0], None, Some(ALICE), 100,
          );
          // Check the second transfer event relating to the actual transfer.
          assert_transfer_event(
            &emitted_events[1], Some(ALICE), Some(BOB), 10,
          );
        }
        }

        E2E tests


        #![allow(unused)]
        fn main() {
        #[ink_e2e::test]
        async fn e2e_transfer(mut client: ink_e2e::Client<C, E>) -> E2EResult<()> {
          let constructor = Erc20Ref::new(total_supply);
          let erc20 = client
                  .instantiate("erc20", &ink_e2e::alice(), constructor, 0, None)
                  .await
                  .expect("instantiate failed");
        
          let mut call = erc20.call::<Erc20>();
          let total_supply_msg = call.total_supply();
          let total_supply_res = client
                  .call_dry_run(&ink_e2e::bob(), &total_supply_msg, 0, None)
                  .await;
          ...
        }
        }

        E2E tests


        #![allow(unused)]
        fn main() {
        #[ink_e2e::test]
        async fn e2e_transfer(mut client: ink_e2e::Client<C, E>) -> E2EResult<()> {
          let constructor = Erc20Ref::new(total_supply);
          let erc20 = client
                  .instantiate("erc20", &ink_e2e::alice(), constructor, 0, None)
                  .await
                  .expect("instantiate failed");
        
          let mut call = erc20.call::<Erc20>();
          let total_supply_msg = call.total_supply();
          let total_supply_res = client
                  .call_dry_run(&ink_e2e::bob(), &total_supply_msg, 0, None)
                  .await;
          ...
        }
        }

        E2E tests


        #![allow(unused)]
        fn main() {
        #[ink_e2e::test]
        async fn e2e_transfer(mut client: ink_e2e::Client<C, E>) -> E2EResult<()> {
          let constructor = Erc20Ref::new(total_supply);
          let erc20 = client
                  .instantiate("erc20", &ink_e2e::alice(), constructor, 0, None)
                  .await
                  .expect("instantiate failed");
        
          let mut call = erc20.call::<Erc20>();
          let total_supply_msg = call.total_supply();
          let total_supply_res = client
                  .call_dry_run(&ink_e2e::bob(), &total_supply_msg, 0, None)
                  .await;
          ...
        }
        }

        E2E pipeline: traps, traps everywhere

        1. Preparing and encoding transaction data (client side)
        2. Signing the transaction (client side)
        3. Sending transaction to a node (client side)
        4. Block and event subscribing (client side)
        5. Transaction pool processing (node side)
        6. Block building (node side)
        7. Block dissemination (node side)
        8. Import queue processing (node side)
        9. Block finalizing (node side)
        10. Block execution (node side)
        11. Transaction execution (runtime side)
        12. Event emitting (node side)
        13. Event capturing (client side)
        14. Event processing (client side)
        15. State fetching via RPC calling (client side)
        16. State report (node side)
        17. State validation (client side)

        E2E pipeline: traps, traps everywhere


        Test core


        1. Preparing and encoding transaction data (given)
        2. Transaction execution (when)
        3. State validation (then)

        quasi-E2E tests


        Interact directly with runtime, skipping node layer.


        quasi-E2E tests


        #![allow(unused)]
        fn main() {
        #[test]
        fn flipping() -> Result<(), Box<dyn Error>> {
          let init_value = Session::<MinimalRuntime>::new(transcoder())?
              .deploy_and(bytes(), "new", &["true".to_string()], vec![])?
              .call_and("flip", &[])?
              .call_and("flip", &[])?
              .call_and("flip", &[])?
              .call_and("get", &[])?
              .last_call_return()
              .expect("Call was successful");
        
          assert_eq!(init_value, ok(Value::Bool(false)));
        
          Ok(())
        }
        }

        Local playing with contracts using drink-cli

        ๐Ÿงฌ Substrate

        The blockchain framework canonical to Polkadot and Parachains covered in depth, at a lower level.

        Introduction to Substrate


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: Introduction to Substrate description: Substrate Overview for web3 builders duration: 60 minutes ---

        Introduction to Substrate


        Before Going Any Further ๐Ÿ›‘

        While I speak, please clone polkadot-sdk, and run cargo build && cargo build --release.

        https://github.com/paritytech/polkadot-sdk/


        About These Lectures and Lecturer

        • Ground-up, low-level, but hands-on.
        • Intentionally avoiding FRAME, but giving you the tools to be successful at it.
        • Narratives above facts all.
        • Interrupts and questions are always welcome.

        What is Substrate?

        Substrate is a Rust framework for building blockchains.

        ---v

        Why Substrate?

        Notes:

        Highlight the multi-chain part.

        ---v

        Why Substrate?

        Notes:

        Polkadot is the biggest bet in this ecosystem against chain maximalism, and Substrate plays a big role in this scenario.

        ---v

        Why Substrate?

        • โ›“๏ธ Future is multi-chain.
        • ๐Ÿ˜ญ Building a blockchain is hard. Upgrading it even harder.
        • ๐Ÿ’ก Framework!
        • ๐Ÿง But which attitude to take?

        Core Philosophies of Substrate ๐Ÿ’ญ

        The pre-substrate way of thinking:

        • ๐Ÿ˜ญ Building a blockchain is hard. Upgrading it even harder.
        • ๐Ÿ’ช๐Ÿป We are going to spend maximal resources at making sure we get it right.

        ---v

        Core Philosophies of Substrate ๐Ÿ’ญ

        But has this worked?

        • ๐Ÿ˜ญ Bitcoin block size debate
        • 2๏ธโƒฃ L2s and beyond
        • ๐Ÿ“ˆ Ethereum gas price

        Notes:

        Bitcoin block size has never been and is an ongoing debate.

        I am not against L2s per se, but it is true that they mostly exist because the underlying protocol is too hard/slow to upgrade itself. ETH Gas prices also show that the underlying protocol cannot meet the demands of today.

        https://en.wikipedia.org/wiki/Bitcoin_scalability_problem https://ycharts.com/indicators/ethereum_average_gas_price

        ---v

        Core Philosophies of Substrate ๐Ÿ’ญ

        The Substrate way of thinking:

        • โ˜ฏ๏ธ Society and technology evolve
        • ๐Ÿฆธ Humans are fallible
        • ๐Ÿง  Best decision of today -> mistake of tomorrow

        ---v

        Core Philosophies of Substrate ๐Ÿ’ญ

        Outcomes of this:

        • ๐Ÿฆ€ Rust
        • ๐Ÿคฉ Generic, Modular and Extensible Design
        • ๐Ÿฆ Governance + Upgradeability

        Notes:

        Think about how each of these links back to "whatever you decide today will be a mistake soon".


        ๐Ÿฆ€ Rust

        • First line of defense: prevent human error when possible.
        • Safe language, no memory safety issues.

        Notes:

        So at least we don't want to deal with human error, and only deal with the fact that we cannot predict the future.

        Memory safety is a fundamental issue in most major system-level programming languages.

        Some such mistakes are impossible to make in Rust.

        ---v

        ๐Ÿฆ€ Rust

        int main() {
            int* x = malloc(sizeof(int));
            *x = 10;
            int* y = x;
            free(x);
            printf("%d\n", *y);  // Accessing memory after it's been freed
        }
        

        fn main() {
            let x = Box::new(10);
            let y = x;
            println!("{}", *y); // โŒ
        }

        Notes:

        another one:

        int* foo() {
            int x = 10;
            return &x;
        }
        
        int main() {
            int* y = foo();
            printf("%d\n", *y); // Accessing memory out of its scope
        }
        

        fn foo() -> &'static i32 {
            let x = 10;
            &x
        }
        
        fn main() {
            let y = foo();
            println!("{}", y); // โŒ
        }

        ---v

        ๐Ÿฆ€ Rust

        Microsoft and Google have each stated that software memory safety issues are behind around 70 percent of their vulnerabilities.

        Notes:

        https://www.nsa.gov/Press-Room/News-Highlights/Article/Article/3215760/nsa-releases-guidance-on-how-to-protect-against-software-memory-safety-issues/#:~:text=Microsoft%20and%20Google%20have%20each,70%20percent%20of%20their%20vulnerabilities.

        ---v

        ๐Ÿฆ€ Rust

        • ๐ŸŽ๏ธ Most Rust abstractions are zero-cost.
        • โฐ Rust has (almost) no "runtime".

        Notes:

        this is not 100% accurate though, rust has a small runtime that is the panic handler etc. Rust for Rustacean's chapter on no_std covers this very well.

        Also, this is a good time to talk about how we use "Runtime" in a different way.


        ๐Ÿคฉ Generic, Modular and Extensible Design

        • Second line of defense.
        • Our execution (possibly thanks to Rust) is perfect, but we can't predict the future.

        Notes:

        this is where a module, generic design is useful. You can change components easily based on the needs of the future.

        ---v

        ๐Ÿคฉ Generic, Modular and Extensible Design

        • Multiple consensus engines (BABE/Grandpa/AURA/PoW/Sassafras)
        • Multiple network protocols (QUIC, TCP)
        • Multiple database implementations (ParityDB, RocksDB)
        • Highly configurable, graph-based transaction-pool.
        • Easy to change primitives: AccountId, Signature, BlockNumber, Header, Hash, and many more.

        Notes:

        FRAME takes this even multiple steps further, but that's for later.

        These are all examples of being generic, modular and extensible at the Substrate level. FRAME takes these even further, but more on that later.

        ---v

        ๐Ÿคฉ Generic, Modular and Extensible Design

        • AlephZero: Custom finality, DAG-based, 1s block time.
        • Moonbeam: Ethereum compatible, build with substrate.
        • HydraDX: Custom transaction pool logic to match DEX orders.
        • Kulupu: Proof of work, custom hashing.

        Notes:

        Substrate has been coded, from the ground up, such that it is easy to have multiple implementations for certain functions. Heavy use of traits and generics is the key to achieve this. As noted, Substrate has a lot of APIs, and optional implementations. You are bound to the API, but not the particular implementation.


        ๐Ÿฆ Governance + Upgradeability

        • Third, final, and non-negotiable line of defense to survive the test of time.

        ---v

        ๐Ÿฆ Governance + Upgradeability

        • We have correct code, and components are easy to swap, replace, and upgrade.
        • What use is that, if we cannot agree on what to replace/upgrade?
        • Governance!
        • What use is governance, if the upgrade cannot be enacted?
        • (trustlessly) Upgradeability!

        Notes:

        Even if we are governable, but we still need "trust" to enact the upgrade, it is hardly any better. In essence, if an upgrade mechanism is not self-enacting, it might as well just reside offchain and be a signaling mechanism.

        ---v

        ๐Ÿฆ Governance + Upgradeability

        • โœ… Governance: Easy
        • ๐Ÿ˜ข Upgradeability: Not so much

        ---v

        ๐Ÿฆ Governance + Upgradeability

        • How does a typical blockchain upgrade itself?

        Notes:

        1. Discussion, offchain signaling
        2. Possibly onchain voting
        3. Hard(ish) Fork

        ---v

        ๐Ÿฆ Governance + Upgradeability

        ---v

        ๐Ÿฆ Governance + Upgradeability

        Notes:

        the problem is that the system is one big monolith protocol. Updating any part of it requires the whole thing to be updated.

        ---v

        ๐Ÿฆ Governance + Upgradeability

        The way to make a protocol truly upgradeable is to design a meta-protocol that is not upgradeable.

        ---v

        ๐Ÿฆ Governance + Upgradeability

        Note:

        In this figure, the meta-protocol, the substrate client, is not forklessly upgrade-able. It can only be upgraded with a fork. The Wasm protocol, though, can be upgraded forklessly.

        ---v

        ๐Ÿฆ Governance + Upgradeability

        ---v

        ๐Ÿฆ Governance + Upgradeability

        • Fixed meta-protocol?
        • ยญ "State machine as stored Wasm" in the Substrate client.
        • inherently upgradeable protocol?
        • Substrate Wasm Runtime

        Substrate Architecture

        ---v

        Substrate (simplified) Architecture

        Runtime (Protocol)

        • Application logic
        • Wasm (maybe FRAME)
        • Stored as a part of your chain state
        • Also known as: STF

        Client (Meta-protocol)

        • Native Binary
        • Executes the Wasm runtime
        • Everything else: Database, Networking, Mempool, Consensus..
        • Also known as: Host

        The Runtime

        • Runtime -> Application Logic.
        • A fancy term: Runtime -> State Transition Function.
        • A technical term: Runtime -> how to execute blocks.

        Notes:

        • I would personally call the runtime STF to avoid confusion with the "runtime" that is the general programming runtime, but kinda too late for that now.
        • Among the definitions of the Wasm runtime, let's recap what the state transition is.
        • The block execution definition will be described more in the Wasm-meta lecture.

        State Transition Function

        State

        Notes:

        entire set of data upon which we want to maintain a consensus. key value. associated with each block.

        ---v

        State Transition Function

        Transition Function

        ---v

        State Transition Function

        $$STF = F(block_{N}, state_{N}, code_{N}): state_{N+1}$$

        ---v

        State Transition Function

        Notes:

        The Wasm runtime in this figure is in fact obtained from the state (see 0x123)

        ---v

        State Transition Function

        ---v

        State Transition Function

        Notes:

        THIS IS HOW A META-PROTOCOL MAKES A SYSTEM UPGRADE-ABLE.

        could we have updated the code in N+1? By default, no because we load the wasm before you even look into the block.

        IMPORTANT: State is not IN THE BLOCK, each state has AN ASSOCIATED BLOCK.

        Keeping the state is 100% optional. You can always re-create the state of block N by re-executing block [0, .., N-1].

        ofc, changing the Wasm code cannot be done by anyone. That's up to governance.


        Full Substrate Architecture


        Positive Consequences of Wasm Runtime ๐Ÿ”ฅ

        ---v

        ๐Ÿค– Deterministic Execution

        • Portable, yet deterministic.

        Notes:

        Wasm's instruction set is deterministic, so all good.

        ---v

        ๐Ÿงฑ Sandboxing

        • Useful when executing untrusted code.
        1. Smart contracts
        2. Parachain runtime

        Notes:

        How can we guarantee that neither enter an infinite loop, or try to access the filesystem?

        ---v

        ๐ŸŒˆ Easier (light)Client Development

        Notes:

        for the case of client, your client only needs to implement a set of host environments, and NOT re-implement the business logic.

        Simply compare the process to create an alternative client for Ethereum, where you need to re-implement the EVM.

        Same applies to light client, as they do not need to deal with the state transition function.

        ---v

        ๐Ÿ˜Ž Forkless Upgrade

        ---v

        ๐Ÿ˜Ž Forkless Upgrade

        ---v

        ๐Ÿ˜Ž Forkless Upgrade

        This update was:

        1. Forkless
        2. Self-enacting

        Notes:

        take a moment to establish that this upgrade is forkless. The runtime is upgraded, but the client is not. In fact, the client didn't need to know about this at all.

        This is what the meta-protocol achieves.


        Negative Consequences of Wasm Runtime

        • ๐Ÿ˜ฉ Constrained resources (memory, speed, host access).
        • ๐ŸŒˆ Client diversification != state-transition diversification

        Notes:

        • 4GB of memory, which we limit even further.
        • Wasm has no allocator+panic handler of its own
        • Can be slower than native, depending on the executor/execution method.
        • Limited access to the host host env, all needs to be done through syscalls.

        Less state-transition diversification, because the runtime is the same for all clients. If there is a bug in it, everyone is affected.


        Consensus <> Runtime ๐Ÿค”

        • Yes, consensus is not a core part of a blockchain runtime. Why?
        • Not part of your STF!
        • The consensus protocol is to your runtime what HTTP is to Facebook.

        Notes:

        comments from Joshy:

        I think this is important. The runtime is the application you want to run.

        Consensus is outside of this application helping us agree what the official state of this runtime is. Last wave I used this analogy.

        Imagine a writers room for some TV show. Writers are sitting around coming up with potential plot points for future episodes. Any of their ideas could work. But eventually they need to agree what the next episode to air actually will be.


        Database <> State ๐Ÿค”

        • State is the entire set of key value data that is associated with one block.
        • Database is the component that allows this be stored in disk. May or may not be key-value.

        Notes:

        State is sometimes called "storage" as well.


        Database <> Runtime ๐Ÿค”

        • Yes, data is stored outside of the runtime. Why?
        • Wasm runtime does not have the means to store it.
        • Yet, the interpretation of the data is up to the runtime.

        ---v

        Database <> Runtime ๐Ÿค”

        ---v

        The Client: Database ๐Ÿค”

        • The database, from the client's PoV, is an untyped, key-value storage.
        • The runtime knows which key/value means what.

        State of Light Clients

        • Client that follows headers, therefore knows state roots and can ask for state-proofs to do more.

        ---v

        State of Light Clients

        • Not only possible, but they can also run as Wasm, in the browser!
        • "Substrate Connect" / SMOLDOT

        Notes:

        What was a light client? follows only block headers, therefore knows state roots, and a few other pieces of information, others send it state proofs if it wishes to do more.

        SMOLDOT is not exactly a substrate client. It is mainly designed to work with Polkadot. But with minimal tweaks, you could make it work for more substrate based chains.

        This has to do with the fact that consensus and a few other bits of the client and runtime are not 100% independent. For example, GRANDPA has a pallet on the runtime side, but is mostly in the client. Now, a client that is configured with GRANDPA can only work with runtimes that are also configured with GRANDPA.


        Communication Paths

        ---v

        Communication Paths

        ---v

        Example: SCALE vs JSON

        • SCALE is an efficient, non-descriptive, binary encoding format, used EXTENSIVELY in the Substrate ecosystem.

        ---v

        Example: SCALE vs JSON

        use parity_scale_codec::{Encode};
        
        #[derive(Encode)]
        struct Example {
        	number: u8,
        	is_cool: bool,
        	optional: Option<u32>,
        }
        
        fn main() {
        	let my_struct = Example {
        		number: 42,
        		is_cool: true,
        		optional: Some(69),
        	};
        	println!("{:?}", my_struct.encode());
        	// [42, 1, 1, 69, 0, 0, 0]
        	println!("{:?}", my_struct.encode().len());
        	// 7
        }

        ---v

        Example: SCALE vs JSON

        use serde::{Serialize};
        
        #[derive(Serialize)]
        struct Example {
        	number: u8,
        	is_cool: bool,
        	optional: Option<u32>,
        }
        
        fn main() {
        	let my_struct = Example {
        		number: 42,
        		is_cool: true,
        		optional: Some(69),
        	};
        	println!("{:?}", serde_json::to_string(&my_struct).unwrap());
        	// "{\"number\":42,\"is_cool\":true,\"optional\":69}"
        	println!("{:?}", serde_json::to_string(&my_struct).unwrap().len());
        	// 42
        }

        Substrate and Polkadot


        Substrate and Smart Contracts

        ---v

        Substrate and Smart Contracts

        A Substrate-Connect extension is syncing a chain who's runtime is executing wasm contracts.

        Question: How many nested Wasm blobs are executing one another?

        ---v

        Substrate and Smart Contracts

        • The browser is executing:
        • a Wasm blob (substrate-connect)
        • that executes a Wasm blob (runtime)
        • that executes a Wasm blob (contract)

        ---v

        Substrate and Smart Contracts

        ---v

        Substrate and Smart Contracts

        • So when should you write with a smart contract (Ink!) and when a Runtime (FRAME)?

        Notes:

        I was asked this yesterday as well. My latest answer is: if you don't need any of the customizations that a blockchain client/runtime gives to you, and the performance of a shared platform is okay for you, then go with a smart contract. If you need more, you need a "runtime" (some kind of chian, parachain or solo)

        An example of customization is that a runtime has access to on_initialize etc.

        Also, a contract can not have fee-less transactions.

        Also, a contract usually depends on a token for gas, while a runtime can be in principle token-less fee-less.


        Technical Freedom vs Ease


        Substrate: The Gaming Console of Blockchains!

        Substrate Client

        Substrate's Wasm Runtime

        Notes:

        Another good analogy: Client is the FPGA, and FRAME/Wasm is the VHDL.


        Lecture Recap

        • Substrate's design stems from 3 core principles:
          • Rust, Generic Design, Upgradeability/Governance
        • Client / Runtime architecture
        • State Transition
        • Positive and negative consequences of Wasm
        • Substrate next to Polkadot and other chains.
        • Substrate for Smart Contracts.

        ---v

        Recap: Substrate Architecture

        ---v

        Recap: ๐Ÿฆ Governance and Upgradeability

        A timeless system must be:

        1. Generic
        2. Governable
        3. Trust-less-ly Upgradeable.

        Substrate's Wasm meta-protocol exactly enables the latest โœ…

        Notes:

        Question: how would you put the meta-protocol of Substrate into words?

        The client is basically a wasm meta-protocol that does only one thing. This meta-protocol is hardcoded, but the protocol itself is flexible.


        Rest of This Module! ๐Ÿ˜ˆ

        Track: Main Lectures

        • Wasm Meta Protocol
        • Substrate Storage

        Track: Aux Lecture

        • TX-Pool
        • Substrate: Show Me The Code
        • Substrate Interactions
        • SCALE

        Track: Graded Activity

        • FRAME-Less

        ---v

        Rest of This Module! ๐Ÿ˜ˆ

        Day 0

        • Introduction โœ… (60m)
        • Wasm Meta Protocol (120+m)
          • Activity: Finding Runtime APIs and Host Functions in Substrate
        • ๐ŸŒญ Lunch Break
        • Show Me The Code (60m)
        • Substrate Interactions (60m)
        • FRAME-Less Activity (60m)

        Notes:

        We are aware that the module is highly skewed in terms of lecture time, but it is intentional and we want to see how it works. This allows you to kickstart with your assignment earlier.

        ---v

        Rest of This Module! ๐Ÿ˜ˆ

        Day 1

        • Transaction Pool (60m)
        • SCALE (60m)
        • Substrate/FRAME Tips and Tricks
        • ๐ŸŒญ Lunch Break
        • FRAME-Less Activity

        ---v

        Rest of This Module! ๐Ÿ˜ˆ

        Day 2

        • Substrate Storage (90m)
        • FRAME-Less Activity
        • ๐ŸŒญ Lunch Break
        • End of Module ๐ŸŽ‰

        Additional Resources! ๐Ÿ˜‹

        Check speaker notes (click "s" ๐Ÿ˜‰)

        Notes:

        Substrate Client (/client/* / sc-*) is licensed under GPL v3.0 with a classpath linking exception.

        • Apache2 allows teams full freedom over what and how they release, and giving licensing clarity to commercial teams.

        • GPL3 ensures any deeper improvements made to Substrate's core logic (e.g. Substrate's internal consensus, crypto or database code) to be contributed back so everyone can benefit.

        • Currently, the Wasm binary spec v1 is used, read more about the new version here: https://webassembly.github.io/spec/core/binary/index.html

        Post Lecture Feedback

        • a takeaway after each section, more clear path (shawn).

        Appendix: What is Wasm Anyways?

        WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable compilation target for programming languages, enabling deployment on the web for client and server applications.

        ---v

        What is Wasm Anyways?

        ---v

        What is Wasm Anyways?

        • Wasm โค๏ธ Web
        • Streaming and rapid compilation.
        • Designed with the concept of host in mind. Sandboxed, permissioned sys-calls.

        Anyone remember "Java Applets"?

        Notes:

        People actually tried sticking things like JVM into the browser (Java Applets), but it didn't work.

        ---v

        How to Write a Wasm Runtime?

        • Any language that can compile to Wasm and exposes a fixed set of functions, to be used by the client.
        • ... But, of course, Substrate comes with a framework to make this developer-friendly, FRAMEโ„ข๏ธ.

        Appendix: More Diagrams of Substrate and Polkadot

        Notes:

        I made these figures recently to explain the relationship between Substrate, Cumulus and Polkadot. They use the most generic term for client and runtime, namely "Host" and "STF" respectively.

        ---v

        Substrate

        ---v

        Polkadot

        ---v

        A Parachain

        Substrate Wasm meta-protocol


        How to use the slides - Full screen (new tab)
        Slides Content
        --- title: Substrate Wasm meta-protocol description: A deeper dive into how the Wasm is meta-protocol functions in substrate. duration: 60 minutes ---

        Substrate Wasm Meta Protocol


        Part 1

        • This is a big lecture, so I divided it into two small parts, that's all ๐Ÿซต๐Ÿป

        It All Began With a Runtime..

        ---v

        It All Began With a Runtime..

        • Personal opinion:

        Substrate technology stack will put "Wasm stored onchain" on the map,
        the same way Ethereum put "smart contracts stored onchain" on the map.

        Notes:

        It is only a matter of time until every blockchain is doing the same thing.

        ---v

        It All Began With a Runtime..

        • The Client / Runtime division is one of the most important design decisions in Substrate.
          • ๐Ÿ‘ฟ Bad: Fixed opinion.
          • ๐Ÿ˜‡ Good: Enables countless other things to not be fixed.

        Notes:

        Recall that the boundary for this division is the state transition


        Substrate: a short recap

        ---v

        Substrate: a short recap

        • Host Functions: Means of a runtime communicating with its host environment, i.e. the Substrate client.

        ---v

        Substrate: a short recap

        • Runtime API: The well-defined functions that a Wasm substrate runtime is providing.

        Notes:

        Building a Wasm module's activity was building something akin to runtime-apis

        ---v

        Substrate: a short recap

        • Database is on the client side, storing an opaque key-value state per block.

        ---v

        Substrate: a short recap

        • Communication language of client/runtime is SCALE:
        flowchart LR B[Known Type eg. `u32`] --Encode--> V["Vec(u8)"] V --Decode-->B

        Learning with Examples

        and some pseudo-code

        Notes:

        During each example, we deduce what host functions and/or runtime APIs are needed.


        Example #1: State

        • The runtime wants to add 10 units to Kian's balance.

        ---v

        Example #1: State

        #![allow(unused)]
        fn main() {
        // the runtime decides what key stores kian's balance.
        key: Vec<u8> = b"kian_balance".to_vec();
        
        // the runtime reads the raw bytes form that key.
        let current_kian_balance_raw: Vec<u8> = host_functions::get(key);
        
        // and needs to know to which type it should be decoded, u128.
        let mut current_kian_balance: u128 = current_kian_balance_raw.decode();
        
        // actual logic.
        current_kian_balance += 10;
        
        // encode this again into an opaque byte-array.
        let new_balance_encoded: Vec<u8> = current_kian_balance.encode();
        
        // write the encoded bytes again.
        host_functions::set(key, new_balance_encoded);
        }

        ---v

        Example #1: State

        • ๐Ÿ’ก The runtime needs host functions to read/write to state.
        #![allow(unused)]
        fn main() {
        fn get(key: Vec<u8>) -> Vec<u8>;
        fn set(key: Vec<u8>, value: Vec<u8>);
        }

        Notes:

        ofc the IO to these functions is all opaque bytes, because the client does not know the state layout.

        ---v

        Example #1: State

        • could we have communicated with the client like this?
        #![allow(unused)]
        fn main() {
        fn set_balance(who: AccountId, amount: u128)
        }

        Notes:

        This would imply that the client would have to know, indefinitely, the types needed for account id and balance. Also, it would have to know the final key for someone's balance.

        ---v

        Example #1: State

        • Exceptions:
        #![allow(unused)]
        fn main() {
        /// The keys known to the client.
        mod well_known_keys {
          const CODE: &[u8] = b":code";
        }
        }

        Notes:

        See https://paritytech.github.io/substrate/master/sp_storage/well_known_keys/index.html

        ---v

        Example #1: State


        Example #2: Block Import

        ---v

        Example #2: Block Import

        • Client's view of the state -> Opaque.
        • Client's view of the transactions? ๐Ÿค”

        Notes:

        Short answer is: anything that is part of the STF definition must be opaque to the client, and is upgradeable, but we will learn this later.

        ---v

        Example #2: Block Import

        • Transactions format is by definition part of the state transition function as well.
        • What about header, and other fields in a typical block?

        Notes:

        as in, do we want to able to update our transactions format as well in a forkless manner? we want the runtime to be able to change its transactions format as well, in a forkless manner.

        The answer to the latter is more involved. The short answer is that these fields like header must be known and established between client and runtime. If you want to alter the header format, that's a hard fork.

        The concept of digest is a means through which additional data can be put in the header without breaking changes, but that is outside the scope of this lecture.

        Yet, as with other primitives, substrate allows you to readily change your header type when you are building your blockchain. This is achieved by a set of traits in sp-runtime. Notably, trait Block and trait Header in this crate define what it means to be a header and block, and as long as you fulfill that, you are good to go.

        Also, substrate provides one set of implementation for all of these types in https://paritytech.github.io/substrate/master/sp_runtime/generic/index.html

        ---v

        ---v

        Example #2: Block Import

        #![allow(unused)]
        fn main() {
        struct ClientBlock {
          header: Header,
          transactions: Vec<Vec<u8>>
        }
        }
        #![allow(unused)]
        fn main() {
        struct RuntimeBlock {
          header: Header,
          transaction: Vec<KnownTransaction>
        }
        }

        Notes:

        this slide is intentionally using the keyword transaction instead of extrinsic.

        ---v

        Example #2: Block Import

        #![allow(unused)]
        fn main() {
        // fetch the block from the outer world.
        let opaque_block: ClientBlock = networking::import_queue::next_block();
        
        // initialize a wasm runtime.
        let code = database::get(well_known_keys::CODE);
        let runtime = wasm::Executor::new(code);
        
        // call into this runtime.
        runtime.execute_block(opaque_block);
        }

        ---v

        Example #2: Block Import

        • ๐Ÿ’ก The client needs a runtime API to ask the runtime to execute the block.
        #![allow(unused)]
        fn main() {
        fn execute_block(opaque_block: ClientBlock) -> Result<_, _> { .. }
        }

        Notes:

        execute_block is the most basic, fundamental runtime API that any substrate based runtime has to implement in order to be called a "blockchain runtime".


        Example #2: Block Import: Something Missing

        #![allow(unused)]
        fn main() {
        // ๐Ÿค”
        let code = database::get(well_known_keys::CODE);
        
        // ๐Ÿค”
        runtime.execute_block(opaque_block);
        }

        Notes:

        • From which block's state do we fetch the code??
        • This probably calls into host_functions::{get/set} internally. What do we return

        ---v

        Example #2: Block Import

        #![allow(unused)]
        fn main() {
        // fetch the block from the outer world.
        let block: ClientBlock = networking::import_queue::next_block();
        
        // get the parent block's state.
        let parent = block.header.parent_hash;
        let mut state = database::state_at(parent);
        
        // initialize a wasm runtime FROM THE PARENT `state`!
        let code = state::get(well_known_keys::CODE);
        let runtime = wasm::Executor::new(code);
        
        // call into this runtime, updates `state`.
        state.execute(|| {
          runtime.execute_block(block);
        });
        
        // create the state of the next_block
        database::store_state(block.header.hash, state)
        }

        Notes:

        • Question: why is state defined as mut?
        • within these snippets, more or less, everything inside state.execute is executed within Wasm.

        ---v

        Example #2: Block Import

        • A state key is only meaningful at a given block.
        • A :code is only meaningful at at given block.
        • ๐Ÿ’ก A runtime (API) is only meaningful when executed at a give block.

        Notes:

        • The same way that Alice's balance value is only meaningful when read at a given block.

        • Based on that:

          • The correct runtime code is loaded.
          • The correct state (and other host functions) is provided.
        • Similarly, almost all RPC operations that interact with the runtime have an Option<Hash> argument. This specifies "at which block to load the runtime and state from".

        ---v

        Example #2: Block Import

        • I can add one more small touch to this to make it more accurate.. ๐ŸคŒ

        ---v

        Example #2: Block Import

        #![allow(unused)]
        fn main() {
        // fetch the block from the outer world.
        let block: ClientBlock = networking::import_queue::next_block();
        
        // get the parent hash. Note that `sp_runtime::traits::Header` provides this.
        let parent = block.header.parent_hash;
        let mut state = database::state_at(parent);
        
        // initialize a wasm runtime FROM THE PARENT `state`!
        let code = state::get(well_known_keys::CODE);
        let runtime = wasm::Executor::new(code);
        
        // call into this runtime, update `state`.
        state.execute(|| {
          // within this, we probably call into `host_functions::set` a lot.
          runtime.execute_block(block);
        
          let new_state_root = host_functions::state_root();
          let claimed_state_root = block.header.state_root;
          assert_eq!(new_state_root, claimed_state_root);
        });
        
        // create the state of the next_block
        database::store_state(block.header.hash, state)
        }

        ---v

        Example #2: Block Import: Recap


        Detour: Extrinsic

        • Previous slides used the term "transactions" in a simplified way. Let's correct it.

        ---v

        Detour: Extrinsic

        %%{init: {'theme': 'dark', 'themeVariables': { 'darkMode': true }}}%% flowchart TD E(Extrinsic) ---> I(Inherent); E --> T(Transaction) T --> ST("Signed (aka. Transaction)") T --> UT(Unsigned)

        ---v

        Detour: Extrinsic

        • An Extrinsic is data that come from outside of the runtime.
        • ยญInherents are data that is put into the block by the block author, directly.
        • ยญ Yes, transactions are a type of extrinsic, but not all extrinsics are transactions.
        • ยญ So, why is it called Transaction Pool and not Extrinsic Pool?

        Notes:

        Extrinsics are just blobs of data which can be included in a block. Inherents are types of extrinsic which are crafted by the block builder itself in the production process. They are unsigned because the assertion is that they are "inherently true" by virtue of getting past all validators. Notionally the origin can be said to be a plurality of validators. Take for example the timestamp set inherent. If the data were sufficiently incorrect (i.e. the wrong time), then the block would not be accepted by enough validators and would not become canonicalized. So the "nobody" origin is actually the tacit approval of the validators. Transactions are generally statements of opinion which are valuable to the chain to have included (because fees are paid or some other good is done). The transaction pool filters out which of these are indeed valuable and nodes share them.


        Example #3: Block Authoring

        ---v

        Example #3: Block Authoring

        ---v

        Example #3: Block Authoring

        ---v

        Example #3: Block Authoring

        ---v

        Example #3: Block Authoring

        ---v

        Example #3: Block Authoring

        Notes:

        The point being, eventually the pool builds a list of "ready transactions".

        ---v

        Example #3: Block Authoring

        ---v

        Example #3: Block Authoring

        ---v

        Example #3: Block Authoring

        ---v

        Example #3: Block Authoring

        ---v

        Example #3: Block Authoring

        ---v

        Example #3: Block Authoring

        ---v

        Example #3: Block Authoring

        ---v

        Example #3: Block Authoring

        ---v

        Example #3: Block Authoring

        #![allow(unused)]
        fn main() {
        // get the best-block, based on whatever consensus rule we have.
        let (best_number, best_hash) = consensus::best_block();
        
        // get the latest state.
        let mut state = database::state_at(best_hash);
        
        // initialize a wasm runtime.
        let code = state::get(well_known_keys::CODE);
        let runtime = wasm::Executor::new(code);
        
        // get an empty client block.
        let mut block: ClientBlock = Default::default();
        
        // repeatedly apply transactions.
        while let Some(next_transaction) = transaction_pool_iter::next() {
          state.execute(|| {
            runtime.apply_extrinsic(next_transaction);
          });
          block.extrinsics.push(next_transaction);
        }
        
        // set the new state root.
        block.header.state_root = state.root();
        }

        Notes:

        • What is the type of next_ext? Vec<u8>
        • Do we actually loop forever until the tx-pool is empty? probably not!

        ---v

        Example #3: Block Authoring

        • Substrate based runtimes are allowed to perform some operations at the beginning and end of each block.
        • โœ‹๐Ÿป And recall that a smart contract could not do this.

        ---v

        Example #3: Block Authoring

        #![allow(unused)]
        fn main() {
        // get the best-block, based on whatever consensus rule we have.
        let (best_number, best_hash) = consensus::best_block();
        
        // get the latest state.
        let mut state = database::state_at(best_hash);
        
        // initialize a wasm runtime.
        let code = state::get(well_known_keys::CODE);
        let runtime = wasm::Executor::new(code);
        
        // get an empty client block.
        let mut block: ClientBlock = Default::default();
        
        // tell this runtime that you wish to start a new block.
        runtime.initialize_block();
        
        // repeatedly apply transactions.
        while let Some(next_ext) = transaction_pool_iter::next() {
          state.execute(|| {
            runtime.apply_extrinsic(next_ext);
          });
          block.extrinsics.push(next_ext);
        }
        
        // tell the runtime that we are done.
        runtime.finalize_block();
        
        // set the new state root.
        block.header.state_root = state.root();
        }

        ---v

        Example #3: Block Authoring

        • What about Inherents?

        ---v

        Example #3: Block Authoring

        #![allow(unused)]
        fn main() {
        // get the best-block, based on whatever consensus rule we have.
        let (best_number, best_hash) = consensus::best_block();
        
        // get the latest state.
        let mut state = database::state_at(best_hash);
        
        // initialize a wasm runtime.
        let code = state::get(well_known_keys::CODE);
        let runtime = wasm::Executor::new(code);
        
        // get an empty client block.
        let mut block: ClientBlock = Default::default();
        
        // tell this runtime that you wish to start a new block.
        runtime.initialize_block();
        
        let inherents: Vec<Vec<u8>> = block_builder::inherents();
        block.extrinsics = inherents;
        
        // repeatedly apply transactions.
        while let Some(next_ext) = transaction_pool_iter::next() {
          state.execute(|| {
            runtime.apply_extrinsic(next_ext);
          });
          block.extrinsics.push(next_ext);
        }
        
        // tell the runtime that we are done.
        runtime.finalize_block();
        
        // set the new state root.
        block.header.state_root = state.root();
        }

        Notes:

        while inherents can in principle come at any point in the block, since FRAME restricts them to come first, we also keep our example aligned.

        Should you wish to see the real version of this, check this crate: https://paritytech.github.io/substrate/master/sc_basic_authorship/index.html

        ---v

        Example #3: Block Authoring

        #![allow(unused)]
        fn main() {
        fn initialize_block(..) { ... }
        // note the opaque extrinsic type.
        fn apply_extrinsic(extrinsic: Vec<u8>) { ... }
        fn finalize_block(..) { ... }
        }

        Notes:

        in fact, the client also builds its inherent list with the help of the runtime.


        BUT WAIT A MINUTE ๐Ÿ˜ฑ

        • if the code changes, all the following can also change:

          • What state key is Kian's balance.
          • What extrinsic format is valid.
        • How on earth is an application (i.e. a wallet) is supposed to survive?

        ---v

        BUT WAIT A MINUTE ๐Ÿ˜ฑ

        • Metadata ๐ŸŽ‰
        #![allow(unused)]
        fn main() {
        fn metadata() -> Vec<u8> { ... }
        }

        Notes:

        Notice the opaque return type.

        In order the address the mentioned issue, metadata must be a runtime API.


        • Metadata contains all the basic information to know about all storage items, all extrinsics, and so on. It will also help a client/app decode them into the right type.
        • Substrate itself doesn't impose what the metadata should be. It is Vec<u8>.
        • FRAME based runtime expose a certain format, which is extensively adopted in the ecosystem.

        ---v

        BUT WAIT A MINUTE ๐Ÿ˜ฑ

        • Recall the fact that "runtime is only meaningful at a certain block".
        • Two different runtimes at block N and N+1 return different metadata โœ….

        Notes:

        By Applications/Clients I really mean anyone/anything. Substrate client doesn't really use metadata because it is dynamically typed, but if needed, it could.


        Radical Upgradeability

        Comes at the cost of radical opaque/dynamic typing.

        Notes:

        I wish you could have both, but not so easy.

        Some personal rant: radical upgrade-ability is the biggest advantage, and arguably one of the main develop-ability problems of the substrate ecosystem. Writing clients, such as block explorers, scanners, and even exchange integration are orders of magnitude harder than a blockchain that has a fixed format and only changes every 18 months at most. That being said, this is a battle that is to me obvious: we simply HAVE to win. When ethereum first introduced smart contracts, everyone probably had the same class of issues. This is the same matter, on a a different level.

        also, as noted in an earlier slide, once you make it work for one chain, it works for many chains.


        Oblivious Client ๐Ÿ™ˆ๐Ÿ™‰

        • The underlying reason why the client is "kept in the dark" is so that it wouldn't need to care about the runtime upgrading from one block to the other.

        ---v

        Oblivious Client ๐Ÿ™ˆ๐Ÿ™‰

        $$STF = F(blockBody_{N}, state_{N}) > state_{N+1}$$

        Anything that is part of the STF is opaque to the client, but it can change forklessly!

        • The `F` itself (your Wasm blob)? It can change!
        • Extrinsic format? It can change!
        • State format? It can change!

        Notes:

        In essence, all components of the STF must be opaque to the client. Vec<u8>. Metadata is there to assist where needed. This is why forkless upgrades are possible in substrate.

        ---v

        Oblivious Client ๐Ÿ™ˆ๐Ÿ™‰

        • What