Building A Blockchain in Rust & Substrate: [A Step-by-Step Guide for Developers]

A beginner-friendly tutorial on how to build hardcore blockchain infrastructure in Substrate, an open source framework. In this self-guided tutorial, you’ll build a gas-less, Bitcoin-like blockchain from scratch. You'll learn that a blockchain is a lot more powerful than just smart contracts. Feel free to repurpose any of this content to host your own workshops! What you will learn: Implement the UTXO ledger model, Bitcoin’s accounting mechanism Change the network transaction pool logic Configure a genesis block & write some tests Deploy your chain & interact with the live node using a web client Requirements: NO blockchain & Rust knowledge needed Basic programming experience Estimated time: 3 hours For a more detailed walkthrough where I thoroughly explain why we’re doing each step, and where we take time to discuss Rust traits, check out this video tutorial accompaniment: Let’s get started! Installation 1. Get the latest stable version of Rust & WebAssembly tools. In your terminal, run the commands: curl https://sh.rustup.rs -sSf | sh rustup update nightly rustup target add wasm32-unknown-unknown —toolchain nightly rustup update stable cargo install —git https://github.com/alexcrichton/wasm-gc # On Windows, download and run rustup-init.exe from https://rustup.rs instead # On Macs: If you counter any issues or use a different OS , check out this . detailed setup guide 2. In a new terminal, also clone your copy of this tutorial's boilerplate code: git https://github.com/substrate-developer-hub/utxo-workshop.git git fetch origin workshop:workshop git checkout workshop /project_base_directory cargo -p utxo-runtime clone # [Optional] Once step 1 installations are completed # Run the following commands to shorten future build time cd test This repo also contains an updated & complete Bitcoin implementation in the branch (as a cheat), so make sure you check out the branch to start from scratch! master workshop Depending on your CPU, 1st-time Rust installations can take up to 10-20 minutes. Let’s use this time now for a crash course how Bitcoin works, as well as explore this developer SDK that we're using! Quick Primer on the UTXO ledger model If you own a bank account, you’re already familiar with the “accounts based” ledger model. This is where your bank account has a total balance that gets credited or debited per transaction. Bitcoin offers a fundamentally different ledger model called UTXO, or Unspent Transaction Outputs. UTXO works like travellers checks in that: Whoever physically has the check can spend the money. In other words, the spending permission is tied to the money, not an account number. You can’t rip up a check and spend its parts. You have to spend the entire check, and receive any change in a new check. In the following example, Bob has a UTXO worth $50. He wants to give Alice $0.5, so he destroys his $50 UTXOs, and creates two new UTXOs of values $0.5 (for Alice) and $49.5 (as change). Image source: https://freedomnode.com/ Cryptography is the underlying mechanism that allows only Bob, and not anyone else, to spend his UTXOs. This information is stored in one of each UTXO’s 3 fields: of where the UTXO exists, sort of like an address pointer in a database A reference hash of the UTXO, e.g. $50 The monetary value The public key of its “owner” The public key corresponds with a secret “private key”, that only the owner would have. So to spend the UTXO, the owner has to cryptographically "sign over" a transaction with his corresponding private key. The signature can be later checked against the "public key" to verify its validity. For example, when Alice spends her UTXO, it would look like this: She creates a new transaction (gray background), supplies her UTXO as input to be spent, and in the field, Alice provides her signature. sigscript Note: Alice is "signing over" the details of the entire transaction. This has the benefit of locking in the transaction output details, to prevent network level tampering. Later on, the blockchain will verify that Alice did indeed authorise all of the details of this entire transaction. We’ll cover the security implications in greater detail in Part 2, when you secure your blockchain against malicious attacks. Quick Primer on Substrate Framework You're using an open-source blockchain framework called . Substrate is and compiles to a binary instruction format called WebAssembly (WAsm). Substrate Rust based Out of the box, you get core blockchain components like a distributed database, a peer-to-peer networking layer, and various consensus mechanisms we can choose from. In this tutorial, we’ll get very familiar with the transaction queue and runtime module layers Let's start coding! Part 1: Build the logic layer of the blockchain 1. In terminal, do a quick Rust compiler check to ensure everything downloaded correctly: cargo check -p utxo-runtime # Don’t worry if this takes a while, just let it run! # You should see a few warnings but no breaking errors 2. Open up the project in your favourite Rust compatible IDE. I recommend IntelliJ or VSCode for their Rust syntax highlighting. 3. Open the subdirectory, which houses the blockchain runtime. Then, open up file, which is where you’ll build most of your Bitcoin UTXO logic. Runtime utxo.rs You’ll see a typical Substrate starter template, with inline comments that explain how to use the SDK. Further down, you should also see where you can write unit tests. 4. Right after dependency import lines, create the data structures needed to represent UTXOs and a UTXO transaction. { inputs: , outputs: , } { outpoint: H256, sigscript: H512, } { value: Value, pubkey: H256, } /// Single transaction to be dispatched #[cfg_attr(feature = , derive(Serialize, Deserialize))] "std" #[derive(PartialEq, Eq, PartialOrd, Ord, Default, Clone, Encode, Decode, Hash, Debug)] pub struct Transaction /// UTXOs to be used as inputs for current transaction pub Vec /// UTXOs to be created as a result of current transaction dispatch pub Vec /// Single transaction input that refers to one UTXO #[cfg_attr(feature = , derive(Serialize, Deserialize))] "std" #[derive(PartialEq, Eq, PartialOrd, Ord, Default, Clone, Encode, Decode, Hash, Debug)] pub struct TransactionInput /// Reference to an UTXO to be spent pub /// Proof that transaction owner is authorized to spend referred UTXO & /// that the entire transaction is untampered pub /// Single transaction output to create upon transaction dispatch #[cfg_attr(feature = , derive(Serialize, Deserialize))] "std" #[derive(PartialEq, Eq, PartialOrd, Ord, Default, Clone, Encode, Decode, Hash, Debug)] pub struct TransactionOutput /// Value associated with this output pub /// Public key associated with this output. In order to spend this output /// owner must provide a proof by hashing the whole `Transaction` and /// signing it with a corresponding private key. pub For a more detailed walkthrough & line-by-line explanation of the above, check out this video accompaniment . 5. Designate what gets stored on the blockchain chain-state. This is done inside a Rust macro called . You'll be storing a hashmap of 256bit pointers to UTXOs as the key, and the UTXO struct itself as the value. Implement the following: decl_storage decl_storage! { Module Utxo { UtxoStore build(|config: &GenesisConfig| { config.genesis_utxos .iter() .cloned() .map(|u| (BlakeTwo256::hash_of(&u), u)) .collect:: >() }): map H256 => ; RewardTotal get(reward_total): Value; } add_extra_genesis { config(genesis_utxos): ; } } trait Store for as /// All valid unspent transaction outputs are stored in this map. /// Initial set of UTXO is populated from the list stored in genesis. Vec Option /// Total reward value to be redistributed among authorities. /// It is accumulated from transactions during block execution /// and then dispersed to validators on block finalization. pub Vec Notice, in addition to configuring storage, we also configured how* this storage will be populated at the genesis block. At block 0, you will be able to seed your blockchain with an existing vector of UTXOs to spend! 6. Create the transaction signatures that will allow users of your Bitcoin blockchain to spend UTXOs. Let's implement the function: spend decl_module! { origin: T::Origin { () = ; (_origin, transaction: Transaction) -> DispatchResult { transaction_validity = Self::validate_transaction(&transaction)?; Self::update_storage(&transaction, transaction_validity.priority )?; Self::deposit_event(Event::TransactionSuccess(transaction)); (()) } } // External functions: callable by the end user pub struct Module for enum Call where fn deposit_event default /// Dispatch a single transaction and update UTXO set accordingly pub fn spend // TransactionValidity{} let as u128 Ok Note, we'll be implementing a few helper functions for this spend transaction logic shortly. 7. Blockchains can also emit events whenever there are on-chain transactions. Set up your blockchain to recognize a event type. TransactionSuccess decl_event!( { TransactionSuccess(Transaction), } ); pub enum Event /// Transaction was executed successfully Note, in step 2, you are already emitting this event after every successful spend transaction. For a more line-by-line explanation of the above, check out this video accompaniment . Part 2: Secure the chain against malicious attacks In Part 1, we scaffolded the basic building of UTXOs. In this section, we’ll get into the cryptographic security of our chain and implement UTXO transaction logic. In fact, there are many vulnerabilities that the Bitcoin blockchain defends against. Inputs and outputs are not empty Each input exists and is used exactly once Each output is defined exactly once and has nonzero value Total output value must not exceed total input value New outputs do not collide with existing ones Replay attacks are not possible Provided input signatures are valid The input UTXO is indeed signed by the owner Transactions are tamperproof 1. Let's now implement the function to ensure these security checks. validate_transaction Module { (transaction: &Transaction) -> { ensure!(!transaction.inputs.is_empty(), ); ensure!(!transaction.outputs.is_empty(), ); { input_set: BTreeMap =transaction.inputs.iter().map(|input| (input, ())).collect(); ensure!(input_set.len() == transaction.inputs.len(), ); } { output_set: BTreeMap = transaction.outputs.iter().map(|output| (output, ())).collect(); ensure!(output_set.len() == transaction.outputs.len(), ); } total_input: Value = ; total_output: Value = ; output_index: = ; simple_transaction = Self::get_simple_transaction(transaction); missing_utxos = ::new(); new_utxos = ::new(); reward = ; input transaction.inputs.iter() { (input_utxo) = ::get(&input.outpoint) { ensure!(sp_io::crypto::sr25519_verify( &Signature::from_raw(*input.sigscript.as_fixed_bytes()), &simple_transaction, &Public::from_h256(input_utxo.pubkey) ), ); total_input = total_input.checked_add(input_utxo.value).ok_or( )?; } { missing_utxos.push(input.outpoint.clone().as_fixed_bytes().to_vec()); } } output transaction.outputs.iter() { ensure!(output.value > , ); hash = BlakeTwo256::hash_of(&(&transaction.encode(), output_index)); output_index = output_index.checked_add( ).ok_or( )?; ensure!(! ::exists(hash), ); total_output = total_output.checked_add(output.value).ok_or( )?; new_utxos.push(hash.as_fixed_bytes().to_vec()); } missing_utxos.is_empty() { ensure!( total_input >= total_output, ); reward = total_input.checked_sub(total_output).ok_or( )?; } (ValidTransaction { requires: missing_utxos, provides: new_utxos, priority: reward , longevity: TransactionLongevity::max_value(), propagate: , }) } } // "Internal" functions, callable by code. impl pub fn validate_transaction Result 'static str // Check basic requirements "no inputs" "no outputs" let "each input must only be used once" let "each output must be defined only once" let mut 0 let mut 0 let mut u64 0 let // Variables sent to transaction pool let mut Vec let mut Vec let mut 0 // Check that inputs are valid for in if let Some "signature must be valid" "input value overflow" else // Check that outputs are valid for in 0 "output value must be nonzero" let 1 "output index overflow" "output already exists" "output value overflow" // If no race condition, check the math if "output value must not exceed input value" "reward underflow" // Returns transaction details Ok as u64 true Oof that's a lot. To get a line by line explanation of what's going on, check out part 2 of the video accompaniment. 2. In part 1, we assumed the use of a few internal helper functions. Namely the step where we actually update the blockchain storage, when our transactions are validated. In the same scope, do the following: impl Module (transaction: &Transaction, reward: Value) -> DispatchResult { new_total = ::get() .checked_add(reward) .ok_or( )?; ::put(new_total); input &transaction.inputs { ::remove(input.outpoint); } index: = ; output &transaction.outputs { hash = BlakeTwo256::hash_of(&(&transaction.encode(), index)); index = index.checked_add( ).ok_or( )?; ::insert(hash, output); } (()) } /// Update storage to reflect changes made by transaction /// Where each utxo key is a hash of the entire transaction and its order in the TransactionOutputs vector fn update_storage // Calculate new reward total let "Reward overflow" // Removing spent UTXOs for in let mut u64 0 for in let 1 "output index overflow" Ok As well as : get_simple_transaction (transaction: &Transaction) -> { trx = transaction.clone(); input trx.inputs.iter_mut() { input.sigscript = H512::zero(); } trx.encode() } // Strips a transaction of its Signature fields by replacing value with ZERO-initialized fixed hash. pub fn get_simple_transaction Vec u8 //&'a [u8] { let mut for in Part 3: Test your code In this short part, you'll learn how to construct a blockchain testing environment, build some initial state prior to each test, as well as use some handy helper functions that are available for testing in Substrate/Rust. 1. Construct your test environment: () -> sp_io::TestExternalities { keystore = KeyStore::new(); alice_pub_key = keystore.write().sr25519_generate_new(SR25519, (ALICE_PHRASE)).unwrap(); t = system::GenesisConfig:: () .build_storage:: () .unwrap(); t.top.extend( GenesisConfig { genesis_utxos: [ TransactionOutput { value: , pubkey: H256::from(alice_pub_key), } ], .. :: () } .build_storage() .unwrap() .top, ); ext = sp_io::TestExternalities::from(t); ext.register_extension(KeystoreExt(keystore)); ext } // This function basically just builds a genesis storage key/value store according to our desired mockup. // We start each test by giving Alice 100 utxo to start with. fn new_test_ext let // a key storage to store new key pairs during testing let Some let mut default vec! 100 Default default // Print the values to get GENESIS_UTXO let mut This function builds a genesis storage key/value store according to the code we wrote back in step 1 during . We simply start each test by giving Alice a UTXO of value 100 to start spending. decl_storage 2. Write a simple unit test, testing a simple transaction () { new_test_ext().execute_with(|| { alice_pub_key = sp_io::crypto::sr25519_public_keys(SR25519)[ ]; transaction = Transaction { inputs: [TransactionInput { outpoint: H256::from(GENESIS_UTXO), sigscript: H512::zero(), }], outputs: [TransactionOutput { value: , pubkey: H256::from(alice_pub_key), }], }; alice_signature = sp_io::crypto::sr25519_sign(SR25519, &alice_pub_key, &transaction.encode()).unwrap(); transaction.inputs[ ].sigscript = H512::from(alice_signature); new_utxo_hash = BlakeTwo256::hash_of(&(&transaction.encode(), )); assert_ok!(Utxo::spend(Origin::signed( ), transaction)); (!UtxoStore::exists(H256::from(GENESIS_UTXO))); (UtxoStore::exists(new_utxo_hash)); ( , UtxoStore::get(new_utxo_hash).unwrap().value); }); } #[test] fn test_simple_transaction let 0 // Alice wants to send herself a new utxo of value 50. let mut vec! vec! 50 let 0 let 0 as u64 0 assert! assert! assert_eq! 50 3. Don't forget the handy constants! hex_literal::hex; ALICE_PHRASE: & = ; GENESIS_UTXO: [ ; ] = hex!( ); // need to manually import this crate since its no include by default use const str "news slush supreme milk chapter athlete soap sausage put clutch what kitten" const u8 32 "79eabcbd5ef6e958c6a7851b36da07691c19bda1835a08f875aa286911800999" 4. Run your test in the console with cargo test -p utxo-runtime And your test should pass, meaning your basic UTXO blockchain is done! Check out for what your current implementation might look like. Stuck at this point? this repo Part 4: Configure transaction pool logic In this section, you’ll change how your network prioritizes incoming transactions & handles an annoying UTXO race condition that Bitcoin experiences. Specifically, you'll learn how to . change the blockchain transaction queueing logic without much code Consider the following race condition, where Alice sends Bob her UTXO A, creating a new UTXO B belonging to Bob. What’s happening behind the scenes is that Alice's transaction starts propagating across the nodes in the network: Immediately, Bob decides to spend this UTXO B, creating a UTXO C. Thus his transaction starts to propagate across the network to nodes that haven’t heard from Alice yet! This can be common, due to regular network latency and other real world constraints. since UTXO B doesn’t yet exist in their blockchain state. But Bob’s transaction IS valid, so this error due to this race condition is not ideal. Nodes that heard from Bob but not from Alice yet, will reject his transaction Ideally, we can queue up valid transactions a network pool and wait until prerequisite conditions are satisfied. 1. Luckily Substrate enables a single API call for you to change transaction ordering logic. Configure the trait as follows: runtime_api::TaggedTransactionQueue sp_transaction_pool::runtime_api::TaggedTransactionQueue Runtime { (tx: ::Extrinsic) -> TransactionValidity { (&utxo::Call::spend( transaction)) = IsSubType:: , Runtime>::is_sub_type(&tx.function) { >::validate_transaction(&transaction) { (e) => { sp_runtime::print(e); (TransactionValidityError::Invalid(InvalidTransaction::Custom( ))); } (tv) => { (tv); } } } Executive::validate_transaction(tx) } } impl for fn validate_transaction as // Extrinsics representing UTXO transaction need some special handling if let Some ref match // Transaction verification failed Err return Err 1 // Race condition, or Transaction is good to go Ok return Ok // Fall back to default logic for non UTXO::execute extrinsics Effectively, we're telling the transaction queue to wait for the hash of a required UTXO to exist, before processing a new transaction that spends this UTXO. The transaction pool will hold on to race-UTXOs until this condition is satisfied for a certain period of time. For a detailed explanation of what's happening, check out accompaniment. part 4 of the video Part 5: Final touches & deploy! In this final section we’re ready to configure our blockchain deployment specifications, designate what gets included in genesis block, and deploy! 1. Step into the subdirectory now, and find the file. src chain_spec.rs 2. In function, append the following configuration to set-up your testnet with seed data / UTXOs. testnet_genesis ( initial_authorities: , root_key: AccountId, endowed_accounts: , endowed_utxos: , _enable_println: ) -> GenesisConfig { GenesisConfig { system: (SystemConfig { code: WASM_BINARY.to_vec(), changes_trie_config: :: (), }), ... utxo: (utxo::GenesisConfig { genesis_utxos: endowed_utxos .iter() .map(|x| utxo::TransactionOutput { value: utxo::Value, pubkey: H256::from_slice(x.as_slice()), }) .collect() }), } } // Dev mode genesis setup fn testnet_genesis Vec Vec Vec bool Some Default default Some 100 as 3. In , make sure to also include the genesis set of public keys that should own these UTXOs. fn load [ get_from_seed:: ( ), get_from_seed:: ( ), ], // Genesis set of pubkeys that own UTXOs vec! "Alice" "Bob" 4. In your terminal, compile and build a release of your blockchain in developer mode: ./scripts/init.sh cargo build --release # Initialize your Wasm Build environment: # Build Wasm and native code: 5. Start your node and your blockchain will start producing blocks: ./target/release/utxo-workshop --dev # If you already modified state, run this to purge the chain ./target/release/utxo-workshop purge-chain --dev For a more detailed walkthrough & line-by-line explanation of the above, check out this accompaniment. final part of the video Congratulations! You’ve now built and deployed a Bitcoin-like blockchain from scratch. Quick UI Demo In this tutorial, you learned how to change the underlying ledger model and built a Bitcoin chain on Substrate. In fact, you can implement any fundamental token model on your chain. You can change how your network prioritizes various transactions i.e. manipulate the networking layer without much code. You can even change validator economics structures by using leftover values to reward your validators. You can easily configure your genesis block. Hopefully this tutorial convinced you to give building blockchain infrastructures a try and to check out ! Substrate Questions? Ping us on Riot chat: https://riot.im/app/#/room/#substrate-technical:matrix.org DM me on Twitter: https://twitter.com/nczhu