Any kind of system that features multiple interacting parts will inevitably have more potential for error. When we increase complexity, we need to proportionately increase our efforts at detecting bugs.
In one of our previous articles, we discussed the benefits of automated bug testing, in particular, the technique that is known as fuzzing (also referred to as fuzz testing). Fuzzing involves testing software by feeding it a large amount of generated inputs, the purpose of which is to capture behavior that could potentially cause vulnerabilities.
A Tezos node is composed of the economic protocol and the shell. The shell provides the environment where the economic protocol runs, and it is in charge of: handling P2P connections, implementing the storage, and the interfaces that connect to end-users (RPCs). TezEdge is the shell written in Rust, and the economic protocol is written in OCaml, between both we have the FFI layer so code from both languages can communicate with each other.
Having fuzzed most parts of our shell implementation, we now turn our attention to the economic protocol implementation used by the Tezos network.
The economic protocol is the logic in charge of applying operations (transactions, smart contract execution, etc) and producing blocks. In Tezos the economic protocol can be updated on-chain via a voting system in the protocol itself. Currently, the Tezos network uses protocol-12 "Ithaca".
Since the economic protocols of Tezos are written in OCaml, we need to utilize a Foreign Function Interface (FFI) layer to ‘glue’ the TezEdge node’s Rust code with OCaml. An FFI can have serious bugs because although it is a very small surface, it involves unsafe code with potential for memory corruption bugs.
Instead of fuzzing the OCaml code directly, we perform fuzzing through TezEdge's operation injection RPC endpoint. This way, the injected operations travel all the way from the RPC endpoint in the shell, to the protocol-runner process and finally to the FFI and the OCaml runtime. This way we cover not just the OCaml code, but also our FFI code.
Fuzzing the economic protocol
The goal of fuzzing operations is to target the economic protocol implementation, in this case the Ithaca protocol. To do so we employ structure-aware fuzzing by implementing encoders for the different operations supported by the protocol:
- Seed nonce revelation
- Double endorsement evidence
- Double baking evidence
- Activate account
- Double pre-endorsement evidence
- Failing noop
- Register global constant
- Set deposits limit
The encoders allow the generation of operations by providing partial information and generating the rest of the data randomly. This way, we can selectively provide valid information for some, all, or none of the fields of each operation.
For some fields, the fuzzer will always generate valid information, one example is the signature field, all operations are signed with the ed25519 key of the bootstrap1 account. Without it, all operations would fail the signature check performed by the protocol.
The operation fuzzer is implemented as a Python script that makes use of the Tezos' Python Execution and Testing Environment. Four nodes and bakers are started in sandboxed mode. This amount of bakers is the minimum required to make progress (bake new blocks). Bootstrap accounts bootstrap2 to bootstrap5 are used by bakers, and bootstrap1 is used as source for operations. Random operations are injected via the injection/operationRPC.
The target node, and the protocol implementation are built with coverage instrumentation and periodic coverage reports are generated. These reports will display coverage information relevant to the protocol implementation (OCaml code). We can find the coverage reports used in our CI here: http://fuzz.tezedge.com/develop/.fuzzing.latest/operation_fuzzer/
This fuzzer is implemented as a Python script that makes use of the Tezos' Python Execution and Testing Environment and allows to craft and inject random (protocol-12, Ithaca) operations.
The fuzzer runs four nodes and four bakers in sandboxed mode, this is the minum required to bake new blocks and do progress. Bootstrap accounts
bootstrap2-5 are used by bakers, and
bootstrap1 is used as source for the randomly generated operations, before injecting any operations protocol Ithaca is activated.
On every iteration the fuzzer will:
- Request via RPC the current block's level.
- Request via RPC the current contract's counter.
- Generate a random operation, sign it, and inject it via the
- Every 100 iterations coverage counters are dumped and coverage reports are generated. Reports are stored in
/var/lib/fuzzing-data/reports/develop/.fuzzing.latest/operation_fuzzer/in the host.
This repository contains the script files needed to deploy and run TezEdge's operations fuzzer in the fuzzing CI.
./deploy.sh. This script will build the fuzzop_ Docker container.
./run.sh. The scrip will run the fuzzop_ container which will listen form XMLRPC requests at address
- The fuzzer can be restarted at any time by sending an XMLRPC request, this can be done by running the script at
scripts/restart_fuzzer.py. In this restart process new code will be pulled and built from TezEdge's
developbranch, this way the fuzzer can be integrated in CI.