Ethereum Network Architecture Analysis

0x00 Introduction

The hotness of the blockchain has been rising in a straight line. The blockchain 2.0, represented by Ethereum, has continuously brought innovation to the traditional industry and promoted the development of blockchain technology.

Blockchain is a new application mode of computer technology such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. This is a typical decentralized application, built on the p2p network;In order to learn and analyze the operating principle of Ethereum, this article will gradually analyze its network architecture and finally let readers have a general understanding of the Ethereum network architecture.

This article focuses on the establishment and interaction of data links, and does not involve modules such as node discovery, block synchronization, and broadcast in the network module.

0x01 Content

• Geth Startup
• Network Architecture
• Shared Key
• RLPXFrameRW Frame
• RLP Encoding
• LES Protocol
• Summary

0x02 Geth Startup

Before introducing the Ethereum network architecture, we first briefly analyze the startup process of Geth to better understand and analyze the following content.

the source code directory of Ethereum is as follows:

tree -d -L 1
.
├── accounts
├── bmt
├── build
├── cmd
├── common
├── consensus
├── console
├── containers
├── contracts
├── core
├── crypto
├── dashboard
├── eth
├── ethclient
├── ethdb
├── ethstats
├── event
├── internal
├── les
├── light
├── log
├── metrics
├── miner
├── mobile
├── node
├── p2p
├── params
├── rlp
├── rpc
├── signer
├── swarm
├── tests
├── trie
├── vendor
└── whisper

35 directories

Initialization

Geth's main() function is very concise, starting the program with app.Run() :

[./cmd/geth/main.go]
func main() {
    if err := app.Run(os.Args); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}

Its simplicity is due to the gopkg.in/urfave/cli.v1 extension, which is used to start the hypervisor and command line parsing, and app is an instance of the extension.

In the Go language, if there is an init() function, the init() function will be called by default, and then the main() function is called.Geth has almost completed all initialization operations in ./cmd/geth/main.go#init(): set the program's subcommand set, set the program entry function, and so on. Let's take a look at the init() function fragment:

[./cmd/geth/main.go]
func init() {
    // Initialize the CLI app and start Geth
    app.Action = geth
    app.HideVersion = true // we have a command to print the version
    app.Copyright = "Copyright 2013-2018 The go-ethereum Authors"
    app.Commands = []cli.Command{
        // See chaincmd.go:
        initCommand,
        importCommand,
        exportCommand,
        importPreimagesCommand,
        ...
    }
    ...
}

In the above code, the value of the app instance is preset.And app.Action = geth is the default function called by app.Run(),and app.Commands saves subcommand instances. By matching command line arguments, you can call different functions (without calling app.Action) to use different Geth functions, such as: opening Geth with console, using Geth to create foundation blocks, etc.

Node startup process

Whether the node is started by the geth() function or other command line arguments, the startup process of the node is roughly the same. Here is geth() as an example:

[./cmd/geth/main.go]
func geth(ctx *cli.Context) error {
    node := makeFullNode(ctx)
    startNode(ctx, node)
    node.Wait()
    return nil
}

The makeFullNode() function will return a node instance, which is then started by startNode().In Geth, each function module is treated as a service, and the normal operation of each service drives the functions of Geth.makeFullNode() Registers the specified service by parsing the command line arguments.Here is the code snippet of makeFullNode() function:

[./cmd/geth/config.go]
func makeFullNode(ctx *cli.Context) *node.Node {
    stack, cfg := makeConfigNode(ctx)

    utils.RegisterEthService(stack, &cfg.Eth)

    if ctx.GlobalBool(utils.DashboardEnabledFlag.Name) {
        utils.RegisterDashboardService(stack, &cfg.Dashboard, gitCommit)
    }

    ...

    // Add the Ethereum Stats daemon if requested.
    if cfg.Ethstats.URL != "" {
        utils.RegisterEthStatsService(stack, cfg.Ethstats.URL)
    }
    return stack
}

Then start each service with startNode() function and run the node. The Geth startup process is as follows:

Geth consists of every normal running, collaborative services:

0x03 Network Architecture

The p2p network is finally started by a call to the main() function. This section provides a detailed analysis of the network architecture.

Three-tier architecture

Ethereum is a decentralized digital currency system that is naturally applicable to the p2p communication architecture and supports multiple protocols on it.In Ethereum, p2p is used as a communication link for the transmission of the upper layer protocol, which can be divided into three layers:

1. The uppermost layer is the specific implementation of each protocol in Ethereum, such as ETH protocol and LES protocol.
2. The second layer is the p2p communication link layer in Ethereum. It is mainly responsible for starting monitoring, processing new joins or maintaining connections, and providing channels for upper layer protocols.
3. The bottom layer is the network IO layer provided by the Go language, which is the encapsulation of the network layer in TCP/IP.

P2P communication link layer

This article directly analyzes the p2p communication link layer. The p2p communication link layer mainly does three things:

After the data of the upper layer protocol is delivered to the p2p layer, it is first encoded by RLP.

The RLP encoded data will be encrypted by the shared key to ensure the security of the data during the communication process.

Finally, the data stream is converted to an RLPXFrameRW frame for easy encrypted transmission and parsing of data.

P2P source code analysis

P2P is also a service in Geth, started by startNode(),p2p is started by its Start() function.Here is the code snippet for the Start() function:

[./p2p/server.go]
func (srv *Server) Start() (err error) {
    ...
    if !srv.NoDiscovery {
        ...
    }
    if srv.DiscoveryV5 {
        ...
    }
    ...
    // listen/dial
    if srv.ListenAddr != "" {
        if err := srv.startListening(); err != nil {
            return err
        }
    }
    ...
    go srv.run(dialer)
    ...
}

parameters of the p2p service are set, and node discovery is started according to the user parameters.Then the p2p service listener is turned on, and finally a separate coroutine is opened for processing the message.The following is divided into two modules: service monitoring and message processing.

Service monitoring

It will enter the service listening process through a call to startListening().Then call the listenLoop function with an infinite loop in the function to handle the accepted connection.Finally, the p2p communication link is established for the normal connection through the SetupConn() function.Call the setupConn() function in SetupConn() to do the specific work. Here is the code snippet for the setupConn() function:

[./p2p/server.go]
func (srv *Server) setupConn(c *conn, flags connFlag, dialDest *discover.Node) error {
    ...
    if c.id, err = c.doEncHandshake(srv.PrivateKey, dialDest); err != nil {
        srv.log.Trace("Failed RLPx handshake", "addr", c.fd.RemoteAddr(), "conn", c.flags, "err", err)
        return err
    }
    ...
    phs, err := c.doProtoHandshake(srv.ourHandshake)
    ...
}

The setupConn() function mainly exchanges keys with the client by the doEncHandshake() function and generates a temporary shared key for this communication encryption,and creates a frame processor RLPXFrameRW.Then call the doProtoHandshake() function to set the rules and transactions to be followed for this communication, including the version number, name, capacity, port number and other information.After the communication link is successfully established and the protocol handshake is completed, the processing flow is transferred to the message processing module.

Here is the process of calling the service listener function:

Message processing

p2p.Start() processes the message by calling the run() function.The run() function waits for a transaction with an infinite loop. For example, after the new connection completes the handshake, it will be responsible for this function.The run() function supports the processing of multiple commands, including the service exit cleanup command, the send handshake message command, the add new node command, and the delete node command. The following is the structure of the run() function:

[./p2p/server.go]
func (srv *Server) run(dialstate dialer) {
    ...
    for {
        select {
        case <-srv.quit: ...
        case n := <-srv.addstatic: ...
        case n := <-srv.removestatic: ...
        case op := <-srv.peerOp: ...
        case t := <-taskdone: ...
        case c := <-srv.posthandshake: ...
        case c := <-srv.addpeer: ...
        case pd := <-srv.delpeer: ...
        }
    }
}

In order to clarify the entire network architecture, this article directly discusses the addpeer branch.When a new node is added to the server nodes, it will enter the branch, generate an instance for the upper layer protocol based on the previous handshake information, and then call runPeer() function, and finally enter the message processing flow through p.run() function.

Let's continue to analyze the p.run() function. It turns on the two coroutines, read data and ping, to process the received message and maintain the connection. Then, by calling the startProtocols() function, the Run() function of the specified protocol is called to enter the processing flow of the specific protocol.

The following is the flow of the message processing function

P2P communication link interaction process

Here we look at the processing flow of the p2p communication link and the encapsulation of the data packets.

0x04 Shared Key

In the process of establishing a p2p communication link, the first step is to specify a shared key. This section mainly resolves the key generation process.

Diffie-Hellman key exchange

The "Diffie-Hellman Key Exchange" technology is used in the p2p network, which is a security protocol.It allows one side to create a key over an insecure channel without any prior information from the other side.

Simply put, the two sides get the public keys by generating random private keys each other.Then the two sides exchange their respective public keys so that both sides can generate the same shared key by their own random private key and the other side's public key.We use this shared key as the key to the symmetric encryption algorithm when communicating.The public and private keys of both A and B can be expressed by the following formula.

ECDH (A's private key, B's public key) == ECDH (B's private key, A's public key)

Shared key generation

In the p2p network, the doEncHandshake() function performs key exchange and shared key generation.Here is the code snippet for this function:

[./p2p/rlpx.go]
func (t *rlpx) doEncHandshake(prv *ecdsa.PrivateKey, dial *discover.Node) (discover.NodeID, error) {
    ...
    if dial == nil {
        sec, err = receiverEncHandshake(t.fd, prv, nil)
    } else {
        sec, err = initiatorEncHandshake(t.fd, prv, dial.ID, nil)
    }
    ...
    t.rw = newRLPXFrameRW(t.fd, sec)
    ..
}

If it listens for the connections as a server, the receiverEncHandshake() function will be called after receiving the new connections. If it initiates the request to the server as a client, the initiatorEncHandshake() function will be called.The two functions are not much different, they will exchange keys and generate a shared key.

For the server, the receiverEncHandshake() function will be called to create the shared key.The following is a snippet of the function:

[./p2p/rlpx.go]
func receiverEncHandshake(conn io.ReadWriter, prv *ecdsa.PrivateKey, token []byte) (s secrets, err error) {
    authPacket, err := readHandshakeMsg(authMsg, encAuthMsgLen, prv, conn)
    ...
    authRespMsg, err := h.makeAuthResp()
    ...
    if _, err = conn.Write(authRespPacket); err != nil {
        return s, err
    }
    return h.secrets(authPacket, authRespPacket)
}

The process of shared key generation:

1. After completing the TCP connection, the client encrypts its public key, the signature containing the temporary public key, and a random value nonce using the server's public key (node_id), and sends the encrypted data to the server.
2. After receiving the data, the server obtains the public key of the client, and obtains the temporary public key from the signature by using an elliptic curve algorithm;The server encrypts its temporary public key and random value nonce with the client's public key and sends it to the client.
3. After the above two steps of key exchange process, the client currently has its own temporary public key and private key and the server's temporary public key, then the elliptic curve algorithm can be used to calculate the shared key; the server can also calculate the shared key in the same way.

The following is a shared key generation diagram:

After the shared key is generated, the client and the server can use it for symmetric encryption to complete the encryption of the communication.

0x05 RLPXFrameRW Frame

After the shared key is generated, the RLPXFrameRW frame processor is also initialized at the same time.The purpose of the RLPXFrameRW frame is to support the multiplex protocol over a single connection.Since the frame-grouped message creates a boundary for the encrypted data stream, it is easier to parse the data and verify the transmitted data.

The RLPXFrameRW frame contains two main functions, the WriteMsg() function for sending data and the ReadMsg() function for reading data.The following is a snippet of the WriteMsg() function:

[./p2p/rlpx.go]
func (rw *rlpxFrameRW) WriteMsg(msg Msg) error {
    ...
    // write header
    headbuf := make([]byte, 32)
    ...
    // write header MAC
    copy(headbuf[16:], updateMAC(rw.egressMAC, rw.macCipher, headbuf[:16]))
    if _, err := rw.conn.Write(headbuf); err != nil {
        return err
    }

    // write encrypted frame, updating the egress MAC hash with
    // the data written to conn.
    tee := cipher.StreamWriter{S: rw.enc, W: io.MultiWriter(rw.conn, rw.egressMAC)}
    if _, err := tee.Write(ptype); err != nil {
        return err
    }
    if _, err := io.Copy(tee, msg.Payload); err != nil {
        return err
    }
    if padding := fsize % 16; padding > 0 {
        if _, err := tee.Write(zero16[:16-padding]); err != nil {
            return err
        }
    }

    // write frame MAC. egress MAC hash is up to date because
    // frame content was written to it as well.
    fmacseed := rw.egressMAC.Sum(nil)
    mac := updateMAC(rw.egressMAC, rw.macCipher, fmacseed)
    _, err := rw.conn.Write(mac)
    return err
}

Combined with the Ethereum RLPX documentation and the above code, we can analyze the structure of the RLPXFrameRW frame.In general, it will generate five packets when sending data once:

header          // contains packet size and packet source protocol
header_mac      // head message authentication
frame           // transferred content
padding         // align frames in bytes
frame_mac       // message authentication

The receiver can parse and verify the packets in the same format.

0x06 RLP Encoding

RLP provides a suitable encoding for arbitrary binary data. It has become the main encoding method for serializing objects in Ethereum, which facilitates the parsing of data structures.And RLP uses fewer bytes than the JSON data format.

In the Ethereum network module, all packets of the upper layer protocol are first encoded by RLP before being handed over to the p2p link;After reading data from the p2p link, it is also necessary to use RLP for decoding.

Here is the encoding rule of RLP in Ethereum.

0x07 LES Protocol Layer

Here,we use the LES protocol as the representative of the upper layer protocols to analyze the working principle of the application protocols in the Ethereum network architecture.

When Geth is initialized, The LES service will be started and initialized by calling the NewLesServer() function, and then the interface functions of the Ethereum sub-protocol is implemented by the NewProtocolManager() function.The les/handle.go file contains most of the logic for LES service interaction.

Looking back above, we can see that the p2p underlying protocol is started by the p.Run() function.So the LES protocol can be started by calling its Run() function:

[./les/handle.go#NewProtocolManager()]
Run: func(p *p2p.Peer, rw p2p.MsgReadWriter) error {
    ...
    select {
        case manager.newPeerCh <- peer:
            ...
            err := manager.handle(peer)
            ...
        case <-manager.quitSync:
            ...
    }
}

We can see that the handle() function contains most of the important processing logics, including the LES protocol handshake and message processing.The code snippet of the handle() function is as follows:

[./les/handle.go]
func (pm *ProtocolManager) handle(p *peer) error {
    ...
    if err := p.Handshake(td, hash, number, genesis.Hash(), pm.server); err != nil {
        p.Log().Debug("Light Ethereum handshake failed", "err", err)
        return err
    }
    ...
    for {
        if err := pm.handleMsg(p); err != nil {
            p.Log().Debug("Light Ethereum message handling failed", "err", err)
            return err
        }
    }
}

In the handle() function, the protocol handshake is first performed by calling the ./les/peer.go#Handshake() function.The server and the client obtain information from each other by exchanging handshake packets, including protocol version, network number, the hash of block header, and the hash of foundation block.The following is the logic of message processing:

[./les/handle.go]
func (pm *ProtocolManager) handleMsg(p *peer) error {
    msg, err := p.rw.ReadMsg()
    ...
    switch msg.Code {
        case StatusMsg: ...
        case AnnounceMsg: ...
        case GetBlockHeadersMsg: ...
        case BlockHeadersMsg: ...
        case GetBlockBodiesMsg: ...
        ...
    }
}

The detailed process for processing a request is as follows:

1. Get the requested data using the RLPXFrameRW frame processor.
2. Decrypt data using a shared secret.
3. Serialize binary data using RLP.
4. Perform corresponding operation by msg.Code.
5. Encode the response data by RLP,then encrypt the encoded result with shared key,and finally send it to the requesting side.

The following is the LES protocol process:

0x08 Summary

Through this article, we have a general understanding of the Ethereum network architecture.For the security, the problems caused by protocol itself are often more serious than the problems caused in local, so we should pay more attention to security issues at the network level.