Merge bitcoin#24748: test/BIP324: functional tests for v2 P2P encryption

bc9283c [test] Add functional test to test early key response behaviour in BIP 324 (stratospher)
ffe6a56 [test] Check whether v2 TestNode performs downgrading (stratospher)
ba73735 [test] Add functional tests to test v2 P2P behaviour (stratospher)
4115cf9 [test] Ignore BIP324 decoy messages (stratospher)
8c054aa [test] Allow inbound and outbound connections supporting v2 P2P protocol (stratospher)
382894c  [test] Reconnect using v1 P2P when v2 P2P terminates due to magic byte mismatch (stratospher)
a94e350 [test] Build v2 P2P messages (stratospher)
bb7bffe [test] Use lock for sending P2P messages in test framework (stratospher)
5b91fb1 [test] Read v2 P2P messages (stratospher)
05bddb2 [test] Perform initial v2 handshake (stratospher)
a049d1b [test] Introduce EncryptedP2PState object in P2PConnection (stratospher)
b89fa59 [test] Construct class to handle v2 P2P protocol functions (stratospher)
8d6c848 [test] Move MAGIC_BYTES to messages.py (stratospher)
595ad4b [test/crypto] Add ECDH (stratospher)
4487b80 [rpc/net] Allow v2 p2p support in addconnection (stratospher)

Pull request description:

  This PR introduces support for v2 P2P encryption(BIP 324) in the existing functional test framework and adds functional tests for the same.

  ### commits overview
  1. introduces a new class `EncryptedP2PState` to store the keys, functions for performing the initial v2 handshake and encryption/decryption.
  3. this class is used by `P2PConnection` in inbound/outbound connections to perform the initial v2 handshake before the v1 version handshake. Only after the initial v2 handshake is performed do application layer P2P messages(version, verack etc..) get exchanged. (in a v2 connection)
      - `v2_state` is the object of class `EncryptedP2PState` in `P2PConnection` used to store its keys, session-id etc.
      - a node [advertising](https://github.com/stratospher/blogosphere/blob/main/integration_test_bip324.md#advertising-to-support-v2-p2p) support for  v2 P2P is different from a node actually [supporting v2 P2P](https://github.com/stratospher/blogosphere/blob/main/integration_test_bip324.md#supporting-v2-p2p) (differ when false advertisement of services occur)
          - introduce a boolean variable `supports_v2_p2p` in `P2PConnection` to denote if it supports v2 P2P.
          - introduce a boolean variable `advertises_v2_p2p` to denote whether `P2PConnection` which mimics peer behaviour advertises V2 P2P support. Default option is `False`.
      - In the test framework, you can create Inbound and Outbound connections to `TestNode`
          1. During **Inbound Connections**, `P2PConnection` is the initiator [`TestNode` <--------- `P2PConnection`]
              - Case 1:
                  - if the `TestNode` advertises/signals v2 P2P support (means `self.nodes[i]` set up with `"-v2transport=1"`), different behaviour will be exhibited based on whether:
                      1. `P2PConnection` supports v2 P2P
                      2. `P2PConnection` does not support v2 P2P
                 - In a real world scenario, the initiator node would intrinsically know if they support v2 P2P based on whatever code they choose to run. However, in the test scenario where we mimic peer behaviour, we have no way of knowing if `P2PConnection` should support v2 P2P or not. So `supports_v2_p2p` boolean variable is used as an option to enable support for v2 P2P in `P2PConnection`.
                - Since the `TestNode` advertises v2 P2P support (using "-v2transport=1"), our initiator `P2PConnection` would send:
                  1. (if the `P2PConnection` supports v2 P2P) ellswift + garbage bytes to initiate the connection
                  2. (if the `P2PConnection` does not support v2 P2P) version message to initiate the connection
             - Case 2:
                  - if the `TestNode` doesn't signal v2 P2P support; `P2PConnection` being the initiator would send version message to initiate a connection.
         2. During **Outbound Connections** [TestNode --------> P2PConnection]
             - initiator `TestNode` would send:
                  - (if the `P2PConnection` advertises v2 P2P) ellswift + garbage bytes to initiate the connection
                  - (if the `P2PConnection` advertises v2 P2P) version message to initiate the connection
            - Suppose `P2PConnection` advertises v2 P2P support when it actually doesn't support v2 P2P (false advertisement scenario)
                 - `TestNode` sends ellswift + garbage bytes
                 - `P2PConnection` receives but can't process it and disconnects.
                 - `TestNode` then tries using v1 P2P and sends version message
                 - `P2PConnection` receives/processes this successfully and they communicate on v1 P2P

  4. the encrypted P2P messages follow a different format - 3 byte length + 1-13 byte message_type + payload + 16 byte MAC
  5. includes support for testing decoy messages and v2 connection downgrade(using false advertisement - when a v2 node makes an outbound connection to a node which doesn't support v2 but is advertised as v2 by some malicious
  intermediary)

  ### run the tests
  * functional test - `test/functional/p2p_v2_encrypted.py` `test/functional/p2p_v2_earlykeyresponse.py`

  I'm also super grateful to @ dhruv for his really valuable feedback on this branch.
  Also written a more elaborate explanation here - https://github.com/stratospher/blogosphere/blob/main/integration_test_bip324.md

ACKs for top commit:
  naumenkogs:
    ACK bc9283c
  mzumsande:
    Code Review ACK bc9283c
  theStack:
    Code-review ACK bc9283c
  glozow:
    ACK bc9283c

Tree-SHA512: 9b54ed27e925e1775e0e0d35e959cdbf2a9a1aab7bcf5d027e66f8b59780bdd0458a7a4311ddc7dd67657a4a2a2cd5034ead75524420d58a83f642a8304c9811

Author: achow101

src/net.cpp

@@ -1827,7 +1827,7 @@ void CConnman::CreateNodeFromAcceptedSocket(std::unique_ptr<Sock>&& sock,
     RandAddEvent((uint32_t)id);
 }
 
-bool CConnman::AddConnection(const std::string& address, ConnectionType conn_type)
+bool CConnman::AddConnection(const std::string& address, ConnectionType conn_type, bool use_v2transport = false)
 {
     AssertLockNotHeld(m_unused_i2p_sessions_mutex);
     std::optional<int> max_connections;
@@ -1860,7 +1860,7 @@ bool CConnman::AddConnection(const std::string& address, ConnectionType conn_typ
     CSemaphoreGrant grant(*semOutbound, true);
     if (!grant) return false;
 
-    OpenNetworkConnection(CAddress(), false, std::move(grant), address.c_str(), conn_type, /*use_v2transport=*/false);
+    OpenNetworkConnection(CAddress(), false, std::move(grant), address.c_str(), conn_type, /*use_v2transport=*/use_v2transport);
     return true;
 }
 

src/net.h

@@ -1189,13 +1189,14 @@ class CConnman
      * @param[in]   address     Address of node to try connecting to
      * @param[in]   conn_type   ConnectionType::OUTBOUND, ConnectionType::BLOCK_RELAY,
      *                          ConnectionType::ADDR_FETCH or ConnectionType::FEELER
+     * @param[in]   use_v2transport  Set to true if node attempts to connect using BIP 324 v2 transport protocol.
      * @return      bool        Returns false if there are no available
      *                          slots for this connection:
      *                          - conn_type not a supported ConnectionType
      *                          - Max total outbound connection capacity filled
      *                          - Max connection capacity for type is filled
      */
-    bool AddConnection(const std::string& address, ConnectionType conn_type) EXCLUSIVE_LOCKS_REQUIRED(!m_unused_i2p_sessions_mutex);
+    bool AddConnection(const std::string& address, ConnectionType conn_type, bool use_v2transport) EXCLUSIVE_LOCKS_REQUIRED(!m_unused_i2p_sessions_mutex);
 
     size_t GetNodeCount(ConnectionDirection) const;
     uint32_t GetMappedAS(const CNetAddr& addr) const;

src/rpc/client.cpp

@@ -303,6 +303,7 @@ static const CRPCConvertParam vRPCConvertParams[] =
     { "sendmsgtopeer", 0, "peer_id" },
     { "stop", 0, "wait" },
     { "addnode", 2, "v2transport" },
+    { "addconnection", 2, "v2transport" },
 };
 // clang-format on
 

src/rpc/net.cpp

@@ -371,6 +371,7 @@ static RPCHelpMan addconnection()
         {
             {"address", RPCArg::Type::STR, RPCArg::Optional::NO, "The IP address and port to attempt connecting to."},
             {"connection_type", RPCArg::Type::STR, RPCArg::Optional::NO, "Type of connection to open (\"outbound-full-relay\", \"block-relay-only\", \"addr-fetch\" or \"feeler\")."},
+            {"v2transport", RPCArg::Type::BOOL, RPCArg::Default{false}, "Attempt to connect using BIP324 v2 transport protocol"},
         },
         RPCResult{
             RPCResult::Type::OBJ, "", "",
@@ -379,8 +380,8 @@ static RPCHelpMan addconnection()
                 { RPCResult::Type::STR, "connection_type", "Type of connection opened." },
             }},
         RPCExamples{
-            HelpExampleCli("addconnection", "\"192.168.0.6:8333\" \"outbound-full-relay\"")
-            + HelpExampleRpc("addconnection", "\"192.168.0.6:8333\" \"outbound-full-relay\"")
+            HelpExampleCli("addconnection", "\"192.168.0.6:8333\" \"outbound-full-relay\" true")
+            + HelpExampleRpc("addconnection", "\"192.168.0.6:8333\" \"outbound-full-relay\" true")
         },
         [&](const RPCHelpMan& self, const JSONRPCRequest& request) -> UniValue
 {
@@ -402,11 +403,16 @@ static RPCHelpMan addconnection()
     } else {
         throw JSONRPCError(RPC_INVALID_PARAMETER, self.ToString());
     }
+    bool use_v2transport = !request.params[2].isNull() && request.params[2].get_bool();
 
     NodeContext& node = EnsureAnyNodeContext(request.context);
     CConnman& connman = EnsureConnman(node);
 
-    const bool success = connman.AddConnection(address, conn_type);
+    if (use_v2transport && !(connman.GetLocalServices() & NODE_P2P_V2)) {
+        throw JSONRPCError(RPC_INVALID_PARAMETER, "Error: Adding v2transport connections requires -v2transport init flag to be set.");
+    }
+
+    const bool success = connman.AddConnection(address, conn_type, use_v2transport);
     if (!success) {
         throw JSONRPCError(RPC_CLIENT_NODE_CAPACITY_REACHED, "Error: Already at capacity for specified connection type.");
     }

test/functional/feature_addrman.py

@@ -8,9 +8,8 @@
 import re
 import struct
 
-from test_framework.messages import ser_uint256, hash256
+from test_framework.messages import ser_uint256, hash256, MAGIC_BYTES
 from test_framework.netutil import ADDRMAN_NEW_BUCKET_COUNT, ADDRMAN_TRIED_BUCKET_COUNT, ADDRMAN_BUCKET_SIZE
-from test_framework.p2p import MAGIC_BYTES
 from test_framework.test_framework import BitcoinTestFramework
 from test_framework.test_node import ErrorMatch
 from test_framework.util import assert_equal

test/functional/feature_reindex.py

@@ -11,7 +11,7 @@
 """
 
 from test_framework.test_framework import BitcoinTestFramework
-from test_framework.p2p import MAGIC_BYTES
+from test_framework.messages import MAGIC_BYTES
 from test_framework.util import assert_equal
 
 

test/functional/p2p_v2_earlykeyresponse.py

@@ -0,0 +1,85 @@
+#!/usr/bin/env python3
+# Copyright (c) 2022 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+import random
+
+from test_framework.test_framework import BitcoinTestFramework
+from test_framework.crypto.ellswift import ellswift_create
+from test_framework.p2p import P2PInterface
+from test_framework.v2_p2p import EncryptedP2PState
+
+
+class TestEncryptedP2PState(EncryptedP2PState):
+    """ Modify v2 P2P protocol functions for testing that "The responder waits until one byte is received which does
+    not match the 16 bytes consisting of the network magic followed by "version\x00\x00\x00\x00\x00"." (see BIP 324)
+
+    - if `send_net_magic` is True, send first 4 bytes of ellswift (match network magic) else send remaining 60 bytes
+    - `can_data_be_received` is a variable used to assert if data is received on recvbuf.
+            - v2 TestNode shouldn't respond back if we send V1_PREFIX and data shouldn't be received on recvbuf.
+              This state is represented using `can_data_be_received` = False.
+            - v2 TestNode responds back when mismatch from V1_PREFIX happens and data can be received on recvbuf.
+              This state is represented using `can_data_be_received` = True.
+    """
+
+    def __init__(self):
+        super().__init__(initiating=True, net='regtest')
+        self.send_net_magic = True
+        self.can_data_be_received = False
+
+    def initiate_v2_handshake(self, garbage_len=random.randrange(4096)):
+        """Initiator begins the v2 handshake by sending its ellswift bytes and garbage.
+        Here, the 64 bytes ellswift is assumed to have it's 4 bytes match network magic bytes. It is sent in 2 phases:
+            1. when `send_network_magic` = True, send first 4 bytes of ellswift (matches network magic bytes)
+            2. when `send_network_magic` = False, send remaining 60 bytes of ellswift
+        """
+        if self.send_net_magic:
+            self.privkey_ours, self.ellswift_ours = ellswift_create()
+            self.sent_garbage = random.randbytes(garbage_len)
+            self.send_net_magic = False
+            return b"\xfa\xbf\xb5\xda"
+        else:
+            self.can_data_be_received = True
+            return self.ellswift_ours[4:] + self.sent_garbage
+
+
+class PeerEarlyKey(P2PInterface):
+    """Custom implementation of P2PInterface which uses modified v2 P2P protocol functions for testing purposes."""
+    def __init__(self):
+        super().__init__()
+        self.v2_state = None
+
+    def connection_made(self, transport):
+        """64 bytes ellswift is sent in 2 parts during `initial_v2_handshake()`"""
+        self.v2_state = TestEncryptedP2PState()
+        super().connection_made(transport)
+
+    def data_received(self, t):
+        # check that data can be received on recvbuf only when mismatch from V1_PREFIX happens (send_net_magic = False)
+        assert self.v2_state.can_data_be_received and not self.v2_state.send_net_magic
+
+
+class P2PEarlyKey(BitcoinTestFramework):
+    def set_test_params(self):
+        self.num_nodes = 1
+        self.extra_args = [["-v2transport=1", "-peertimeout=3"]]
+
+    def run_test(self):
+        self.log.info('Sending ellswift bytes in parts to ensure that response from responder is received only when')
+        self.log.info('ellswift bytes have a mismatch from the 16 bytes(network magic followed by "version\\x00\\x00\\x00\\x00\\x00")')
+        node0 = self.nodes[0]
+        self.log.info('Sending first 4 bytes of ellswift which match network magic')
+        self.log.info('If a response is received, assertion failure would happen in our custom data_received() function')
+        # send happens in `initiate_v2_handshake()` in `connection_made()`
+        peer1 = node0.add_p2p_connection(PeerEarlyKey(), wait_for_verack=False, send_version=False, supports_v2_p2p=True)
+        self.log.info('Sending remaining ellswift and garbage which are different from V1_PREFIX. Since a response is')
+        self.log.info('expected now, our custom data_received() function wouldn\'t result in assertion failure')
+        ellswift_and_garbage_data = peer1.v2_state.initiate_v2_handshake()
+        peer1.send_raw_message(ellswift_and_garbage_data)
+        peer1.wait_for_disconnect(timeout=5)
+        self.log.info('successful disconnection when MITM happens in the key exchange phase')
+
+
+if __name__ == '__main__':
+    P2PEarlyKey().main()

test/functional/p2p_v2_encrypted.py

@@ -0,0 +1,134 @@
+#!/usr/bin/env python3
+# Copyright (c) 2022 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+"""
+Test encrypted v2 p2p proposed in BIP 324
+"""
+from test_framework.blocktools import (
+    create_block,
+    create_coinbase,
+)
+from test_framework.p2p import (
+    P2PDataStore,
+    P2PInterface,
+)
+from test_framework.test_framework import BitcoinTestFramework
+from test_framework.util import (
+    assert_equal,
+    assert_greater_than,
+    check_node_connections,
+)
+from test_framework.crypto.chacha20 import REKEY_INTERVAL
+
+
+class P2PEncrypted(BitcoinTestFramework):
+    def set_test_params(self):
+        self.num_nodes = 2
+        self.extra_args = [["-v2transport=1"], ["-v2transport=1"]]
+
+    def setup_network(self):
+        self.setup_nodes()
+
+    def generate_blocks(self, node, number):
+        test_blocks = []
+        last_block = node.getbestblockhash()
+        tip = int(last_block, 16)
+        tipheight = node.getblockcount()
+        last_block_time = node.getblock(last_block)['time']
+        for _ in range(number):
+            # Create some blocks
+            block = create_block(tip, create_coinbase(tipheight + 1), last_block_time + 1)
+            block.solve()
+            test_blocks.append(block)
+            tip = block.sha256
+            tipheight += 1
+            last_block_time += 1
+        return test_blocks
+
+    def create_test_block(self, txs):
+        block = create_block(self.tip, create_coinbase(self.tipheight + 1), self.last_block_time + 600, txlist=txs)
+        block.solve()
+        return block
+
+    def run_test(self):
+        node0, node1 = self.nodes[0], self.nodes[1]
+        self.log.info("Check inbound connection to v2 TestNode from v2 P2PConnection is v2")
+        peer1 = node0.add_p2p_connection(P2PInterface(), wait_for_verack=True, supports_v2_p2p=True)
+        assert peer1.supports_v2_p2p
+        assert_equal(node0.getpeerinfo()[-1]["transport_protocol_type"], "v2")
+
+        self.log.info("Check inbound connection to v2 TestNode from v1 P2PConnection is v1")
+        peer2 = node0.add_p2p_connection(P2PInterface(), wait_for_verack=True, supports_v2_p2p=False)
+        assert not peer2.supports_v2_p2p
+        assert_equal(node0.getpeerinfo()[-1]["transport_protocol_type"], "v1")
+
+        self.log.info("Check outbound connection from v2 TestNode to v1 P2PConnection advertised as v1 is v1")
+        peer3 = node0.add_outbound_p2p_connection(P2PInterface(), p2p_idx=0, supports_v2_p2p=False, advertise_v2_p2p=False)
+        assert not peer3.supports_v2_p2p
+        assert_equal(node0.getpeerinfo()[-1]["transport_protocol_type"], "v1")
+
+        # v2 TestNode performs downgrading here
+        self.log.info("Check outbound connection from v2 TestNode to v1 P2PConnection advertised as v2 is v1")
+        peer4 = node0.add_outbound_p2p_connection(P2PInterface(), p2p_idx=1, supports_v2_p2p=False, advertise_v2_p2p=True)
+        assert not peer4.supports_v2_p2p
+        assert_equal(node0.getpeerinfo()[-1]["transport_protocol_type"], "v1")
+
+        self.log.info("Check outbound connection from v2 TestNode to v2 P2PConnection advertised as v2 is v2")
+        peer5 = node0.add_outbound_p2p_connection(P2PInterface(), p2p_idx=2, supports_v2_p2p=True, advertise_v2_p2p=True)
+        assert peer5.supports_v2_p2p
+        assert_equal(node0.getpeerinfo()[-1]["transport_protocol_type"], "v2")
+
+        self.log.info("Check if version is sent and verack is received in inbound/outbound connections")
+        assert_equal(len(node0.getpeerinfo()), 5)  # check if above 5 connections are present in node0's getpeerinfo()
+        for peer in node0.getpeerinfo():
+            assert_greater_than(peer['bytessent_per_msg']['version'], 0)
+            assert_greater_than(peer['bytesrecv_per_msg']['verack'], 0)
+
+        self.log.info("Testing whether blocks propagate - check if tips sync when number of blocks >= REKEY_INTERVAL")
+        # tests whether rekeying (which happens every REKEY_INTERVAL packets) works correctly
+        test_blocks = self.generate_blocks(node0, REKEY_INTERVAL+1)
+
+        for i in range(2):
+            peer6 = node0.add_p2p_connection(P2PDataStore(), supports_v2_p2p=True)
+            assert peer6.supports_v2_p2p
+            assert_equal(node0.getpeerinfo()[-1]["transport_protocol_type"], "v2")
+
+            # Consider: node0 <-- peer6. node0 and node1 aren't connected here.
+            # Construct the following topology: node1 <--> node0 <-- peer6
+            # and test that blocks produced by peer6 will be received by node1 if sent normally
+            # and won't be received by node1 if sent as decoy messages
+
+            # First, check whether blocks produced be peer6 are received by node0 if sent normally
+            # and not received by node0 if sent as decoy messages.
+            if i:
+                # check that node0 receives blocks produced by peer6
+                self.log.info("Check if blocks produced by node0's p2p connection is received by node0")
+                peer6.send_blocks_and_test(test_blocks, node0, success=True)  # node0's tip advances
+            else:
+                # check that node0 doesn't receive blocks produced by peer6 since they are sent as decoy messages
+                self.log.info("Check if blocks produced by node0's p2p connection sent as decoys aren't received by node0")
+                peer6.send_blocks_and_test(test_blocks, node0, success=False, is_decoy=True)  # node0's tip doesn't advance
+
+            # Then, connect node0 and node1 using v2 and check whether the blocks are received by node1
+            self.connect_nodes(0, 1, peer_advertises_v2=True)
+            self.log.info("Wait for node1 to receive all the blocks from node0")
+            self.sync_all()
+            self.log.info("Make sure node0 and node1 have same block tips")
+            assert_equal(node0.getbestblockhash(), node1.getbestblockhash())
+
+            self.disconnect_nodes(0, 1)
+
+        self.log.info("Check the connections opened as expected")
+        check_node_connections(node=node0, num_in=4, num_out=3)
+
+        self.log.info("Check inbound connection to v1 TestNode from v2 P2PConnection is v1")
+        self.restart_node(0, ["-v2transport=0"])
+        peer1 = node0.add_p2p_connection(P2PInterface(), wait_for_verack=True, supports_v2_p2p=True)
+        assert not peer1.supports_v2_p2p
+        assert_equal(node0.getpeerinfo()[-1]["transport_protocol_type"], "v1")
+        check_node_connections(node=node0, num_in=1, num_out=0)
+
+
+if __name__ == '__main__':
+    P2PEncrypted().main()

test/functional/p2p_v2_transport.py

@@ -7,8 +7,7 @@
 """
 import socket
 
-from test_framework.messages import NODE_P2P_V2
-from test_framework.p2p import MAGIC_BYTES
+from test_framework.messages import MAGIC_BYTES, NODE_P2P_V2
 from test_framework.test_framework import BitcoinTestFramework
 from test_framework.util import (
     assert_equal,

test/functional/rpc_net.py

@@ -244,7 +244,10 @@ def test_addnode_getaddednodeinfo(self):
     def test_service_flags(self):
         self.log.info("Test service flags")
         self.nodes[0].add_p2p_connection(P2PInterface(), services=(1 << 4) | (1 << 63))
-        assert_equal(['UNKNOWN[2^4]', 'UNKNOWN[2^63]'], self.nodes[0].getpeerinfo()[-1]['servicesnames'])
+        if self.options.v2transport:
+            assert_equal(['UNKNOWN[2^4]', 'P2P_V2', 'UNKNOWN[2^63]'], self.nodes[0].getpeerinfo()[-1]['servicesnames'])
+        else:
+            assert_equal(['UNKNOWN[2^4]', 'UNKNOWN[2^63]'], self.nodes[0].getpeerinfo()[-1]['servicesnames'])
         self.nodes[0].disconnect_p2ps()
 
     def test_getnodeaddresses(self):

test/functional/test_framework/messages.py

@@ -75,6 +75,13 @@
 
 DEFAULT_MEMPOOL_EXPIRY_HOURS = 336  # hours
 
+MAGIC_BYTES = {
+    "mainnet": b"\xf9\xbe\xb4\xd9",   # mainnet
+    "testnet3": b"\x0b\x11\x09\x07",  # testnet3
+    "regtest": b"\xfa\xbf\xb5\xda",   # regtest
+    "signet": b"\x0a\x03\xcf\x40",    # signet
+}
+
 def sha256(s):
     return hashlib.sha256(s).digest()