Skip to content

Add CKB_TX_MESSAGE_ALL specification proposal #446

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Add CKB_TX_MESSAGE_ALL specification proposal #446

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
009965e
Add CIGHASH_ALL specification proposal
xxuejie Feb 8, 2025
c1aa79f
Fix wording
xxuejie Feb 8, 2025
865ebc2
Add explanation for using 32-bit integer to denote length
xxuejie Feb 8, 2025
00d2bcc
Review fixes
xxuejie Feb 10, 2025
2013873
Rename the spec to CKB_TX_MESSAGE_ALL
xxuejie Feb 25, 2025
e7ad835
Change the spec to skip lock field's length
xxuejie Mar 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions rfcs/0000-ckb-tx-message-all/0000-ckb-tx-message-all.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
---
Number: "0000"
Category: Standards Track
Status: Proposal
Author: Xuejie Xiao <[email protected]>
Created: 2025-02-05
---

# CKB_TX_MESSAGE_ALL

This document defines a new message calculation scheme used by CKB lock scripts to guard against malleable attacks.

## Rationale

Unlike most blockchains out there, CKB does not formally define signature verification flow in CKB transactions. Instead, a CKB transaction is considered to be valid when all lock scripts in its input cells, as well as all type scripts in its input & output cells succeed in [execution](https://github.com/nervosnetwork/rfcs/blob/master/rfcs/0003-ckb-vm/0003-ckb-vm.md).

Nonetheless, a transaction must not be malleable, meaning a transaction shall not be tampered with after someone creates it in the first place. By convention, the lock scripts in CKB guard against malleable attacks: a typical lock script, running for a series of input cells forming a particular script group, would calculate a `message` by accessing the transaction it runs upon, then fetches a signature from one of the designated witness field. It then runs a signature verification process to validate the signature against the `message`, and only succeeds when the signature passes the verification. With this mechanism, any tampering on the transaction itself will result in a different `message`, resulting in a failure of the verification process, leading to a failure of the execution of lock scripts.

The exact way to calculate such a `message` processes enough challenge, since the `message` shall capture enough information so the transaction is safe from any tampering, while the `message` shall not cover too much data to obscure interoperability.

Historically, a particular `message` calculation algorithm has been [introduced](https://github.com/nervosnetwork/ckb-system-scripts/blob/934166406fafb33e299f5688a904cadb99b7d518/c/secp256k1_blake160_sighash_all.c#L149-L219) by lock scripts included in CKB's genesis blocks, and used since then. Many other locks from the community have also adopted a similar workflow. However, this workflow has only since existed in part of a script's implementation. It has never been properly documented. On the other hand, certain pitfalls of this very workflow arise as we learn more about coding for CKB's particular environment:

* While the current workflow assumes the first witness of current executed input cells [script group](https://github.com/nervosnetwork/rfcs/blob/master/rfcs/0022-transaction-structure/0022-transaction-structure.md) is a [WitnessArgs](https://github.com/nervosnetwork/ckb/blob/a6733e6af5bb0da7e34fb99ddf98b03054fa9d4a/util/types/schemas/blockchain.mol#L104-L108) structure serialized in the [molecule](https://github.com/nervosnetwork/rfcs/blob/master/rfcs/0008-serialization/0008-serialization.md) serialization format, this particular assumption is not enforced, and there is code that [exploits](https://github.com/cryptape/quantum-resistant-lock-script/blob/22de5369b60b1e59bb698927c143d9efbe8527a9/c/ckb-sphincsplus-lock.c#L67-L80) this oversight for certain gains. We do believe this can be a problem as future standards arise.
* The current workflow covers the whole [Transaction](https://github.com/nervosnetwork/rfcs/blob/master/rfcs/0022-transaction-structure/0022-transaction-structure.md) structure, as well as all witnesses from the current script group. However, the `Transaction` structure only contains pointer to all the consumed input cells, it does not cover any contents of the input cells, e.g., the CKBytes stored in each input cell, or any input cell's data. This makes it harder to design a proper offline signing protocol. If we dig through the literature, the Bitcoin community actually made the same choice early, but later [came up](https://en.bitcoin.it/wiki/BIP_0143) with an updated design, that signs actual contents of each input UTXOs as well. We do believe a message that covers all input cells' contents can definitely bring merits to future CKB wallets & applications.

As a result, this document aims to propose `CKB_TX_MESSAGE_ALL`, a properly defined message calculation scheme used by CKB lock scripts to ensure transactions are not malleable.

The name is intentionally chosen to be different from `SIGHASH_ALL`, so as to avoid any confusions. The latter has been used in many CKB code to represent the old workflow to calculate a signing message. The new name consists of 3 parts:

* `CKB_TX` denotes a prefix, serving a namespace since we are defining specification for CKB's transaction structure.
* `MESSAGE` denotes that we are generating a message for signing purpose. As we shall see below, current specification really defines a way of concatenating data, while the most likely outcome for the concatenated bytes, will be a hashing function generating fixed length bytes, it is not always the case that a hash will be generated. Some use cases are perfectly fine with the concatenated bytes. Given those thoughts, `MESSAGE` will be a more suitable term, since it does not always refer to a hash.
* `ALL` as a suffix, denotes that we try to hash all components related to current transaction and script group. `WHOLE` is a different term we considered while coming up the specification, but in the end we picked `ALL`, since we are not necessarily hashing the `whole` transaction, but only `all` the parts that make sense to a script group.

## Specification

For a CKB transaction, `CKB_TX_MESSAGE_ALL` utilities the following workflow:

* The first witness field of the current running script group, must be a valid `WitnessArgs` structure serialized in the molecule serialization format, with compatible mode turned off. The message calculation workflow fails if molecule validation fails.
* The byte concatenation of all the following fields is then calculated, following the exact same order defined here:
+ 32-byte transaction hash returned by [Load Transaction Hash](https://github.com/nervosnetwork/rfcs/blob/bd5d3ff73969bdd2571f804260a538781b45e996/rfcs/0009-vm-syscalls/0009-vm-syscalls.md#load-transaction-hash) syscall.
+ For each input cell of the current transaction in sequential order:
* The full [CellOutput](https://github.com/nervosnetwork/ckb/blob/a6733e6af5bb0da7e34fb99ddf98b03054fa9d4a/util/types/schemas/blockchain.mol#L44-L48) structure of current input cell serialized in the molecule serialization format, which is also the full content returned by [Load Cell](https://github.com/nervosnetwork/rfcs/blob/bd5d3ff73969bdd2571f804260a538781b45e996/rfcs/0009-vm-syscalls/0009-vm-syscalls.md#load-cell) syscall, given the correct `index` and `source`.
* The length of current input cell data, packed in little-endian encoded unsigned 32-bit integer.
* The full cell data of current input cell, or the full content returned by [Load Cell Data](https://github.com/nervosnetwork/rfcs/blob/bd5d3ff73969bdd2571f804260a538781b45e996/rfcs/0009-vm-syscalls/0009-vm-syscalls.md#load-cell-data) syscall, given the correct `index` and `source`.
+ The length of the whole `input_type` field (a `BytesOpt` structure) from the first witness field in current script group, packed in little-endian encoded unsigned 32-bit integer.
+ The whole `input_type` field (a `BytesOpt` structure) from the first witness field in current script group.
+ The length of the whole `output_type` field (a `BytesOpt` structure) from the first witness field in current script group, packed in little-endian encoded unsigned 32-bit integer.
+ The whole `output_type` field (a `BytesOpt` structure) from the first witness field in current script group.
+ Starting from the second witness field in current script group, for each witness in sequential order:
* The length of the witness field, packed in little-endian encoded unsigned 32-bit integer.
* The full witness field, or the full content returned by [Load Witness](https://github.com/nervosnetwork/rfcs/blob/bd5d3ff73969bdd2571f804260a538781b45e996/rfcs/0009-vm-syscalls/0009-vm-syscalls.md#load-witness) syscall, given the correct `index` and `source`.
+ Starting from the first witness that do not have an input cell of the same index(e.g., assuming a transaction has 5 input cells in total, the counting here starts from index 5 of witnesses), for each witness in sequential order:
* The length of the witness field, packed in little-endian encoded unsigned 32-bit integer.
* The full witness field, or the full content returned by [Load Witness](https://github.com/nervosnetwork/rfcs/blob/bd5d3ff73969bdd2571f804260a538781b45e996/rfcs/0009-vm-syscalls/0009-vm-syscalls.md#load-witness) syscall, given the correct `index` and `source`.
* As an optional step, a cryptographic hashing algorithm can be leveraged to convert the above concatenated bytes into a hash of 32 bytes or more.

### Notable Points

There are several notable points worth mentioning regarding the above specification:

* The first witness of current running script group must be a valid [WitnessArgs](https://github.com/nervosnetwork/ckb/blob/81a1b9a1491edca0bc42c12d8bf0f715a055a93f/util/gen-types/schemas/blockchain.mol#L114-L118) structure serialized in the molecule serialization format. This has now become an enforced rule, it is not an assumption that can be exploited or ignored.
* The content of all input cells are covered by the message calculation workflow, making it much easier to design an offline signing scheme.
* Witness length is packed in 32-bit unsigned integers, while 64-bit unsigned integers were used in older workflow. Notice that all CKB data structures, including `CellOutput`, cell data, witness, etc., will first be serialized in molecule serialization format. Note molecule uses 32-bit integer to denote the length of a structure, this means that we will never have a `CellOutput` / cell data / witness structure that is bigger than 4GB, and there is no point in representing the length in 64-bit integers.
* A different concatenation/hashing design is introduced for the first witness of the current script group, discarding the original zero-filled design. We believe this new solution can contribute to a more optimized implementation, both in terms of runtime cycles and binary size.
Copy link

@Hanssen0 Hanssen0 May 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds weird to me since:

  1. Bad compatibility: Considering our attempt on CoBuild, having a different witness layout seems possible in the future. This enforcement brings a serious problem to compatibility.

  2. Confusion of abstraction levels: For me, the zero-filled design comes with the algorithm instead of the message specification. Almost all signing algorithms in the past used SIGHASH_ALL, but they filled the lock with different dummy values (65 0s for Secp256k1, 85 0s for EVM, 572 0s for Nostr), and of course, we can leave the lock empty for performance.
    So if CKB_TX_MESSAGE_ALL is at the same abstraction level as SIGHASH_ALL, I don't think it should force the algorithm to put what dummy value here (even more, if the algorithm wanted to put some meaningful value to the lock to aid validation, it would not be able to do so.）

  3. Uncertain optimization: Check a possible implementation. If a program wants to process this logic in a relatively readable manner, it may introduce additional decoding work when computing the message. Even if we merge this decoding step with the decoding necessary for signature verification, I don't think it will be a performance optimization compared with the original method of calculating the message without decoding. Notice that replacing the zero-fill design with an always-empty design already leads to substantial performance gains.

Copy link
Contributor

@phroi phroi May 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fully agree on the bad [future] compatibility of the proposed witness layout, important things that are not accounted for:

  1. CoBuild-alike protocols
  2. An hypothetical meta protocol proposed by @janx:

(weird but good for brain-storming) create a type/type script composing standard, e.g. abandon lock script completely, use type script for everything, every cell uses a 'meta' type script which can read cell data to determine what script/logic should be invoked and invoke them in turn.

  1. Non-signature based locks protocols such as the protocol proposed by @janx (which would use always-successful locks) or the ones implemented in iCKB.

While I really appreciate the inclusion of input cell metadata in the message, it also seems like a missed opportunity for an improved witness layout.

Love & Peace, Phroi

Copy link
Contributor Author

@xxuejie xxuejie May 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is some misunderstanding here.

First of all, CKB_TX_MESSAGE_ALL DOES NOT introduce any incompatible changes whatsoever. It it compatible with all existing locks & types, it uses WitnessArgs structure exactly in the same way as all lock scripts I know of today, it also is perfectly compatible with cobuild. There are no breaking changes. In fact, I don't think I can move forward with any new CKB design that is not compatible with existing CKB designs, a powerful killer app might be able to do so, but I just can't.

What this paragraph describes, is simply a chicken-egg problem: the first witness in the script group of a lock script by convention is a WitnessArgs structure, the lock field of this WitnessArgs, is a signature. However, prior to generating the signature, we need to either skip the contents in this lock field, or use zeros to fill the space occupied by the signature.

We could definitely debate which is the majority design choice, but historically there are certain lock scripts which will simply skip the lock field, an example can be found here. There was a time where Rust based lock script favors this workflow due to poorly-designed public APIs in the molecule libraries.

On the other hand, even if we stick to a zero-filled design, it is not always as simple as "fill everything in lock field as zeros", lock field can be variable in length(making it harder to do the filling in Rust using constant memory consumption). In addition, lock field can contain more than signatures, what if a lock field contains multisig paramters or other user request parameters for lock script? In this sense, the lock field might mix signatures with other parameter values together. It might be quite troublesome to pinpoint the exact bytes for all the signatures, and zero-fill them instead. I personally consider the following 2 as 2 separate tasks:

  • Generating a signing message from a single CKB transaction
  • Validate a signature given a signing message

With spawn deployed in mainnet, I expect there are lock scripts(and potentially type scripts) which simply generates a signing message, and relies on other scripts on chain for the actual signature validation. Assuming a zero-filled design, there might be leaky abstraction, where a signing message algorithm must know signature formats to know what exact bytes it should zero-fill in the lock field.

Given all those past concerns, I'm proposing a different solution here: instead of zero-filling signatures, CKB_TX_MESSAGE_ALL simply do not sign lock field at all, it leaves the task to a lock script to fully validate the data in lock field, whether it is a plain signature, or contains more user request parameters.

Again, the changed solution only applies to how a signature is calculated. It DOES NOT change witness layout at all. We continue to build WitnessArgs structure in the first witness of a script group, a signature continues to live in the lock field of the WitnessArgs structure. This new design is indeed compatible with cobuild and all existing designs I personally know of. CKB_TX_MESSAGE_ALL simply states the following requirements in the specification level:

  • The first witness in the current script group must be a valid WitnessArgs structure.
  • The input_type field of the first witness in the current script group will be signed in the signature
  • The output_type field of the first witness in the current script group will be signed in the signature

The lock field will simply be ignored by the signature calculation process. However all data will continue to be placed in the position where they were placed before. No incompatible changes are introduced.

Finally, let's talk performance.

I must say that personally, I consider the performance of on-chain CKB scripts to be of more importance than off-chain servers. On-chain CKB scripts are running in a much more constraint place,and the performance metrics of those scripts are directly connected to fees paid by each CKB transaction. On the other hand it is a much simpler task to scale off-chain servers, people have been saying for almost 20 years that developers cost more than machines, when it is possible it makes more sense just to add more machines.

I would argue that the new design here, where lock field is ignored rather than zero-filled, is more performant in terms of on-chain CKB scripts. Ignoring lock field leads to super optimal on-chain script implementation, right now a C implementation does this, assuming molecule's Rust implementation introduces lazy reader based validator(which I assume is really a general feature request), we could do the exact same optimal implementation in Rust code as well.

For zero-filled implementation, I can only think of 2 possible implementations:

  • We read the full witness into a CKB script's memory space, and zero fill lock field stored in memory. However, this solution could run into issues very quickly. For example, a buggy implementation can be found here, this code assumes that an entire witness could fit in a CKB script's heap space. Theoretically, a witness can be almost as large as a CKB block, which will be ~600KB, however, ckb-std in current implementation, does not allocate enough heap space for such big a block. We can certainly debate the likelyhood, but there will be cases where current CKB script cannot handle a CKB transaction which should run perfectly fine. Even such code is fixed, reading full witness all at once can waste quite a lot of memory in a precious on-chain memory space. I doubt if this is a good idea.
  • We use a lazy reader based solution, we also divide the full lock field in chunks, we load each chunk at a time, and do zero-filling as requested. However, I have yet to see one implementation which is willing to go through all those troubles.

Given those thoughts, I do consider current design in this proposal an optimization in on-chain scripts. As for off-chain servers, I'm willing to bet that decoding one more WitnessArgs when signing a transaction does not affect performance in any noticeable ways.

And just to recap: I do believe the newly introduced design here tackles 3 potential issues:

  • Different implementation workflows (some ignore lock field, some do zero-filling) due to missing specification
  • A leaky abstraction would be introduced since signing message calculation algorithms might need to learn about signature formats
  • Zero-filling has performance concerns for on-chain scripts.

In the meantime, no breaking change is introduced, the new design is perfectly compatible will all existing proposals.

Copy link

@Hanssen0 Hanssen0 May 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much for the explanation, now I understand better about the sentence We believe this new solution can contribute to a more optimized implementation, both in terms of runtime cycles and binary size.. Also, the spawn example is a good example of wanting to compute a message without knowing the signature algorithm, which makes everything more reasonable.

Let me sort this out - we actually have three ways of message hashing:

  1. SIGHASH_ALL + (Usually) WitnessArgs.lock Zero-Fill - This method cannot correctly handle dynamic length signatures.
  2. CoBuild's SighashAll - I think it's more like CoBuild.TxMessageAll, since it also requires all input info.
  3. TX_MESSAGE_ALL - A minor upgrade to SIGHASH_ALL.

Here, both 1 and 3 are designed specifically for WitnessArgs. If we want a CoBuild compatible lock script, we should just use 2.

Copy link
Contributor Author

@xxuejie xxuejie May 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I think you understand them correctly, tho I do want to mention that for 1, we actually have a series of slightly-different-but-generally-the-same workflow, not just one. Some might not do zero-fill(such as the example I mentioned earlier).

I was hoping that in the future we can all converge on 2 and 3, and gradually deprecate 1.

Copy link
Contributor

@phroi phroi May 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment makes everything clear, so much so that you may consider providing this context in the RFC Rationale Section.

The vibe I get is that gently pushing the ecosystem from SIGHASH_ALL to CKB_TX_MESSAGE_ALL may also very well be what enables CoBuild adoption, thanks to CCC likely support for both and that, once you start making changes in your codebase, you are open to make more.

I just wonder if it is possible to streamline more 2 and 3.

Love & Peace, Phroi

Copy link
Contributor Author

@xxuejie xxuejie May 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally, I think the adoption of CKB_TX_MESSAGE_ALL is much easier than CoBuild, since the scope is much smaller and easier for integration. In addition, it touches almost no UX part at all, in most cases it's just developers switching to a new set of APIs, which can be quickly picked up.

CoBuild, on the other hand, is a heavy component that requires all parties to participate. And it really is unclear when / if there will be enough adoption in current CKB community.

Copy link
Contributor

@phroi phroi May 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right!! That said, I also forecast that the synergy between CoBuild, SSRI and spawn will drive adoption 🤗


## Examples

Following the defined spec above, a [series of libraries, CKB scripts and utilities](https://github.com/xxuejie/ckb-tx-message-all-test-vector-utils) have been developed as a demonstration and inspiration. For example:

* A [Rust module](https://github.com/xxuejie/ckb-tx-message-all-test-vector-utils/blob/8f1378ba5b7f0cb0fc2eb78c342e0bdf757ee92c/crates/ckb-tx-message-all-utils/src/ckb_tx_message_all_in_ckb_vm.rs) calculates `CKB_TX_MESSAGE_ALL` message with the help of [ckb-std](https://docs.rs/ckb-std/latest/ckb_std/) to provide CKB-related APIs in CKB-VM environment. It is also designed in a generic way, which makes it compatible with different kinds of hashers;
* Another [Rust module](https://github.com/xxuejie/ckb-tx-message-all-test-vector-utils/blob/8f1378ba5b7f0cb0fc2eb78c342e0bdf757ee92c/crates/ckb-tx-message-all-utils/src/ckb_tx_message_all_from_mock_tx.rs) also calculates `CKB_TX_MESSAGE_ALL` message in a generic way. But it was designed to take the whole CKB [Transaction](https://docs.rs/ckb-gen-types/0.119.0/ckb_gen_types/packed/struct.Transaction.html) as input. Certainly, the CKB Transaction structure is missing the actual contents for all input cells, a user can either provide [MockTransaction](https://docs.rs/ckb-mock-tx-types/latest/ckb_mock_tx_types/struct.MockTransaction.html) instead, or simply provide the contents for input cells.
* A [C header-only implementation](https://github.com/xxuejie/ckb-tx-message-all-test-vector-utils/blob/8f1378ba5b7f0cb0fc2eb78c342e0bdf757ee92c/contracts/c-assert-ckb-tx-message-all/ckb_tx_message_all.h) is also provided to calculate `CKB_TX_MESSAGE_ALL` message, also in a generic way to support different kinds of hashers, in CKB-VM compatible environments.

All of the above Rust & C implementations have been carefully written, well optimized, and extensively tested. They are considered to be usable in production environments.

A [utility](https://github.com/xxuejie/ckb-tx-message-all-test-vector-utils/tree/main/crates/native-test-vector-generator) is also provided so one can manually generate as many test vectors as one wish. Each test vector includes a tx file that can be accepted and executed in [ckb-debugger](https://github.com/nervosnetwork/ckb-standalone-debugger), as well as the generated `CKB_TX_MESSAGE_ALL` message, together with enough information to generate such message.