Merge #43: Update 03 20 2023

20dd79c rust doc fixes (sanket1729)
a300697 CI fixes for MSRV and benches (sanket1729)
0ac0326 Add int tests to workflow (sanket1729)
91f5c2d Move elementsd test to separate directory (sanket1729)
25ccd25 Removed previous mistakenly checked in integration tests (sanket1729)
2f84d32 Add README build docs for std and no-std (Steve Myers)
d1c71e3 Add build error if both std and no-std features disabled (Steve Myers)
7118c84 Add breaking tests (sanket1729)
0203377 Update multi_a to not include n wrapper (sanket1729)
3502477 Serde for DescriptorPublicKey (Scott Robinson)
a0b001b hexify for better error message (Harshil Jani)
76fb99c Fix the CI failing due to `which` crate (Harshil Jani)
ebe3245 Fixing the Inclusivity of Z in StrKeyTranslator (Harshil Jani)
360eed7 Add some compiler documentation (sanket1729)
3fcc84c Update TapTree::iter docs (sanket1729)
1e30000 Make checksum pub (sanket1729)
cf05787 Remove fixed TODO (sanket1729)
e94efbf Clarify that alloy tests don't actually test timelock mixing (sanket1729)
62219d4 Fix CI (sanket1729)
0b482cd Use hash_newtype macro instead of the hash256 module (Tobin C. Harding)
4eaf261 Fix CI (sanket1729)
842ae93 Use p2wpkh_script_code from rust-bitcoin (Tobin C. Harding)
7296f8e Introduce `max_weight_to_satisfy` (志宇)
1477dbb DefiniteDescriptorKey: provide additional methods for converting to a DescriptorPublicKey (Andrew Poelstra)
2c67681 Add base64 feature (Tobin C. Harding)
f0f7fd6 fix formatting after #507 (Andrew Poelstra)
eeb823f Remove DummyKey (Tobin C. Harding)
ebaa31b Enable usage of -D rustdoc::broken-intra-doc-links (Tobin C. Harding)
16b6c76 Improve docs on enumerate_policy_tree (Tobin C. Harding)
90dc766 Do trivial cleanup to main lib docs (Tobin C. Harding)
b069239 Take self by value (Tobin C. Harding)
29b6272 Introduce SPDX identifiers (Tobin C. Harding)
8c48538 Clean up DO_NO_STD usage (Tobin C. Harding)
cc32761 Use 1.58.0 directly (Tobin C. Harding)
1a098ad Fix spelling mistake it CI job (Tobin C. Harding)
5705902 Remove DO_NO_STD from fuzz job (Tobin C. Harding)
8b85649 Remove MSRV local variable (Tobin C. Harding)
2edc456 Remove --all flag (Tobin C. Harding)
1a09248 Remove flags from shebang line (Tobin C. Harding)
fb4c33b Use if let and panic (Tobin C. Harding)
b481cba Use allow attribute (Tobin C. Harding)
6b6f808 Use any combinator (Tobin C. Harding)
6088c25 Use {} instead of explicit return (Tobin C. Harding)
06c9617 Remove unnecessary cast (Tobin C. Harding)
eb38507 Do not use bool in assert_eq macro (Tobin C. Harding)
3c46e4c Use expect_err (Tobin C. Harding)
9c377b2 Remove useless use of format (Tobin C. Harding)
bf44938 Remove unneeded calls to clone (Tobin C. Harding)
227a3a9 Remove identical if blocks (Tobin C. Harding)
45091a8 Remove unneeded explicit lifetimes (Tobin C. Harding)
2e70cd0 Do not manually convert bool to integer (Tobin C. Harding)
c4ab53e Use &str not &String (Tobin C. Harding)
b81f08c Return result of function call (Tobin C. Harding)
fc37f41 Use unwrap_or_default (Tobin C. Harding)
f0820ce Remove redundant explicit references (Tobin C. Harding)
64c6e51 Fix pattern matching ref/deref (Tobin C. Harding)
a923855 Do not use redundant closures (Tobin C. Harding)
4ab7e7a Use terse struct construction form (Tobin C. Harding)
9d1a984 Remove 'static from &str (Tobin C. Harding)
b9df24c Clear block can be collapsed warnings (Tobin C. Harding)
98e47d5 Remove support for 1.29 (Tobin C. Harding)
ae0f3d9 desc: check for mismatch in the number of derivation paths between multipath keys (Antoine Poinsot)
131bb5b descriptors: BIP389 multipath descriptors support (Antoine Poinsot)
cf401b6 desc keys: add a method to get single-path keys from multipath keys (Antoine Poinsot)
b5eb317 desc keys: unit tests for multipath key expressions (Antoine Poinsot)
34472bb desc keys: implement BIP389 multipath key expressions (Antoine Poinsot)
e122255 desc keys: make 'at_derivation_index' return an error instead of panicing (Antoine Poinsot)
ccd7ef1 desc keys: make parse_xkey_deriv a standalone function (Antoine Poinsot)
fb3cb90 desc keys: rename parse_xkey_origin to parse_key_origin (Antoine Poinsot)
60732fb desc keys: make origin key parsing a standalone function (Antoine Poinsot)
8e781ca Derive comparison traits for all secret key types (Scott Robinson)
a113e35 Release 9.0.0 (sanket1729)
63ceb5b Add psbt example for sign and finalize (sanket1729)
8f86992 Don't run rawpkh logic through pkh (sanket1729)
815fd1c Bug fix: pkh->pk lookup API (sanket1729)
43abc43 Use expr_raw_pk_h for Terminal::RawPkH (sanket1729)
480c3fd replaced if else return with mapping (BufferOverflow)

Pull request description:

  Includes all updates from the previous 30 PRs since last merge.

ACKs for top commit:
  apoelstra:
    ACK 20dd79c

Tree-SHA512: 029e76111789c66cb1472f342d3eed0625bd46715f832d6f071435dfa9117444b860b235f4bde5543551c687a548f61a49844221c5ae98732a9034c6f69ad3d4

Author: apoelstra

.github/workflows/rust.yml

@@ -6,10 +6,6 @@ jobs:
   lint_fuzz_stable:
     name: Lint + Fuzz
     runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        rust:
-          - 1.58.0
     steps:
       - name: Checkout Crate
         uses: actions/checkout@v2
@@ -19,7 +15,7 @@ jobs:
         uses: actions-rs/toolchain@v1
         with:
           profile: minimal
-          toolchain: ${{ matrix.rust }}
+          toolchain: 1.58.0
           override: true
       - name: Running fuzzer
         env:
@@ -48,6 +44,23 @@ jobs:
           DO_DOCS: true
         run: ./contrib/test.sh
 
+  Int-tests:
+    name: Integration tests
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Crate
+        uses: actions/checkout@v2
+      - name: Checkout Toolchain
+        uses: actions-rs/toolchain@v1
+        with:
+          profile: minimal
+          toolchain: stable
+          override: true
+      - name: Running integration tests
+        env:
+          DO_BITCOIND_TESTS: true
+        run: ./contrib/test.sh
+
   Tests:
     name: Tests
     runs-on: ubuntu-latest

CHANGELOG.md

@@ -1,3 +1,10 @@
+# 9.0.0 - November 5, 2022
+
+- Fixed a bug dealing with dissatisfying pkh inside thresh
+- Changed the signature of `Satisfier::lookup_raw_pkh_pk` API. Only custom implementations
+  of `Satisfier` need to be updated. The psbt APIs are unchanged.
+- Fixed a bug related to display of `raw_pk_h`. These descriptors are experimental
+  and only usable by opting via `ExtParams` while parsing string.
 # 8.0.0 - October 20, 2022
 
 This release contains several significant API overhauls, as well as a bump

Cargo.toml

@@ -17,20 +17,24 @@ trace = []
 unstable = []
 serde = ["actual-serde", "bitcoin/serde"]
 rand = ["bitcoin/rand"]
+base64 = ["bitcoin/base64"]
 
 [dependencies]
 bitcoin = "0.29.1"
 elements = "0.21.0"
-bitcoin-miniscript = { package = "miniscript", version = "9.0" }
+bitcoin-miniscript = { package = "miniscript", version = "9.0", git = "https://github.com/rust-bitcoin/rust-miniscript", rev = "1351c20a5ead4f308e18bce88cc01983c7d0e876" }
 
 # Do NOT use this as a feature! Use the `serde` feature instead.
 actual-serde = { package = "serde", version = "1.0", optional = true }
 
 [dev-dependencies]
 serde_json = "1.0"
-elementsd = {version = "0.6.0", features=["0_21_0","bitcoind_22_0"]}
 actual-rand = { package = "rand", version = "0.8.4"}
+serde_test = "1.0.147"
+bitcoin = { version = "0.29.2", features = ["base64"] }
 secp256k1 = {version = "0.24.0", features = ["rand-std"]}
+actual-base64 = { package = "base64", version = "0.13.0" }
+
 
 [[example]]
 name = "htlc"
@@ -51,3 +55,7 @@ name = "xpub_descriptors"
 [[example]]
 name = "taproot"
 required-features = ["compiler"]
+
+[[example]]
+name = "psbt_sign_finalize"
+required-features = ["base64"]

README.md

@@ -29,6 +29,11 @@ coins in a given Bitcoin transaction
 More information can be found in [the documentation](https://docs.rs/elements-miniscript)
 or in [the `examples/` directory](https://github.com/ElementsProject/elements-miniscript/tree/master/examples)
 
+## Building
+
+The cargo feature `std` is enabled by default. At least one of the features `std` or `no-std` or both must be enabled.
+
+Enabling the `no-std` feature does not disable `std`. To disable the `std` feature you must disable default features. The `no-std` feature only enables additional features required for this crate to be usable without `std`. Both can be enabled without conflict.
 
 ## Minimum Supported Rust Version (MSRV)
 This library should always compile with any combination of features on **Rust 1.41.1**.
@@ -52,6 +57,14 @@ architectural mismatches. If you have any questions or ideas you want to discuss
 please join us in
 [##miniscript](https://web.libera.chat/?channels=##miniscript) on Libera.
 
-# Release Notes
+
+## Release Notes
 
 See [CHANGELOG.md](CHANGELOG.md).
+
+
+## Licensing
+
+The code in this project is licensed under the [Creative Commons CC0 1.0
+Universal license](LICENSE). We use the [SPDX license list](https://spdx.org/licenses/) and [SPDX
+IDs](https://spdx.dev/ids/).

bitcoind-tests/Cargo.toml

@@ -0,0 +1,13 @@
+[package]
+name = "bitcoind-tests"
+version = "0.1.0"
+authors = ["sanket1729 <[email protected]>"]
+edition = "2018"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+elements-miniscript = {path = "../"}
+elementsd = {version = "0.6.0", features=["0_21_0","bitcoind_22_0"]}
+actual-rand = { package = "rand", version = "0.8.4"}
+secp256k1 = {version = "0.24.0", features = ["rand-std"]}

tests/data/random_ms.txt renamed to bitcoind-tests/tests/data/random_ms.txt

@@ -1,7 +1,7 @@
 pub extern crate elements_miniscript;
 
 use std::str::FromStr;
-
+use miniscript::elements;
 use elements::encode::{deserialize, serialize_hex};
 use elements::hashes::hex::FromHex;
 use elements::BlockHash;

tests/setup/mod.rs renamed to bitcoind-tests/tests/setup/mod.rs

@@ -20,9 +20,10 @@
 use std::collections::HashMap;
 use std::str::FromStr;
 
-use bitcoin::hashes::hex::ToHex;
-use bitcoin::hashes::{hash160, ripemd160, sha256, Hash};
-use bitcoin::secp256k1;
+use miniscript::{elements, bitcoin};
+use elements::hashes::hex::ToHex;
+use elements::hashes::{hash160, ripemd160, sha256, Hash};
+use elements::secp256k1_zkp as secp256k1;
 use elements::hashes::hex::FromHex;
 use elements::{confidential, encode, AddressParams, BlockHash};
 use miniscript::descriptor::{SinglePub, SinglePubKey};

tests/setup/test_util.rs renamed to bitcoind-tests/tests/setup/test_util.rs

@@ -3,11 +3,12 @@
 //! Arith expression fragment integration tests
 //!
 
+use miniscript::elements;
 use elements::pset::PartiallySignedTransaction as Psbt;
 use elements::sighash::SigHashCache;
 use elements::taproot::{LeafVersion, TapLeafHash};
 use elements::{
-    self, confidential, pset as psbt, secp256k1_zkp as secp256k1, sighash, OutPoint, Script,
+    confidential, pset as psbt, secp256k1_zkp as secp256k1, sighash, OutPoint, Script,
     Sequence, TxIn, TxOut, Txid,
 };
 use elementsd::ElementsD;
@@ -44,7 +45,8 @@ pub fn test_desc_satisfy(cl: &ElementsD, testdata: &TestData, desc: &str) -> Vec
 
     let definite_desc = test_util::parse_test_desc(&desc, &testdata.pubdata)
         .unwrap()
-        .at_derivation_index(0);
+        .at_derivation_index(0)
+        .unwrap();
 
     let derived_desc = definite_desc.derived_descriptor(&secp).unwrap();
     let desc_address = derived_desc.address(&PARAMS).unwrap(); // No blinding

tests/test_arith.rs renamed to bitcoind-tests/tests/test_arith.rs

@@ -7,12 +7,13 @@
 use std::fs::File;
 use std::io::{self, BufRead};
 use std::path::Path;
+use miniscript::{elements, bitcoin};
 
 use bitcoin::hashes::{sha256d, Hash};
 use bitcoin::secp256k1::{self, Secp256k1};
 use elements::pset::PartiallySignedTransaction as Psbt;
 use elements::{
-    self, confidential, pset as psbt, secp256k1_zkp, AssetIssuance, LockTime, OutPoint, Script,
+    confidential, pset as psbt, secp256k1_zkp, AssetIssuance, LockTime, OutPoint, Script,
     Sequence, TxIn, TxInWitness, TxOut, TxOutWitness, Txid,
 };
 use elements_miniscript as miniscript;

tests/test_cpp.rs renamed to bitcoind-tests/tests/test_cpp.rs

@@ -3,11 +3,12 @@
 //! CheckSigFromStack integration tests
 //!
 
+use miniscript::{elements, bitcoin};
 use elements::pset::PartiallySignedTransaction as Psbt;
 use elements::sighash::SigHashCache;
 use elements::taproot::{LeafVersion, TapLeafHash};
 use elements::{
-    self, confidential, pset as psbt, secp256k1_zkp as secp256k1, sighash, OutPoint, Script,
+    confidential, pset as psbt, secp256k1_zkp as secp256k1, sighash, OutPoint, Script,
     Sequence, TxIn, TxOut, Txid,
 };
 use elementsd::ElementsD;
@@ -44,7 +45,8 @@ pub fn test_desc_satisfy(cl: &ElementsD, testdata: &TestData, desc: &str) -> Vec
 
     let definite_desc = test_util::parse_test_desc(&desc, &testdata.pubdata)
         .unwrap()
-        .at_derivation_index(0);
+        .at_derivation_index(0)
+        .unwrap();
 
     let derived_desc = definite_desc.derived_descriptor(&secp).unwrap();
     let desc_address = derived_desc.address(&PARAMS).unwrap(); // No blinding

tests/test_csfs.rs renamed to bitcoind-tests/tests/test_csfs.rs

@@ -6,12 +6,13 @@
 
 use std::{error, fmt};
 
+use miniscript::{elements, bitcoin};
 use elements::hashes::{sha256d, Hash};
 use elements::pset::PartiallySignedTransaction as Psbt;
 use elements::sighash::SigHashCache;
 use elements::taproot::{LeafVersion, TapLeafHash};
 use elements::{
-    self, confidential, pset as psbt, secp256k1_zkp as secp256k1, sighash, OutPoint, SchnorrSig,
+    confidential, pset as psbt, secp256k1_zkp as secp256k1, sighash, OutPoint, SchnorrSig,
     Script, Sequence, TxIn, TxOut, Txid,
 };
 use elementsd::ElementsD;
@@ -77,7 +78,8 @@ pub fn test_desc_satisfy(
 
     let definite_desc = test_util::parse_test_desc(&descriptor, &testdata.pubdata)
         .map_err(|_| DescError::DescParseError)?
-        .at_derivation_index(0);
+        .at_derivation_index(0)
+        .unwrap();
 
     let derived_desc = definite_desc.derived_descriptor(&secp).unwrap();
     let desc_address = derived_desc.address(&PARAMS); // No blinding
@@ -384,6 +386,10 @@ fn test_descs(cl: &ElementsD, testdata: &TestData) {
     let result = test_desc_satisfy(cl, testdata, "tr(X!,{pk(X1!),pk(X2!)})");
     assert_eq!(result, Err(DescError::PsbtFinalizeError));
 
+    // Test 10: Test taproot desc with ZERO known keys
+    let result = test_desc_satisfy(cl, testdata, "tr(X!,j:multi_a(3,X1!,X2,X3,X4))");
+    assert_eq!(result, Err(DescError::DescParseError));
+
     // Test 11: Test taproot with insufficient known keys
     let result = test_desc_satisfy(cl, testdata, "tr(X!,{pk(X1!),multi_a(3,X2!,X3,X4)})");
     assert_eq!(result, Err(DescError::PsbtFinalizeError));

tests/test_desc.rs renamed to bitcoind-tests/tests/test_desc.rs

@@ -3,11 +3,12 @@
 //! Arith expression fragment integration tests
 //!
 
+use miniscript::elements;
 use elements::pset::PartiallySignedTransaction as Psbt;
 use elements::sighash::SigHashCache;
 use elements::taproot::{LeafVersion, TapLeafHash};
 use elements::{
-    self, confidential, pset as psbt, secp256k1_zkp as secp256k1, sighash, OutPoint, Script,
+    confidential, pset as psbt, secp256k1_zkp as secp256k1, sighash, OutPoint, Script,
     Sequence, TxIn, TxOut, Txid,
 };
 use elementsd::ElementsD;
@@ -44,7 +45,8 @@ pub fn test_desc_satisfy(cl: &ElementsD, testdata: &TestData, desc: &str) -> Vec
 
     let definite_desc = test_util::parse_test_desc(&desc, &testdata.pubdata)
         .unwrap()
-        .at_derivation_index(0);
+        .at_derivation_index(0)
+        .unwrap();
 
     let derived_desc = definite_desc.derived_descriptor(&secp).unwrap();
     let desc_address = derived_desc.address(&PARAMS).unwrap(); // No blinding

tests/test_introspect.rs renamed to bitcoind-tests/tests/test_introspect.rs

@@ -1,39 +1,23 @@
-#!/bin/sh -ex
+#!/bin/sh
 
-set -e
+set -ex
 
-FEATURES="compiler serde rand"
-
-cargo update -p serde --precise 1.0.142
-cargo update -p serde_derive --precise 1.0.142
+FEATURES="compiler serde rand base64"
 
 cargo --version
 rustc --version
 
-# Work out if we are using a nightly toolchain.
-MSRV=false
+# Pin dependencies required to build with Rust 1.41.1
 if cargo --version | grep "1\.41\.0"; then
-    MSRV=true
-fi
-
-if cargo --version | grep "1\.47\.0"; then
-    cargo update -p once_cell --precise 1.13.1
-fi
-
-# form_urlencoded 1.1.0 breaks MSRV.
-if [ "$MSRV" = true ]; then
-    cargo update -p url --precise 2.2.2
-    cargo update -p form_urlencoded --precise 1.0.1
-    cargo update -p once_cell --precise 1.13.1
-    cargo update -p bzip2 --precise 0.4.2
-    cargo update -p which --precise 4.3.0
+    cargo update -p serde --precise 1.0.156
+    cargo update -p syn --precise 1.0.107
 fi
 
 # Format if told to
 if [ "$DO_FMT" = true ]
 then
     rustup component add rustfmt
-    cargo fmt --all -- --check
+    cargo fmt -- --check
 fi
 
 # Fuzz if told to
@@ -47,6 +31,15 @@ then
     exit 0
 fi
 
+# Test bitcoind integration tests if told to (this only works with the stable toolchain)
+if [ "$DO_BITCOIND_TESTS" = true ]; then
+    cd bitcoind-tests
+    cargo test --verbose
+
+    # Exit integration tests, do not run other tests.
+    exit 0
+fi
+
 # Defaults / sanity checks
 cargo test
 
@@ -69,6 +62,7 @@ then
     cargo run --example verify_tx > /dev/null
     cargo run --example xpub_descriptors
     cargo run --example taproot --features=compiler
+    cargo run --example psbt_sign_finalize --features=base64
 fi
 
 # Bench if told to (this only works with the nightly toolchain)
@@ -79,7 +73,7 @@ fi
 
 # Build the docs if told to (this only works with the nightly toolchain)
 if [ "$DO_DOCS" = true ]; then
-    RUSTDOCFLAGS="--cfg docsrs" cargo doc --all --features="$FEATURES"
+    RUSTDOCFLAGS="--cfg docsrs" cargo +nightly rustdoc --features="$FEATURES" -- -D rustdoc::broken-intra-doc-links
 fi
 
 exit 0