update #4
Open
update #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change is part of the effort to add utxocache support to btcd. mapslice allows the caller to allocate a fixed amount of memory for the utxo cache maps without the mapslice going over that fixed amount of memory. This is useful as we can have variable sizes (1GB, 1.1GB, 2.3GB, etc) while guaranteeing a memory limit.
This change is part of the effort to add utxocache support to btcd. The fresh flag indicates that the entry is fresh and that the parent view (the database) hasn't yet seen the entry. This is very useful as a performance optimization for the utxo cache as if a fresh entry is spent, we can simply remove it from the cache and don't bother trying to delete it from the database.
This change is part of the effort to add utxocache support to btcd. Getting the memory usage of an entry is very useful for the utxo cache as we need to know how much memory all the cached entries are using to guarantee a cache usage limit for the end user.
This change is part of the effort to add utxocache support to btcd. The utxoStateConsistency indicates what the last block that the utxo cache got flush at. This is useful for recovery purposes as if the node is unexpectdly shut down, we know which block to start rebuilding the utxo state from.
This change is part of the effort to add utxocache support to btcd. dbPutUtxoView handled putting and deleting new/spent utxos from the database. These two functinalities are refactored to their own functions: dbDeleteUtxoEntry and dbPutUtxoEntry. Refactoring these out allows the cache to call these two functions directly instead of having to create a view and saving that view to disk.
This change is part of the effort to add utxocache support to btcd. connectBlock may have an empty utxoviewpoint as the block verification process may be using the utxo cache directly. In that case, a nil utxo viewpoint will be passed in. Just return early on a nil utxoviewpoint.
This change is part of the effort to add utxocache support to btcd. Require the caller to pass in the utxoBucket as the caller may be fetching many utxos in one loop. Having the caller pass it in removes the need for dbFetchUtxoEntry to grab the bucket on every single fetch.
This change is part of the effort to add utxocache support to btcd. fetchInputUtxos had mainly 2 functions: 1: Figure out which outpoints to fetch 2: Call fetchUtxosMain to fetch those outpoints Functionality for (1) is refactored out to fetchInputsToFetch. This is done to allow fetchInputUtxos to use the cache to fetch the outpoints as well in a later commit.
The implemented utxocache implements connectTransactions just like utxoviewpoint and can be used as a drop in replacement for connectTransactions. One thing to note is that unlike the utxoViewpoint, the utxocache immediately deletes the spent entry from the cache. This means that the utxocache is unfit for functions like checkConnectBlock where you expect the entry to still exist but be marked as spent. disconnectTransactions is purposely not implemented as using the cache during reorganizations may leave the utxo state inconsistent if there is an unexpected shutdown. The utxoViewpoint will still have to be used for reorganizations.
This change is part of the effort to add utxocache support to btcd. utxo cache is now used by the BlockChain struct. By default it's used and the minimum cache is set to 250MiB. The change made helps speed up block/tx validation as the cache allows for much faster lookup of utxos. The initial block download in particular is improved as the db i/o bottleneck is remedied by the cache.
PruneBlocks used to delete files immediately before the database transaction finished. By making the prune atomic, we can guarantee that the database flush will happen before the utxo cache is flushed, ensuring that the utxo cache is never in an irrecoverable state.
flushNeededAfterPrune returns true if the utxocache needs to be flushed after the pruning of the given slice of block hashes. For the utxo cache to be recoverable while pruning is enabled, we need to make sure that there exists blocks since the last utxo cache flush. If there are blocks that are deleted after the last utxo cache flush, the utxo set is irrecoverable. The added method provides a way to tell if a flush is needed.
If the prune will delete block past the last flush hash of the utxocache, the cache will need to be flushed first to avoid a case where the utxocache is irrecoverable. The newly added code adds this flush logic to connectBlock.
The testing function in export_test.go is changed to just export.go so that callers outside the ffldb package will be able to call the function. The main use for this is so that the prune code can be triggered from the blockchain package. This allows testing code to have less than 1.5GB worth of blocks to trigger the prune.
blockchain, cmd, netsync, main: Add utxocache
btcutil: update to chaincfg/chainhash/v1.1.0
* fix: enhance code comments
A fix for a bug introduced by #2168 Previously, config.Host worked in the following way: 1. Documented as supporting ip addresses only 2. In fact supported "host/path" syntax 3. Did not support "scheme" prefixes, i.e. https:// Not sure this is the desired approach, probably the best thing would have been to extend config to contain "Scheme" and "Path" fields as well. However, this was the way it worked. 1. Now Host can contain scheme prefixes "unix://..." 2. Host can no longer contain ".../path" This PR solves this behavior while maintaining support of the "unix://" flow as well. For some reason, "scheme" is named "network" in #2168 - I did not change that. Also remove disambiguation in "network:address:port", where it parsed "myhost:8888" as network:address instead address:port.
Fix non-root hosts failing on resolving DNS
rpctest: make sure to `WaitForShutdown`
rpcclient: safe read and write to batch
wire: add bitcoin network magic for default SigNet
Adds support for the PSBT_GLOBAL_XPUB type as defined in BIP-0174.
psbt: add support for PSBT_GLOBAL_XPUB type
deps: bump go version
Signed-off-by: finaltrip <[email protected]>
refactor: using slices.Contains to simplify the code
chore: update license
github: upgrade the CI
multi:use t.TempDir replace os.MkdirTemp to simplify test temp dir manager
multi: remove deprecated io/ioutil
docs: update guidelines and links
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.