Skip to content

Releases: filecoin-project/boost

v2.1.1

30 Nov 15:16
e9d18ac
Compare
Choose a tag to compare

Overview

We're excited to introduce Boost v2.1.1. This release focuses on reducing the resource usage utilisation for boostd-data service.

Compatibility

This release is compatible with Lotus versions v1.24.0 and v1.25.0. For the latest compatibility details, refer to the Boost and Lotus compatibility matrix.

⚠️ Important Upgrade Notice

👉 Before updating, carefully read the release notes and documentation. Storage providers will need to set up a database service and undergo a database migration process. Incorrect procedures can lead to permanent data loss.

⭐ Highlights

  • The high CPU usage by boostd-data service is now fixed
  • Deal indexing has been improved to avoid deals getting stuck in "Announcing" state
  • Boost UI should not display the momentary searching message: %!w(<nil>) message anymore

What's Changed

Full Changelog: v2.1.0...v2.1.1

v2.1.0

24 Nov 15:32
3233c51
Compare
Choose a tag to compare

Overview

We're excited to introduce Boost v2.1.0, a significant update that transforms how Boost manages indices for user data. This version phases out the DAG store in favor of the new Local Index Directory (LID). Alongside this, we've added new features and improvements, and squashed some bugs to enhance your experience.

Compatibility

This release is compatible with Lotus versions v1.24.0 and v1.25.0. For the latest compatibility details, refer to the Boost and Lotus compatibility matrix.

⚠️ Important Upgrade Notice

👉 Before updating, carefully read the release notes and documentation. Storage providers will need to set up a database service and undergo a database migration process. Incorrect procedures can lead to permanent data loss.

Highlights

  • New HTTP Gateway (Frisbii): booster-http now uses Frisbii, a trustless gateway. It no longer supports serving unixfs files, requiring an additional bifrost-gateway binary for such files.
  • Boost UI Enhancements: The UI now allows SPs to check payload CIDs for pieceCIDs, aiding in retrieval testing and debugging.
  • Boostd-data Metrics: Emission of metrics for each API method, with updated dashboards to monitor boostd-data performance and service uptime.
  • Index Provider Upgrades: Switch from Graphsync to HTTP-libp2p protocol and UI improvements for better retrieval and indexing information display.
  • Multiple Miners to Single LID: Connect multiple miners to a single LID instance for unified retrieval and deal direction.
  • Manual PSD (Publish Storage Deal): Manual control over PSD message publishing, with a configuration variable for enabling it.
  • Graphsync Improvements: Better connection termination to resolve stalled retrievals.
  • New IPNI UI Page: Dedicated page for IPNI retrievals with detailed information and sync status.

Improvements

  • Performance Enhancements: Reduced devnet build time, parallelization in index migration, batched write operations, and optimized deal filter mechanism.
  • UI and Config File Tweaks: Improved UI elements, added CORS handler for booster-http, and generated commented config file on boostd init.
  • Database Support: Interfaces for LevelDB and YugabyteDB, catering to different data storage scales.
  • Extended Deal Durations: Support for making deals up to 3.5 years long.
  • Timeout Configurations: Updated and customizable CQL client timeouts.

Notable Changes

  • Graphsync Server and Client Updates: Enhanced response handling for identity CID requests and improved multipart downloads for online deals.
  • Security Enhancements: Critical security issue resolved by upgrading to grpc v1.56.3.
  • Boostd-data Command Improvements: Added CQL timeout configuration.
  • Monitoring Stack Update: Revised dashboards shipped with the Docker-based monitoring stack.
  • Database Migration Support: Added support for parallel processing in database migration.

Upgrading to Boost v2.1.0

For a smooth transition to Boost v2.1.0, follow the detailed instructions available in the Boost Documentation Tutorials - How to Migrate to Boost v2.
We're confident that these updates will enhance your experience with Boost, and we look forward to your feedback! 🚀

What's Changed

Read more

v2.1.0-rc3

01 Nov 17:38
1622d0c
Compare
Choose a tag to compare
v2.1.0-rc3 Pre-release
Pre-release

Overview

We're happy to announce the Boost v2.1.0-rc3 release. Boost v2 revamps the way Boost handles indices for user data, introducing the Local Index Directory (LID) and removing the DAG store. In addition, there are a number of new features including bug fixes, increased flexibility on LID setup, and more.

This release is compatible with the Lotus v1.24.0-rc1 and v1.25.0-rc1 release. See the Boost and Lotus compatibility matrix for up to date information to plan your upgrade.

👉 ⚠️ ❗ ❗ Before updating, please read the release notes and documentation carefully, as storage providers will need to setup a database service and go through a database migration process. Incorrect process can result in permanent data loss ❗ ❗ ⚠️

⭐ Highlights

New HTTP Gateway (Frisbii)

  1. The booster-http binary has been updated to use Frisbii as a trustless gateway.
  2. Serving unixfs file is no longer supported by booster-http. Users would need to run an additional bifrost-gateway binary to serve unixfs formatted files.
  3. The new gateway implementation should allow retrieval verifiability by default.

Boostd-data metrics

  1. The boostd-data service emits metrics for each API method.
  2. The dashboards shipped with the docker based monitoring stack now include these metrics.
  3. The updated dashboards can be used to understand the boostd-data performance, as well as service uptime for all Boost services.
  4. Documentation on how to deploy monitoring has been updated to include all the new changes.

🏗️ Improvements

  1. CQL client timeout has been updated to 60 seconds from the original default of 6 seconds.
  2. Users can now tweak the CQL timeout further using the --CQLtimeoutflag when starting theboostd-data` command.
  3. Boost now supports making 3.5 year long deals.
  4. Write operations for index insertion in YugabyteDB are now batched with the size of 10k entries per batch, which will improve performance for large write operations.

What's Changed

  • chore(deps): bump @babel/traverse from 7.18.0 to 7.23.2 in /react by @dependabot in #1759
  • fix: empty IPNI UI due to lack if latest Ad CID by @LexLuthr in #1766
  • fix: handle isexist error for IsDirectDeal column in migration by @LexLuthr in #1761
  • Create support ticket template by @LexLuthr in #1749
  • feat: spans and debug lines for piece directory by @nonsense in #1773
  • fix: Remove line forcing for logs in issue template by @LexLuthr in #1770
  • feat: add cql timeout to boostd-data command by @LexLuthr in #1772
  • booster-http refactor - trustless only, improved compression, logging & e2e testing w/ trusted by @rvagg in #1676
  • fix: awaiting publish confirmation count by @LexLuthr in #1778
  • fix: update deal filters to use new values by @LexLuthr in #1779
  • fix: add CQL timeout to lid-migrate command by @LexLuthr in #1783
  • Resolve critical security issue by Upgrading to grpc v1.56.3 by @masih in #1785
  • boostx: add venus to stats cmd by @nonsense in #1786
  • fix: use new MaxSectorExpirationExtension by @LexLuthr in #1788
  • boostd-data: yugabyte migrate fix missing cql timeout by default by @nonsense in #1790
  • feat: add boostd-data metrics by @LexLuthr in #1784
  • chore: release v2.1.0-rc3 by @LexLuthr in #1794
  • fix: speed up ydb insert by @ischasny in #1793
  • yugabyte/service: parallel batch inserts for car indices by @nonsense in #1795

Full Changelog: v2.1.0-rc2...v2.1.0-rc3

v2.1.0-rc2

18 Oct 20:58
fa9fc84
Compare
Choose a tag to compare
v2.1.0-rc2 Pre-release
Pre-release

Overview

We're happy to announce the Boost v2.1.0-rc2 release. Boost v2 revamps the way Boost handles indices for user data, introducing the Local Index Directory (LID) and removing the DAG store. In addition, there are a number of new features including bug fixes, increased flexibility on LID setup, and more.

This release is compatible with the Lotus v1.24.0-rc1 and v1.25.0-rc1 release. See the Boost and Lotus compatibility matrix for up to date information to plan your upgrade.

👉 ⚠️ ❗ ❗ Before updating, please read the release notes and documentation carefully, as storage providers will need to setup a database service and go through a database migration process. Incorrect process can result in permanent data loss ❗ ❗ ⚠️

⭐ Highlights

Boost UI will allow SPs to check the payload CIDs for a pieceCID. These CID are useful for retrieval testing and debugging.

Screenshot 2023-09-27 at 7 18 09 PM

Index Provider

  1. The index-provider has switched from Graphsync data transfer protocol to HTTP-libp2p protocol.
  2. The Retrieval page in the UI has been split up so that only retrieval deal information is shown on the existing page, and all indexing information is shown on the Indexer page.
  3. The indexer UI in Boost displays the lag between the latest local advertisement and the sync status of the network indexers.
Screenshot 2023-10-03 at 3 20 43 PM

🏗️ Improvements

  1. The devnet build time has been reduced from 40+ minutes to <5 minutes.
  2. The migrate-lid command now support parallelisation of index migration to speed up the over all migration time required for switching from Boost v1 to v2.
  3. boostd init generates a commented out config file
  4. The deal filter mechanism has been redesigned to run in parallel. This should speed up the deal acceptance time and reduce client timeouts.
  5. Graphsync server has been updated to respond correctly for identity CID requests to reduce the overall retrieval failure rates.
  6. The deal download has been redesigned to perform multi-part downloads to reduce the download time for online deals(like aria2c).
  7. The pieceReader is closed correctly after generating index to fix the issue of dangling deleted files on Lotus AP workers.

What's Changed

New Contributors

Full Changelog: v2.1.0-rc1...v2.1.0-rc2

v2.1.0-rc1

30 Aug 17:47
0cd9d5d
Compare
Choose a tag to compare
v2.1.0-rc1 Pre-release
Pre-release

Overview

We're happy to announce the Boost v2.1.0-rc1 release. Boost v2 revamps the way Boost handles indices for user data, introducing the Local Index Directory (LID) and removing the DAG store. In addition, there are a number of new features including bug fixes, increased flexibility on LID setup, and more.

The Local Index Directory requires a database for persistence. After reviewing various options and consulting the community, we are providing interfaces to two databases - LevelDB (for SPs holding less than 1 PiB of user data) and YugabyteDB (for SPs holding more than 1 PiB of user data). ❗ Please take your future growth into account before choosing a database ❗

👉 ⚠️ ❗ ❗ Before updating, please read the release notes and documentation carefully, as storage providers will need to setup a database service and go through a database migration process. Incorrect process can result in permanent data loss ❗ ❗ ⚠️

Highlights

Multiple Miners to Single LID

  • Storage providers running multiple miners can connect all of their boostd, booster-http and booster-bitswap instances to the same LID instance.
  • All boostd, booster-http and booster-bitswap instances can serve retrievals from any miner connected to the LID instance.

mm-cluster

  • New storage deals coming to a boostd will be directed to its dedicated miner.

mm-storage-deal

Graphsync Improvements

The Graphsync library should now terminate the connection properly. This should permanently fix the stalled Graphsync retrievals issue that some storage providers have experienced.

Manual PSD (Publish Storage Deal)

This new feature allows SPs to manually send PSD messages. Once this feature is turned on, Boost will no longer automatically send PSD messages unless explicitly prompted by the user. The feature can be turned on with the following config variable:

[Dealmaking]
        // When set to true, the user is responsible for publishing deals manually.
	// The values of MaxDealsPerPublishMsg and PublishMsgPeriod will be
	// ignored, and deals will remain in the pending state until manually published.
	ManualDealPublish bool

More details about how to use this feature can be found in the Boost docs.

New UI page for IPNI interactions

  • IPNI retrievals are now separated out from the data retrievals page.
  • Latest local and remote advertisement CIDs can be used to check the sync status.
  • Publisher details and other config can be found in the UI.
IPNI UI

⚠️ Users will now see Boost v2 documentation by default. To refer to Boost v1 documentation, you can select v1 in the dropdown on the header.

⚠️ We will no longer be backporting any fixes to v1 releases.

🏗️ Additional Improvements

  • The miner ID is now displayed on the top of all UI pages.
  • The boost binaries will now show the network name to avoid confusion.
./boost --version
boost version 2.0.0-rc1+calibnet+git.8f88bde.dirty
  • Disabled index-provider will not crash boostd process
  • Booster-http now has a CORS handler for IPFS gateway as well as full piece download path.
  • Cached pieceReader allows faster retrievals speeds for the same content.
  • Boost now uses the Lotus pieceReader enhancements, resulting in a 3x improvement in read performance.

Upgrading to Boost v2

You can find detailed instructions on how to upgrade to Boost v2 via the Boost Documentation Tutorials - How to Migrate to Boost v2.

What's Changed

New Contributors

Full Changelog: v2.0.0...v2.1.0-rc1

v1.7.5

08 Aug 09:56
Compare
Choose a tag to compare

Overview

We're happy to announce the Boost v1.7.5 release.

This release is compatible with the Lotus v1.23.0 release. See the Boost and Lotus compatibility matrix for up to date information to plan your upgrade.

⚠️ If you have not yet upgraded to Boost 1.7.0 or later, please see the 1.7.0 release notes before upgrading. ⚠️

Improvements

  • Fixes the boostd crash when index-provider is disabled in the configuration
  • Boost deals will not be filtered out based on ExpectedSealDuration and MaxDealStartDelay values
  • MaxDealStartDelay will now only be applicable for online deals
  • StartEpochSealingBuffer will now be used from [Dealmaking] section for boost deals instead of [LotusDealmaking] section ❗

Full Changelog: v1.7.4...v1.7.5

v2.0.0

31 Jul 16:56
Compare
Choose a tag to compare

Overview

We're happy to announce the Boost v2.0.0 release. This release revamps the way Boost handles indices for user data, introducing the Local Index Directory (LID) and removing the DAG store.

The Local Index Directory requires a database for persistence. After reviewing various options and consulting the community, we are providing interfaces to two databases - LevelDB (for SPs holding less than 1 PiB of user data) and YugabyteDB (for SPs holding more than 1 PiB of user data). ❗ Please take your future growth into account before choosing a database ❗

👉 ⚠️ ❗ ❗ Before updating, please read the release notes and documentation carefully, as storage providers will need to setup a database service and go through a database migration process. Incorrect process can result in permanent data loss ❗ ❗ ⚠️

Highlights

Local Index Directory (LID)

The Local Index Directory (LID) manages and stores indices of deal data so that it can be retrieved by a content identifier (CID).

Currently this task is performed by the DAG store component. The DAG store keeps its indices on disk on a single machine. LID replaces the DAG store and introduces a horizontally scalable backend database for storing the data - YugabyteDB.

LID is designed to provide a more intuitive experience for the user, by surfacing problems and providing various repair tools. It makes it easy to understand what data is index and retrievable, and what data is archived and requires unsealing to make it retrievable.

Screenshot 2023-06-14 at 13 13 54 Screenshot 2023-06-14 at 13 14 32 Screenshot 2023-06-14 at 13 14 55

Legacy storage deals are deprecated

Boost will no longer support legacy deals by default. Users will get an error message stating the same when they try to make deals using deal protocols older than mk2.0
To avoid breaking the deal flow, we have a new configuration option to enable them back. We highly recommend switching to new deals as this option will be removed in future releases.

[Dealmaking]
  # Whether to enable legacy deals on the Boost node or not. We recommend keeping
  # them disabled. These will be completely deprecated soon.
  #
  # type: bool
  #EnableLegacyStorageDeals = false

⚠️ Boost v2 documentation will now be default. Users can select v1 from the dropdown if they wish to refer to Boost v1 documentation.

⚠️ Boost will no longer be backporting any fixes to v1 releases

Upgrading to Boost v2

You can find detailed instructions on how to upgrade to Boost v2 via the Boost Documentation Tutorials - How to Migrate to Boost v2.

Full Changelog: v1.7.4...v2.0.0

v2.0.0-rc1

11 Jul 09:17
f69945e
Compare
Choose a tag to compare
v2.0.0-rc1 Pre-release
Pre-release

Overview

We're happy to announce the Boost v2.0.0-rc1 release. This release revamps the way Boost handles indices for user data, introducing the Local Index Directory (LID) and removing the DAG store.

The Local Index Directory requires a database for persistence. After reviewing various options and consulting the community, we are providing interfaces to two databases - LevelDB (for SPs holding less than 1 PiB of user data) and YugabyteDB (for SPs holding more than 1 PiB of user data). ❗ Please take your future growth into account before choosing a DB type ❗

👉 ⚠️ ❗ ❗ Before updating, please read the release notes and documentation carefully, as storage providers will need to setup a database service and go through a database migration process. Incorrect process can result in permanent data loss ❗ ❗ ⚠️

Highlights

Local Index Directory (LID)

The Local Index Directory (LID) manages and stores indices of deal data so that it can be retrieved by a content identifier (CID).

Currently this task is performed by the DAG store component. The DAG store keeps its indices on disk on a single machine. LID replaces the DAG store and introduces a horizontally scalable backend database for storing the data - YugabyteDB.

LID is designed to provide a more intuitive experience for the user, by surfacing problems and providing various repair tools. It makes it easy to understand what data is index and retrievable, and what data is archived and requires unsealing to make it retrievable.

Screenshot 2023-06-14 at 13 13 54 Screenshot 2023-06-14 at 13 14 32 Screenshot 2023-06-14 at 13 14 55

Upgrading to Boost v2

You can find detailed instructions on how to upgrade to Boost v2 via the Boost Documentation Tutorials - How to Migrate to Boost v2.

Full Changelog: v1.7.3...v2.0.0-rc1

v1.7.4

10 Jul 08:24
52cb7cd
Compare
Choose a tag to compare

Overview

We're happy to announce the Boost v1.7.4 release.

This release is compatible with the Lotus v1.23.0 release. See the Boost and Lotus compatibility matrix for up to date information to plan your upgrade.

⚠️ If you have not yet upgraded to Boost 1.7.0 or later, please see the 1.7.0 release notes before upgrading. ⚠️

Improvements

  • Fixes the UI bug alerting about all stuck messages instead of local stuck messages

What's Changed

New Contributors

Full Changelog: v1.7.3...v1.7.4

v1.7.3

12 Jun 09:07
188d5f0
Compare
Choose a tag to compare

Overview

We're happy to announce the Boost v1.7.3 release.
This release is compatible with the Lotus v1.23.0 and v.23.1-rcX release. See the Boost and Lotus compatibility matrix for up to date information to plan your upgrade.

⚠️ If you have not yet upgraded to Boost 1.7.0 or later, please see the 1.7.0 release notes before upgrading. ⚠️

🌟 Highlights

Option to disable tagging of wallet funds

  • By default, Boost tags funds for each deal with the value defined by MaxPublishDealsFee, and untags the funds when the deal is published. This is to prevent SPs from accepting too many deals and then not being able to publish them due to a lack of funds.
  • Some SPs would prefer to manage their wallet funds themselves.
  • Tagging can now be disabled by setting the following variable in Boost config
   	// Whether to enable tagging of funds. If enabled, each time a deal is
	// accepted boost will tag funds for that deal so that they cannot be used
	// for any other deal.
	FundsTaggingEnabled bool

HTTP announcements to the IPNI

  • Boost can now announce advertisements over HTTP to the IPNI
  • Once enabled, it should reduce the request load over Graphsync and improve performance of data retrievals
  • New configuration to enable HTTP announcements
    PurgeCacheOnStart = true
    
      [IndexProvider.Announce]
        # Make a direct announcement to a list of indexing nodes over http.
        # Note that announcements are already made over pubsub regardless
        # of this setting.
        #
        # type: bool
        # env var: LOTUS_INDEXPROVIDER_ANNOUNCE_ANNOUNCEOVERHTTP
        AnnounceOverHttp = true
    
        # The list of URLs of indexing nodes to announce to.
        #
        # type: []string
        # env var: LOTUS_INDEXPROVIDER_ANNOUNCE_DIRECTANNOUNCEURLS
        DirectAnnounceURLs = ["https://cid.contact/ingest/announce", "http://localhost:3001"]
    
      [IndexProvider.HttpPublisher]
        # If not enabled, requests are served over graphsync instead.
        #
        # type: bool
        # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_ENABLED
        #Enabled = true
    
        # Set the public hostname / IP for the index provider listener.
        # eg "82.129.73.111"
        # This is usually the same as the for the boost node.
        #
        # type: string
        # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_PUBLICHOSTNAME
        #PublicHostname = "xx.xx.xx.xx"
    
        # Set the port on which to listen for index provider requests over HTTP.
        # Note that this port must be open on the firewall.
        #
        # type: int
        # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_PORT
       # Port = 3104
    

Fix for stalled Graphsync retrievals

  • When a deal is announced, the network indexer queries boost to get the cids of each block in the deal data
  • There was an issue with the code that could cause retrievals to stall.
  • Requests made via the network indexers, like cid.contact, are now routed through the simplified graphsync retrieval pathways as opposed to the legacy retrievals which is prone to stalling, and is less performant.
  • Stalled Graphsync retrieval cancellation code has been refactored for faster execution and efficiency

🏗️ Improvements

Managing Mpool

  • Deal page will display the number of epochs elapsed since PSD message was sent. This should help SPs to quickly find PSD messages stuck in the local mpool by looking at the number of epochs elapsed vs message confidence (10 epochs)
  • Mpool section in WebUI displays the method parameters for each message

Improved configuration and defaults for internet interactions

  • Users can specify a listen address for booster-http instead of listening on all available addresses
  • WebUI now listens on localhost by default to avoid accidentally exposing it to the internet

Deal making

  • Better deal-acceptance timeout implementation to avoid timeout due to the execution time of the external deals filters
  • A listen address can be provided for GraphQL service. This should mitigate security risks for publicly exposed Boost nodes.
  • Offline deals waiting for data can be cancelled from the UI.
  • Boost will now fail the deal if a snap-deal sector reverts to Proving at any point of the sealing process. Earlier, Boost would incorrectly mark such deals as successfully sealed.
  • Boost now supports local commp calculation for 64 GiB sectors.

Others

  • Indexer should be able to skip bad advertisements when processing the Ad chain from a Boost node
  • All the available Boost APIs can now be listed using rpc.discover or Filecoin.Discover method
  • Boost now has a config command similar to lotus config command with additional flag --diff to generate a diff between current and default config

📣 Deprecation Notice: Disabling payments via built in payment channels actor in Boost 📣

The Boost team will be disabling payments via built in payment channels actor by default. There has been limited activity on these payment channels on the entire network. We are planning on disabling this in an effort to simplify the code base for existing retrieval use cases and improve retrieval reliability.

A few items to note:

  • The main use case for payment channels are for micropayments where small payments are made as data is sent back. With payment channels you do not need to keep sending messages to mainnet.
  • There is a PLN team actively working on payment channels, and is starting to look into integration with the Filecoin network.
  • You can still make payments for retrieval deals on mainnet, but you will need to negotiate payment terms for a retrieval deal with a SP directly.

What to expect:

  1. Boost maintainers will update default behavior to use new retrieval code (optimized for retrievals without payment channels).
  2. Please report any issues you have in #boost-help, so we can track and monitor SP impact.
  3. If you have any questions or concerns, please raise them in Boost discussion, or reach out in #boost-help

What's Changed

Read more