Adds Programmatic Storage section.

SUMMARY.md

@@ -122,6 +122,10 @@
   * [Client contract tutorial](smart-contracts/developing-contracts/client-contract-tutorial.md)
   * [Verify a contract](smart-contracts/developing-contracts/verify-a-contract.md)
   * [Best practices](smart-contracts/developing-contracts/best-practices.md)
+* [Programmatic storage](smart-contracts/programmatic-storage/README.md)
+  * [RaaS](smart-contracts/programmatic-storage/raas.md)
+  * [RaaS interfaces](smart-contracts/programmatic-storage/raas-interfaces.md)
+  * [Direct deal-making](smart-contracts/programmatic-storage/direct-deal-making.md)
 * [Advanced](smart-contracts/advanced/README.md)
   * [Wrapped FIL](smart-contracts/advanced/wrapped-fil.md)
   * [Oracles](smart-contracts/advanced/oracles.md)

smart-contracts/developing-contracts/using-raas.md

@@ -0,0 +1,72 @@
+---
+description: >-
+  This page gives a tutorial on how to use RaaS from a smart contract.
+---
+
+# Using RaaS
+
+RaaS refers to replication, renewal and repair as a service, for data stored in storage deals on Filecoin. You can read more here.
+
+In this article, we will cover how to create storage deals, with replication, renew and repair requirements, via a smart contract. This utilizes the [RaaS Starter Kit](https://github.com/filecoin-project/raas-starter-kit).
+
+## Interacting with the Smart Contract
+
+First, you need to:
+
+- EITHER start an instance of the BaseInterface by deploying a contract that inherits from `IAggregatorOracle` (you can do so via. `yarn deploy` in the [RaaS Starter Kit](https://github.com/filecoin-project/raas-starter-kit))
+- OR use an existing instance of the FullInterface located at
+  - Calibration Testnet: `0x6ec8722e6543fB5976a547434c8644b51e24785b`
+
+Interact with the smart contract by submitting a CID of your choice to the `submit` function. This will create a new deal request that will be picked up by the RaaS services.
+
+```javascript
+// contractInstance is the address of the contract you deployed or the FullInterface address above.
+const dealStatus = await ethers.getContractAt("DealStatus", contractInstance);
+// Submit the CID of the file you want to upload to the Filecoin network in the following way.
+await dealStatus.submit(ethers.utils.toUtf8Bytes(newJob.cid));
+```
+
+The [RaaS Starter Kit](https://github.com/filecoin-project/raas-starter-kit) provides you with a frontend that allows you to upload your file to Lighthouse, get a CID for the uploaded file, then seamlessly submit the CID to the smart contract (accessible via `yarn start`).
+
+Before that, you need to know how to register the various RaaS workers. Note that RaaS functionality will NOT function automatically if deals are only created using submit function.
+
+## Add Replication, Renewal, Repair Workers
+
+You can add workers to perform replication, renewal, and repair jobs by having them listen to the `SubmitAggregatorRequest`. The methods for doing so differ between the Base and Full interfaces.
+
+If you are running a base interface (specifically, the one in the [RaaS Starter Kit](https://github.com/filecoin-project/raas-starter-kit)), there's an event listener inside the RaaS service node that you can use to listen for new deal requests.
+
+This event listener performs processing for each job submitted to the contract to add RaaS service workers and eventually to call `complete` on the contract.
+
+```javascript
+// Initialize the listener for the Deal Creation event
+async function initializeDealCreationListener() {
+  const dealStatus = await ethers.getContractAt(contractName, contractInstance);
+
+  /// Logic for handling SubmitAggregatorRequest events
+  function handleEvent(transactionId, cid) {
+    console.log(
+      `Received SubmitAggregatorRequest event: (Transaction ID: ${transactionId}, CID: ${cid})`,
+    );
+    // ... other code to handle the event emission
+
+    (async () => {
+      // ... other code
+
+      // After processing this event, reattach the event listener
+      if (dealStatus.listenerCount("SubmitAggregatorRequest") === 0) {
+        dealStatus.once("SubmitAggregatorRequest", handleEvent);
+      }
+    })();
+  }
+
+  // Start listening to the first event and recursively handle the next events
+  if (dealStatus.listenerCount("SubmitAggregatorRequest") === 0) {
+    dealStatus.once("SubmitAggregatorRequest", handleEvent);
+  }
+}
+```
+
+To use this, call `yarn service` in the terminal and proceed through the frontend, as you normally would when uploading a file. Register the workers using the autocompleted CID that appears in the box.
+
+If you want to register the workers manually for a job that you didn't upload, paste in the known CID of your file into the box and register the jobs.

smart-contracts/programmatic-storage/README.md

@@ -0,0 +1,47 @@
+---
+description: >-
+    This section covers how to programmatically store data on the Filecoin network.
+---
+
+# Programmatic storage
+
+Programmatic storage is a concept where storage operations are controlled, automated, or orchestrated through code. Processes like reading, writing, and data management can all happen automatically and without human intervention.
+
+## Manual storage
+
+A client must go through several steps to manually store something with a storage provider on Filecoin without using the FVM. These steps involve interacting with the Filecoin network, negotiating a deal with a storage provider, and performing various other operations to ensure the data is stored correctly and securely. We discuss how storage works in the [Basics section](../../basics/how-storage-works/README.md) but let's quickly summarize the process here:
+
+1. **Identify a storage provider (SP)**: Clients must identify storage providers who will store their data. Clients usually pick their storage providers based on geographic location and bandwidth limitations, although other factors may impact their decision.
+1. **Negotiate a deal**: Clients negotiate deals with storage providers. These deals include terms like data size, price, deal duration, and collateral. The deal-making process happens off-chain, and once both parties agree to the terms of a deal, that deal is published on-chain for the rest of the network to see and validate. To successfully store data on Filecoin, clients need to have enough FIL to cover the deal's cost.
+1. **Data preparation**: Before storing files on Filecoin, clients must import them into their local Filecoin node, which produces a data CID (Content Identifier) - an ID uniquely describing the content. Then, the data is transferred to the storage provider.
+1. **Deal publication**: The deal is published on-chain, making the storage provider publicly accountable for the deal.
+1. **Sealing and proving storage**: The storage provider stores the data in sectors. Sectors are the basic units of provable storage where storage providers store clients’ data and generate Proof-of-Spacetime (PoSt) on behalf of the Filecoin network. Sectors have standard sizes and a lifetime that storage providers can extend before reaching the end of the lifetime.
+
+## Programmatic storage
+
+The [Filecoin virtual machine (FVM)](../filecoin-evm-runtime/README.md) is not designed to directly interact with data on Filecoin. Instead, it operates on blockchain state data, which includes metadata about stored data. The FVM's role is to compute over this state data and create logic around the storage and retrieval process. Storage and retrieval providers on the Filecoin network handle the actual process of storing and retrieving data.
+
+Developers can use FVM actors (also called smart-contracts) to implement various use cases related to data storage and retrieval on the Filecoin network. For instance, they can automate the process of renewing and repairing storage deals, replicating data across multiple storage providers for redundancy, or implement access control for data.
+
+This process for programmatically storing data on Filecoin differs depending on how much data you need to store. Anything less than 4 GiB of data is considered _small_, while anything 4 GiB or above is considered _large_. Lets cover the process for both _small_ and _large_ storage deals:
+
+### Small-scale storage
+
+Storage deals for less than 4 GiB of data are considered _small_ on the Filecoin network. This is because storage providers use sectors of either 32 GiB or 64 GiB. Because of these comparably large sector sizes, SPs are incentivized to accept storage deals closer to this cap size. An SP will receive more FIL rewards for storing a 31 GiB file rather than a 200 KiB file. Clients will have difficulty finding a storage provider to accept deals for small amounts of data.
+
+To get around this problem, these developers can _aggregate_ their data into bundles. The deal-aggregation process bundles a collection of small storage deals with many others to create one large deal.
+
+Checkout out [lighthouse.storage](https://lighthouse.storage/documentation) for an example of a deal aggregator.
+
+### Large-scale storage
+
+Storage deals for more than 4 GiB don't suffer from the limitations of small-scale storage deals. Large-scale deals can be processed and sent to an SP directly without aggregating data.
+
+The client contract performs _direct_ deal making to automatically finds an SP to store your data. There are other considerations an SP must make when deciding to pick up your data deal proposals from the client contract. For example, some SPs may only deal with clients with whom they have an existing relationship with. For more information regarding client contracts and how they work, checkout the [client contract tutorial](../developing-contracts/client-contract-tutorial.md).
+
+1. **Upload data to a data depot**: Data depots hold your data for the duration of this process so that you don't have to maintain your connection to the network. Once an SP verifiably holds your data, the data depot discards your data. After uploading your data to a depot, the depot will give you all the information you need to fill out the client contract.
+1. **Deploy a client contract**: The deal client contract coordinates the handshaking required on FVM to load your data into the Filecoin network.
+1. **Create a deal-proposal payload**: Once your data is ready in the data depot and your client contract has been deployed, you can create your deal-proposal payload. This payload contains everything a storage provider needs to download and store a copy of your data.
+1. **Deal publication and activation**: An SP picks up your contract and downloads the data from the data depot.
+
+Check out the [Deal Making Starter Kit](https://github.com/filecoin-project/fvm-starter-kit-deal-making) for additional information on how this process works.

smart-contracts/programmatic-storage/direct-deal-making.md

@@ -0,0 +1,48 @@
+---
+description: >-
+  This page explains the direct deal-making process in regards to the Filecoin network.
+---
+
+# Direct deal-making
+
+There are services like deal aggregators that allow clients to create storage deals programmatically. However, aggregators are better suited to small-scale deals of under 4 GiB of total data. For those wishing to programmatically store larger blocks of data, _direct deal-making_ is a more suitable process.
+
+## Steps
+
+We're going to cover the basic steps of _direct deal-making_.
+
+### Package the data
+
+The client packages the data they want to store into a CAR file. They then upload this to a hosting intermediary service and generate a CID. [Storage on-ramp services](../../basics/how-storage-works/storage-onramps.md) are available to act as this intermediary.
+
+### On-chain process
+
+Using the CID and URL provided by the hosting service, the client can deploy the client contract and propose a storage deal.
+
+Once the client contract is deployed, an event is emitted on-chain, and storage providers running [Boost](https://boost.filecoin.io/experimental-features/fvm-contract-deals) will receive the deal proposal.
+
+### Storage providers
+
+Each storage provider will decide whether to take the proposal and pick up the deal.
+
+There are a variety of factors influencing whether or not a storage provider will accept the deal, including if there is a pre-existing relationship between the client and the storage provider, whether the deal is financially attractive to the storage provider if they can accommodate the size of the data, or whether clients have datacap.
+
+If the storage provider accepts the deal through Boost, that provider will download the CAR file from the web server and publish the storage deal on-chain. A client can check that the deal was successfully published on the blockchain via [block explorers](../../networks/mainnet/explorers.md).
+
+A storage deal published on-chain denotes the _intent_ for a storage provider to make a storage deal. But this _intent_ does not mean that the deal is _complete_. The deal is only complete once a storage provider has sealed the data into sectors and the deal ID is published on-chain.
+
+#### Timelines
+
+The timelines for the published storage deals and the deal ID confirmation are inherently undefined. Storage providers do not necessarily publish storage deals once they come in. Depending on the size of the data, the storage provider may wait until they have enough data from other deals to fill a sector.
+
+Deals made on the Calibration testnet will be picked-up within 24 hours.
+
+## Review
+
+Here is a diagram showing each of these steps in turn:
+
+![](../../.gitbook/assets/smart-contracts-programmatic-storage-direct-deal-making-diagram.png)
+
+## Next steps
+
+For a complete tutorial on how to use the client contract to initiate a deal from FVM, see our [client contract tutorial](../developing-contracts/client-contract-tutorial.md). A full explanation of the client contract in the context of a direct deal-making starter kit can be found [here](https://github.com/filecoin-project/fvm-starter-kit-deal-making/tree/main).

smart-contracts/programmatic-storage/raas-interfaces.md

@@ -0,0 +1,62 @@
+---
+description: >-
+  This page explains the specifications for RaaS base and full interfaces.
+---
+
+# RaaS interfaces
+
+RaaS has a base and full interface to enable replication, renew and repair of storage deals.
+
+## Self-hosted RaaS
+
+RaaS refers to replication, renewal and repair as a service, for data stored in storage deals on Filecoin. Developers can leverage the self-hosted RaaS to provide RaaS features, within their storage solution, using the RaaS Starter Kit.
+
+Self-hosted RaaS requires 4 components:
+
+- The Client who has data to upload
+- An Aggregator platform (a type of storage solution) that receives the Client’s data and makes a storage deal on Filecoin
+- The RaaS node, hosted by the developer, that checks if the storage deal requires replication, renewal and/or repair
+- The RaaS DealStatus smart contract that the RaaS node executes checks with
+
+### In the example of replication:
+
+1. The client generates a `CID` for the data and requests the RaaS node to store data.
+1. The RaaS node takes the client’s data and makes the storage deal onto Filecoin.
+1. The client registers a replication job to the RaaS node and defines the number of replicas of their data to store, by calling the `/register_jobs` API on the RaaS node (e.g. “This data needs to have a minimum of 10 copies on the network”).
+1. The RaaS node periodically checks the client data’s `CID` for its deal status on Filecoin.
+
+    - The RaaS node requests deal status with the DealStatus smart contract, via `getActiveDeals(CID)` and checks if the client’s data is stored with the accurate number of replicas.
+    - The DealStatus smart contract returns the information of active deals to the RaaS node.
+
+1. If the number of replicas does not match the client’s requirements, the RaaS node is notified.
+1. The RaaS node fetches the data via its `CID` and resubmit a request to create new storage deals (repeat step 2).
+1. When the client requests for retrieval of data, it queries the RaaS node, which will fetch the data from the storage provider on Filecoin or provide an IPFS pinned copy (depends on how RaaS node is setup to store the data).
+
+### In the example of renewal:
+
+1. The client generates a `CID` for the data and requests the RaaS node to store data.
+1. The RaaS node takes the client’s data and makes the storage deal onto Filecoin.
+1. The client registers a renewal job to the RaaS node and defines the renewal threshold for the data’s storage deal, by calling the `/register_jobs` API on the RaaS node (e.g. renew storage deals that are 1 month away from expiry).
+1. The RaaS node periodically checks the client data’s `CID` for its deal status on Filecoin.
+
+    - The RaaS node requests deal status with the DealStatus smart contract, via `getExpiringDeals(CID)` and checks if any of its active deals is expiring.
+    - The DealStatus smart contract returns the information of expiring deals to the RaaS node.
+
+1. If deals with the client’s data are expiring, the RaaS node is notified.
+1. The RaaS node fetches the data via its `CID` and resubmit a request to create new storage deals (repeat step 2).
+1. When the client requests for retrieval of data, it queries the RaaS node, which will fetch the data from the storage provider on Filecoin or provide an IPFS pinned copy (depends on how RaaS node is setup to store the data).
+
+### In the example of repair
+
+1. The client generates a `CID` for the data and requests the RaaS node to store data.
+1. The RaaS node takes the client’s data and makes the storage deal onto Filecoin.
+1. The client registers a repair job to the RaaS node and defines the repair threshold for the data’s storage deal, by calling the `/register_jobs` API on the RaaS node (e.g. “this deal needs repairing if it is not proven active for X epochs”).
+1. The RaaS node periodically checks with [Lotus](https://lotus.filecoin.io/lotus/get-started/what-is-lotus/), if the deal ID and corresponding miner, is actively being proven by the miner on Filecoin. The node calls `StateMarketStorageDeal`, with provided deal and miner IDs.
+1. If the deal ID and corresponding deal sector are not being actively proven for X epochs, the deal will require repairing.
+1. The RaaS node fetches the data via its `CID` and resubmit a request to create new storage deals (repeat step 2).
+
+Check out the [RaaS starter kit](https://github.com/filecoin-project/raas-starter-kit) to build your own RaaS interface.
+
+## Aggregator-hosted RaaS
+
+The aggregator-hosted RaaS refers to a solution that requires an aggregator to host all the components of RaaS and provides a seamless interface to the client. [Lighthouse.storage](https://www.lighthouse.storage/) is the first FVM project to provide an aggregator-hosted RaaS, via its SDK.