Skip to content

Add alternative provider retrieval check #132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
aedec80
Add network wide retrieval check
pyropy Apr 9, 2025
233cc1f
Use status code instead of boolean retrieval flag
pyropy Apr 9, 2025
83e7f31
Simplify name for network wide measurements
pyropy Apr 9, 2025
afe30dd
Refactor code for picking random provider
pyropy Apr 9, 2025
23ee203
Add network retrieval protocol field
pyropy Apr 9, 2025
4bc1076
Add basic test for testing network retrieval
pyropy Apr 9, 2025
63424ff
Refactor function for picking random providers
pyropy Apr 9, 2025
8a94f4e
Only return providers in case of no valid advert
pyropy Apr 9, 2025
c4350b6
Convert network stats to object inside stats obj
pyropy Apr 10, 2025
edfdef1
Format testNetworkRetrieval func
pyropy Apr 10, 2025
dbf0fd7
Refactor queryTheIndex function
pyropy Apr 10, 2025
d33f276
Handle case when no random provider is picked
pyropy Apr 10, 2025
97bee91
Test function for picking random providers
pyropy Apr 10, 2025
4b6d0bc
Rename network retrieval to alternative provider check
pyropy Apr 11, 2025
97fcc28
Update logging to reflect metric name change
pyropy Apr 11, 2025
5121a49
Update logging to reflect metric name change
pyropy Apr 11, 2025
4065784
Rename providers field to alternativeProviders
pyropy Apr 11, 2025
74f06e9
Rename testNetworkRetrieval to checkRetrievalFromAlternativeProvider
pyropy Apr 11, 2025
ea8cce4
Return retrieval stats from checkRetrievalFromAlternativeProvider
pyropy Apr 11, 2025
f9afe34
Update lib/spark.js
pyropy Apr 11, 2025
9959b50
Update lib/spark.js
pyropy Apr 11, 2025
a2da050
Rename functions to match new metric name
pyropy Apr 11, 2025
9759d80
Merge branch 'add/network-wide-retrieval-check' of github.com:filecoi…
pyropy Apr 11, 2025
820e8a3
Pick alternative provider using supplied randomness
pyropy Apr 15, 2025
5b13287
Replace custom rng implementation with Prando
pyropy Apr 15, 2025
3c14f84
Fix typos
pyropy Apr 15, 2025
fe0f1f5
Merge remote-tracking branch 'origin/main' into add/network-wide-retr…
pyropy Apr 28, 2025
ad8a8e8
Lint fix
pyropy Apr 28, 2025
31019d0
Add ID to Provider
pyropy Apr 28, 2025
3710910
Filter out bitswap providers before picking random provider
pyropy Apr 28, 2025
c61a196
Update lib/ipni-client.js
pyropy Apr 29, 2025
d1f62fa
Update lib/spark.js
pyropy Apr 29, 2025
05bd1c2
Update lib/spark.js
pyropy Apr 29, 2025
3451eff
Rename random to alternative provider
pyropy Apr 29, 2025
8b8db36
Merge branch 'add/network-wide-retrieval-check' of github.com:Checker…
pyropy Apr 29, 2025
59d3d22
Simplify pseudo-rng
pyropy Apr 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 32 additions & 9 deletions lib/ipni-client.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,22 @@ import {
assertOkResponse,
} from '../vendor/deno-deps.js'

/**
* @typedef {{
* peerId: string
* address: string
* protocol: string
* contextId: string
* }} Provider
*/

/**
* @param {string} cid
* @param {string} providerId
* @returns {Promise<{
* indexerResult: string
* provider?: { address: string; protocol: string }
* provider?: Provider
* alternativeProviders?: Provider[]
* }>}
*/
export async function queryTheIndex(cid, providerId) {
Expand Down Expand Up @@ -38,9 +48,8 @@ export async function queryTheIndex(cid, providerId) {
}

let graphsyncProvider
const alternativeProviders = []
for (const p of providerResults) {
if (p.Provider.ID !== providerId) continue

const [protocolCode] = decodeVarint(decodeBase64(p.Metadata))
const protocol = {
0x900: 'bitswap',
Expand All @@ -52,22 +61,32 @@ export async function queryTheIndex(cid, providerId) {
const address = p.Provider.Addrs[0]
if (!address) continue

const provider = {
peerId: p.Provider.ID,
address: formatProviderAddress(p.Provider.ID, address, protocol),
contextId: p.ContextID,
protocol,
}

if (p.Provider.ID !== providerId) {
alternativeProviders.push(provider)
continue
}

switch (protocol) {
case 'http':
return {
indexerResult: 'OK',
provider: { address, protocol },
provider,
}

case 'graphsync':
if (!graphsyncProvider) {
graphsyncProvider = {
address: `${address}/p2p/${p.Provider.ID}`,
protocol,
}
graphsyncProvider = provider
}
}
}

if (graphsyncProvider) {
console.log('HTTP protocol is not advertised, falling back to Graphsync.')
return {
Expand All @@ -79,7 +98,7 @@ export async function queryTheIndex(cid, providerId) {
console.log(
'All advertisements are from other miners or for unsupported protocols.',
)
return { indexerResult: 'NO_VALID_ADVERTISEMENT' }
return { indexerResult: 'NO_VALID_ADVERTISEMENT', alternativeProviders }
}

async function getRetrievalProviders(cid) {
Expand All @@ -90,3 +109,7 @@ async function getRetrievalProviders(cid) {
const result = await res.json()
return result.MultihashResults.flatMap((r) => r.ProviderResults)
}

function formatProviderAddress(peerId, address, protocol) {
return protocol === 'http' ? address : `${address}/p2p/${peerId}`
}
172 changes: 155 additions & 17 deletions lib/spark.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
/* global Zinnia */

/** @import {Provider} from './ipni-client.js' */
import { ActivityState } from './activity-state.js'
import {
SPARK_VERSION,
Expand Down Expand Up @@ -44,18 +45,20 @@ export default class Spark {

async getRetrieval() {
const retrieval = await this.#tasker.next()
if (retrieval) {
console.log({ retrieval })
const { retrievalTask } = retrieval
if (retrievalTask) {
console.log({ retrievalTask })
}

return retrieval
}

async executeRetrievalCheck(retrieval, stats) {
async executeRetrievalCheck({ retrievalTask, stats, randomness }) {
console.log(
`Calling Filecoin JSON-RPC to get PeerId of miner ${retrieval.minerId}`,
`Calling Filecoin JSON-RPC to get PeerId of miner ${retrievalTask.minerId}`,
)
try {
const peerId = await this.#getIndexProviderPeerId(retrieval.minerId)
const peerId = await this.#getIndexProviderPeerId(retrievalTask.minerId)
console.log(`Found peer id: ${peerId}`)
stats.providerId = peerId
} catch (err) {
Expand All @@ -78,29 +81,45 @@ export default class Spark {
}

console.log(
`Querying IPNI to find retrieval providers for ${retrieval.cid}`,
)
const { indexerResult, provider } = await queryTheIndex(
retrieval.cid,
stats.providerId,
`Querying IPNI to find retrieval providers for ${retrievalTask.cid}`,
)
const { indexerResult, provider, alternativeProviders } =
await queryTheIndex(retrievalTask.cid, stats.providerId)
stats.indexerResult = indexerResult

const providerFound =
indexerResult === 'OK' || indexerResult === 'HTTP_NOT_ADVERTISED'
if (!providerFound) return
const noValidAdvertisement = indexerResult === 'NO_VALID_ADVERTISEMENT'

// In case index lookup failed we will not perform any retrieval
if (!providerFound && !noValidAdvertisement) return

// In case we fail to find a valid advertisement for the provider
// we will try to perform network wide retrieval from other providers
if (noValidAdvertisement) {
console.log(
'No valid advertisement found. Trying to retrieve from an alternative provider...',
)
stats.alternativeProviderCheck =
await this.checkRetrievalFromAlternativeProvider({
alternativeProviders,
randomness,
cid: retrievalTask.cid,
})
return
}

stats.protocol = provider.protocol
stats.providerAddress = provider.address

await this.fetchCAR(
provider.protocol,
provider.address,
retrieval.cid,
retrievalTask.cid,
stats,
)
if (stats.protocol === 'http') {
await this.testHeadRequest(provider.address, retrieval.cid, stats)
await this.testHeadRequest(provider.address, retrievalTask.cid, stats)
}
}

Expand Down Expand Up @@ -220,6 +239,51 @@ export default class Spark {
}
}

async checkRetrievalFromAlternativeProvider({
alternativeProviders,
randomness,
cid,
}) {
if (!alternativeProviders.length) {
console.info('No alternative providers found for this CID.')
return
}

const validAlternativeProviders = alternativeProviders.filter(
(p) => p.protocol !== 'bitswap',
)

if (!validAlternativeProviders.length) {
console.warn(
'No providers serving the content via HTTP or Graphsync found. Skipping network-wide retrieval check.',
)
return
}

const alternativeProvider = pickRandomProvider(
validAlternativeProviders,
randomness,
)
if (!alternativeProvider) {
console.warn(
'Failed to pick a alternative provider. Skipping network-wide retrieval check.',
)
return
}

const alternativeProviderRetrievalStats = newAlternativeProviderCheckStats()
alternativeProviderRetrievalStats.providerId = alternativeProvider.peerId

await this.fetchCAR(
alternativeProvider.protocol,
alternativeProvider.address,
cid,
alternativeProviderRetrievalStats,
)

return alternativeProviderRetrievalStats
}

async submitMeasurement(task, stats) {
console.log('Submitting measurement...')
const payload = {
Expand All @@ -246,8 +310,8 @@ export default class Spark {
}

async nextRetrieval() {
const retrieval = await this.getRetrieval()
if (!retrieval) {
const { retrievalTask, randomness } = await this.getRetrieval()
if (!retrievalTask) {
console.log(
'Completed all tasks for the current round. Waiting for the next round to start.',
)
Expand All @@ -256,9 +320,11 @@ export default class Spark {

const stats = newStats()

await this.executeRetrievalCheck(retrieval, stats)
await this.executeRetrievalCheck({ retrievalTask, randomness, stats })

const measurementId = await this.submitMeasurement(retrieval, { ...stats })
const measurementId = await this.submitMeasurement(retrievalTask, {
...stats,
})
Zinnia.jobCompleted()
return measurementId
}
Expand Down Expand Up @@ -335,6 +401,17 @@ export function newStats() {
carChecksum: null,
statusCode: null,
headStatusCode: null,
alternativeProviderCheck: null,
}
}

function newAlternativeProviderCheckStats() {
return {
statusCode: null,
timeout: false,
endAt: null,
carTooLarge: false,
providerId: null,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this also have byteLength, carChecksum and headStatusCode?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or are we consciously omitting them? If so, could you please add a code comment?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if we're supposed to have them; I think it wouldn't be a big deal to add those fields.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't have them it means we could have a successful retrieval (using the alternative provider method) but not know the byte length, car checksum and head status code. @bajtos wdyt?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It depends on what do we want to use the alternative retrieval check measurement for.

As I understand it, we want to calculate network-wide RSR for retrievals that include alternative providers so that we can show this RSR on the leaderboard. I don't see how we need byteLength, carChecksum or headStatusCode for that.

I'd say YAGNI, exclude these fields for now, and wait until we need them.

}
}

Expand Down Expand Up @@ -418,3 +495,64 @@ function mapErrorToStatusCode(err) {
// Fallback code for unknown errors
return 600
}

/**
* Picks a random provider based on their priority and supplied randomness.
*
* Providers are prioritized in the following order:
*
* 1. HTTP Providers with Piece Info advertised in their ContextID.
* 2. Graphsync Providers with Piece Info advertised in their ContextID.
* 3. HTTP Providers.
* 4. Graphsync Providers.
*
* @param {Provider[]} providers
* @param {string} randomness
* @returns {Provider | undefined}
*/
export function pickRandomProvider(providers, randomness) {
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pickRandomProvider now picks random provider based on the priority rather then weight and generated pseudo-random number.

const filterByProtocol = (items, protocol) =>
items.filter((provider) => provider.protocol === protocol)

const pickRandomItem = (items) => {
if (!items.length) return undefined
const randomValue = BigInt('0x' + randomness)
const ix = Number(randomValue % BigInt(items.length))
return items[ix]
}

const providersWithPieceInfoContextID = providers.filter((p) =>
p.contextId.startsWith('ghsA'),
)

// Priority 1: HTTP providers with ContextID containing PieceCID
const httpProvidersWithPieceInfoContextID = filterByProtocol(
providersWithPieceInfoContextID,
'http',
)
if (httpProvidersWithPieceInfoContextID.length) {
return pickRandomItem(httpProvidersWithPieceInfoContextID, randomness)
}

// Priority 2: Graphsync providers with ContextID containing PieceCID
const graphsyncProvidersWithPieceInfoContextID = filterByProtocol(
providersWithPieceInfoContextID,
'graphsync',
)
if (graphsyncProvidersWithPieceInfoContextID.length) {
return pickRandomItem(graphsyncProvidersWithPieceInfoContextID, randomness)
}

// Priority 3: HTTP providers
const httpProviders = filterByProtocol(providers, 'http')
if (httpProviders.length) return pickRandomItem(httpProviders, randomness)

// Priority 4: Graphsync providers
const graphsyncProviders = filterByProtocol(providers, 'graphsync')
if (graphsyncProviders.length) {
return pickRandomItem(graphsyncProviders, randomness)
}

// No valid providers found
return undefined
}
12 changes: 7 additions & 5 deletions lib/tasker.js
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ export class Tasker {
#remainingRoundTasks
#fetch
#activity
#randomness

/**
* @param {object} args
Expand All @@ -37,10 +38,11 @@ export class Tasker {
this.#remainingRoundTasks = []
}

/** @returns {Task | undefined} */
/** @returns {Promise<{ retrievalTask?: RetrievalTask; randomness: string }>} */
async next() {
await this.#updateCurrentRound()
return this.#remainingRoundTasks.pop()
const retrievalTask = this.#remainingRoundTasks.pop()
return { retrievalTask, randomness: this.#randomness }
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We somehow need to export the round randomness so I have opted for returning object with randomness attribute from the next function.

Maybe adding the randomness property to the retrieval task wouldn't be a bad thing either.

}

async #updateCurrentRound() {
Expand Down Expand Up @@ -76,13 +78,13 @@ export class Tasker {
console.log(' %s retrieval tasks', retrievalTasks.length)
this.maxTasksPerRound = maxTasksPerNode

const randomness = await getRandomnessForSparkRound(round.startEpoch)
console.log(' randomness: %s', randomness)
this.#randomness = await getRandomnessForSparkRound(round.startEpoch)
console.log(' randomness: %s', this.#randomness)

this.#remainingRoundTasks = await pickTasksForNode({
tasks: retrievalTasks,
maxTasksPerRound: this.maxTasksPerRound,
randomness,
randomness: this.#randomness,
stationId: Zinnia.stationId,
})

Expand Down
Loading
Loading