Add HTTP HEAD results. CheckerNetwork/spark-checker#104

[juliangruber]

Closes CheckerNetwork/spark-checker#104
Blocked by CheckerNetwork/spark-stats#322
Blocked by #31 (without tooltips, overlapping lines are impossible to read)

[cloudflare-workers-and-pages]

Deploying spark-dashboard with    Cloudflare Pages

Latest commit:	0ddc4b3
Status:	✅  Deploy successful!
Preview URL:	https://855411ae.spark-dashboard.pages.dev
Branch Preview URL:	https://add-http-head.spark-dashboard.pages.dev

View logs

[juliangruber]

The metric currently is at 0, which is why it doesn't show up yet

[NikolasHaimerl]

The metric currently is at 0, which is why it doesn't show up yet

Should we wait for a final review until enough data is available so we can see the visualization?

[bajtos]

FWIW, the Cloudflare Pages build is failing:

16:34:38.222 | ✘ [ERROR] Error: Pages only supports files up to 25 MiB in size
-- | --
16:34:38.222 |  
16:34:38.222 | _file/data/spark-miner-rsr-summaries.ef3ecf71.json is 27.1 MiB in size

[bajtos]

I think we need to find a more compact format. The current array of objects with verbose properties introduces too much overhead.

Here is what I was thinking about (conceptually):

//-- a shared helper --//

const FIELDS = [
  'ttfb_ms',
  'success_rate',
  'total',
  // etc.
]

// convert from key:value object to an array of values
function packSparkMinerRates(record) {
  return FIELDS.map(f => record[f])
}

// convert from array of values to key:value object
function unpackSparkMinerRates(data) {
  return Object.fromEntries(FIELDS.map(f => [f, data[f]]))
}
    

//-- in data-fetching JS file --//
process.stdout.write(JSON.stringify(summary.map(r => packSparkMinerRates(r)))

//-- in provider/[provider.md] --//
const rsrData = FileAttachment(
  `../data/${observable.params.provider}-spark-rsr-summary.json`,
).json().map(r => unpackSparkMinerRates(r))

Alternatively, can we use a more efficient storage format like CSV?

https://observablehq.com/documentation/data/files/file-attachments#csv-files

(We need to figure out how to convert JSON fetched from spark-stats API into a CSV file.)

@juliangruber @NikolasHaimerl @pyropy Thoughts?

[pyropy]

I think we need to find a more compact format. The current array of objects with verbose properties introduces too much overhead.

Alternatively, can we use a more efficient storage format like CSV?

https://observablehq.com/documentation/data/files/file-attachments#csv-files

(We need to figure out how to convert JSON fetched from spark-stats API into a CSV file.)

@bajtos I think doing we should do both as that would ship lowest amount of data to the client.

I found this library json2csv which seems to have good amount of weekly downloads that offers, as the name suggest, converting JSON to CSV but also it offers filtering (you can specify json fields that get converted to columns). Here's the library docs.