Run fails on proof param download

[ZenGround0]

ERROR src/main.rs:72: Command failed, poison file left in place for safety: Failed to download proof parameters: 
Error: "Failed to download proof parameters: "

[ZenGround0]

I was running on main which is many commits behind. @redpanda-f please merge the best current branch to main to avoid this extra piece of friction

[ZenGround0]

Reopening because this fails on feat/redpanda/v1-rc

[ZenGround0]

 INFO src/commands/start/genesis/proof_parameters.rs:40: ⬇ Downloading proof parameters (this may take a while)...
 WARN src/utils/retry.rs:92: Proof parameters download failed (attempt 1/6), retrying in 4 seconds...
 WARN src/utils/retry.rs:92: Proof parameters download failed (attempt 2/6), retrying in 4 seconds...
^[[B^[[B⠁ Downloaded 0.0 MB (0.00 MB/s)                                                                                                                                           WARN src/utils/retry.rs:92: Proof parameters download failed (attempt 3/6), retrying in 4 seconds...
 WARN src/utils/retry.rs:92: Proof parameters download failed (attempt 4/6), retrying in 4 seconds...
 WARN src/utils/retry.rs:92: Proof parameters download failed (attempt 5/6), retrying in 4 seconds...
^[[A⠁ Downloaded 0.0 MB (0.00 MB/s)                                                                                                                                              Error: "Failed to download proof parameters: "

[redpanda-f]

This is flaky by default and unfortunately, not sure what much can be done on this end, unless lotus downloader is also fixed. A few things I have figured out is:

• If you are being a ISP NAT and a lot of people are using GH, this seems to be an issue
• This is intermittent, there is a good chance that this passes a couple hours later.
• To combat this issue, we currently host a self-packaged tarball on S3. This works for CI but is not "authoritative", i.e. would still prefer to go with lotus' traditional route of downloading (which this one is, and is equally flaky as original)
• Please refer to how CI does it:

  foc-localnet/.github/workflows/ci.yml

  Lines 232 to 240 in 1b898fe

  	- name: "EXEC: {Download proof parameters from S3}, independent"
  	run: |
  	mkdir -p ~/.foc-localnet/docker/volumes/cache/filecoin-proof-parameters/
  	curl -L https://fil-proof-params-2k-cache.s3.us-east-2.amazonaws.com/filecoin-proof-params-2k.tar -o /tmp/filecoin-proof-params-2k.tar
  	tar -xf /tmp/filecoin-proof-params-2k.tar -C ~/.foc-localnet/docker/volumes/cache/filecoin-proof-parameters/
  	rm /tmp/filecoin-proof-params-2k.tar
  	ls -lath ~/.foc-localnet/docker/volumes/cache/filecoin-proof-parameters/
  	PROOF_PARAMS_HASH=$(find ~/.foc-localnet/docker/volumes/cache/filecoin-proof-parameters -type f -exec sha256sum {} \; | cut -d' ' -f1 | sort | sha256sum | cut -d' ' -f1)
  	echo "Downloaded proof parameters with hash: $PROOF_PARAMS_HASH"

  (P.S. This is again, non-authoritative way of doing things)

[ZenGround0]

I am not convinced we are doing everything we can given that I don't have these problems downloading lotus or curio. I can look into what is happening differently there.

This works for CI but is not "authoritative", i.e. would still prefer to go with lotus' traditional route of downloading

Because of content addressing we should feel empowered to add the S3 bucket as a second path. True authority is only at the level of data.

[ZenGround0]

I am observing lotus param-fetch 2048 immediately working on my machine whereas the cargo run --start immediately fails on proof param fetching. I am pretty sure this is not a lotus param-fetch issue.

Both were tried within a minute of each other so I don't think this is a bad-luck-in-service-stability kind of issue. There is probably something simple going wrong in the docker startup process.