CI/Nightly End-to-End Validation of FOC as a whole

[BigLep]

Done Criteria

• CI pipeline (or scheduled workflow) exists that:
  • Pulls or builds the latest "release" tags and "main" branches from each relevant repo (more details below)
  • Spins up the DevNet via foc-localnet.
  • Runs at least:
    • The basic deal + retrieval scenario that happens in https://github.com/FilOzone/foc-devnet/blob/main/.github/workflows/ci.yml
    • Optionally one or more additional scenarios
  • Tears everything down cleanly.
• Runs on a schedule (e.g. nightly) and is visible to the team.
• On failure:
  • CI run is clearly marked as failed.
  • Logs are preserved for debugging.
  • It's clear what branches/tags/commits were used so that a developer can reproduce the failure.
  • GitHub issue is created in the repo for visibiltiy for the team. (We can do something similar to what Lotus does for its periodic "very expensive tests")

Why Important

• Detects cross-repo breakages before they reach calibnet/mainnet hopefully.
• Provides an early warning system for regressions in core flows.
• Supports GA and post-GA stability by continuously validating the “golden paths”.

##User/Customer

• Primary: Entire FOC engineering team.

Notes

• Start with nightly-only; maybe we'll consider per-PR jobs later for critical components.
• Keep logs and artifacts lightweight but sufficient for root-cause analysis.
• This issue was clarified based on discussion in Nightly CI runs, needs clarity #17

What to run against

We want two sets of runs:

1. Latest set of tagged releases
2. Latest changes in "main" branches.

Specifically:

repo	"master" branch	latest release tag
curio	pdpv0	pdp/vX
lotus	master	vX
filecoin-services	main	vX
synapse-sdk	master	synapse-sdk-vX

Where to run this

We'll do this in a new repo to now avoid more clutter to existing "central" repos like synapse-sdk. We'll do this in a new repo "foc-integ-tests" that houses both various integration test scenarios (where an example scenario will be created following #7 ) and the ci yaml files to periodically run these scenarios.

[redpanda-f]

v1-rc that is presented in #16 presents a synapse E2E test as part of cluster start flow and validates as follows:

 INFO src/commands/start/synapse_test_e2e/synapse_test_step.rs:119: Created random test file at /home/rvagg/.foc-localnet/run/26jan06-1622_ZanyBug/random_test_file.txt
 INFO src/commands/start/synapse_test_e2e/synapse_test_step.rs:200: Executing test script in container...
 INFO src/commands/start/synapse_test_e2e/synapse_test_step.rs:212: ✓ Synapse E2E Test completed successfully
 INFO src/commands/start/step/mod.rs:286: ✓ Execution completed
 INFO src/commands/start/step/mod.rs:288: Running post-execution verification...
 INFO src/commands/start/step/mod.rs:290: ✓ Post-execution verification passed
 INFO src/commands/start/step/mod.rs:293: Step 'Synapse E2E Test' completed in 97.807127384s
 INFO src/commands/start/step/mod.rs:501: ✓ Epoch 8 completed successfully
 INFO src/commands/start/mod.rs:434: Cluster started successfully!

This is deemed enough for now for local testing. Further, more extensive tests can be added but will not form as the core "cluster starting" workflow of foc-localnet.

I will open newer issues that breaks this problem down to actionable pieces, but would close this one in particular.