Skip to content

Latest commit

 

History

History
83 lines (68 loc) · 10.8 KB

confidence.md

File metadata and controls

83 lines (68 loc) · 10.8 KB

What you can validate against Feint

One question, answered in your vocabulary rather than this project's: what can I reasonably test against this emulator?

Every row carries an anchor. A yes names the suite or the assertion that proves it, a conditional row names the capability it keys on, and a no links the section of limits.md that says why. A row with nothing to point at does not belong here, and a test refuses one (TestEveryConfidenceRowCarriesItsProof).

This is the coarse view, for a human deciding whether to try. The fine view is per operation and generated: routes.md prints what every mounted operation has earned — the generated block below carries the count — and coverage/evidence.json is the record it prints.

409 operations mounted, 384 driven by a real client in the recorded run. The 25 that are not each state why at their route, and routes.md prints the reason under the pack that owns it.

The table

What you want to validate Verdict Proof, or the limit
Terraform and OpenTofu syntax, plan and lifecycle yes mise run conformance: apply, an empty second plan, destroy — conformance.md
A CLI workflow end to end (scw, octl, exo) yes the same run drives all three against one emulator — conformance.md
Request and response shapes, field by field yes, with a stated ceiling every answer is checked against the provider's own API description, and what a schema cannot catch is named in conformance.md
That your client reads a field the cloud really returns yes, where a recording exists recorded shapes arbitrate; the blind spot is counted in the run's own output — conformance.md
Resource dependency order: what refuses to be deleted under what yes, while the relation is synchronous the refusal assertions in each suite; a relation the real cloud holds for a while AFTER a delete is not reproduced, and that is stated with the measurement in limits.md
Error handling: that your code decodes a refusal yes each suite demands refusals and the emulator refuses to close the bracket when none happened — conformance.md
Cloud-init, an SSH login, a security group that filters with a runtime FEINT_VM=incus mise run conformance and mise run conformance:ssh; keys on capabilities.firewall, whose bounds are stated in limits.md
Two VPCs that cannot reach each other OVN only keys on capabilities.isolation, never on a mode name — limits.md
A security group with a stateless rule OVN only keys on capabilities.stateless_rules: Incus applies allow-stateless only on OVN networks, so a bridge mode refuses such a group by name and logs a WARN; the refused version is not written, and a stateful version written earlier under the same name stays attached as it was — limits.md
A load balancer that distributes real connections OVN, and per pack keys on the conjunction of capabilities.balancing (the runtime can) and enforced.balancing (this pack hands its balancers over) — each half has been true while the other was false — limits.md
That what the API claims is really on the host: a running machine exists, a claimed firewall left rule sets, a claimed balancer distributes OVN, per provider mise run conformance:witness reads the claims off each pack's API and the witnesses off the runtime with incus, and a pack that claims no dataplane is skipped by name — the gate #475, #481, #483 and #484 share (#486); what a green control-plane run does not prove is stated in conformance.md
That a stack's machines actually do something: a service that listens, answers over the published address and survives a restart; a port a rule opens answering while a port no rule opens refuses, in one pass OVN, per stack mise run conformance:functional reads what each stack declares in its own proof.json and probes it from another machine or from the station, never through the host: a harness reaching the machine with incus exec would have found none of the four defects it was built for (#503). The bounds it skips by name are the routed NIC's and the same-subnet one, both in limits.md
Reaching a private address from the host that runs the emulator depends on the mode keys on capabilities.private_from_host — limits.md
A kernel-level test: sysctl, modules, the boot path with a VM runtime FEINT_VM=incus-vm, keying on capabilities.own_kernel: a container shares the host kernel and cannot carry any of it — limits.md
That a bad image or instance-type id is rejected with serve --strict-catalog, against the catalogue you declared off by default, and the default is the decision; each pack refuses in its own shape, and which of those shapes were recorded is in the table — limits.md
Provider quotas, real capacity, prices no limits.md
Authentication, IAM permissions, a signature that must fail no limits.md
Waiting behaviour: a resource that stays provisioning for a while partly, with serve --consistency eventual a Scaleway server walks the states fr-par walks, one per read (#637); an Outscale volume is creating then available and a snapshot in-queue then completed, and a snapshot taken during creating meets the measured 409 InvalidVolumeState (#124); immediate is still the default — limits.md
Retry and backoff against 429s, 5xx or a control-plane outage yes, on the operations you arm PUT /_feint/faults names an operation and a status, off by default, and deterministic under a client that does not parallelise — the schedule counts arrivals, limits.md; what an injected refusal does NOT count as is stated in conformance.md
S3 bytes through declared explicit doors opt-in, AWS CLI 2.32.17, scw 2.62.0, exo 1.95.1 upload/list/download/delete mise run conformance:s3:mounted drives private Scaleway buckets, Exo multipart/range downloads (6,340,608 bytes), AWS restart/resume/completion, and Exo dangling-upload abort/bucket removal; actual bytes, dates and inventories are checked. Normal Exoscale bucket creation is not certified — object-storage-routing.md
Hardcoded Scaleway Terraform Object Storage endpoints opt-in, Terraform with the published provider 2.82.0 through feint run TestScalewayTerraformStorageThroughFeintRunOnTheHostNetwork, on the host network with no sudo and no namespace: apply complete, 139,264 identical bytes read back from the local byte store by sha256, an empty second plan, a complete teardown. Not run on a pull request — it needs a provider mirror, so it is a measurement you reproduce rather than a gate, and OpenTofu is not covered — object-storage-routing.md
A Terraform run against Exoscale yes, from provider v0.71.0 the published provider drives this pack since #644; an older one, which splits between this emulator and a paying account, is refused by user agent naming the floor — limits.md
Control-plane multi-zone or multi-region behaviour on Exoscale no one zone per process, selectable with FEINT_EXOSCALE_ZONE, and the reason is the client — limits.md
Traffic actually flowing through a gateway or a NAT no records move, packets do not — limits.md
That a run stays on your machine yes, for what Feint serves the APIs it serves answer locally; a client that composes its own endpoint for a product outside that scope leaves, and the four measured paths are in limits.md — feint run closes the first of them since 0.15.0

How to read a verdict

yes means a real client drove it in the run that produced coverage/evidence.json, and that the suite asserted something about the answer. It does not mean every field of every response is right: two defects sat behind a driven axis for months, which is why the record publishes seven independent axes and never adds them into a score.

with a runtime means the control plane answers without one, and the claim only holds when machines are real. Ask the emulator rather than guessing:

curl -s localhost:4599/_feint/health | jq .capabilities

A capability nobody declares reads as absent, deliberately, so a check skips itself instead of asserting what no driver promised.

no is a decision with a reason, not a gap waiting to be filled by accident. Each one links the section that states the cost. Several would change with work somebody could do — fault injection, an opt-in identifier check, observable transient states — and those are issues, not omissions.

What this page is not

It is not a coverage percentage, and it deliberately carries no score. A single number lets a weak claim ride on a strong one; that is the arithmetic version of the overstatement this whole project exists to avoid. If you want the fine view, it is per operation and it is generated.