Skip to content

test-architecture.md

Latest commit

 

History

History
351 lines (257 loc) · 17 KB

test-architecture.md

File metadata and controls

351 lines (257 loc) · 17 KB
Raw

Four-Layer Test Architecture for VIP

This document describes how VIP structures its tests into four layers. The key rule is that each layer only communicates with the one directly below it -- a .feature file doesn't call httpx, and the httpx client has no idea what Gherkin is.

The Four Layers

Layer 1: Test           →  what we're testing (the Gherkin scenario)
Layer 2: DSL            →  how we express it (step definitions + fixtures)
Layer 3: Driver Port    →  what we need from the system (protocols/interfaces)
Layer 4: Driver Adapter →  how we interact with the system (API calls, browser clicks)

graph TD
    subgraph "Layer 1: Test"
        F[".feature files<br/>(Gherkin scenarios)"]
    end

    subgraph "Layer 2: DSL"
        S["Step definitions<br/>(given / when / then)"]
        FX["Fixtures<br/>(vip.fixtures)"]
        S
        FX
    end

    subgraph "Layer 3: Driver Port"
        CC["ConnectClient"]
        WC["WorkbenchClient"]
        PC["PackageManagerClient"]
    end

    subgraph "Layer 4: Driver Adapter"
        API["httpx<br/>(API calls)"]
        UI["Playwright<br/>(browser)"]
    end

    F --> S
    S --> CC
    S --> WC
    S --> PC
    CC --> API
    CC --> UI
    WC --> API
    WC --> UI
    PC --> API
Loading

Skip a layer and you end up with feature files full of HTTP status codes, or step definitions that break every time an endpoint changes.

Layer 1: The Test (.feature files)

Feature files are pure business scenarios. They contain no implementation details -- no URLs, no HTTP calls, no CSS selectors.

@connect
Feature: Connect authentication
  Scenario: User can log in via the web UI
    Given Connect is accessible at the configured URL
    When a user navigates to the Connect login page
    And enters valid credentials
    Then the user is successfully authenticated
    And the Connect dashboard is displayed

What's not here:

  • No URLs or endpoints
  • No HTTP status codes
  • No CSS selectors or page structure
  • No database queries
  • No setup or teardown logic

The scenario describes what the user experiences. How the system delivers that is entirely somebody else's problem.

Channel annotations

VIP supports running the same scenario through different channels using product marker tags (@connect, @workbench, @package_manager). The same business scenario can be verified through both the API and the UI when step definitions support both paths.

Tagging rules

Every feature file must have a product marker tag (@connect, @workbench, or @package_manager). The tag controls auto-skip: when a product is not configured, all scenarios with its tag are skipped automatically. Forgetting the tag breaks this mechanism and causes confusing failures.

Version gating

Use the min_version marker for features that only exist in certain product versions:

@pytest.mark.min_version(product="connect", version="2024.09.0")

This skips the test when the deployed version is older than the specified minimum, so the same test suite works across multiple product releases. Versions are parsed and compared with vip.version.ProductVersion, which understands Posit's calendar versioning scheme (YYYY.MM.patch plus -dev/-daily/-preview/+build suffixes).

When the deployed or required version cannot be determined at all (unconfigured, or unparseable), the test is skipped and flagged with a distinct "N/A (version)" report status rather than run optimistically -- a version gap should be visible in the report, not hidden behind a possibly-spurious pass.

min_version can only skip a whole test. When a UI change requires switching behavior or selectors rather than skipping outright -- for example, a redesigned dialog that closes via Escape instead of a Cancel button -- use a versioned page-object class plus a get_<page>(version) factory in src/vip_tests/workbench/pages/. Subclasses hold only the delta from the version they change in (cumulative inheritance), and behavior differences that aren't expressible as a selector change use a strategy dict keyed by version threshold, mirroring vip.idp's IdP strategy dict. See src/vip_tests/workbench/pages/homepage.py (Homepage_2026_05, get_homepage, get_new_session_dialog_close_strategy) for the pattern.

Layer 2: The DSL (step definitions + fixtures)

Step definitions are where the fluent API lives. They translate Gherkin steps into actions using fixtures and driver ports.

Given -- setting up the world

@given("Connect is accessible at the configured URL")
def connect_accessible(connect_client):
    """Guard step -- ensures the product is available."""
    if connect_client is None:
        pytest.skip("Connect is not configured")

Given steps collect preconditions. They use fixtures (like connect_client, vip_config) to access configuration and verify the world is ready. When a precondition isn't met, they call pytest.skip() rather than fail -- this marks the test as skipped with a clear reason instead of producing a confusing assertion error.

When -- performing the action

@when("a user navigates to the Connect login page")
def navigate_to_login(page, connect_url):
    page.goto(f"{connect_url}/__login__")

When steps perform the action under test. They call the system through driver ports (API clients or Playwright pages). The target_fixture parameter passes results to subsequent steps.

Then -- verifying the outcome

@then("the user is successfully authenticated")
def user_authenticated(page, connect_url):
    assert "/__login__" not in page.url

Then steps verify the outcome. Importantly, they should verify actual system state, not just the action's return value. Make a separate call to fetch current state when possible.

Fail, warn, or skip

A Then step has three outcomes, and picking the wrong one is how a suite loses its signal:

Outcome Use when Example
Fail The deployment is wrong and an administrator can fix it x-powered-by leaks a proxy's version — suppress it at the proxy
Warn (warnings.warn) The finding is real and worth recording, but nothing in the deployment's control can change it Package Manager's own server header carries its version and has no setting to suppress it
Skip The thing under test isn't present or configured, so there was nothing to verify No OpenVSX repository is configured

Two failure modes to watch for, both of which have bitten this suite:

  • A check that always fails. Advice the product cannot satisfy turns a whole category red on every stock deployment and trains people to skim past it. Warn instead — the exposure stays on the record for a hardening baseline that cares.
  • A check that can never fail. If every branch of a Then step warns or skips, it is not a check. Whenever you downgrade one branch to a warning, confirm some other branch can still fail (see no_version_headers in security/test_https.py).

Skips carry the same burden of accuracy as failures. A skip reason states why there was nothing to verify, so it must be true: test_repos.py used to report "package not available — repo may not be synced yet" after probing only the first repo whose name matched, when a synced mirror sitting beside it served the package fine. Probe every candidate before concluding anything, and name all of them in the reason.

Fixtures as glue

Pytest fixtures are the glue between layers. They provide:

  • Configuration: vip_config, connect_url, test_username
  • Clients: connect_client, workbench_client, pm_client
  • Browser state: page, browser_context_args
  • Feature flags: email_enabled, monitoring_enabled

VIP's core fixtures live in src/vip/fixtures.py, not a conftest.py. pytest scopes conftest.py fixtures by directory ancestry, which would make them invisible to a test extension collected from outside src/vip_tests (see "Writing a Test Extension" below) — so vip.plugin registers vip.fixtures as part of VIP's own pytest plugin instead, making every fixture and shared "Given" step available everywhere vip is installed, regardless of where a test lives on disk. src/vip_tests/conftest.py still defines a handful of fixtures deliberately kept out of that global plugin (autouse Connect content-cleanup) — see that file's docstring for why.

Layer 3: The Driver Port (protocols/interfaces)

Driver ports define what the DSL needs from the system without saying how. In VIP, these are the client interfaces in src/vip/clients/.

# src/vip/clients/connect.py -- the port (interface)
class ConnectClient:
    """Client interface for interacting with the Connect API in tests."""

    def current_user(self) -> dict: ...
    def list_content(self) -> list[dict]: ...
    def deploy_content(self, bundle: bytes, name: str) -> dict: ...

Key rules for driver ports:

  • String identifiers and parameters for maximum flexibility (allows invalid values in negative tests); non-string payloads like binary bundles are fine
  • Return dicts, not custom model objects
  • No product SDK dependencies -- use raw HTTP
  • Minimal surface -- add methods only when tests need them

The port is the contract between the DSL and the system. It never changes when you switch between API and UI testing.

Layer 4: The Driver Adapter (implementations)

This is where the same interface gets different implementations.

API adapter (httpx)

class ConnectClient:
    def __init__(self, base_url: str, *, api_key: str | None = None):
        headers = {"Authorization": f"Key {api_key}"} if api_key else {}
        self._client = httpx.Client(base_url=base_url, headers=headers)

    def current_user(self) -> dict:
        resp = self._client.get("/v1/user")
        resp.raise_for_status()
        return resp.json()

Straightforward HTTP calls that map requests and responses.

UI adapter (Playwright)

@when("a user navigates to the Connect login page")
def navigate_to_login(page, connect_url):
    page.goto(f"{connect_url}/__login__")

@when("enters valid credentials")
def enter_credentials(page, test_username, test_password):
    page.fill("[name='username']", test_username)
    page.fill("[name='password']", test_password)
    page.click("[data-automation='login-panel-submit']")

Same business action, completely different implementation. The UI adapter uses Playwright to open a browser, navigate pages, fill forms, and click buttons.

VIP's Layer Mapping

Layer 4-Layer Concept VIP Implementation
1. Test Scenario / specification .feature files with @product tags
2. DSL Fluent API / step builders pytest_bdd step definitions in .py files
3. Driver Port Interface / protocol Client classes in src/vip/clients/
4. Driver Adapter API / UI implementation httpx clients + Playwright page interactions

Why This Architecture Matters

VIP's layer structure is an application of Valentina Jemuović's architecture behind acceptance tests to how we test Posit Team deployments.

In practice, it means changes stay local:

  • Business rules change? Update the .feature files. Step definitions and clients stay the same.
  • API endpoint changes? Update the API client. Features, steps, and UI tests don't move.
  • UI redesign? Update the Playwright steps. Features, API steps, and clients don't move.
  • New feature? Add new .feature + .py pairs and extend clients. Everything existing stays untouched.

The .feature files only need to change when the requirements change.

Tuning timeouts on slow hardware

VIP ships with generous defaults, but on small QA VMs some operations — launching a Workbench session, deploying content to Connect — can take much longer than on production hardware. Set the VIP_TIMEOUT_SCALE environment variable to multiply every operation timeout uniformly:

VIP_TIMEOUT_SCALE=3 vip verify --connect-url https://connect.example.com

This scales:

  • All Playwright wait/expect timeouts (Workbench session start, IDE load, page load, …)
  • IdP login-form and network-idle waits
  • API polling deadlines (wait_for_task, wait_for_system_check)
  • The default httpx client timeout (30 s)
  • The pytest subprocess budget (--test-timeout)

Deliberately not scaled:

  • The 5-minute MFA prompt window — a human wait, not a server operation
  • UI busy-loop pacing sleeps (e.g. wait_for_timeout(500)) — scaling poll intervals only slows the loop without preventing any timeout
  • Explicit values: callers that pass an explicit timeout value opt out of scaling

Values < 1.0 are valid and useful for speeding up CI smoke checks (VIP_TIMEOUT_SCALE=0.5).

Fast confidence checks with --basic

vip verify --basic runs only the core subset of tests, excluding detailed or long-running checks tagged @slow. Today the @slow set covers the heavier Workbench checks — IDE extension installation, job execution, Git operations, and publishing to Connect — so a --basic run still exercises auth, IDE launch, sessions, runtime versions, packages, data sources, and Chronicle.

--basic composes with --categories: vip verify --categories workbench --basic runs the Workbench category minus its @slow scenarios. The mechanism is product-agnostic — tag any feature @slow to exclude it from basic runs.

Adding a New Test: Layer-by-Layer Checklist

  1. Layer 1 -- Feature file: Write the Gherkin scenario with a @product tag. Focus on business intent, not implementation.

  2. Layer 2 -- Step definitions: Implement given/when/then functions. Reuse existing steps where possible. Use target_fixture to pass state between steps.

  3. Layer 3 -- Driver port: Check if the client already has the method you need. If not, add a method to the appropriate client in src/vip/clients/.

  4. Layer 4 -- Driver adapter: Implement the client method using httpx (API) or add Playwright steps (UI). Keep each adapter focused on one way of talking to the system.

Anti-Patterns to Avoid

  • Leaking implementation into features: Don't put URLs, status codes, or selectors in .feature files.
  • Skipping the port layer: Don't make HTTP calls directly in step definitions. Go through a client.
  • Fat step definitions: If a step definition is more than ~10 lines, it's doing too much. Push logic down to the client layer.
  • Shared mutable state: Use target_fixture to pass state between steps, not module-level globals.
  • Testing implementation, not behavior: Scenarios should describe what the user experiences, not how the system works internally.

Writing a Test Extension

VIP's extension mechanism lets you load custom test directories alongside the built-in suite:

vip verify --config vip.toml --extensions ./my-custom-tests

Or in vip.toml:

[general]
extension_dirs = ["./my-custom-tests"]

Use vip scaffold to generate a ready-to-run reference implementation. Run vip scaffold --list to see the available templates:

vip scaffold --list
vip scaffold --template minimal --output ./my-custom-tests
vip scaffold --template cross-product --output ./my-custom-tests

--template defaults to cross-product (the pre-existing behavior of vip scaffold --output DIR is unchanged). Two canonical templates ship with VIP:

  • minimal (examples/custom_tests/) — a single-scenario HTTP health check against your own configured product; the best starting point for a new extension
  • cross-product (examples/cross_product_validation/) — a full GxP validation example that verifies R/Python runtime versions and package installability across Connect and Workbench

Both follow the same four-layer architecture as the built-in suite. Every scaffolded directory also gets an AGENTS.md, generated from a single shared source (examples/_shared/AGENTS.md), documenting the extension contract: the auto-skip rules, min_version gating, and an enumerated inventory of the public fixtures, registered markers, and client entry points an extension may use. It's the reference an AI coding assistant (or a human) should read before writing a new extension — a selftest guards it against drifting from the real fixtures and markers.

Key requirement for auto-skip to work in extensions: apply @pytest.mark.connect and/or @pytest.mark.workbench decorators directly on every @scenario function. With pytest-bdd, scenario-level Gherkin tags (e.g. @connect on a Scenario: block) do become pytest markers and participate in auto-deselect via item.get_closest_marker(). However, feature-level tags apply to every scenario in the file — if you tag the whole Feature: with both @connect and @workbench, every scenario requires both products, which causes incorrect deselection when only one product is configured. The safe, portable pattern is to tag each Scenario: with only the product(s) it actually uses, or to apply @pytest.mark.connect/@pytest.mark.workbench directly on the @scenario decorator function in Python.