Skip to content

README.md

Latest commit

 

History

History
303 lines (254 loc) · 10.9 KB

README.md

File metadata and controls

303 lines (254 loc) · 10.9 KB

Jido Integration

Tooling-root, non-umbrella Elixir monorepo for the final V2 integration platform.

The repo root owns workspace tooling, quality gates, and repo-level guides only. Runtime code, connector code, and proof surfaces live in child packages and top-level apps. We do not restore root examples/ or reference_apps/.

Workspace Model

jido_integration/
  mix.exs                    # tooling/workspace root only
  README.md                  # repo architecture + command index
  AGENTS.md                  # working contract for future agents
  lib/                       # root Mix tasks and workspace helpers only
  test/                      # root tooling tests only
  docs/                      # repo-level architecture and operational guides
  core/
    platform/               # public facade package (`:jido_integration_v2`)
    contracts/              # shared public structs and behaviours
    control_plane/          # connector registry + durable run/trigger truth
    conformance/            # reusable connector conformance engine
    auth/                   # install, connection, credential, and lease truth
    ingress/                # trigger normalization and durable admission
    policy/                 # pre-attempt policy and shed decisions
    direct_runtime/         # direct capability execution
    runtime_asm_bridge/     # integration-owned `asm` Harness driver projection
    session_kernel/         # reusable session execution
    stream_runtime/         # reusable stream execution
    store_local/            # restart-safe local durability tier
    store_postgres/         # database-backed durable tier
    dispatch_runtime/       # async queue, retry, replay, recovery
    webhook_router/         # hosted route lifecycle and ingress bridge
  connectors/
    github/                # direct GitHub connector + live acceptance runbook
    notion/                # direct Notion connector + package-local live proofs
    codex_cli/             # Harness-routed session connector via `asm`
    market_data/           # Harness-routed stream connector via `asm`
  apps/
    trading_ops/           # cross-runtime operator proof
    devops_incident_response/ # hosted webhook + async recovery proof

Final V2 Surface

  • core/platform owns the public app identity :jido_integration_v2 and the stable facade module Jido.Integration.V2.
  • Jido.Integration.V2 exposes typed invocation, connector discovery, auth lifecycle calls, durable review lookups, and target lookup through a single public surface.
  • connector packages author rich manifests in core/contracts; executable capabilities remain available as a derived projection for runtime lookup and invocation.
  • core/conformance owns reusable connector review logic behind the root mix jido.conformance connector acceptance command.
  • non-direct runtime ownership stays below the integration layer. Jido.Harness is the stable runtime-driver contract seam; the authored asm driver is implemented by core/runtime_asm_bridge, which projects into agent_session_manager above cli_subprocess_core.
  • core/dispatch_runtime and core/webhook_router stay as child packages. Hosted async and webhook behavior does not move back into the root or the facade package.
  • Durability is explicit and package-owned:
    • core/auth and core/control_plane still ship in-memory defaults.
    • core/store_local is the restart-safe local durability tier.
    • core/store_postgres is the shared database-backed durable tier.
  • Child packages depend on each other only through explicit path: deps.
  • No child package depends on the repo root.

Public API Highlights

The stable public entrypoint is Jido.Integration.V2.

Key calls:

  • connector discovery:
    • connectors/0
    • capabilities/0
    • fetch_connector/1
    • fetch_capability/1
  • auth lifecycle:
    • start_install/3
    • complete_install/2
    • fetch_install/1
    • connection_status/1
    • request_lease/2
    • rotate_connection/2
    • revoke_connection/2
  • invocation:
    • InvocationRequest.new!/1
    • invoke/1
    • invoke/3
  • durable review truth:
    • fetch_run/1
    • fetch_attempt/1
    • events/1
    • run_artifacts/1
    • fetch_artifact/1
  • target announcement and lookup:
    • announce_target/1
    • fetch_target/1
    • compatible_targets/1

Hosted webhook routing and async replay are intentionally separate public package APIs:

  • Jido.Integration.V2.DispatchRuntime
  • Jido.Integration.V2.WebhookRouter

Current Proof Surface

Runtime families proved in-tree:

  • :direct
    • github.issue.list
    • github.issue.fetch
    • github.issue.create
    • github.issue.update
    • github.issue.label
    • github.issue.close
    • github.comment.create
    • github.comment.update
    • notion.users.get_self
    • notion.search.search
    • notion.pages.create
    • notion.pages.retrieve
    • notion.pages.update
    • notion.blocks.list_children
    • notion.blocks.append_children
    • notion.data_sources.query
    • notion.comments.create
  • :session
    • codex.exec.session
  • :stream
    • market.ticks.pull

Reference apps:

  • apps/trading_ops
    • proves one operator-visible workflow across stream, session, and direct runtimes
    • keeps trigger admission in core/ingress
    • keeps durable review truth in core/control_plane
  • apps/devops_incident_response
    • proves hosted webhook registration, async dispatch, dead-letter, replay, and restart recovery
    • keeps webhook behavior app-local instead of widening connectors/github

The current surface also proves:

  • connectors execute through short-lived auth leases, not durable credential truth
  • public invocation binds auth through connection_id; credential_ref remains internal execution plumbing
  • session and stream connector packages depend on jido_harness for the shared seam; they do not take direct agent_session_manager or cli_subprocess_core package dependencies
  • Notion OAuth control flows stay in the auth/install lifecycle instead of the normal invoke surface
  • InvocationRequest is the typed public invoke object
  • conformance runs from the root while connector evidence stays package-local
  • connector READMEs carry runtime/auth posture, package verification, live proof status, and package-boundary notes as part of connector review
  • local durability, async queue state, and webhook route state are all explicit opt-in packages

Guide Index

Repo-level guides in docs/:

Package and app runbooks:

  • core/platform/README.md
  • core/conformance/README.md
  • core/store_local/README.md
  • core/dispatch_runtime/README.md
  • core/webhook_router/README.md
  • connectors/github/README.md
  • connectors/github/docs/live_acceptance.md
  • connectors/notion/README.md
  • connectors/notion/docs/live_acceptance.md
  • apps/trading_ops/README.md
  • apps/devops_incident_response/README.md

Validation Prerequisites

The monorepo test and CI surface now includes packages that wire core/store_postgres in :test.

mix mr.test and mix ci therefore expect a reachable Postgres test store. Before calling the repo blocked on Postgres reachability, run:

mix mr.pg.preflight

That check validates the canonical core/store_postgres test tier only. The repo still supports the other two durability tiers in parallel:

  • in-memory defaults in core/auth and core/control_plane
  • core/store_local for restart-safe local durability
  • core/store_postgres for the shared database-backed tier

The default test configuration uses:

  • JIDO_INTEGRATION_V2_DB_HOST=127.0.0.1
  • JIDO_INTEGRATION_V2_DB_PORT=5432
  • JIDO_INTEGRATION_V2_DB_NAME=jido_integration_v2_test
  • JIDO_INTEGRATION_V2_DB_USER=postgres
  • JIDO_INTEGRATION_V2_DB_PASSWORD=postgres

If your local environment exposes Postgres through a socket directory instead of TCP, set JIDO_INTEGRATION_V2_DB_SOCKET_DIR.

core/store_postgres/README.md is the canonical home for the durable-tier test configuration.

Monorepo Commands

Run these from the repo root:

mix test
mix monorepo.deps.get
mix monorepo.format
mix monorepo.compile
mix monorepo.test
mix monorepo.credo --strict
mix monorepo.dialyzer
mix monorepo.docs
mix mr.pg.preflight
mix quality
mix docs.all
mix ci
mix jido.conformance Jido.Integration.V2.Connectors.GitHub
mix jido.conformance Jido.Integration.V2.Connectors.MarketData
mix jido.conformance Jido.Integration.V2.Apps.DevopsIncidentResponse.GitHubIssueConnector
mix jido.conformance Jido.Integration.V2.Connectors.Notion
mix jido.integration.new acme_crm --runtime-class direct
mix jido.integration.new analyst_cli --runtime-class session --runtime-driver asm

mix ci is the main acceptance gate.

Trigger-capable proof surfaces now include both connector-local common poll proofs and app-local hosted webhook proofs. Run root conformance against the module that actually owns the published ingress_definitions/0 evidence.

Shortcuts

The root mix.exs also defines mr.* aliases for the same monorepo task surface:

mix mr.deps.get
mix mr.format
mix mr.compile
mix mr.test
mix mr.pg.preflight
mix mr.credo --strict
mix mr.dialyzer
mix mr.docs

These are shortcuts for the corresponding mix monorepo.* commands above.

The repo no longer carries its own monorepo runner implementation. The monorepo.* commands are root aliases to the generic mix blitz.workspace <task> runner from the blitz Hex package.

Workspace policy now lives in the root mix.exs under :blitz_workspace:

  • project discovery comes from the configured projects list
  • task behavior comes from the configured tasks list
  • concurrency weights come from parallelism.base
  • the default machine scaler comes from parallelism.multiplier: :auto
  • optional per-task overrides still let the repo pin exceptional tasks
  • --max-concurrency N and JIDO_MONOREPO_MAX_CONCURRENCY still override the current invocation directly

The intended model is:

  • task weight lives in base
  • machine size lives in multiplier
  • hard per-run overrides still win when you need them

For mix monorepo.test and mix ci, each package test run gets its own isolated Postgres database name derived from the base JIDO_INTEGRATION_V2_DB_NAME. That keeps DB-backed package tests parallel without cross-package resets colliding in the shared store, and the child mix test processes run with --color so ExUnit ANSI output is preserved.