Skip to content

Commit 50d2285

Browse files
rycerzesrycerzes
authored
Merge pull request #19 from 3xcaffeine/feat/openenv-yaml-manifest
2 parents da556f4 + ba30222 commit 50d2285

7 files changed

Lines changed: 486 additions & 2 deletions

File tree

‎SkyRL‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Subproject commit fa866e767302846da1df779e80a19acbc3a4563f

‎openenv.yaml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
spec_version: 1
2+
name: frontier-swe-openenv
3+
type: package
4+
runtime: fastapi
5+
version: "0.1.0"
6+
7+
description: >
8+
Frontier SWE — OpenEnv. A family of long-horizon software-engineering
9+
environments deployed as Hugging Face Spaces, sharing one OpenEnv-shaped
10+
FastAPI server (Gym-style /reset, /step, /state plus FastMCP tools) and a
11+
multi-layer rubric (gate checks, hidden-test verifier, LLM diff review,
12+
LLM plan review) that produces a normalised [0, 1] reward. Each task
13+
ships as its own GHCR image and is mirrored to a dedicated HF Space.
14+
15+
repo: https://github.com/3xcaffeine/frontier-swe-openenv
16+
17+
# Shared package primitives. Each task Space inherits these and overrides
18+
# the per-task subset in its own openenv.yaml.
19+
package:
20+
app: frontier_swe_env.server.app:app
21+
port: 8000
22+
client: frontier_swe_env.client:FrontierSweEnv
23+
base_classes:
24+
server: openenv.core.env_server.create_app
25+
client: openenv.core.EnvClient
26+
api:
27+
gym_style:
28+
- POST /reset
29+
- POST /step
30+
- GET /state
31+
- GET /health
32+
mcp:
33+
- POST /mcp # OpenEnv POST-only JSON-RPC
34+
- ANY /tools/mcp # FastMCP Streamable HTTP (POST + GET/SSE)
35+
tools:
36+
- submit_plan
37+
- submit_subtask
38+
- get_status
39+
- advance
40+
reserved_tool_names_avoided:
41+
- reset
42+
- step
43+
- state
44+
- close
45+
46+
# Tasks bundled in this repo. Each entry points at its per-Space manifest
47+
# and the live HF Space URL; judges should pull the per-task openenv.yaml
48+
# from the Space they are evaluating.
49+
tasks:
50+
- name: notebook-compression
51+
manifest: spaces/notebook/openenv.yaml
52+
hf_space: https://huggingface.co/spaces/rycerzes/frontier-swe-notebook
53+
image: ghcr.io/3xcaffeine/frontier-swe-openenv/frontier-swe-notebook:latest
54+
domain: systems / compression
55+
- name: postgres-sqlite-wire-adapter
56+
manifest: spaces/postgres/openenv.yaml
57+
hf_space: https://huggingface.co/spaces/rycerzes/frontier-swe-postgres
58+
image: ghcr.io/3xcaffeine/frontier-swe-openenv/frontier-swe-postgres:latest
59+
domain: systems / databases / Zig
60+
- name: dependent-type-checker
61+
manifest: spaces/type-checker/openenv.yaml
62+
hf_space: https://huggingface.co/spaces/rycerzes/frontier-swe-type-checker
63+
image: ghcr.io/3xcaffeine/frontier-swe-openenv/frontier-swe-dependent-type-checker:latest
64+
domain: programming languages / type theory
65+
- name: libexpat-to-x86asm
66+
manifest: spaces/libexpat-to-x86asm/openenv.yaml
67+
hf_space: https://huggingface.co/spaces/rycerzes/frontier-swe-libexpat-to-x86asm
68+
image: ghcr.io/3xcaffeine/frontier-swe-openenv/frontier-swe-libexpat-to-x86asm:latest
69+
domain: systems programming / x86-64 assembly / XML
70+
71+
# Default landing task for `openenv pull frontier-swe-openenv` style tooling
72+
# until per-task selection is wired through.
73+
default_task: dependent-type-checker

‎scripts/prepare_hf_space.py‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
"""Assemble the push payload for an HF Space.
22
33
Given a task name, produce a directory that can be force-pushed to the Space:
4-
- Dockerfile and README.md are lifted from ``spaces/<task>/`` to the payload root
5-
(HF requires both at the root for Docker Spaces).
4+
- Dockerfile, README.md, and openenv.yaml are lifted from ``spaces/<task>/``
5+
to the payload root (HF requires Dockerfile + README at the root for Docker
6+
Spaces; openenv.yaml goes there so judges pulling the Space see a valid
7+
manifest at the URL root).
68
- The sibling ``spaces/<other-task>/`` subtree is dropped to reduce Space size.
79
- ``.gitattributes`` is preserved so HF correctly interprets the LFS-tracked
810
bundle.
@@ -34,12 +36,19 @@ def prepare(task: str, out: Path, repo_root: Path) -> None:
3436
if not space_src.is_dir():
3537
raise SystemExit(f"expected {space_src} to exist")
3638

39+
# Required: HF Docker Spaces need Dockerfile + README.md at the root.
3740
for name in ("Dockerfile", "README.md"):
3841
src = space_src / name
3942
if not src.is_file():
4043
raise SystemExit(f"missing {src}")
4144
shutil.copy2(src, out / name)
4245

46+
# Optional: lift openenv.yaml to the root so judges pulling the Space URL
47+
# see a valid OpenEnv manifest at the top level. Missing is non-fatal.
48+
manifest = space_src / "openenv.yaml"
49+
if manifest.is_file():
50+
shutil.copy2(manifest, out / "openenv.yaml")
51+
4352
shutil.rmtree(out / "spaces")
4453
print(out)
4554

‎spaces/libexpat-to-x86asm/openenv.yaml‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
spec_version: 1
2+
name: frontier-swe-libexpat-to-x86asm
3+
type: space
4+
runtime: fastapi
5+
app: frontier_swe_env.server.app:app
6+
port: 8000
7+
version: "0.1.0"
8+
9+
description: >
10+
Frontier SWE — libexpat to x86-64 Assembly. An OpenEnv-shaped FastAPI
11+
service hosting a long-horizon task: reimplement libexpat 2.6.4 in x86-64
12+
assembly and produce /app/asm-port/libexpat.so exporting the expat C ABI.
13+
The verifier builds reference C libexpat, runs the upstream test suite and
14+
benchmarks, and writes /logs/verifier/reward.json (0.5 correctness + 0.5
15+
performance, hard-fail to 0.0 on anti-cheat or missing .so). Scored with
16+
gate checks, structured L1, LLM code/plan review, and weighted episode reward.
17+
18+
repo:
19+
source: https://github.com/3xcaffeine/frontier-swe-openenv
20+
task_directory: tasks/libexpat-to-x86asm
21+
22+
environment:
23+
task_name: libexpat-to-x86asm
24+
workspace_dir: /app/asm-port
25+
build_command: "true"
26+
episode_timeout_s: 3600
27+
max_attempts_per_subtask: 3
28+
l1_score_mode: reward_json_score
29+
l1_timeout_s: 1500
30+
reward_json_path: /logs/verifier/reward.json
31+
reward_json_score_field: score
32+
reward_json_score_anchors: [0.0, 1.0]
33+
reward_json_score_higher_is_better: true
34+
task_domain: systems programming / x86-64 assembly / XML parsing
35+
cpus: 4
36+
memory_mb: 8192
37+
38+
rubric:
39+
type: composite
40+
layers:
41+
- name: gate_checks
42+
kind: shell
43+
script: /app/gate_checks.sh
44+
output: GATE_SCORE=N/M
45+
gates:
46+
- /app/asm-port writable, expat.h present
47+
- nasm, as, and ld on PATH
48+
- encrypted gcc + libexpat bundles staged
49+
- name: l1_tests
50+
kind: structured_reward
51+
score_mode: reward_json_score
52+
reward_json_path: /logs/verifier/reward.json
53+
score_field: score
54+
anchors: [0.0, 1.0]
55+
higher_is_better: true
56+
hard_fail_signals:
57+
- score == 0.0 (no .so, anti-cheat, verifier infra failure, zero correctness)
58+
- name: l2_code_review
59+
kind: llm_judge
60+
model_env: FSWE_GRADER_MODEL
61+
api_url_env: FSWE_GRADER_API_URL
62+
api_key_env: FSWE_GRADER_API_KEY
63+
dimensions:
64+
[completeness, correctness, robustness, forward_compatibility]
65+
- name: l3_plan_review
66+
kind: llm_judge
67+
model_env: FSWE_GRADER_MODEL
68+
api_url_env: FSWE_GRADER_API_URL
69+
api_key_env: FSWE_GRADER_API_KEY
70+
- name: episode_aggregator
71+
kind: weighted_blend
72+
output_field: observation.episode_reward
73+
74+
tools:
75+
- name: submit_plan
76+
description: Propose a subtask plan for the episode (PLANNING -> EXECUTING).
77+
parameters:
78+
- name: subtasks
79+
type: list[dict]
80+
required: true
81+
- name: submit_subtask
82+
description: Submit the current subtask for L1 + L2 scoring.
83+
parameters:
84+
- name: subtask_id
85+
type: str
86+
required: true
87+
- name: get_status
88+
description: Return the current episode status snapshot (phase, scores, time remaining).
89+
- name: advance
90+
description: Freeze the current subtask score and advance to the next subtask.
91+
92+
metrics:
93+
observation:
94+
- observation.phase
95+
- observation.current_subtask
96+
- observation.frozen_scores
97+
- observation.time_remaining_s
98+
- observation.plan_score
99+
- observation.subtask_feedback
100+
- observation.episode_reward
101+
reward:
102+
- reward.gate_score
103+
- reward.l1_test_score
104+
- reward.l1_blended
105+
- reward.l2_code_review
106+
- reward.l3_plan_review
107+
- reward.episode_reward

‎spaces/notebook/openenv.yaml‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
spec_version: 1
2+
name: frontier-swe-notebook
3+
type: space
4+
runtime: fastapi
5+
app: frontier_swe_env.server.app:app
6+
port: 8000
7+
version: "0.1.0"
8+
9+
description: >
10+
Frontier SWE — Notebook Compression. An OpenEnv-shaped FastAPI service that
11+
hosts a long-horizon software engineering task: build a fit/compress/decompress
12+
pipeline for Jupyter notebooks. Agents plan subtasks, edit code in a Linux
13+
workspace, then submit for multi-layer rubric scoring (gate checks, hidden
14+
test verifier, LLM diff review, LLM plan review). The verifier writes a
15+
structured reward.json the rubric layer normalises into a [0, 1] reward.
16+
17+
repo:
18+
source: https://github.com/3xcaffeine/frontier-swe-openenv
19+
task_directory: tasks/notebook-compression
20+
21+
# Task-level defaults (overridable via FSWE_TASK_MODE=demo|training and env vars).
22+
environment:
23+
task_name: notebook-compression
24+
workspace_dir: /app
25+
build_command: ""
26+
episode_timeout_s: 3600
27+
max_attempts_per_subtask: 2
28+
l1_score_mode: reward_json
29+
reward_json_path: /logs/verifier/reward.json
30+
task_domain: systems / compression
31+
cpus: 8
32+
memory_mb: 32768
33+
34+
# Composite multi-layer rubric. Final episode reward is a weighted blend of
35+
# all layers, normalised to [0, 1].
36+
rubric:
37+
type: composite
38+
layers:
39+
- name: gate_checks
40+
kind: shell
41+
script: /app/gate_checks.sh
42+
output: GATE_SCORE=N/M (parsed by frontier_swe_env.rubrics.gate_checks)
43+
- name: l1_tests
44+
kind: structured_reward
45+
score_mode: reward_json
46+
reward_json_path: /logs/verifier/reward.json
47+
- name: l2_code_review
48+
kind: llm_judge
49+
model_env: FSWE_GRADER_MODEL
50+
api_url_env: FSWE_GRADER_API_URL
51+
api_key_env: FSWE_GRADER_API_KEY
52+
dimensions:
53+
[completeness, correctness, robustness, forward_compatibility]
54+
- name: l3_plan_review
55+
kind: llm_judge
56+
model_env: FSWE_GRADER_MODEL
57+
api_url_env: FSWE_GRADER_API_URL
58+
api_key_env: FSWE_GRADER_API_KEY
59+
- name: episode_aggregator
60+
kind: weighted_blend
61+
output_field: observation.episode_reward
62+
63+
# MCP tools exposed by this environment. None of these collide with OpenEnv's
64+
# reserved tool names (reset, step, state, close).
65+
tools:
66+
- name: submit_plan
67+
description: Propose a subtask plan for the episode (PLANNING -> EXECUTING).
68+
parameters:
69+
- name: subtasks
70+
type: list[dict]
71+
required: true
72+
- name: submit_subtask
73+
description: Submit the current subtask for L1 + L2 scoring.
74+
parameters:
75+
- name: subtask_id
76+
type: str
77+
required: true
78+
- name: get_status
79+
description: Return the current episode status snapshot (phase, scores, time remaining).
80+
- name: advance
81+
description: Freeze the current subtask score and advance to the next subtask.
82+
83+
# Observation fields surfaced to the agent on every /step response.
84+
metrics:
85+
observation:
86+
- observation.phase
87+
- observation.current_subtask
88+
- observation.frozen_scores
89+
- observation.time_remaining_s
90+
- observation.plan_score
91+
- observation.subtask_feedback
92+
- observation.episode_reward
93+
reward:
94+
- reward.gate_score
95+
- reward.l1_test_score
96+
- reward.l1_blended
97+
- reward.l2_code_review
98+
- reward.l3_plan_review
99+
- reward.episode_reward

0 commit comments

Comments
 (0)