Skip to content

Commit 4660a9c

Browse files
divshekhardivshekhar
committed
v1.3.2: docs updated
1 parent 2a050bd commit 4660a9c

68 files changed

Lines changed: 8182 additions & 442 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 10 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -12,36 +12,27 @@
1212
### Features
1313

1414
1. **Tool-Level Structural Safety Validation (`syrin test`)** - New default behavior for `syrin test` command that validates tool contracts through sandboxed execution.
15-
- **Tool Unit Contracts**: Define behavioral guarantees for each tool in YAML files (`tools/<tool-name>.yaml`)
16-
- **Sandboxed Execution**: Tools are tested in isolated environments with resource limits and I/O monitoring
17-
- **Behavioral Observation**: Detects side effects, non-determinism, output explosions, hidden dependencies, and unbounded execution
18-
- **Process Reuse**: Optimized for performance - MCP server started once, all tools tested, then closed (critical for 100+ tools)
19-
- **Synthetic Input Generation**: Automatically generates test inputs from JSON Schema definitions
20-
- **Contract-Defined Tests**: Support for explicit test cases in contract files
21-
- **CI-Friendly**: JSON output format and proper exit codes for CI/CD integration
22-
23-
2. **New Behavioral Error Rules (E012-E016)**:
24-
- **E012: Side Effect Detected** - Tool attempts filesystem writes to project files
25-
- **E013: Non-Deterministic Output** - Tool produces different outputs for same input
26-
- **E014: Output Explosion** - Tool output exceeds declared size limit
27-
- **E015: Hidden Dependency** - Tool calls other tools without declaring them
28-
- **E016: Unbounded Execution** - Tool execution timed out or failed to terminate
15+
16+
2. **New Behavioral Error Rules**:
17+
- **E500: Side Effect Detected** - Tool attempts filesystem writes to project files
18+
- **E301: Output Explosion** - Tool output exceeds declared size limit
19+
- **E403: Unbounded Execution** - Tool execution timed out or failed to terminate
2920

3021
3. **New Behavioral Warning Rules (W021-W023)**:
31-
- **W021: Weak Schema** - Contract schema is too loose or doesn't match MCP tool schema
32-
- **W022: High Entropy Output** - Tool output has high entropy (random, unpredictable)
33-
- **W023: Unstable Defaults** - Tool behavior changes significantly with default values
22+
- **W110: Weak Schema** - Contract schema is too loose or doesn't match MCP tool schema
23+
- **W300: High Entropy Output** - Tool output has high entropy (random, unpredictable)
24+
- **W301: Unstable Defaults** - Tool behavior changes significantly with default values
3425

3526
4. **Enhanced `syrin test` Command**:
3627
- **Default Mode**: Tool validation (new default behavior)
3728
- **Connection Testing**: Available via `--connection` flag (legacy behavior)
38-
- **Options**: `--tool`, `--strict`, `--json`, `--mcp-root`, `--timeout`, `--memory-limit`, `--max-output-size`, `--determinism-runs`
29+
- **Options**: `--tool`, `--strict`, `--json`, `--ci`
3930
- **Strict Mode**: `--strict` flag treats warnings as errors
4031
- **JSON Output**: `--json` flag for CI integration
32+
- **CI Mode**: `--ci` flag for CI Mode.
4133

4234
5. **Configuration Enhancements**:
4335
- New `check` section in `syrin.yaml` for tool validation configuration
44-
- Configurable parameters: `timeout_ms`, `memory_limit_mb`, `mcp_root`, `tools_dir`, `max_output_size_kb`, `determinism_runs`, `strict_mode`
4536

4637
### Improvements
4738

‎docs/CI/best-practices.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
---
2+
title: "CI Best Practices"
3+
description: "Best practices for using Syrin in CI/CD pipelines"
4+
weight: 5
5+
---
6+
7+
## CI Best Practices
8+
9+
Follow these best practices to get the most out of Syrin in your CI/CD pipelines.
10+
11+
## 1. Run Analysis and Tests Separately
12+
13+
Run static analysis first (faster), then tests (more comprehensive):
14+
15+
```bash
16+
# Fast: Static analysis
17+
syrin analyse --ci
18+
19+
# Comprehensive: Runtime tests
20+
syrin test --ci
21+
```
22+
23+
**Why**: Static analysis is faster and catches structural issues early. Tests are more comprehensive but take longer.
24+
25+
## 2. Use Strict Mode
26+
27+
Enable strict mode to catch warnings:
28+
29+
```bash
30+
syrin analyse --ci --strict
31+
syrin test --ci --strict
32+
```
33+
34+
**Why**: Warnings indicate potential issues that should be addressed. Strict mode ensures they don't slip through.
35+
36+
## 3. Store Results as Artifacts
37+
38+
Save JSON results for later analysis:
39+
40+
```bash
41+
syrin analyse --json > analysis.json
42+
syrin test --json > test-results.json
43+
```
44+
45+
**Why**: JSON results can be parsed programmatically, stored for historical analysis, and used for reporting.
46+
47+
## 4. Fail Fast on Errors
48+
49+
Exit immediately on errors:
50+
51+
```bash
52+
syrin analyse --ci || exit 1
53+
syrin test --ci || exit 1
54+
```
55+
56+
**Why**: Don't waste CI time if there are blocking errors. Fail fast and fix issues immediately.
57+
58+
## 5. Set Appropriate Timeouts
59+
60+
Configure timeouts for long-running tests:
61+
62+
```yaml
63+
# syrin.yaml
64+
check:
65+
timeout_ms: 60000 # 60 seconds for CI
66+
```
67+
68+
**Why**: CI environments may have different performance characteristics. Set timeouts based on your CI environment.
69+
70+
## 6. Use CI Mode
71+
72+
Always use `--ci` flag in CI environments:
73+
74+
```bash
75+
syrin analyse --ci
76+
syrin test --ci
77+
```
78+
79+
**Why**: CI mode provides minimal output suitable for automated pipelines and proper exit codes.
80+
81+
## 7. Parallel Execution
82+
83+
Run analysis and tests in parallel when possible:
84+
85+
```yaml
86+
jobs:
87+
analyse:
88+
runs-on: ubuntu-latest
89+
steps:
90+
- run: syrin analyse --ci
91+
92+
test:
93+
runs-on: ubuntu-latest
94+
steps:
95+
- run: syrin test --ci
96+
```
97+
98+
**Why**: Parallel execution reduces total CI time.
99+
100+
## 8. Cache Dependencies
101+
102+
Cache npm dependencies to speed up builds:
103+
104+
```yaml
105+
- uses: actions/cache@v3
106+
with:
107+
path: ~/.npm
108+
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
109+
```
110+
111+
**Why**: Faster builds mean faster feedback.
112+
113+
## See Also
114+
115+
- [CI Platform Setup](/ci/setup/)
116+
- [CI Workflows](/ci/workflows/)
117+
- [Reporting Results](/ci/reporting/)

‎docs/CI/how-syrin-helps.md‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
---
2+
title: "How Syrin Helps Write Safe MCPs"
3+
description: "Understanding how Syrin makes your MCP tools safer"
4+
weight: 6
5+
---
6+
7+
## How Syrin Helps Write Safe MCPs
8+
9+
Syrin provides comprehensive validation and safety checks that catch issues before they reach production.
10+
11+
## 1. Static Analysis (`syrin analyse`)
12+
13+
Catches issues **before** execution:
14+
15+
- **Schema Errors**: Missing output schemas, type mismatches
16+
- **Contract Issues**: Underspecified inputs, circular dependencies
17+
- **Best Practices**: Generic descriptions, missing examples
18+
19+
**Benefits**:
20+
21+
- Fast feedback (no execution required)
22+
- Catches structural issues early
23+
- Prevents runtime failures
24+
25+
## 2. Runtime Testing (`syrin test`)
26+
27+
Validates actual tool behavior:
28+
29+
- **Side Effects**: Detects filesystem writes to project files
30+
- **Output Validation**: Ensures output matches schema
31+
- **Execution Limits**: Validates timeouts and output size limits
32+
33+
**Benefits**:
34+
35+
- Catches behavioral issues
36+
- Validates actual tool execution
37+
- Ensures tools meet contracts
38+
39+
## 3. Comprehensive Error Detection
40+
41+
Syrin detects 20+ error types:
42+
43+
- **E100-E110**: Schema and contract errors
44+
- **E200**: Input validation errors
45+
- **E300-E301**: Output validation errors
46+
- **E400-E403**: Execution errors
47+
- **E500**: Behavioral errors
48+
- **E600**: Test framework errors
49+
50+
## 4. Warning Detection
51+
52+
Syrin identifies 12+ warning types:
53+
54+
- **W100-W110**: Schema and contract warnings
55+
- **W300-W301**: Output validation warnings
56+
57+
## 5. Contract Validation
58+
59+
Ensures tools match their contracts:
60+
61+
- Input/output schema validation
62+
- Guarantee enforcement (side effects, limits)
63+
- Dependency verification
64+
- Test expectation matching
65+
66+
## Safety Benefits
67+
68+
### Prevents Production Failures
69+
70+
- Catches tools that crash or hang
71+
- Validates timeout limits
72+
- Ensures proper error handling
73+
74+
### Prevents Security Issues
75+
76+
- Detects filesystem mutations
77+
- Validates side effect declarations
78+
- Ensures isolation
79+
80+
### Prevents Agent Confusion
81+
82+
- Validates tool descriptions
83+
- Ensures schema completeness
84+
- Catches ambiguous tool definitions
85+
86+
### Prevents Cost Overruns
87+
88+
- Validates output size limits
89+
- Catches output explosions
90+
- Ensures efficient tool design
91+
92+
## See Also
93+
94+
- [Error Rules Documentation](/errors/)
95+
- [Warning Rules Documentation](/warnings/)
96+
- [Writing Test Cases](/testing/writing-test-cases/) - Tool contract documentation
97+
- [Testing Documentation](/testing/)

‎docs/CI/index.md‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
---
2+
title: "Continuous Integration"
3+
description: "How Syrin helps you write safe MCPs through CI integration"
4+
weight: 1
5+
---
6+
7+
## Automate safety checks
8+
9+
Syrin is designed to integrate seamlessly into CI/CD pipelines, providing automated validation and safety checks for MCP tools before they reach production.
10+
11+
## Why CI Integration Matters
12+
13+
MCP tools are critical infrastructure for AI agents. Issues that slip through can cause:
14+
15+
- **Production Failures**: Tools that crash or hang
16+
- **Security Vulnerabilities**: Tools that mutate project state
17+
- **Agent Confusion**: Ambiguous or poorly defined tools
18+
- **Cost Overruns**: Tools that produce excessive output
19+
20+
Syrin catches these issues **before** they reach production through automated testing and analysis.
21+
22+
## Documentation Sections
23+
24+
### [CI Platform Setup](/ci/setup/)
25+
26+
Complete setup guides for popular CI platforms:
27+
28+
- [GitHub Actions](/ci/setup/github-actions/)
29+
- [GitLab CI](/ci/setup/gitlab-ci/)
30+
- [CircleCI](/ci/setup/circleci/)
31+
32+
### [CI Best Practices](/ci/best-practices/)
33+
34+
Learn how to configure Syrin effectively in CI:
35+
36+
- Running analysis and tests
37+
- Using strict mode
38+
- Storing results
39+
- Setting timeouts
40+
- Failing fast on errors
41+
42+
### [How Syrin Helps](/ci/how-syrin-helps/)
43+
44+
Understand how Syrin makes your MCPs safer:
45+
46+
- Static analysis benefits
47+
- Runtime testing benefits
48+
- Error and warning detection
49+
- Contract validation
50+
51+
### [CI Workflows](/ci/workflows/)
52+
53+
Real-world workflow examples:
54+
55+
- Pre-commit hooks
56+
- Pull request checks
57+
- Release validation
58+
59+
### [Reporting Results](/ci/reporting/)
60+
61+
How to report and visualize CI results:
62+
63+
- GitHub Actions annotations
64+
- GitLab CI reports
65+
- Custom reporting
66+
67+
## Quick Start
68+
69+
Add Syrin to your CI pipeline:
70+
71+
```bash
72+
# Install Syrin
73+
npm install -g @syrin/cli
74+
75+
# Run static analysis
76+
syrin analyse --ci
77+
78+
# Run tests
79+
syrin test --ci
80+
```
81+
82+
## See Also
83+
84+
- [Testing Documentation](/testing/)
85+
- [Error Rules Documentation](/errors/)
86+
- [Warning Rules Documentation](/warnings/)
87+
- [Writing Test Cases](/testing/writing-test-cases/) - Tool contract documentation

0 commit comments

Comments
 (0)