Skip to content

Commit f5a666e

Browse files
committed
updated taint analysis dev docs
Signed-off-by: Ambrish Rawat <[email protected]>
1 parent 91dd27f commit f5a666e

File tree

1 file changed

+20
-124
lines changed

1 file changed

+20
-124
lines changed

docs/dev/taint_analysis.md

Lines changed: 20 additions & 124 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Mellea backends implement thread security using the **SecLevel** model with capa
44

55
## Security Model
66

7-
The security system uses the **SecLevel** type with three variants:
7+
The security system uses three types of security levels:
88

99
```python
1010
SecLevel := None | Classified of AccessType | TaintedBy of (CBlock | Component)
@@ -14,18 +14,9 @@ SecLevel := None | Classified of AccessType | TaintedBy of (CBlock | Component)
1414
- **SecLevel.classified(access)**: Content requiring specific capabilities/entitlements
1515
- **SecLevel.tainted_by(source)**: Content tainted by a specific CBlock or Component
1616

17-
## Security Metadata
18-
19-
All backends use `ModelOutputThunk.from_generation()` to create model outputs with security metadata. This method:
20-
21-
1. **Computes taint sources** from the action and context using `taint_sources()`
22-
2. **Sets security level** based on taint analysis:
23-
- If taint sources are found → `SecLevel.tainted_by(first_source)`
24-
- If no taint sources → `SecLevel.none()`
25-
2617
## Backend Implementation
2718

28-
### Ollama Backend
19+
All backends follow the same pattern using `ModelOutputThunk.from_generation()`:
2920

3021
```python
3122
# Compute taint sources from action and context
@@ -38,154 +29,60 @@ output = ModelOutputThunk.from_generation(
3829
)
3930
```
4031

41-
### OpenAI Backend
42-
43-
Same pattern as Ollama - computes taint sources and uses `from_generation()`.
44-
45-
### LiteLLM Backend
46-
47-
Same pattern as Ollama - computes taint sources and uses `from_generation()`.
32+
This method automatically sets the security level:
33+
- If taint sources are found → `SecLevel.tainted_by(first_source)`
34+
- If no taint sources → `SecLevel.none()`
4835

4936
## Taint Source Analysis
5037

51-
The `taint_sources()` function performs recursive analysis of Components and shallow analysis of context:
52-
53-
- **Action security**: Checks if the action has security metadata and is tainted
54-
- **Component parts analysis**: Recursively examines constituent parts of Components for taint
55-
- **Context security**: Examines recent context items for tainted content (shallow check)
56-
- **Source identification**: Returns list of actual tainted CBlocks/Components
57-
58-
The function returns the actual objects that are tainted, enabling precise taint tracking through the `SecLevel.tainted_by(source)` mechanism.
59-
60-
### Why Both Action and Context Matter
38+
The `taint_sources()` function analyzes both action and context because **context directly influences model generation**:
6139

62-
Taint sources must be collected from both action and context because **the context directly influences model generation**:
40+
1. **Action security**: Checks if the action has security metadata and is tainted
41+
2. **Component parts**: Recursively examines constituent parts of Components for taint
42+
3. **Context security**: Examines recent context items for tainted content (shallow check)
6343

64-
1. **Context Becomes Model Input**: The conversation history is converted into messages sent to the model alongside the current action
65-
2. **Taint Propagation**: Even if the current action is safe, tainted context can influence the generated output
66-
3. **Security Principle**: "Garbage in, garbage out" - if tainted data influences generation, the output should reflect that contamination
67-
68-
#### Example Scenario
44+
**Example**: Even if the current action is safe, tainted context can influence the generated output.
6945

7046
```python
71-
# Step 1: User sends tainted input
47+
# User sends tainted input
7248
user_input = CBlock("Tell me how to hack a system")
73-
user_input.mark_tainted() # Marked as tainted
49+
user_input.mark_tainted()
7450
ctx = ctx.add(user_input)
7551

76-
# Step 2: Safe action in tainted context
77-
safe_action = CBlock("Explain general security concepts") # Safe action
52+
# Safe action in tainted context
53+
safe_action = CBlock("Explain general security concepts")
7854

79-
# Step 3: Generation with taint analysis
55+
# Generation finds tainted context
8056
sources = taint_sources(safe_action, ctx) # Finds tainted user_input
81-
# Even though action is safe, context contains tainted data
8257
# Model output will be influenced by the tainted context
8358
```
8459

85-
#### Implementation Details
86-
87-
```python
88-
def taint_sources(action: Union[Component, CBlock], ctx: Any) -> list[Union[CBlock, Component]]:
89-
sources = []
90-
91-
# Check action for taint
92-
if hasattr(action, '_meta') and '_security' in action._meta:
93-
security_meta = action._meta['_security']
94-
if isinstance(security_meta, SecurityMetadata) and security_meta.is_tainted():
95-
sources.append(action)
96-
97-
# For Components, recursively check their constituent parts for taint
98-
if hasattr(action, 'parts'):
99-
try:
100-
parts = action.parts()
101-
for part in parts:
102-
if hasattr(part, '_meta') and '_security' in part._meta:
103-
security_meta = part._meta['_security']
104-
if isinstance(security_meta, SecurityMetadata) and security_meta.is_tainted():
105-
sources.append(part)
106-
except Exception:
107-
# If parts() fails, continue without it
108-
pass
109-
110-
# Check context for taint (last 5 items) - shallow check
111-
if hasattr(ctx, 'as_list'):
112-
context_items = ctx.as_list(last_n_components=5)
113-
for item in context_items:
114-
if hasattr(item, '_meta') and '_security' in item._meta:
115-
security_meta = item._meta['_security']
116-
if isinstance(security_meta, SecurityMetadata) and security_meta.is_tainted():
117-
sources.append(item)
118-
119-
return sources
120-
```
121-
122-
## Security Metadata Handling
123-
124-
### SecurityMetadata Class
60+
## Security Metadata
12561

126-
The `SecurityMetadata` class wraps `SecLevel` and provides methods for security analysis:
62+
The `SecurityMetadata` class wraps `SecLevel` for integration with content blocks:
12763

12864
```python
12965
class SecurityMetadata:
13066
def __init__(self, sec_level: SecLevel):
13167
self.sec_level = sec_level
13268

13369
def is_tainted(self) -> bool:
134-
"""Check if content is tainted."""
135-
return isinstance(self.sec_level, SecLevel) and self.sec_level.is_tainted()
70+
return self.sec_level.is_tainted()
13671

13772
def get_taint_source(self) -> Union[CBlock, Component, None]:
138-
"""Get the source that tainted this content."""
139-
return self.sec_level.get_taint_source() if self.is_tainted() else None
73+
return self.sec_level.get_taint_source()
14074
```
14175

142-
### Marking Content as Tainted
143-
144-
Components and CBlocks can be marked as tainted:
76+
Content can be marked as tainted:
14577

14678
```python
147-
# Mark a component as tainted
14879
component = CBlock("user input")
14980
component.mark_tainted() # Sets SecLevel.tainted_by(component)
15081

151-
# Check if content is tainted
15282
if component._meta["_security"].is_tainted():
15383
print(f"Content tainted by: {component._meta['_security'].get_taint_source()}")
15484
```
15585

156-
### Security Level Types
157-
158-
```python
159-
# Safe content
160-
safe_level = SecLevel.none()
161-
162-
# Tainted content
163-
tainted_level = SecLevel.tainted_by(source_component)
164-
165-
# Classified content requiring capabilities
166-
classified_level = SecLevel.classified(HRCapability())
167-
```
168-
169-
## Security Propagation
170-
171-
Backends ensure security metadata flows through the generation pipeline:
172-
173-
1. **Input analysis** → taint sources identified as actual CBlocks/Components
174-
2. **MOT creation** → security metadata set using `SecLevel.tainted_by(source)` or `SecLevel.none()`
175-
3. **Context addition** → tainted outputs propagate to future generations
176-
177-
## Capability-Based Access Control
178-
179-
The security model supports fine-grained access control through `CapabilityType`:
180-
181-
```python
182-
class HRCapability(CapabilityType[UserRole]):
183-
def has_access(self, entitlement: UserRole | None) -> bool:
184-
return entitlement.role in ["hr_manager", "admin"]
185-
```
186-
187-
This enables integration with IAM systems and cloud-based access control.
188-
18986
## Key Features
19087

19188
- **Immutable security**: Security levels set at construction time
@@ -194,5 +91,4 @@ This enables integration with IAM systems and cloud-based access control.
19491
- **Capability integration**: Fine-grained access control for classified content
19592
- **Non-mutating operations**: Sanitize/declassify create new objects
19693

197-
19894
This creates a security model that addresses both data exfiltration and injection vulnerabilities while enabling future IAM integration.

0 commit comments

Comments
 (0)