Release 0.2.0: multilingual detection, configurable output scanner, ANN index

Major improvements to detection quality and reliability:

- Add injection patterns for Chinese, Japanese, Korean, Arabic, Hindi (11 languages)
- Generalize script-mixing detector from Khmer-only to all non-Latin scripts
- Add indirect injection patterns (HTML comments, confused deputy, URL payloads)
  gated behind SecurityPolicy.detect_indirect_injection flag
- Make OutputScanner weights configurable via OutputScannerConfig
- Reduce false positives: exempt SHA hashes, keyword-gate base64, JWT heuristic
- Add optional hnswlib ANN index for O(log n) memory search (fast-memory extra)
- Fix MCP server to initialize with working memory bank (learn_threat now stores)
- Add TextEmbedder.using_fallback property with degradation warnings
- Add public AdversarialMemoryBank.add_threat_batch() method
- Replace 46 repetitive test fixtures with 28 diverse attack patterns
- Add MCP integration tests for learn -> assess round-trip

181 tests passing, 0 lint errors.

Made-with: Cursor

Author: denial-web

CHANGELOG.md

@@ -2,6 +2,35 @@
 
 All notable changes to agent-immune are documented here.
 
+## [0.2.0] — 2026-04-07
+
+### Added
+
+- **Multilingual injection detection** — 12 new patterns for Chinese, Japanese, Korean, Arabic, and Hindi. Total: 11 languages.
+- **Generalized script-mixing detector** — any non-Latin script (CJK, Arabic, Devanagari, Hangul) mixed with English imperatives now triggers detection (previously Khmer-only).
+- **Indirect injection patterns** — HTML comment injection, markdown comment injection, confused deputy attacks, URL-embedded payloads. Gated behind `SecurityPolicy.detect_indirect_injection` flag.
+- **Configurable output scanner** — new `OutputScannerConfig` model with per-category weights (PII, credentials, base64, hex, etc.). Passed via `SecurityPolicy.output_scanner_config`.
+- **Reduced false positives** — output scanner now exempts SHA-256/512 hex hashes, requires threat keywords in decoded base64, and distinguishes bare JWT tokens from documented examples.
+- **Optional ANN index** — `hnswlib`-backed HNSW index for memory bank search, reducing query time from O(n) to O(log n). Install via `pip install 'agent-immune[fast-memory]'`. Falls back to NumPy when not installed.
+- **Public batch API** — `AdversarialMemoryBank.add_threat_batch()` for bulk loading. `train_from_corpus` now uses the public API instead of private internals.
+- **MCP server memory** — `build_mcp()` now initializes with a working embedder and memory bank. `learn_threat` actually stores patterns. Fallback embedder status surfaced in tool responses.
+- **Fallback embedder warnings** — `TextEmbedder.using_fallback` property; logs WARNING when hash-based fallback is active. Memory bank warns about degraded matching quality.
+
+### Changed
+
+- `DecompositionResult` now includes `indirect_hits` field.
+- Volume anomaly condition in output scanner explicitly parenthesized for clarity.
+- Test fixtures diversified: replaced 46 repetitive jailbreak variants with 28 genuinely distinct attack patterns across multiple categories and languages.
+- Russian injection pattern updated to handle post-homoglyph-normalization text.
+
+### Fixed
+
+- MCP `learn_threat` tool now correctly stores entries (was silently returning `stored: false` due to missing memory bank).
+
+### Stats
+
+- **181 tests**, 0 lint errors, 11 languages supported.
+
 ## [0.1.1] — 2026-04-07
 
 ### Added

README.md

@@ -4,7 +4,7 @@
 [![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://python.org)
 [![Coverage 94%](https://img.shields.io/badge/coverage-94%25-brightgreen.svg)](tests/)
 [![License Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-green.svg)](LICENSE)
-[![179 tests](https://img.shields.io/badge/tests-179%20passing-brightgreen.svg)](tests/)
+[![181 tests](https://img.shields.io/badge/tests-181%20passing-brightgreen.svg)](tests/)
 [![Glama](https://glama.ai/mcp/servers/denial-web/agent-immune/badges/card.svg)](https://glama.ai/mcp/servers/denial-web/agent-immune)
 
 Adaptive threat intelligence for AI agent security: **semantic memory**, **multi-turn escalation**, **output scanning**, **rate limiting**, and **prompt hardening** — designed to complement deterministic governance stacks (e.g. [Microsoft Agent OS](https://github.com/microsoft/agent-governance-toolkit)), not replace them.
@@ -39,9 +39,11 @@ findings : cred_aws, cred_password_assign
 ## Install
 
 ```bash
-pip install -e ".[dev]"          # core + tests (regex-only, no GPU)
-pip install -e ".[memory,dev]"   # + sentence-transformers for semantic memory
-pip install 'agent-immune[mcp]'  # Model Context Protocol server (stdio / HTTP)
+pip install agent-immune                    # core (regex-only, no GPU)
+pip install 'agent-immune[memory]'          # + sentence-transformers for semantic memory
+pip install 'agent-immune[mcp]'             # Model Context Protocol server (stdio / HTTP)
+pip install 'agent-immune[fast-memory]'     # + hnswlib for fast ANN search at scale
+pip install 'agent-immune[all]'             # everything
 ```
 
 Python **3.9+** required; 3.11+ recommended. The MCP stack targets **Python 3.10+** (see the `mcp` package).
@@ -104,8 +106,15 @@ if immune.output_blocks(scan):
 
 ```python
 from agent_immune import AdaptiveImmuneSystem, SecurityPolicy
-
-strict = SecurityPolicy(allow_threshold=0.20, review_threshold=0.45, output_block_threshold=0.50)
+from agent_immune.core.models import OutputScannerConfig
+
+strict = SecurityPolicy(
+    allow_threshold=0.20,
+    review_threshold=0.45,
+    output_block_threshold=0.50,
+    detect_indirect_injection=True,
+    output_scanner_config=OutputScannerConfig(pii_weight=0.5, credential_weight=0.6),
+)
 immune = AdaptiveImmuneSystem(policy=strict)
 ```
 
@@ -191,8 +200,10 @@ Run `PYTHONPATH=src python demos/demo_full_lifecycle.py` to reproduce this on yo
 |------------|-------------------|--------------|
 | Keyword injection | Blocked | Blocked |
 | Rephrased attack | **Often missed** | **Caught** via semantic memory |
+| Multilingual injection | English-only rules | **11 languages** (EN, DE, ES, FR, HR, RU, ZH, JA, KO, AR, HI) |
+| Indirect injection | Not detected | HTML comments, confused deputy, URL payloads |
 | Multi-turn escalation | Not tracked | Detected via session trajectory |
-| Output exfiltration | Rarely scanned | PII, creds, prompt leak, encoded blobs |
+| Output exfiltration | Rarely scanned | PII, creds, prompt leak, encoded blobs (configurable weights) |
 | Learns from incidents | Manual rule updates | `immune.learn()` — instant semantic coverage |
 | Rate limiting | Separate system | Built-in circuit breaker |
 | Prompt hardening | DIY | `PromptHardener` with role-lock, sandboxing, output guard |
@@ -259,7 +270,7 @@ python bench/run_benchmarks.py
 | [deepset/prompt-injections](https://huggingface.co/datasets/deepset/prompt-injections) | 662 | 1.000 | 0.342 | 0.510 | 0.0 | 0.12 ms |
 | Combined | 847 | 1.000 | 0.521 | 0.685 | 0.0 | 0.12 ms |
 
-Zero false positives across all datasets. Multilingual patterns cover English, German, Spanish, French, Croatian, and Russian.
+Zero false positives across all datasets. Multilingual patterns cover English, German, Spanish, French, Croatian, Russian, Chinese, Japanese, Korean, Arabic, and Hindi.
 
 ### With adversarial memory
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "agent-immune"
-version = "0.1.1"
+version = "0.2.0"
 description = "Adaptive threat intelligence for AI agent security — semantic memory, multi-turn escalation, output scanning, rate limiting, and prompt hardening."
 readme = "README.md"
 license = { text = "Apache-2.0" }
@@ -46,13 +46,17 @@ memory = [
 mcp = [
     "mcp[cli]>=1.0",
 ]
+fast-memory = [
+    "hnswlib>=0.7",
+]
 bench = [
     "datasets>=2.14",
     "pandas>=2.0",
 ]
 all = [
     "agent-immune[memory]",
     "agent-immune[mcp]",
+    "agent-immune[fast-memory]",
 ]
 dev = [
     "pytest>=7.0",

src/agent_immune/__init__.py

@@ -8,7 +8,7 @@
 from agent_immune.observability import MetricsCollector
 from agent_immune.rate_limiter import CircuitBreaker
 
-__version__ = "0.1.1"
+__version__ = "0.2.0"
 
 __all__ = [
     "AdaptiveImmuneSystem",

src/agent_immune/core/decomposer.py

@@ -20,6 +20,17 @@
     re.I,
 )
 
+_NON_LATIN_RE = re.compile(
+    r"[\u0600-\u06ff"   # Arabic
+    r"\u0900-\u097f"    # Devanagari (Hindi)
+    r"\u1780-\u17ff"    # Khmer
+    r"\u3040-\u309f"    # Hiragana
+    r"\u30a0-\u30ff"    # Katakana
+    r"\u4e00-\u9fff"    # CJK Unified Ideographs
+    r"\uac00-\ud7af"    # Hangul Syllables
+    r"]"
+)
+
 _QUOTE_REGIONS = re.compile(
     r"(`{1,3}[^`]+`{1,3}|\"[^\"]{0,500}\"|'[^']{0,500}')",
     re.DOTALL,
@@ -36,7 +47,8 @@ def _inside_quoted(text: str, start: int, end: int) -> bool:
 class InputDecomposer:
     """Scan normalized text for threat patterns and produce scores plus redacted clean text."""
 
-    def __init__(self) -> None:
+    def __init__(self, detect_indirect_injection: bool = True) -> None:
+        self._detect_indirect = detect_indirect_injection
         self._injection: List[Tuple[str, re.Pattern[str], str]] = [
             (
                 "injection_ignore",
@@ -217,7 +229,67 @@ def __init__(self) -> None:
             ),
             (
                 "injection_ru_forget",
-                re.compile(r"забудьте\s+все", re.I),
+                re.compile(r"з[аa]б[уy]дьт[еe]\s+вс[еe]|з[аa]б[уy]дьт[еe]\s+в[сc][еe]", re.I),
+                "high",
+            ),
+            # Chinese
+            (
+                "injection_zh_ignore",
+                re.compile(r"忽略(所有|之前的?|以上的?)(指令|指示|规则|说明)"),
+                "high",
+            ),
+            (
+                "injection_zh_forget",
+                re.compile(r"忘记(之前|以前|上面)(的)?(指令|指示|规则|内容|一切)"),
+                "high",
+            ),
+            (
+                "injection_zh_you_are_now",
+                re.compile(r"你现在是.{0,20}(无限制|黑客|恶意|管理员|不受约束)"),
+                "high",
+            ),
+            # Japanese
+            (
+                "injection_ja_ignore",
+                re.compile(r"(すべての|以前の|上記の)(指示|指令|ルール|命令)を(無視|忘れ)"),
+                "high",
+            ),
+            (
+                "injection_ja_forget",
+                re.compile(r"(以前|これまで)の(指示|指令|ルール)を忘れ"),
+                "high",
+            ),
+            # Korean
+            (
+                "injection_ko_ignore",
+                re.compile(r"(모든|이전의?|위의)\s*(지시|지침|규칙|명령)을?\s*(무시|잊어)"),
+                "high",
+            ),
+            (
+                "injection_ko_forget",
+                re.compile(r"(이전|위)\s*(지시|지침)를?\s*잊어"),
+                "high",
+            ),
+            # Arabic
+            (
+                "injection_ar_ignore",
+                re.compile(r"تجاهل\s+(جميع\s+)?التعليمات"),
+                "high",
+            ),
+            (
+                "injection_ar_forget",
+                re.compile(r"(انس[َ]?|تناس[َ]?)\s+(جميع\s+)?التعليمات\s*(السابقة)?"),
+                "high",
+            ),
+            # Hindi
+            (
+                "injection_hi_ignore",
+                re.compile(r"(सभी|पिछले|पहले\s+के)\s*(निर्देशों?|नियमों?)\s*को\s*(अनदेखा|नज़रअंदाज़)\s*कर"),
+                "high",
+            ),
+            (
+                "injection_hi_forget",
+                re.compile(r"पिछले\s*(निर्देश|नियम|आदेश)\s*भूल\s*जाओ"),
                 "high",
             ),
             (
@@ -316,12 +388,43 @@ def __init__(self) -> None:
             ("delim_chatml", re.compile(r"<\|im_start\|>|<\|system\|>|<\|redacted_im_end\|>", re.I), "high"),
             ("delim_dash", re.compile(r"---\s*system\s*---", re.I), "medium"),
         ]
+        self._indirect: List[Tuple[str, re.Pattern[str], str]] = [
+            (
+                "indirect_html_comment",
+                re.compile(r"<!--\s*.{0,60}?(ignore|forget|override|new\s+instructions?|system\s+prompt)\s*.{0,60}?-->", re.I),
+                "medium",
+            ),
+            (
+                "indirect_md_comment",
+                re.compile(r"\[//\]:\s*#\s*\(.{0,100}?(ignore|forget|override|new\s+instructions?)\)", re.I),
+                "medium",
+            ),
+            (
+                "indirect_confused_deputy",
+                re.compile(
+                    r"\b(the\s+user\s+has\s+authorized|as\s+per\s+the\s+admin'?s?\s+request|"
+                    r"the\s+administrator\s+has\s+approved|per\s+user\s+instructions?\s*,?\s+override|"
+                    r"the\s+system\s+owner\s+wants\s+you\s+to)\b",
+                    re.I,
+                ),
+                "low",
+            ),
+            (
+                "indirect_url_payload",
+                re.compile(
+                    r"https?://[^\s]*[#?&][^\s]*(ignore|forget|override|system\s*prompt|new\s*instruct)",
+                    re.I,
+                ),
+                "medium",
+            ),
+        ]
 
         self._weight_injection = 0.35
         self._weight_exfil = 0.40
         self._weight_secret = 0.25
         self._weight_escalation = 0.20
         self._weight_delimiter = 0.15
+        self._weight_indirect = 0.15
 
     def decompose(self, norm: NormalizationResult) -> DecompositionResult:
         """
@@ -344,6 +447,7 @@ def decompose(self, norm: NormalizationResult) -> DecompositionResult:
             "secret": [],
             "escalation": [],
             "delimiter": [],
+            "indirect": [],
         }
         payload_spans: List[Tuple[int, int]] = []
 
@@ -359,7 +463,7 @@ def scan_group(
                     start, end = m.span()
                     inside = _inside_quoted(text, start, end)
                     mult = 0.15 if inside else 1.0
-                    sev_f = 1.0 if sev == "high" else 0.6
+                    sev_f = 1.0 if sev == "high" else (0.3 if sev == "low" else 0.6)
                     score_acc += sev_f * mult
                     bucket.append(PatternHit(
                         pattern_idx=idx,
@@ -377,26 +481,34 @@ def scan_group(
         sec_hits = scan_group(self._secret, "secret")
         esc_hits = scan_group(self._escalation, "escalation")
         _ = scan_group(self._delimiter, "delimiter")
+        ind_hits = 0.0
+        if self._detect_indirect:
+            ind_hits = scan_group(self._indirect, "indirect")
 
         injection_hits = hit_buckets["injection"]
         delimiter_hits = hit_buckets["delimiter"]
+        indirect_hits = hit_buckets["indirect"]
 
         pattern_linear = min(
             1.0,
             min(1.0, inj_hits * 0.22) * self._weight_injection
             + min(1.0, exf_hits * 0.25) * self._weight_exfil
             + min(1.0, sec_hits * 0.3) * self._weight_secret
             + min(1.0, esc_hits * 0.35) * self._weight_escalation
-            + (min(1.0, len(delimiter_hits) * 0.5) * self._weight_delimiter),
+            + (min(1.0, len(delimiter_hits) * 0.5) * self._weight_delimiter)
+            + min(1.0, ind_hits * 0.25) * self._weight_indirect,
         )
-        total_threat_hits = sum(len(hit_buckets[c]) for c in ("injection", "exfiltration", "secret", "escalation"))
+        total_threat_hits = sum(len(hit_buckets[c]) for c in ("injection", "exfiltration", "secret", "escalation", "indirect"))
         hit_boost = min(0.6, 0.18 * (total_threat_hits + len(delimiter_hits)))
 
         khmer_chars = len(_KHMER_RE.findall(text))
         khmer_ratio = khmer_chars / max(1, len(text))
+        non_latin_chars = len(_NON_LATIN_RE.findall(text))
+        non_latin_ratio = non_latin_chars / max(1, len(text))
         language_mixing_score = 0.0
-        if khmer_ratio > 0.10 and _ENGLISH_IMPERATIVES.search(text):
-            language_mixing_score = min(1.0, khmer_ratio + 0.3)
+        mixing_ratio = max(khmer_ratio, non_latin_ratio)
+        if mixing_ratio > 0.10 and _ENGLISH_IMPERATIVES.search(text):
+            language_mixing_score = min(1.0, mixing_ratio + 0.3)
             pattern_linear = min(1.0, pattern_linear + 0.2)
 
         injection_score = min(1.0, pattern_linear + language_mixing_score * 0.25 + hit_boost)
@@ -415,6 +527,7 @@ def scan_group(
             secret_hits=hit_buckets["secret"],
             escalation_hits=hit_buckets["escalation"],
             delimiter_hits=delimiter_hits,
+            indirect_hits=indirect_hits,
             payload_spans=merged_spans,
         )
         logger.debug("decompose score=%s hits=%s", injection_score, total_threat_hits)

src/agent_immune/core/models.py

@@ -36,6 +36,8 @@ class SecurityPolicy(BaseModel):
     memory_confirm_threshold: float = Field(default=0.90, ge=0.0, le=1.0, description="Similarity to confirmed memory entry that forces BLOCK")
     memory_review_threshold: float = Field(default=0.82, ge=0.0, le=1.0, description="Similarity that upgrades to REVIEW when combined with patterns")
     escalation_upgrade: bool = Field(default=True, description="Whether escalation detection upgrades action severity")
+    detect_indirect_injection: bool = Field(default=True, description="Enable low-severity indirect injection patterns (confused deputy, HTML comments)")
+    output_scanner_config: Optional[OutputScannerConfig] = Field(default=None, description="Override output scanner weights; uses defaults if None")
     max_sessions: int = Field(default=10000, ge=1, description="LRU cap on session accumulator registry")
 
     model_config = ConfigDict(frozen=True)
@@ -74,12 +76,28 @@ class DecompositionResult(BaseModel):
     secret_hits: List[PatternHit] = Field(default_factory=list)
     escalation_hits: List[PatternHit] = Field(default_factory=list)
     delimiter_hits: List[PatternHit] = Field(default_factory=list)
+    indirect_hits: List[PatternHit] = Field(default_factory=list)
     payload_spans: List[Tuple[int, int]] = Field(default_factory=list)
 
     @property
     def all_hits(self) -> List[PatternHit]:
         """All pattern hits across categories."""
-        return self.injection_hits + self.exfiltration_hits + self.secret_hits + self.escalation_hits + self.delimiter_hits
+        return self.injection_hits + self.exfiltration_hits + self.secret_hits + self.escalation_hits + self.delimiter_hits + self.indirect_hits
+
+
+class OutputScannerConfig(BaseModel):
+    """Per-category weights and thresholds for the output scanner."""
+
+    pii_weight: float = Field(default=0.35, ge=0.0, le=1.0, description="Score increment per PII match")
+    credential_weight: float = Field(default=0.45, ge=0.0, le=1.0, description="Score increment per credential match")
+    leak_weight: float = Field(default=0.40, ge=0.0, le=1.0, description="Score increment for system prompt leak phrases")
+    base64_weight: float = Field(default=0.25, ge=0.0, le=1.0, description="Score increment for decoded base64 blobs")
+    hex_weight: float = Field(default=0.20, ge=0.0, le=1.0, description="Score increment for long hex blobs")
+    data_uri_weight: float = Field(default=0.25, ge=0.0, le=1.0, description="Score increment for data URIs")
+    url_exfil_weight: float = Field(default=0.20, ge=0.0, le=1.0, description="Score increment for long URL query strings")
+    volume_weight: float = Field(default=0.15, ge=0.0, le=1.0, description="Score increment for volume anomalies")
+
+    model_config = ConfigDict(frozen=True)
 
 
 class OutputScanResult(BaseModel):