refactor: restructure source tree by concern, declarative request validation

CPU recognizer reaches GPU accuracy parity (FUNSD-50 F1 82.28% -> 85.52%,
GPU 85.33%): single-warp crop straight to recognizer input with the shared
32px width floor (was a 320px stretch + double resample), det preprocess in
BGR per model spec. Correctness fixes: logger OOB write, batch-det aspect
distortion, GPU vertical-text rec bucketing, gRPC lifecycle/shutdown,
PPM mid-flush delivery, table/markdown HTML escaping.

Split oversized modules into single-concern TUs and regroup directories
(table/{slanext,vlm}, formula/{ppformulanet,vlm,routing}, pdf/{render,text},
pipeline/{ocr,pdf}, engine/trt, backend_routing, markdown). Join duplicated
logic into shared helpers (pipeline result moves, reading-order assignment,
image-size guards, curl/base64/trim utilities, single slanext host decoder).

Request validation runs on a declarative per-endpoint spec: routing
overrides, text=0 and embed=0 that cannot be honored fail loud; other
unsupported params are tolerated with X-Ignored-Params/X-Deprecation
response headers (removal planned for v4) or rejected under
TURBO_OCR_STRICT_QUERY_PARAMS=1. Parameter classification reads the URL
query string only (POST bodies are no longer parsed as parameters).

Tests grow to 294 cases; docs updated.

Author: nataell95

.github/workflows/ci.yml

@@ -50,7 +50,7 @@ jobs:
           cmake --build build-cpu --target turboocr-cpu-server -j"$(nproc)"
       - name: Endpoint matrix smoke test
         run: |
-          HTTP_PORT=18080 GRPC_PORT=18051 HOST=127.0.0.1 OCR_MODEL=tiny \
+          PORT=18080 GRPC_PORT=18051 BIND_HOST=127.0.0.1 OCR_MODEL=tiny \
             ./build-cpu/turboocr-cpu-server &
           SERVER_PID=$!
           for i in $(seq 1 60); do

.gitignore

@@ -2,6 +2,7 @@
 /build_cpu/
 /build-cpu/
 /build-coverage/
+/build-*/
 /predeploy/
 __pycache__/
 .pytest_cache/

CMakeLists.txt

@@ -38,7 +38,7 @@ Parsed by `server::parse_query_options()` in
     emitted only when non-empty — text-only pages produce a response
     indistinguishable from the pre-feature shape. See
     `emit_pipeline_result_json` in
-    `include/turbo_ocr/common/serialization.h:567`.
+    `include/turbo_ocr/common/serialization/serialization_emit.h`.
 
 !!! warning "LAYOUT_DISABLED"
     Requesting `layout=1`, `reading_order=1`, `as_blocks=1`, `tables=1`,
@@ -515,19 +515,19 @@ counterpart of PP-StructureV3 `save_to_markdown`. See
 [Faithful Markdown export](../output_markdown.md) for the serialization rules.
 
 - **Body**: raw image bytes (same decoders as `/ocr/raw`).
-- **`embed` query** (default `true`): `embed=1` inlines figure/chart crops as
-  base64 `data:` URIs (self-contained `.md`); `embed=0` emits
-  `![](assets/blockN.png)` file-reference links (write the crops yourself).
+- **`embed` query** (default `true`): figure/chart crops are always inlined as
+  base64 `data:` URIs (self-contained `.md`). `embed=1` is accepted explicitly;
+  `embed=0` (file-reference links) is **rejected with
+  `400 INVALID_PARAMETER`** — the asset PNGs would be written to the server's
+  filesystem where an HTTP client cannot retrieve them. File-reference export
+  is available to library consumers via `render_markdown_with_assets`.
 - **Requires layout**: against a server started with `DISABLE_LAYOUT=1` the
   request returns `400 LAYOUT_DISABLED`.
 - **Response**: `text/markdown; charset=utf-8`.
 
 ```bash
 # self-contained markdown (images inline as data URIs)
 curl --data-binary @page.png http://localhost:8000/ocr/markdown > page.md
-
-# file-reference image links
-curl --data-binary @page.png 'http://localhost:8000/ocr/markdown?embed=0'
 ```
 
 Error codes: `EMPTY_BODY`, `LAYOUT_DISABLED`, `IMAGE_DECODE_FAILED`,

docs/api/http.md

@@ -184,6 +184,81 @@ At startup the server emits one structured INFO line (`Effective server
 config`) containing every resolved value — a single grep target for
 post-mortems. Recorded warnings are logged immediately after.
 
+
+## Expert / subsystem knobs
+
+These are read directly by their subsystem (not via `ServerConfig`, so they
+do not appear in `--print-config`). Defaults are tuned; override only with a
+measured reason.
+
+### Recognition / detection tuning
+
+| Variable | Default | Description |
+|---|---|---|
+| `REC_BATCH_N` | `32` | Recognition batch size per inference call. |
+| `REC_BUCKET_STEP` | `16` | CPU recognizer: snap crop widths UP to this step so batches pad each crop by at most step-1 columns. |
+| `REC_ZEROCOPY` | `1` | CPU recognizer: zero-copy batch view into ORT (`0` = copy path). |
+| `REC_SELFTEST` | `0` | CPU recognizer: one-shot batch-consistency self-test on first batch. |
+| `SIMD_CTC` | `1` | SIMD CTC argmax decode (`0` = scalar fallback). |
+| `DET_OPT_BATCH` | `8` | Batch dimension the det TRT profile is optimized for. |
+| `TURBO_DET_FUSED_PRE` | `1` | Fused GPU resize+normalize det preprocess (`0` = OpenCV path). |
+| `GPU_CCL` | `1` | Det post-process: `2` all-GPU JFA, `1` GPU CCL + CPU contours, `0` CPU contours. |
+| `GPU_BOX_THRESH` | model default | Override DB box threshold on the GPU path. |
+| `GPU_UNCLIP_SCALE` | `1.0` | Multiplier on the DB unclip ratio (GPU path). |
+| `CLS_BATCH` | `32` | Angle-classifier batch size. |
+| `MAX_IMAGE_PIXELS_MP` | `128` | Decompression-bomb cap: max decoded image area in megapixels. |
+| `MAX_BATCH_PIXELS_MP` | `2048` | Aggregate pixel cap across one /ocr/batch request. |
+
+### ONNX Runtime (CPU / formula backends)
+
+| Variable | Default | Description |
+|---|---|---|
+| `ORT_EP` | `cpu` | Execution provider for the CPU engine (`cpu` / `coreml`). |
+| `ORT_NUM_THREADS` | auto | Intra-op thread count per ORT session. |
+| `ORT_GLOBAL_THREADS` | auto | Shared global thread-pool size (with `ORT_SHARED_POOL=1`). |
+| `ORT_SHARED_POOL` | `1` | One shared ORT thread pool across sessions instead of per-session pools. |
+| `ORT_REC_OPT_CAP` | unset | Cap ORT graph-optimization level for the recognizer. |
+| `DISABLE_COREML` / `COREML_FLAGS` | unset | macOS CoreML EP opt-out / flags. |
+
+### Structure stages (tables / formulas / VLM sidecar)
+
+| Variable | Default | Description |
+|---|---|---|
+| `TABLE_CROP_MODE` | `layout` | `detunion` snaps each table region to the tight AABB of its det boxes. |
+| `TABLE_CROP_MARGIN` | `0.03` | Fractional expansion per table-region side before structure decode. |
+| `TABLE_MATCH_INTER` | `1` | Cell matcher: intersection-based OCR-fragment assignment. |
+| `TABLE_MATCH_FALLBACK` | `1` | Cell matcher: nearest-cell fallback for unmatched fragments. |
+| `TABLE_CLS_TRT`, `TABLE_SLANEXT_DICT`, `TABLE_SLANEXT_DECODER_BIN`, `TABLE_SLANEXT_WIRELESS_ENCODER_ONNX`, `TURBO_OCR_TABLE_DICT_PATH` | bundled paths | Override individual SLANeXt model/dict file locations. |
+| `PPFNS_CHUNK` | `0` | PP-FormulaNet-S decode chunk size (0 = single pass). |
+| `PPFNS_DROP_COLLAPSE` | `1` | Guard that drops collapsed (repeating) formula decodes. |
+| `VLM_BACKEND` | `pool` | `legacy` selects the per-request curl path instead of the shared async pool. |
+| `VLM_GLOBAL_CONCURRENCY` | `50` | Max in-flight VLM crop requests across the whole process. |
+| `VLM_MAX_RETRIES` | `2` | Retries per VLM crop on transient transport failures. |
+| `VLM_PNG_THREADS` | `4` | Threads PNG-encoding crops before VLM submit. |
+| `VLLM_BASE_URL` / `VLLM_MODEL` | `http://localhost:8000` / `PaddleOCR-VL-1.6-0.9B` | VLM sidecar endpoint and model id. |
+| `VLLM_FORMULA_PROMPT` / `VLLM_FORMULA_BATCH` / `VLLM_FORMULA_TIMEOUT_S` / `VLLM_FORMULA_MAX_TOKENS` | `Formula Recognition:` / `8` / `30` / `512` | Formula sidecar request shape. |
+| `VLLM_TABLE_BASE_URL` / `VLLM_TABLE_MODEL` / `VLLM_TABLE_PROMPT` / `VLLM_TABLE_BATCH` / `VLLM_TABLE_TIMEOUT_S` / `VLLM_TABLE_MAX_TOKENS` | formula equivalents / `Table Recognition:` / `8` / `60` / `4096` | Table sidecar request shape (falls back to the `VLLM_*` values). |
+| `TURBO_ROUTING_CONFIG` | env-synthesized | Path to a routing table JSON replacing the env-derived backend routing. |
+| `TURBO_ALLOW_ADHOC_BACKENDS` | `0` | Allow per-request backends outside the routing table. |
+
+### Server / PDF / misc
+
+| Variable | Default | Description |
+|---|---|---|
+| `BIND_HOST` | `0.0.0.0` | Bind address override. |
+| `GRPC_BATCH_GLOBAL_WORKERS` | `16` | Process-wide ceiling on extra gRPC batch fanout threads (each RPC keeps one guaranteed worker). |
+| `BATCH_FANOUT_GLOBAL_WORKERS` | `64` | Process-wide ceiling on extra CPU `/ocr/batch` fanout threads (each request keeps one guaranteed worker). |
+| `FINALIZE_DEFERRED_TIMEOUT_MS` | request timeout | Await budget for deferred (async VLM) structure results. |
+| `PDF_RENDER_REPLY_TIMEOUT_MS` | `120000` | Cap on waiting for a PDF daemon reply. |
+| `FASTPDF2PNG_PATH` | bundled | Path to the fastpdf2png daemon binary. |
+| `NVJPEG_DEVICE_COPY` | `1` | nvJPEG page-image encode keeps data device-side. |
+| `LAYOUT_KEEP_NESTED_CHILDREN` | `0` | Keep child layout blocks nested inside their parents. |
+| `TURBO_LAYOUT_DEBUG` | `0` | Verbose layout-stage debug output. |
+| `TURBO_OCR_STRICT_QUERY_PARAMS` | `0` | Opt-in: set `1` to reject with 400 any unknown parameter AND any known parameter the endpoint does not support. Default tolerates both (v3.4-compatible) — DEPRECATED: tolerated requests get an `X-Ignored-Params` + `X-Deprecation` response header and v4 rejects them with 400. Routing overrides, `text=0`, and `embed=0` on endpoints that cannot honor them are ALWAYS a 400 — ignoring those would falsify the response. |
+| `TURBO_OCR_DISABLE_MALLOC_REAPER` | `0` | Disable the periodic malloc_trim reaper thread. |
+| `ENABLE_TIMING` / `PROFILE_STAGES` | `0` | Per-stage timing output / CPU-path stage profiler. |
+| `TOCR_LOG_RATELIMIT` | `10:1000` | Per-call-site log rate limit `N[:WINDOW_MS]`; `0` disables. |
+
 !!! info "See also"
     - [Build → Docker](docker.md) — image env vars and the nginx front
       (`TURBO_OCR_PORT`, `MAX_BODY_MB`).

docs/build/config.md

@@ -0,0 +1,35 @@
+# Pre-commit checks
+
+Install once per clone:
+
+```bash
+bash scripts/install_hooks.sh
+```
+
+The hook (`scripts/git-hooks/pre-commit`) runs static checks on **staged
+files only** and is hard-capped at **20 seconds** — cheap greps first, then a
+time-boxed cppcheck with whatever budget remains.
+
+Each check maps to a defect class that has actually shipped in this repo:
+
+| Check | Why it exists |
+|---|---|
+| No `std::cout` / `std::cerr` in `src/` | Diagnostics bypassed the structured logger's level control and per-site rate limiting — request-path prints were a log-flood surface. |
+| No raw `getenv` in `src/`, `include/` | Config knobs were parsed by hand-pasted `to_int` copies with drifting semantics; `turbo_ocr::env` (common/env_utils.h) is the one parser. |
+| No `std::stoi` family | An unguarded `std::stoi` on a daemon reply threw `std::invalid_argument` past the error taxonomy. Use `std::from_chars` with explicit handling. |
+| No `file(GLOB)` in CMake | Explicit source lists only. |
+| No orphaned `CMakeLists.txt` | Four dead ones misled contributors about the build layout. |
+| cppcheck (warning/performance/portability) | General static net, staged `.cpp` only, time-boxed. |
+
+Deliberate exceptions are annotated in-line
+(`// pre-commit-allow-stream`, `// pre-commit-allow-getenv`,
+`// pre-commit-allow-stoi`) or, for a one-off commit, `git commit --no-verify`
+with the reason in the commit message.
+
+## File-length ratchet (max 500 lines)
+
+Any staged `.cpp/.h/.cu/.cuh` over 500 lines fails the commit. The limit is a
+ratchet: pre-existing oversized files trip it the first time they are touched,
+which is when they get split. Escape hatch for the rare genuinely-unsplittable
+file: `// pre-commit-allow-length` within the first 5 lines, next to a short
+justification.

docs/dev/pre_commit.md

@@ -84,7 +84,7 @@ classDiagram
   PaddleCls --> TrtEngine
 ```
 
-`compute_crop_transform` (in `turbo_ocr/common/perspective.h`) produces the
+`compute_crop_transform` (in `turbo_ocr/common/geometry/perspective.h`) produces the
 `M_inv` 3×3 matrix that lets a single CUDA kernel (`cuda_batch_roi_warp`) warp
 the quad directly into the classifier's fixed `80×160` input slot without an
 intermediate crop allocation.

docs/models/classification.md

@@ -137,11 +137,20 @@ void register_ocr_markdown_route_gpu(server::WorkPool &pool,
               "DISABLE_LAYOUT=1)"));
           return;
         }
-        // ?embed=0 -> file-ref image links (assets written by an out-of-band
-        // exporter); default true -> self-contained base64 data: URIs.
-        bool embed = true;
-        if (auto p = req->getParameter("embed"); p == "0" || p == "false")
-          embed = false;
+        // HTTP is always self-contained base64 data: URIs. The file-ref mode
+        // (embed_images=false) writes asset PNGs to the SERVER's filesystem —
+        // unreachable for an HTTP client — so ?embed=0 is rejected with a loud
+        // 400 INVALID_PARAMETER instead of being silently overridden. File-ref
+        // markdown remains available to library/CLI consumers of
+        // render_markdown_with_assets.
+        if (auto p = req->getParameter("embed"); p == "0" || p == "false") {
+          callback(server::error_response(
+              drogon::k400BadRequest, "INVALID_PARAMETER",
+              "embed=0 (file-ref markdown) is not supported over HTTP; "
+              "assets are always embedded as data: URIs"));
+          return;
+        }
+        const bool embed = true;
 
         server::submit_work(pool, std::move(callback),
             [req, &dispatcher, &decode, embed](server::DrogonCallback &cb) {

docs/output_markdown.md

@@ -27,7 +27,7 @@
 // openai backend at a proprietary/account-gated endpoint is the operator's
 // explicit choice.
 
-namespace turbo_ocr::routing {
+namespace turbo_ocr::backend_routing {
 
 enum class Kind { Local, Openai };
 
@@ -110,4 +110,4 @@ const char *kind_name(Kind k) noexcept;
 const char *parser_name(Parser p) noexcept;
 bool        parse_parser(const std::string &s, Parser &out) noexcept;
 
-} // namespace turbo_ocr::routing
+} // namespace turbo_ocr::backend_routing

…clude/turbo_ocr/routing/routing_config.h …rbo_ocr/backend_routing/routing_config.hinclude/turbo_ocr/routing/routing_config.h renamed to include/turbo_ocr/backend_routing/routing_config.h include/turbo_ocr/routing/routing_config.h renamed to include/turbo_ocr/backend_routing/routing_config.h

@@ -7,7 +7,7 @@
 #include <opencv2/core.hpp>
 
 #include "turbo_ocr/engine/cpu_engine.h"
-#include "turbo_ocr/common/box.h"
+#include "turbo_ocr/common/geometry/box.h"
 
 namespace turbo_ocr::classification {