Logickos

Deterministic reasoning verification for LLMs using multi-agent decomposition and graph-theoretic auditing.

---

Abstract

Large Language Models increasingly rely on Chain-of-Thought reasoning to solve complex tasks. While this improves apparent performance, the reasoning process remains opaque, stochastic, and unverifiable. Existing evaluation approaches primarily rely on LLM-as-a-Judge — where one language model evaluates the reasoning of another. This is fundamentally unreliable: both models share probabilistic failure modes and are susceptible to fluent but logically inconsistent reasoning, making this approach unsuitable for high-stakes domains requiring formal correctness guarantees.

We present Logickos, a deterministic multi-agent framework for structural verification of LLM-generated reasoning traces. Our system employs a two-phase pipeline. In the first phase, a decomposition agent parses reasoning traces into atomic propositions and constructs a Directed Acyclic Graph (DAG) encoding logical dependencies between claims. In the second phase, fixed graph-theoretic algorithms — cycle detection, orphan node identification, and premise reachability analysis — are applied to formally audit the reasoning structure. Crucially, no LLM is involved in the verification phase, eliminating the probabilistic failure modes inherent to LLM-as-a-Judge approaches. Based on these checks, the system computes a Reasoning Consistency Score (RCS), a mathematically grounded scalar measure of logical validity. We evaluate Logickos on established reasoning benchmarks and demonstrate that it identifies structural reasoning failures that LLM-based evaluators consistently miss, particularly in cases where logically inconsistent reasoning is expressed with high fluency and surface-level coherence. Logickos is released as an open-source Python library, enabling plug-and-play reasoning audits for any LLM pipeline.

---

How It Works

LLM Reasoning Trace
        ↓
[ Decomposition Agent ]
  Parses trace into atomic propositions
  Extracts logical dependencies
        ↓
[ DAG Construction ]
  Nodes → atomic propositions
  Edges → logical dependencies
        ↓
[ Graph-Theoretic Audit ]
  ├── Cycle Detection       → catches circular logic
  ├── Orphan Node ID        → catches unsupported assumptions
  └── Reachability Analysis → catches reasoning drift
        ↓
[ Reasoning Consistency Score (RCS) ]
  Mathematically grounded scalar ∈ [0, 1]

---

Why Not LLM-as-a-Judge?

	LLM-as-a-Judge	Logickos
Affected by fluency	Yes	No
Shares failure modes with evaluated model	Yes	No
Mathematically grounded	No	Yes
Deterministic output	No	Yes
Catches circular logic formally	No	Yes
Suitable for high-stakes domains	Unreliable	Yes

---

Audit Checks

Cycle Detection Identifies circular reasoning where proposition A depends on B which depends back on A. Caught via DFS-based cycle detection on the DAG.

Orphan Node Identification Flags propositions that appear in the reasoning with no supporting premises — claims that came from nowhere.

Premise Reachability Analysis Verifies that the final conclusion is logically reachable from the stated premises through valid dependency edges.

---

Reasoning Consistency Score (RCS)

The RCS is a scalar value between 0 and 1 computed from the three audit checks:

RCS = w1 × (cycle score) + w2 × (orphan score) + w3 × (reachability score)

• 1.0 — fully consistent reasoning structure
• 0.0 — fundamentally broken reasoning structure

---

Roadmap

• Core decomposition agent
• DAG construction pipeline
• Cycle detection algorithm
• Orphan node identification
• Premise reachability analysis
• RCS scoring function
• DAG visualisation
• Benchmark evaluation (GSM8K, StrategyQA, ARC)
• pip package release
• Research paper

---