Bug Description
gbrain eval longmemeval can only run in --keyword-only mode. Without that flag, every question fails with AI gateway is not configured, making hybrid (keyword + vector) search impossible in the benchmark.
Root Cause
src/eval/longmemeval/harness.ts::createBenchmarkBrain() creates an in-memory PGLite engine but never calls configureGateway(). The benchmark harness is intentionally hermetic (doesn't touch the user's real brain), but it also doesn't bridge the user's embedding provider config into the in-memory brain.
The call chain:
createBenchmarkBrain() → engine.connect({}) → engine.initSchema() → done (no gateway)
- Later,
hybridSearch() → embed() → requireConfig() → throws "AI gateway is not configured"
cli.ts always calls configureGateway(buildGatewayConfig(config)) before connectEngine(), but the eval path bypasses connectEngine() entirely.
Steps to Reproduce
# This works (keyword-only)
gbrain eval longmemeval dataset.jsonl --retrieval-only --keyword-only --limit 5
# This fails — every question throws "AI gateway is not configured"
gbrain eval longmemeval dataset.jsonl --retrieval-only --limit 5
Expected Behavior
gbrain eval longmemeval should support hybrid search by reading the user's embedding provider config from ~/.gbrain/config.json and process.env, then passing it through to configureGateway() inside the in-memory benchmark brain.
Proposed Fix
Two files need changes:
1. src/eval/longmemeval/harness.ts
import type { AIGatewayConfig } from '../../core/ai/types.ts';
export async function createBenchmarkBrain(
gatewayConfig?: AIGatewayConfig
): Promise<PGLiteEngine> {
const engine = new PGLiteEngine();
// Configure gateway BEFORE initSchema — initSchema reads embedding dims
// from the gateway to size the pgvector column.
if (gatewayConfig) {
const { configureGateway } = await import('../../core/ai/gateway.ts');
configureGateway(gatewayConfig);
}
await engine.connect({});
await engine.initSchema();
return engine;
}
export async function withBenchmarkBrain<T>(
fn: (engine: PGLiteEngine) => Promise<T>,
gatewayConfig?: AIGatewayConfig,
): Promise<T> {
const engine = await createBenchmarkBrain(gatewayConfig);
// ...
}
2. src/commands/eval-longmemeval.ts
Before withBenchmarkBrain(), build AIGatewayConfig from the user's config:
import { loadConfig } from '../core/config.ts';
import type { AIGatewayConfig } from '../core/ai/types.ts';
let gatewayConfig: AIGatewayConfig | undefined;
if (!opts.keywordOnly) {
const userConfig = loadConfig();
if (userConfig?.embedding_model) {
gatewayConfig = {
embedding_model: userConfig.embedding_model,
embedding_dimensions: userConfig.embedding_dimensions,
base_urls: userConfig.provider_base_urls,
env: { ...process.env as Record<string, string | undefined> },
};
} else {
// Fall back to keyword-only if no embedding config available
opts.keywordOnly = true;
}
}
await withBenchmarkBrain(async (engine) => { ... }, gatewayConfig);
Key detail: configureGateway() must be called before initSchema(), otherwise the pgvector column defaults to 1536 dims (OpenAI default) and every embed call fails with dimension mismatch.
Verified Results
Tested with DashScope text-embedding-v3 (1024 dims), 50 LongMemEval temporal-reasoning questions:
| Mode |
Recall |
| keyword-only (before fix) |
22/50 (44.0%) |
| hybrid (after fix) |
50/50 (100.0%) |
Environment
- gbrain version: 0.32.5
- Engine: PGLite (in-memory)
- Embedding provider: DashScope text-embedding-v3 (1024d)
Bug Description
gbrain eval longmemevalcan only run in--keyword-onlymode. Without that flag, every question fails withAI gateway is not configured, making hybrid (keyword + vector) search impossible in the benchmark.Root Cause
src/eval/longmemeval/harness.ts::createBenchmarkBrain()creates an in-memory PGLite engine but never callsconfigureGateway(). The benchmark harness is intentionally hermetic (doesn't touch the user's real brain), but it also doesn't bridge the user's embedding provider config into the in-memory brain.The call chain:
createBenchmarkBrain()→engine.connect({})→engine.initSchema()→ done (no gateway)hybridSearch()→embed()→requireConfig()→ throws "AI gateway is not configured"cli.tsalways callsconfigureGateway(buildGatewayConfig(config))beforeconnectEngine(), but the eval path bypassesconnectEngine()entirely.Steps to Reproduce
Expected Behavior
gbrain eval longmemevalshould support hybrid search by reading the user's embedding provider config from~/.gbrain/config.jsonandprocess.env, then passing it through toconfigureGateway()inside the in-memory benchmark brain.Proposed Fix
Two files need changes:
1.
src/eval/longmemeval/harness.ts2.
src/commands/eval-longmemeval.tsBefore
withBenchmarkBrain(), buildAIGatewayConfigfrom the user's config:Key detail:
configureGateway()must be called beforeinitSchema(), otherwise the pgvector column defaults to 1536 dims (OpenAI default) and every embed call fails with dimension mismatch.Verified Results
Tested with DashScope
text-embedding-v3(1024 dims), 50 LongMemEval temporal-reasoning questions:Environment