Skip to content

LongMemEval harness: missing configureGateway() prevents hybrid search #940

Description

@ccxu1990

Bug Description

gbrain eval longmemeval can only run in --keyword-only mode. Without that flag, every question fails with AI gateway is not configured, making hybrid (keyword + vector) search impossible in the benchmark.

Root Cause

src/eval/longmemeval/harness.ts::createBenchmarkBrain() creates an in-memory PGLite engine but never calls configureGateway(). The benchmark harness is intentionally hermetic (doesn't touch the user's real brain), but it also doesn't bridge the user's embedding provider config into the in-memory brain.

The call chain:

  1. createBenchmarkBrain()engine.connect({})engine.initSchema()done (no gateway)
  2. Later, hybridSearch()embed()requireConfig()throws "AI gateway is not configured"

cli.ts always calls configureGateway(buildGatewayConfig(config)) before connectEngine(), but the eval path bypasses connectEngine() entirely.

Steps to Reproduce

# This works (keyword-only)
gbrain eval longmemeval dataset.jsonl --retrieval-only --keyword-only --limit 5

# This fails — every question throws "AI gateway is not configured"
gbrain eval longmemeval dataset.jsonl --retrieval-only --limit 5

Expected Behavior

gbrain eval longmemeval should support hybrid search by reading the user's embedding provider config from ~/.gbrain/config.json and process.env, then passing it through to configureGateway() inside the in-memory benchmark brain.

Proposed Fix

Two files need changes:

1. src/eval/longmemeval/harness.ts

import type { AIGatewayConfig } from '../../core/ai/types.ts';

export async function createBenchmarkBrain(
  gatewayConfig?: AIGatewayConfig
): Promise<PGLiteEngine> {
  const engine = new PGLiteEngine();
  // Configure gateway BEFORE initSchema — initSchema reads embedding dims
  // from the gateway to size the pgvector column.
  if (gatewayConfig) {
    const { configureGateway } = await import('../../core/ai/gateway.ts');
    configureGateway(gatewayConfig);
  }
  await engine.connect({});
  await engine.initSchema();
  return engine;
}

export async function withBenchmarkBrain<T>(
  fn: (engine: PGLiteEngine) => Promise<T>,
  gatewayConfig?: AIGatewayConfig,
): Promise<T> {
  const engine = await createBenchmarkBrain(gatewayConfig);
  // ...
}

2. src/commands/eval-longmemeval.ts

Before withBenchmarkBrain(), build AIGatewayConfig from the user's config:

import { loadConfig } from '../core/config.ts';
import type { AIGatewayConfig } from '../core/ai/types.ts';

let gatewayConfig: AIGatewayConfig | undefined;
if (!opts.keywordOnly) {
  const userConfig = loadConfig();
  if (userConfig?.embedding_model) {
    gatewayConfig = {
      embedding_model: userConfig.embedding_model,
      embedding_dimensions: userConfig.embedding_dimensions,
      base_urls: userConfig.provider_base_urls,
      env: { ...process.env as Record<string, string | undefined> },
    };
  } else {
    // Fall back to keyword-only if no embedding config available
    opts.keywordOnly = true;
  }
}
await withBenchmarkBrain(async (engine) => { ... }, gatewayConfig);

Key detail: configureGateway() must be called before initSchema(), otherwise the pgvector column defaults to 1536 dims (OpenAI default) and every embed call fails with dimension mismatch.

Verified Results

Tested with DashScope text-embedding-v3 (1024 dims), 50 LongMemEval temporal-reasoning questions:

Mode Recall
keyword-only (before fix) 22/50 (44.0%)
hybrid (after fix) 50/50 (100.0%)

Environment

  • gbrain version: 0.32.5
  • Engine: PGLite (in-memory)
  • Embedding provider: DashScope text-embedding-v3 (1024d)

Metadata

Metadata

Assignees

No one assigned

    Labels

    fix-neededConfirmed/likely bug worth fixingp1P1: important bug / high-priority fix

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions