Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 49 additions & 2 deletions docs/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -1713,8 +1713,8 @@ Sources
-------

Sources connect read-only external knowledge bases to Signet recall without
turning them into ordinary saved memories. Obsidian is the currently supported
source kind.
turning them into ordinary saved memories. Supported source kinds include
Obsidian vaults and Discord guilds.

### GET /api/sources

Expand Down Expand Up @@ -1772,6 +1772,52 @@ chunk embeddings to its own database.
}
```

### POST /api/sources/discord

Add or update a Discord source. The daemon indexes configured Discord sources
on startup through the Discord REST API v10 using a bot token stored in Signet
Secrets. The source remains read-only; Signet stores only derived source
chunks, embeddings, and graph rows.

**Request body**

```json
{
"guildIds": ["123456789012345678"],
"tokenRef": "DISCORD_BOT_TOKEN",
"name": "Team Discord",
"channelFilter": ["general", "987654321098765432"],
"maxMessagesPerChannel": 1000,
"includeThreads": true,
"since": "2026-01-01T00:00:00.000Z"
}
```

`guildIds` and `tokenRef` are required. `channelFilter`, `maxMessagesPerChannel`,
`includeThreads`, and `since` are optional. `maxMessagesPerChannel` must be an
integer from `1` through `10000`; `since` must be a valid ISO date.

**Response**

```json
{
"source": {
"id": "discord:abc123",
"kind": "discord",
"settings": {
"guildIds": ["123456789012345678"],
"tokenRef": "DISCORD_BOT_TOKEN",
"maxMessagesPerChannel": 1000,
"includeThreads": true,
"since": "2026-01-01T00:00:00.000Z"
}
},
"created": true,
"indexed": 0,
"queued": false
}
```

### DELETE /api/sources/:sourceId

Remove a source config and purge Signet-owned source artifacts, graph rows,
Expand Down Expand Up @@ -4489,6 +4535,7 @@ silently disappear from the API reference.
| GET | `/api/sources` | platform/daemon/src/routes/sources-routes.ts |
| POST | `/api/sources/pick-directory` | platform/daemon/src/routes/sources-routes.ts |
| POST | `/api/sources/obsidian` | platform/daemon/src/routes/sources-routes.ts |
| POST | `/api/sources/discord` | platform/daemon/src/routes/sources-routes.ts |
| DELETE | `/api/sources/:sourceId` | platform/daemon/src/routes/sources-routes.ts |
| GET | `/api/knowledge/entities` | platform/daemon/src/routes/knowledge-routes.ts |
| POST | `/api/knowledge/entities/:id/pin` | platform/daemon/src/routes/knowledge-routes.ts |
Expand Down
40 changes: 36 additions & 4 deletions docs/SOURCES.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Sources"
description: "Connect read-only knowledge bases like Obsidian vaults directly into Signet recall."
description: "Connect read-only knowledge bases like Obsidian vaults and Discord servers directly into Signet recall."
order: 9
section: "Core Concepts"
---
Expand All @@ -10,7 +10,7 @@ Sources

Sources are external knowledge bases that Signet can read, index, and recall from without turning them into ordinary saved memories.

The first Sources connector is **Obsidian**. Point Signet at an Obsidian vault and the daemon mounts that vault as a read-only knowledge base: Markdown files become searchable artifacts, the vault structure becomes graph topology, and heading-aware chunks participate in semantic recall.
Sources currently support **Obsidian** vaults and **Discord** servers. Point Signet at an Obsidian vault and the daemon mounts that vault as a read-only knowledge base: Markdown files become searchable artifacts, the vault structure becomes graph topology, and heading-aware chunks participate in semantic recall. Add a Discord server and Signet indexes guild, channel, thread, and conversation context through the live Discord REST API.

The important rule is simple: **the source stays canonical**. Signet reads from the vault. It does not edit notes, rewrite frontmatter, create files, or move anything inside the source directory.

Expand All @@ -22,6 +22,7 @@ Saved memories are durable facts that Signet owns. Sources are different: they a
Use Sources when you want Signet to recall from:

- an Obsidian vault;
- Discord guilds and channels;
- a local folder of Markdown knowledge;
- documentation or research notes that should stay under their original editor/workflow;
- future cloud, code, or document connectors.
Expand Down Expand Up @@ -50,6 +51,33 @@ Signet intentionally skips vault metadata and local agent scratch space:
- `.trash/`
- `.hermes/`

Discord v1
----------

Discord Sources v1 indexes opted-in guilds through Discord's REST API v10 using a bot token stored in Signet Secrets:

```bash
signet sources add discord --guild-id 123456789012345678 --token-ref DISCORD_BOT_TOKEN --name "Team Discord"
signet sources add discord --guild-id 123456789012345678 --token-ref DISCORD_BOT_TOKEN --channel-filter general --since 2026-01-01
signet sources list
signet sources remove discord:...
```

The token reference is resolved through Signet Secrets at daemon sync time. Do not paste raw Discord bot tokens into source config. The bot must have access to the guilds and channels you want indexed.

Discord source settings:

| Setting | Description |
|---------|-------------|
| `guildIds` / `--guild-id` | One or more Discord guild IDs to index. |
| `tokenRef` / `--token-ref` | Signet secret reference containing the Discord bot token. |
| `channelFilter` / `--channel-filter` | Optional channel names or IDs to include. |
| `maxMessagesPerChannel` / `--max-messages` | Maximum messages fetched per channel or thread. Defaults to `1000`; capped at `10000`. |
| `includeThreads` / `--no-threads` | Include active and public archived threads by default. |
| `since` / `--since` | Optional ISO date lower bound. Signet converts it to a Discord snowflake message bound. |

Discord source graph rows use stable Discord IDs for canonical keys and display names only as labels. That keeps participants stable across renames and avoids merging unrelated users who share a display name.

What gets indexed
-----------------

Expand Down Expand Up @@ -103,7 +131,7 @@ This means a recall can return either a whole source artifact or a tighter sourc
Recall behavior
---------------

When you recall against Signet, Obsidian source results can appear alongside native memories. Source hits are labeled so callers can tell them apart:
When you recall against Signet, source results can appear alongside native memories. Source hits are labeled so callers can tell them apart:

```json
{
Expand All @@ -113,6 +141,8 @@ When you recall against Signet, Obsidian source results can appear alongside nat
}
```

Discord source chunk results use `type = "source_discord_chunk"` and source-owned chunk IDs that include guild, channel, and optional thread identifiers.

For whole-file artifact hits, the content includes a visible header like:

```text
Expand Down Expand Up @@ -161,6 +191,7 @@ The daemon exposes the Sources lifecycle under `/api/sources`:
|--------|------|-------------|
| `GET` | `/api/sources` | List configured sources. |
| `POST` | `/api/sources/obsidian` | Add/update an Obsidian vault source and index it. |
| `POST` | `/api/sources/discord` | Add/update a Discord source. The daemon indexes it on startup. |
| `DELETE` | `/api/sources/:sourceId` | Remove a source config and purge Signet-owned source rows. |
| `POST` | `/api/sources/pick-directory` | Development/browser fallback for choosing a local directory. |

Expand All @@ -169,9 +200,10 @@ The desktop shell uses native folder selection through IPC. The daemon picker ro
Limitations in v1
-----------------

- Obsidian is the first supported source connector.
- Sources are local/operator-managed. Permissions and RBAC are intentionally out of scope for v1.
- Signet does not write back to Obsidian.
- Signet does not write to Discord.
- Discord indexing depends on bot visibility and Discord API rate limits.
- Rename handling is delete + add.
- Non-Markdown Obsidian attachments are not indexed by the Obsidian v1 source path.

Expand Down
5 changes: 5 additions & 0 deletions platform/core/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -221,18 +221,23 @@ export type {
WorkspaceSourceRepoSyncResult,
} from "./workspace-source-repo";
export {
addDiscordSource,
addObsidianSource,
DEFAULT_DISCORD_MAX_MESSAGES,
DEFAULT_OBSIDIAN_EXCLUDE_GLOBS,
getAgentsDir,
getSourcesConfigPath,
loadSourcesConfig,
markSourceIndexed,
parseDiscordSettings,
removeSource,
saveSourcesConfig,
} from "./sources-config";
export type {
AddDiscordSourceInput,
AddObsidianSourceInput,
AddSourceResult,
DiscordSourceSettings,
RemoveSourceResult,
SignetSourceEntry,
SignetSourceKind,
Expand Down
145 changes: 145 additions & 0 deletions platform/core/src/sources-config.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,12 @@ import { tmpdir } from "node:os";
import { join } from "node:path";
import {
DEFAULT_OBSIDIAN_EXCLUDE_GLOBS,
addDiscordSource,
addObsidianSource,
getSourcesConfigPath,
loadSourcesConfig,
markSourceIndexed,
parseDiscordSettings,
removeSource,
} from "./sources-config";

Expand Down Expand Up @@ -152,4 +154,147 @@ describe("sources-config", () => {
if (removed.ok === true) throw new Error("expected removeSource to fail");
expect(removed.error).toContain("not found");
});

describe("discord source", () => {
it("adds a Discord source with guild IDs and token ref", () => {
const agentsDir = tmp();
const result = addDiscordSource(
{
guildIds: ["123456789012345678"],
tokenRef: "DISCORD_BOT_TOKEN",
name: "My Discord Server",
now: "2026-01-01T00:00:00.000Z",
},
agentsDir,
);

expect(result.ok).toBe(true);
if (result.ok === false) throw new Error(result.error);
expect(result.created).toBe(true);
expect(result.source.kind).toBe("discord");
expect(result.source.mode).toBe("read-only");
expect(result.source.name).toBe("My Discord Server");
expect(result.source.root).toBe("");
expect(result.source.settings).toBeDefined();
expect((result.source.settings as { guildIds: string[] }).guildIds).toEqual(["123456789012345678"]);
expect((result.source.settings as { tokenRef: string }).tokenRef).toBe("DISCORD_BOT_TOKEN");
});

it("rejects missing guild IDs", () => {
const agentsDir = tmp();
const result = addDiscordSource({ guildIds: [], tokenRef: "TOKEN" }, agentsDir);
expect(result.ok).toBe(false);
if (result.ok === true) throw new Error("expected failure");
expect(result.error).toContain("guild ID");
});

it("rejects missing token ref", () => {
const agentsDir = tmp();
const result = addDiscordSource({ guildIds: ["123456789012345678"], tokenRef: "" }, agentsDir);
expect(result.ok).toBe(false);
if (result.ok === true) throw new Error("expected failure");
expect(result.error).toContain("token");
});

it("rejects obvious raw Discord bot tokens", () => {
const agentsDir = tmp();
const result = addDiscordSource(
{
guildIds: ["123456789012345678"],
tokenRef: "MTEyMzQ1Njc4OTAxMjM0NTY3OA.c29tZWJvdA.abcdefghijklmnopqrstuvwxYZ0123456789",
},
agentsDir,
);
expect(result.ok).toBe(false);
if (result.ok === true) throw new Error("expected failure");
expect(result.error).toContain("secret reference");
});

it("rejects invalid guild ID format", () => {
const agentsDir = tmp();
const result = addDiscordSource({ guildIds: ["not-a-number"], tokenRef: "TOKEN" }, agentsDir);
expect(result.ok).toBe(false);
if (result.ok === true) throw new Error("expected failure");
expect(result.error).toContain("Invalid Discord guild ID");
});

it("deduplicates by sorted guild IDs", () => {
const agentsDir = tmp();
const first = addDiscordSource(
{ guildIds: ["111111111111111111"], tokenRef: "TOKEN", name: "Server A", now: "2026-01-01T00:00:00.000Z" },
agentsDir,
);
const second = addDiscordSource(
{ guildIds: ["111111111111111111"], tokenRef: "TOKEN2", name: "Server B", now: "2026-01-02T00:00:00.000Z" },
agentsDir,
);

expect(first.ok).toBe(true);
expect(second.ok).toBe(true);
if (second.ok === false) throw new Error(second.error);
expect(second.created).toBe(false);
expect(second.source.name).toBe("Server B");
expect(loadSourcesConfig(agentsDir).sources).toHaveLength(1);
});

it("parseDiscordSettings returns defaults for empty input", () => {
const settings = parseDiscordSettings(undefined);
expect(settings.guildIds).toEqual([]);
expect(settings.tokenRef).toBe("");
expect(settings.maxMessagesPerChannel).toBe(1000);
expect(settings.includeThreads).toBe(true);
});

it("parseDiscordSettings preserves valid fields", () => {
const settings = parseDiscordSettings({
guildIds: ["123"],
tokenRef: "MY_TOKEN",
channelFilter: ["general"],
maxMessagesPerChannel: 500,
includeThreads: false,
since: "2026-01-01",
});
expect(settings.guildIds).toEqual(["123"]);
expect(settings.tokenRef).toBe("MY_TOKEN");
expect(settings.channelFilter).toEqual(["general"]);
expect(settings.maxMessagesPerChannel).toBe(500);
expect(settings.includeThreads).toBe(false);
expect(settings.since).toBe("2026-01-01T00:00:00.000Z");
});

it("rejects invalid Discord message bounds and since dates", () => {
const agentsDir = tmp();
const invalidMax = addDiscordSource(
{ guildIds: ["123456789012345678"], tokenRef: "TOKEN", maxMessagesPerChannel: -1 },
agentsDir,
);
expect(invalidMax.ok).toBe(false);
if (invalidMax.ok === true) throw new Error("expected max message validation failure");
expect(invalidMax.error).toContain("maxMessagesPerChannel");

const invalidSince = addDiscordSource(
{ guildIds: ["123456789012345678"], tokenRef: "TOKEN", since: "not-a-date" },
agentsDir,
);
expect(invalidSince.ok).toBe(false);
if (invalidSince.ok === true) throw new Error("expected since validation failure");
expect(invalidSince.error).toContain("since");
});

it("removes a Discord source by id", () => {
const agentsDir = tmp();
const added = addDiscordSource(
{ guildIds: ["123456789012345678"], tokenRef: "TOKEN", now: "2026-01-01T00:00:00.000Z" },
agentsDir,
);
expect(added.ok).toBe(true);
if (added.ok === false) throw new Error(added.error);

const removed = removeSource(added.source.id, agentsDir);
expect(removed.ok).toBe(true);
if (removed.ok === false) throw new Error(removed.error);
expect(removed.source.id).toBe(added.source.id);
expect(loadSourcesConfig(agentsDir).sources).toEqual([]);
});
});
});
Loading
Loading