Add a source of messages

A source brings messages into Pepper. Once they are in Pepper's database (the store), the brief, Ask and the dashboard read them like any others, and they go through the same extraction as the chats you connect, so the model sees them. This example is a part (your own folder in src/parts/, see Make Pepper yours) that reads the notes file NOTES_FILE points to on your Mac and turns each line into a message.

import { readFileSync, existsSync } from "node:fs";
import { db, now } from "../../store/db.js";
import { registerSource, type Source } from "../../ingest/source.js";
import type { Part } from "../contract.js";

const FILE = process.env.NOTES_FILE ?? "";

export const source: Source = {
  name: "notes",
  configured: () => !!FILE,
  check: async () => {
    if (!existsSync(FILE)) throw new Error(`${FILE} is missing`);
    return FILE;
  },
  poll: async () => {
    const lines = readFileSync(FILE, "utf8").split("\n").filter(Boolean);
    const at = now();
    db.prepare("INSERT OR IGNORE INTO chats(source, chat_id, account_id, network, title, type, last_activity, updated_at) VALUES('notes','notes','notes','Notes','My notes','single',?,?)").run(at, at);
    const add = db.prepare("INSERT OR IGNORE INTO messages(source, message_id, chat_id, account_id, sender_name, is_sender, timestamp, type, text, ingested_at) VALUES('notes',?,'notes','notes','Me',1,?,'TEXT',?,?)");
    let inserted = 0;
    lines.forEach((text, i) => { inserted += Number(add.run(String(i), at, text, at).changes); });
    return { accounts: 1, chats: 1, seen: lines.length, inserted, updated: 0, deleted: 0, newest: at };
  },
};
registerSource(source);

export const part: Part = { name: "notes", about: "reads my notes file" };

Save it as src/parts/notes/index.ts, add notes: () => import("./notes/index.js"), to src/parts/mine.ts, put the file's full path in .env as NOTES_FILE, and restart Pepper. is_sender is 1 because the notes are yours: that is how Pepper tells your own words from other people's.

registerSource is the line that makes Pepper run poll on a schedule to pick up new lines. Each line is stored once, by its position in the file, and gets the time it was picked up: this example only ever adds lines, so editing, removing or reordering a line doesn't change what was stored. To show the source on the Connections page as well, also call registerConnection, with source set to the same object and check calling the source's check, as src/ingest/tactiq.ts does for the Tactiq meeting recorder. Without that, doctor never runs the file check above.

A part can't reach the network: npm run check scans the code and rejects a part that imports or calls anything that would, because a part reading your messages should not be able to send them anywhere. It is a check on the source code, not a wall around the running part. A source that needs a new online service belongs in Pepper's own code (the core), not in a part, so bring it up as an issue on the Pepper repository.