Skip to main content

Ingestion

External systems write into Oliver by sending normalized POST /events payloads to the Go API. The backend should be running before any ingestion pipeline pushes events.

There are two ingestion paths:

  • pipelines/: finite Python fetch jobs for sources that can be polled.
  • extensions/agent-hooks: local npm/Node collector for live Claude/Codex hook events.

Batch Pipelines​

Pipeline jobs live at the repository root in pipelines/. They fetch source records, normalize each record into an Oliver event, and post to the configured backend API. They do not import Go backend packages or call Go batch jobs.

Dry runs do not need backend credentials. Pushing to the current multi-tenant API requires a bearer token and workspace slug:

cd backend
go run ./cmd/issue-token \
  --kind system \
  --display-name "Pipeline Runner" \
  --workspace oliver \
  --workspace-name "Oliver"

Use the printed token as OLIVER_TOKEN and the workspace slug as OLIVER_WORKSPACE for clients that push events.

Current gap: the Python runner's push path still needs to be migrated to send those auth headers. Use --dry-run to validate pipeline output against the multi-tenant backend until that code path is updated.

From the repository root:

python3 pipelines/run_all.py --list
python3 pipelines/run_all.py --dry-run
python3 pipelines/run_all.py --api http://127.0.0.1:8081

Current finite fetch pipelines:

  • notion-pages
  • google-docs
  • slack-updates
  • github-updates

For per-source environment variables and examples, see pipelines/README.md.

Local Claude/Codex Agent Hooks​

Use the npm collector for live developer-agent events. It installs repo-local Claude and Codex hook config and stores native hook payloads in .oliver/agent-hooks.

The uploader normalizes queued payloads into local_agent_event records and posts them to authenticated POST /events. Browser login saves the local token and uses the shared default workspace/project for the MVP.

Install the CLI, register hooks in the target repo, and sign in. The CLI defaults to production; add --env local (or export OLIVER_ENV=local) to target a local stack:

npm install -g olivergraph
oliver install --env local
oliver login --env local

Environment variables still work for headless/dev fallback:

export OLIVER_API=http://127.0.0.1:8081
export OLIVER_TOKEN=usr_...
export OLIVER_WORKSPACE=default

extensions/agent-hooks/.env.example is the directory-local template for those variables.

You can try a manual flush to inspect the current uploader behavior:

oliver flush

The periodic uploader runs every five minutes by default, or every OLIVER_AGENT_PUSH_INTERVAL_SECONDS seconds when that environment variable is set.

oliver start

The older Go hook listener is still available for direct HTTP hook setups and dry-run inspection:

cd backend

go run ./cmd/agent-hook-listener \
  --project oliver \
  --port 8765 \
  --dry-run \
  --verbose

Run the compatibility listener:

go run ./cmd/agent-hook-listener \
  --project oliver \
  --port 8765 \
  --verbose

For step-by-step Claude/Codex setup, verification, privacy notes, and troubleshooting, see Local Agent Hooks.

Event Shape​

All ingestion paths emit the same backend event shape:

  • kind: source-specific event kind, such as notion_event or local_agent_event
  • session_id: stable source/external-id grouping
  • intent: short source/title line
  • output: source body, message text, or payload digest
  • targets: project, source, and source-specific target IDs
  • idempotency_key: stable per source record/update for safe retries

Add --dry-run when changing a pipeline or hook listener setup so you can inspect the exact event payloads before writing to the backend.

Last updated on