# Mind Layer

Give an agent persistent memory without a Sonzai account or hosted subscription.
Run one Go process with an embedded database. Store facts locally, retrieve them
by keyword, and optionally use your own model-provider key for semantic retrieval,
fact extraction and replies.

This standalone edition reuses Sonzai's BM25 and multilingual tokenization code.
Its persistence, API and provider integration are new. It does not include the
full hosted platform. The [capability matrix](capabilities.md) records the exact
scope and the tests that support it.

## Ask your agent about your project

Copy this into ChatGPT, Claude, or your coding agent. It will read the guide,
ask about what you're building, and suggest useful features with links to the docs.

Read https://docs.sonz.ai/agent-setup.md and suggest how Mind Layer could help my project.

[How to use this with your agent](use-with-your-agent.md).

## Try an evolving character

Run `make demo`, open `http://127.0.0.1:8090`, and choose interactions with Moss.
The example changes fictional traits, personality and replies while retaining
shared memories. Chat and inspect the context behind each reply. Offline recall needs no provider; open-ended dialogue uses your configured model. [Open the character guide](character-demo.md).

![Character evolution](assets/character-evolution.svg)

## Quick start

From the repository root, with Go 1.26 installed:

```sh
go run ./cmd/mind-layer
```

In another terminal:

```sh
python3 examples/quickstart.py
```

No key is needed for this example. Facts persist in `data/memory.db` across restarts.
The initial Go build needs network access to download dependencies. Runtime
keyword memory operations do not make network calls.

## A conversation ends; the memory stays

1. Store: “The user is vegetarian and cooks dinner for two people.”
2. Later, query: “vegetarian dinner”.
3. Supply the retrieved fact to your agent, or call the optional chat endpoint.

For paraphrases such as “What should guide my meal suggestions?”, configure an
embedding model and use semantic or hybrid search. Keyword search alone cannot
reliably connect phrases that use different words; the benchmark demonstrates this.

## Ownership and cost

The source is Apache-2.0. There is no Sonzai payment, license key, license server,
metering or telemetry dependency. You own and operate the database. Optional
provider usage is billed by your provider; a compatible local model server can
run without a paid API. Hardware and hosting remain your responsibility.

Continue with [Configuration](configuration.md), [API](api.md),
[Capabilities](capabilities.md), [Benchmarks](benchmarks.md), and
[Data lifecycle](data-lifecycle.md).


# Ask your agent about Mind Layer

Find out where persistent memory could help your product. Copy the instruction
below into ChatGPT, Claude, or your coding agent, and tell it what you're building.
It will use the docs to suggest useful features and a small first integration.

Read https://docs.sonz.ai/agent-setup.md and suggest how Mind Layer could help my project.

## What happens next

1. Your agent reads the guide and asks about your project where context is missing.
2. It recommends relevant features, explains costs and limitations, and links to the documentation.
3. You choose whether to move from a proposal to implementation.

This starts a conversation with your agent. It does not install a ChatGPT or Claude
integration or send your project to Sonzai. If your agent cannot open links, paste
the [Markdown guide](https://docs.sonz.ai/agent-setup.md) and the
[documentation bundle](https://docs.sonz.ai/llms-full.txt) into the conversation.
Use a project description and fictional examples; no credentials or customer data
are needed.

Prefer to explore directly? Start with [Configuration](configuration.md),
[Tested capabilities](capabilities.md), [REST API](api.md), or the
[character demo](character-demo.md).


# Evolving character project

Moss is a fictional workshop companion. The project demonstrates how an
application can evolve a character using durable Mind Layer memories and
personality configuration. The core does not infer personality changes by itself.

## Run

```sh
make demo
```

Open `http://127.0.0.1:8090`. No other server or API key is needed. Choose
interactions individually or click **Run the five-interaction story**. The page
shows the character's traits, dialogue, a live trend graph, causal explanations,
and the remembered event behind each change. Use **Recall shared experiences**
to query the actual stored memories.

![Moss changes after interactions](assets/character-evolution.svg)

The recorded story starts at trust 20, confidence 30, curiosity 45. Encouragement,
teaching and a kept promise raise these values. Dismissal lowers them. An apology
restores only part of the loss: the final state is **44 / 48 / 65**. These are
explicit fictional game rules, not scientifically validated personality scores.

## Persistent chat

**Talk to Moss** saves your message, reply, and a context receipt together as one
Mind Layer memory record. Use **Inspect this reply** to see the retrieved user
messages/story events, personality snapshot, and IDs of the recent turns used.
Those receipts describe the actual request context, not a model-generated
explanation or hidden reasoning. Offline mode shows local recall; `-live` enables
open-ended dialogue through the configured provider.

Chat records live in a separate `<agent>-chat/<user>` scope in the same demo
database. Their user messages rebuild a small BM25 index on each request. The
three highest-ranked matches from story memory and chat recall are supplied to
the model, along with the last four complete conversation turns. Scores from the
two corpora are ranking hints, not calibrated confidence. Generated replies are
preserved for conversational continuity but are never indexed as user facts.

The chat scope stores full receipts for inspection, so the general memory API
must not be exposed over this dedicated demo database. A rejected provider call
does not store half a turn. Stale turn numbers return HTTP 409; reload to recover
if another browser sent a message or a response was lost. The demo allows 100 chat
turns, messages up to 700 UTF-8 bytes, and replies up to 2,500 bytes. Context receipts
must also fit the core's 8 KiB memory-record limit. Start a new database for a new
conversation. Data is local but not application-encrypted; provider-enabled chat
sends messages and selected context to your provider. Its charges and retention apply.

Try the fictional greenhouse example in the README. A later correction remains
an explicit dated user message; the model is instructed to prefer it. This is
not a general contradiction-resolution system or a promise that every correction
will always be retrieved. Free text does not modify the character's trait policy.

Reproduce the structural checks (fresh temporary database; no personal data):

```sh
make chat-eval
# With configured provider variables; incurs provider usage:
python3 scripts/run_chat_eval.py --live
```

The recorded offline and live results are `benchmarks/results/chat-offline.json`
and `benchmarks/results/chat-live.json`. They check complete turns, exact restart
persistence, original fact and correction retrieval, mode labels, and unchanged
story traits. Five synthetic messages are not a general answer-quality benchmark.

## What changes and why

| Interaction | Trust delta | Confidence delta | Curiosity delta |
| --- | --- | --- | --- |
| Encourage | +12 | +15 | +5 |
| Teach | +5 | +5 | +15 |
| Keep promise | +20 | +10 | +5 |
| Dismiss | -25 | -20 | -10 |
| Repair | +12 | +8 | +5 |

Values clamp to 0–100. At trust 30 Moss starts warming up; at 50 it becomes a
trusting teammate. The kept-promise response changes with the current trust level.
All default dialogue is authored/scripted and labeled as such in the UI.

The application policy lives in `examples/character/character.go`. Each event is
an explicit fact in the dedicated `moss/demo-user` scope. `Store.UpdateScope`
commits that memory and the newly rendered personality in one transaction.
Traits and the chart are reconstructed by replaying the event history; no hidden
state file is required. Revision checks reject duplicate/stale requests.

## Persistence and fresh stories

The default database is `data/character-demo.db`. Reloading the page or restarting
the process preserves the character. To start another story without deleting the
old one, choose a new database path:

```sh
go run ./cmd/character-demo -data data/moss-second-story.db
```

The demonstration allows up to 100 story interactions and 100 chat turns. It binds only to a loopback IP,
rejects cross-origin requests, and does not expose the general memory API. The
demo database must remain separate from other application data.

## Optional model-generated dialogue

With a configured direct provider (`MIND_LAYER_BASE_URL`, `MIND_LAYER_API_KEY`,
`MIND_LAYER_CHAT_MODEL`), start:

```sh
go run ./cmd/character-demo -live
```

The separate **Generate a model reply** button under Ask memory sends the question, current personality and retrieved
memories directly to that provider. It does not infer trait deltas, store the
question or rewrite the event history. This keeps model creativity separate from
the deterministic evolution policy. Provider charges and retention apply.
The demo does not use embeddings; its memory queries are lexical in both modes.
Without `-live`, provider variables are ignored and the entire example is offline.

### Observed live replies

The same synthetic question was sent to Gemini 2.5 Flash before interaction,
after the kept promise, after dismissal and after repair. Initially Moss said it
had no record of a promise. After the promise it recalled the sensor and proposed
integrating it. After dismissal its reply expressed uncertainty about contributing.
After repair it recalled the encouragement and lesson and cautiously offered help.

These four qualitative observations are recorded in
`benchmarks/results/character-live.json`. They demonstrate the actual provider
path using changing persisted state; they are not a statistical evaluation of
personality quality. The UI marks model-generated replies separately from the
offline scripted replies.

## Test and reproduce

```sh
go test -race ./examples/character
make demo-scenario
# Regenerate the authored scenario receipt and graph:
python3 scripts/run_character_scenario.py --record
.venv/bin/python scripts/build_docs.py
```

The scenario command uses a fresh temporary database and exercises the real
executable. Its checked-in receipt is `benchmarks/results/character-scenario.json`.
Tests prove the exact deltas, memory recall, persistence, scope isolation, bounds,
invalid/stale request rejection, concurrent revision handling and HTTP behavior.
`TestAtomicScopeUpdate` proves personality and event memory stay together across
restart and that rejected updates do not partially apply.


# Tested capabilities

This is a scoped standalone edition, not hosted-platform parity. Each supported
claim below maps to named tests. `make verify` requires all of them to pass
with the race detector. Mock-provider tests check integration contracts; the
separate live smoke and benchmark reports measure actual provider behavior.

| Capability | Provider requirement | Tests |
| --- | --- | --- |
| Store and retrieve explicit memories without provider keys or a Sonzai account | none | `TestNoProviderRequired` |
| Memories and personality survive database close and reopen | none | `TestDurableMemory` |
| Separate agent/user scopes for search, export and deletion | none | `TestScopeIsolation`, `TestSemanticAndHybridRecall` |
| BM25 keyword retrieval with stemming and no-match empty results | none | `TestLexicalRecall` |
| Direct-provider embeddings, cosine retrieval and reciprocal-rank hybrid search | embedding model | `TestSemanticAndHybridRecall`, `TestModelMismatchFailsClosed` |
| Update by memory ID and deduplicate exact trimmed text | none; embedding calls when configured | `TestExplicitUpdatesAndDeduplication` |
| Export a scope and logically delete individual memories or the whole scope | none | `TestExportAndDeletion` |
| Persist a manually configured personality and include it in retrieved context | none | `TestPersonalityContext`, `TestDurableMemory` |
| Extract facts from supplied text and answer with selected memories through the configured provider | chat model; optional embedding model | `TestExtractionAndGroundedChat`, `TestExtractionFailureDoesNotPersist` |
| REST memory, personality, retrieval, context, ingest, chat, export and delete endpoints | depends on endpoint | `TestHTTPMemoryLifecycle`, `TestHTTPProviderEndpoints` |
| Loopback defaults, owner-token authentication, browser-origin rejection and public-bind checks | none | `TestHTTPAuthenticationAndBrowserIsolation`, `TestHTTPRejectsInvalidInput` |
| Direct provider transport with URL validation, error redaction and no redirect following | mock provider for tests | `TestProviderURLValidation`, `TestProviderErrorRedaction`, `TestProviderRejectsRedirect`, `TestInvalidProviderOutput` |
| Concurrent in-process access and validation before batch writes | none | `TestConcurrentMemoryAccess`, `TestInputValidationAndAtomicBatch` |
| Commit application-driven personality changes with event memories atomically | none; embeddings if configured | `TestAtomicScopeUpdate` |
| Moss example evolves through explicit interactions, recalls them, persists after restart and rejects stale turns | none; optional generated dialogue | `TestCharacterEvolution`, `TestCharacterRestartAndIsolation`, `TestCharacterConcurrentRevision`, `TestCharacterBoundsAndInvalidAction`, `TestCharacterHTTP` |
| Persistent character chat with retrieved-context receipts, restart recovery, user isolation, and optional direct-provider replies; free text does not infer trait changes | none for offline recall; chat model for open-ended dialogue | `TestChatPersistenceRecallAndIsolation`, `TestChatProviderContextAndFailureAtomicity`, `TestChatHTTPAndValidation`, `TestChatConcurrentRevision` |

## Relationship to existing hosted docs

The following inventory maps the top-level English pages from the existing
Mind Layer docs to this release. It does not import their hosted API claims,
screenshots or examples. Nested hosted guides and reference endpoints are
not included unless this edition's API and tests explicitly document them.

| Existing page | Status | Standalone scope |
| --- | --- | --- |
| Advance Time | not included | Not implemented or promised by this standalone edition. |
| Agent Insights | not included | Not implemented or promised by this standalone edition. |
| Architecture | limited subset | Replaced by the standalone single-process architecture. |
| Pattern 6: Hermes | not included | Not implemented or promised by this standalone edition. |
| Pattern 1: Managed Agent Runtime | not included | Not implemented or promised by this standalone edition. |
| Pattern 2: MCP | not included | Not implemented or promised by this standalone edition. |
| Pattern 3: OpenClaw | not included | Not implemented or promised by this standalone edition. |
| Pattern 5: Standalone Memory (Batch) | limited subset | Library batch writes are atomic; no hosted batch processing service. |
| Pattern 4: Standalone Memory (Real-Time) | limited subset | New local REST API, incompatible with the hosted SDK/API. |
| Conversations | limited subset | Single non-streaming reply with retrieved context; no managed conversation runtime. |
| Custom State | not included | Not implemented or promised by this standalone edition. |
| Custom Tools | not included | Not implemented or promised by this standalone edition. |
| Emotions & Mood | not included | Not implemented or promised by this standalone edition. |
| Events & Multi-Agent Dialogue | not included | Not implemented or promised by this standalone edition. |
| Generation | not included | Not implemented or promised by this standalone edition. |
| Sonzai Mind Layer | limited subset | Replaced by the tested standalone capability list. |
| Instances | limited subset | Agent/user namespace isolation only; no hosted workspace provisioning. |
| Inventory | not included | Not implemented or promised by this standalone edition. |
| Knowledge Analytics | not included | Not implemented or promised by this standalone edition. |
| Knowledge Base | limited subset | Plain text can be inserted as facts; no file ingestion or graph-building pipeline. |
| Memory | limited subset | Explicit facts, BM25 and optional semantic/hybrid retrieval only. |
| Multiplayer Memory | not included | Not implemented or promised by this standalone edition. |
| Notifications (Polling) | not included | Not implemented or promised by this standalone edition. |
| Omnichannel Messaging | not included | Not implemented or promised by this standalone edition. |
| Organization-Global Knowledge Base | not included | Not implemented or promised by this standalone edition. |
| Personality System | limited subset | Manually configured personality text only; no automatic Big Five evolution. |
| Priming | not included | Not implemented or promised by this standalone edition. |
| Proactive Messaging | not included | Not implemented or promised by this standalone edition. |
| Scheduled Reminders | not included | Not implemented or promised by this standalone edition. |
| Self-Improvement (Post-Processing) | not included | Not implemented or promised by this standalone edition. |
| Sessions | limited subset | Session provenance labels only; no session lifecycle or automatic end processing. |
| Shared Memory | not included | Not implemented or promised by this standalone edition. |
| User Personas | limited subset | Manual per-scope personality/context only; no inferred user persona. |
| Voice | not included | Not implemented or promised by this standalone edition. |
| Wakeups | not included | Not implemented or promised by this standalone edition. |
| Webhooks | not included | Not implemented or promised by this standalone edition. |

## Explicit limitations

No automatic personality evolution, graph reasoning, semantic contradiction
resolution, memory decay/consolidation jobs, shared organization memory,
proactive messages, voice, external tool execution, MCP server, or hosted SDK
compatibility is claimed. Personality is configured text. Sessions are labels.
Single-owner authentication is not end-user authorization. Logical deletion
is not secure erasure. This release includes no production-scale certification.


# Configuration

## Offline memory

```sh
go run ./cmd/mind-layer -addr 127.0.0.1:8080 -data data/memory.db
```

Explicit storage, BM25 search, personality, context, export and deletion work
without any provider. Data is stored in one Bolt database file. A second process
cannot open the same database for writing; the open attempt times out. Multiple
requests in one process are supported. Horizontal scaling is outside this edition.

## Direct providers

| Environment variable | Purpose |
| --- | --- |
| `MIND_LAYER_BASE_URL` | Trusted OpenAI-compatible API base URL, without `/chat/completions` or `/embeddings` |
| `MIND_LAYER_API_KEY` | Your provider credential; optional for local servers |
| `MIND_LAYER_CHAT_MODEL` | Explicit model for JSON fact extraction and replies |
| `MIND_LAYER_EMBEDDING_MODEL` | Explicit model for stored/query vectors |
| `MIND_LAYER_TOKEN` | Optional service-owner token; 32+ characters required for public binding |

The server reads process environment variables. It does not automatically load
`.env` files. Set a base URL and at least one model to enable a provider. Model
defaults are deliberately not guessed. HTTPS is required except for a loopback
IP endpoint such as `http://127.0.0.1:11434/v1`. Redirects are refused.

### Gemini example, verified with a synthetic smoke test

```sh
export MIND_LAYER_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai
export MIND_LAYER_API_KEY="$GEMINI_API_KEY"
export MIND_LAYER_CHAT_MODEL=gemini-2.5-flash
export MIND_LAYER_EMBEDDING_MODEL=gemini-embedding-001
go run ./cmd/mind-layer -data data/semantic.db
```

This uses Google's documented [OpenAI-compatible API](https://ai.google.dev/gemini-api/docs/openai).
It calls Google directly. Provider availability and charges depend on your account.

### Other compatible providers

Set the base URL, credential and models supplied by your provider. The adapter
uses [chat completions](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create)
and [float embeddings](https://developers.openai.com/api/reference/resources/embeddings/methods/create).
Extraction requires `response_format: {"type":"json_object"}`. Support varies
between compatible implementations. The protocol is contract-tested locally;
only the recorded Gemini configuration has been live-tested in this preparation.

## Embedding lifecycle

Configuring an embedding model makes new writes embed their text before commit.
Embedding errors fail the write. Extraction embeds the returned facts when enabled.
Lexical search remains offline; semantic and hybrid queries call the embedding
provider. Chat calls the chat model after retrieval. No background jobs run.

Vectors retain their provider-base/model identity. Semantic/hybrid search rejects
a scope containing missing or incompatible vectors. Reinsert those memories
under the same IDs with the current model, or use a fresh database. There is no
automatic re-embedding job. A model alias changing behavior under the same name
cannot be detected; use stable model versions where available.

## Remote access

Bind to loopback unless your application needs remote access. For remote access,
set an unpredictable owner token of at least 32 characters and use TLS at a trusted
reverse proxy. The token grants access to every scope. It does not authenticate
individual end users. Browser-origin requests are rejected; use your backend.


# REST API

Base URL: `http://127.0.0.1:8080`. All POST/PUT requests require
`Content-Type: application/json`. If configured, send the service owner token
as `Authorization: Bearer <owner-token>` from your backend.

Every memory operation requires an explicit scope:

```json
{"scope":{"agent":"assistant","user":"synthetic-user"}}
```

Agent/user IDs accept 1–128 ASCII letters, digits, underscores, dots and hyphens.
They are namespaces, not authentication claims. The caller must authorize them.
POST/PUT request bodies have a 1 MiB limit and reject unknown fields or trailing JSON.

## Store or update memory

`POST /v1/memories`

```json
{
  "scope":{"agent":"assistant","user":"synthetic-user"},
  "id":"diet",
  "text":"The user is vegetarian and cooks dinner for two people.",
  "session":"first-conversation"
}
```

Returns the stored memory. A supplied ID updates that memory inside this scope.
Omit the ID to derive one from trimmed text: exact duplicates collapse, but
semantic duplicates and contradictions are not automatically reconciled. Text
must be nonempty and at most 8,192 bytes. Session labels are optional provenance,
at most 128 bytes; they do not create a managed conversation runtime.

## Search and build context

`POST /v1/search` returns scored memories. `POST /v1/context` returns the same
retrieval results plus the scope's configured personality.

```json
{
  "scope":{"agent":"assistant","user":"synthetic-user"},
  "query":"vegetarian dinner",
  "mode":"lexical",
  "limit":5
}
```

Modes: `lexical` (default), `semantic`, `hybrid`. Limit defaults to 5, range 1–100.
Queries are nonempty and at most 8,192 bytes. Semantic/hybrid require embeddings.
Hybrid combines lexical and vector ranks with reciprocal rank fusion (constant 60).
Scores differ across modes and are not calibrated probabilities. Semantic retrieval
can return irrelevant positive-similarity hits; there is no calibrated abstention gate.

## Configure personality

`PUT /v1/personality`

```json
{"scope":{"agent":"assistant","user":"synthetic-user"},"personality":"Be warm, concise and practical."}
```

Personality is explicitly configured text (up to 8,192 bytes), persisted per scope.
There is no Big Five inference, automatic trait evolution or mood simulation.

## Extract and chat

`POST /v1/ingest` uses the configured chat provider to extract up to 20 durable
facts from supplied text. The raw transcript is not stored by this endpoint.
Provider retention is separate from local retention.

```json
{"scope":{"agent":"assistant","user":"synthetic-user"},"text":"User: I am vegetarian and cook for two.","session":"first-conversation"}
```

`POST /v1/chat` accepts the same input as `/v1/search`. It returns
`{"answer":"...","context":{"personality":"...","memories":[]}}`.
It is a single non-streaming completion using retrieved context, not a managed
multi-turn conversation. It does not automatically store the query or response.
Call ingest explicitly for conversations you choose to remember. Extraction can
be wrong; review facts and use explicit ID updates for corrections.

## Export and delete

| Method and path | Result |
| --- | --- |
| `GET /v1/export?agent=assistant&user=synthetic-user` | All live memories, vectors and personality for the scope |
| `DELETE /v1/memories?agent=assistant&user=synthetic-user&id=diet` | Delete one live record and its search entry |
| `DELETE /v1/scope?agent=assistant&user=synthetic-user` | Delete all live records, indexes and personality for the scope |
| `GET /healthz` | Process liveness; not a provider connectivity test |

Deletion is logical, not secure disk erasure. Export can contain personal data
you intentionally supplied; treat exports as private. There is no bulk restore
endpoint, but explicit memories can be reinserted with their IDs through the API.

Successful operations return HTTP 200. Invalid inputs return 400; unauthorized
owner-token requests 401; rejected browser/host requests 403; a missing memory
delete returns 404. Store/provider failures return a generic 500 without provider
response bodies. Unsupported methods/routes use standard HTTP 405/404 responses.


# Benchmarks

These are measured diagnostics for this standalone edition. They are not scores
for the full hosted platform and do not establish general memory quality.

## Retrieval quality: synthetic-v1

The unchanged authored dataset contains 12 synthetic facts and
12 queries, including exact terms, inflection, paraphrases and one
two-evidence query. Both runs use k=3. No real users or customer data are
included. The query labels are used only for scoring, not ingestion or retrieval.

| Retrieval mode | Mean recall@3 | MRR@3 | Provider |
| --- | --- | --- | --- |
| Most recently inserted 3 facts baseline | 25.00% | Not measured | None |
| BM25 lexical | 75.00% | 0.7500 | None |
| Hybrid BM25 + embeddings | 100.00% | 0.9583 | gemini-embedding-001 |

Recall@3 is the fraction of required evidence memories retrieved, averaged over
queries. MRR@3 is the reciprocal rank of the first relevant memory, or zero if
none is retrieved. A high score on this tiny authored set is not a statistically
reliable quality estimate. Lexical search misses several paraphrases; hybrid
retrieval recovers them in this run. There is no answer-generation judge here.

Dataset SHA-256: `297811d32167a59cfb1fe7bd1bfcdc03633c5b8cc1606e13dfff1d810fcd489d`.
Lexical measured: `2026-09-10T19:56:11Z`. Hybrid measured: `2026-09-10T19:57:06Z`.
Environment: darwin/arm64, 14 logical CPUs,
go1.26.0. Models and machines can change results.

## Local latency

The separate scale test contains 1,000 authored synthetic records,
with 30 sequential warmed-up keyword queries in one process.
It includes database reads and retrieval, without HTTP or provider calls.

| Measurement | Result |
| --- | --- |
| p50 query latency | 3.267 ms |
| p95 query latency | 4.380 ms |
| 1,000 individual durable writes | 9.275 seconds |

This is not cold-start latency, concurrent throughput, a vector-index scaling
test, or a provider-latency claim. Multilingual tokenizers load models lazily.
Semantic search scans the selected scope's vectors; it is not an ANN service.

## Reproduce

```sh
go run ./cmd/benchmark
# Requires configured provider environment and incurs provider usage:
go run ./cmd/benchmark -mode hybrid -live -iterations 0
# Optional separate provider smoke test (synthetic data only):
MIND_LAYER_LIVE_TEST=1 go test -run '^TestLiveProvider$' -count=1 -v .
```

Raw per-query results live in `benchmarks/results/lexical.json` and
`benchmarks/results/hybrid.json`. The fixture is `benchmarks/synthetic.json`.
The live smoke report is `benchmarks/results/live-smoke.json`; it records one
passed Gemini extraction/retrieval/answer test, not a quality leaderboard score.
No token-price estimates or competitor numbers are invented.

## Character chat smoke eval

`make chat-eval` runs five authored messages through a real local HTTP server,
then restarts it to verify exact chat receipts. It checks original fact and
correction retrieval, mode labels, and unchanged story traits. To enable model
replies explicitly, run `python3 scripts/run_chat_eval.py --live` with provider
configuration. This can incur provider charges.

Recorded results are `benchmarks/results/chat-offline.json` and
`benchmarks/results/chat-live.json`. Read the unedited live responses to assess
correction handling and unknown-fact honesty. The automated checks measure
structure and context selection; they do not judge answer quality. This small
fixture is newly authored for the standalone chat path. Existing hosted monolith
eval suites depend on services outside this release and are not represented by
these scores.

## Public benchmarks: not yet measured

| Benchmark | Why relevant | This edition's status |
| --- | --- | --- |
| [LongMemEval](https://github.com/xiaowu0162/LongMemEval) | Long-term memory QA, updates, temporal reasoning, multi-session evidence and abstention | Not run; no score claimed |
| [LoCoMo](https://github.com/snap-research/locomo) | Long conversational histories with QA and evidence annotations | Not run; no score claimed |

For a comparable public result, pin the dataset revision and split, include every
evaluation case (including abstention), document ingestion/chunking and retrieval
settings, and use the benchmark's official scoring protocol. Report provider
models, token usage/cost, timing and all failures. Retrieval recall alone must not
be presented as answer accuracy. External datasets and generated transcripts
must remain outside the public source tree; do not bundle them as fixtures.


# Data lifecycle and architecture

```text
Your backend / Python example
        |
        v
Local REST service -> scoped Bolt database + BM25 index
        |
        +-- optional direct provider: embeddings / extraction / replies
```

The service has no Sonzai control-plane dependency. The Bolt database stores facts,
session labels, timestamps, personality text and optional vectors. In-memory BM25
indexes are rebuilt from persisted records at startup. Writes use database
transactions and update the index only after a successful commit. One process owns
one database; readers and writers in that process are synchronized.

## What leaves the machine

| Operation | External data flow |
| --- | --- |
| Offline write, lexical search, personality, context, export, delete | None |
| Write with embedding model configured | Fact text to configured provider |
| Semantic/hybrid search | Query text to configured provider |
| Ingest | Supplied transcript to chat provider; extracted facts to embedding provider if configured |
| Chat | Query, retrieved memories and personality to chat provider; query embedding if semantic/hybrid |

No background telemetry, Sonzai callbacks, billing, or license validation run.
Provider keys stay in process memory and are not stored in the database. There
are no automatic retries, so rate limits fail the operation rather than silently
creating repeated charges. Request timeouts are bounded.

## User-controlled memory

Use explicit scope IDs; authorize them in your application's backend. Use stable
memory IDs for corrections. Exact text deduplication is supported, but semantic
deduplication, temporal reasoning and contradiction resolution are outside this
edition. If facts change, update or delete the old fact explicitly.

Only supplied facts or model-extracted facts are stored. The ingest endpoint does
not save the raw transcript, and chat does not save conversation history. Models
can still extract inaccurate facts or follow adversarial instructions. The API
does not execute model tools; review extracted data before relying on it.

## Deletion and backups

Delete removes live records and index entries. It does not securely overwrite Bolt
free pages, filesystem snapshots, exported files, backups, or provider logs.
Use encrypted disks and manage retention outside this process. Never commit
database files or exports to the public repository. This is a local ownership
model, not a compliance certification or a guarantee that stored data lacks PII.