Iriis

1. Overview

Iriis is a single-user agent system made of two cooperating services. The agent is a long-running conversational agent with persistent memory, tool use across cloud and desktop, background work, an autonomy scheduler and outbound calling. Vision is a real-time camera assistant that streams frames from a phone, runs cheap local perception on every frame and calls a vision-language model on demand. The agent can call Vision as a tool, and Vision can hand work back to the agent.

Design goals, in priority order:

• Correctness of side effects: the system never reports an action it did not perform.
• Bounded autonomy: unattended operation is allowed only for actions that are reversible or addressed to the owner.
• Latency: first spoken sentence in about a second for chat; a few seconds for a camera answer.
• Survivability: any single model, provider or client can fail without taking the system down.
• Cost: runs on free or near-free tiers; every request is sized for rate-limited providers.

Non-goals: multi-tenancy, fine-tuning, and any model training. All improvement happens in context.

2. Architecture

Tool providers are composed: the core exposes one catalogue that merges cloud servers and whatever the desktop agent advertises on connect. When both offer a server with the same name, the desktop copy wins while it is online and the cloud copy covers when it is not. The model never sees which side executed a call.

3. Model layer

Two adapters sit between the API and the loop. A sentence chunker splits the token stream at sentence boundaries so speech starts before generation ends. A tool-call guard recovers tool calls that some models emit as text markup instead of structured calls, so a formatting quirk does not leak into what the user sees or hears.

4. The turn pipeline

handle(message):
    attachments = parse "[attachment file:N ...]" lines
    context     = inspect_file(N, question) for each       # before the model runs
    system      = build_system_prompt(persona, mood, goals, memories(query),
                                      playbook_rules(query), open_loops, background_work)
    tools       = route(message + previous user turn)       # <= 64 tools, plus more_tools()
    for step in range(MAX_ITERATIONS):                      # 10 in chat, 20 in background work
        stream  = llm(system, history, turn, tools)
        speak each completed sentence
        if no tool calls:
            if claims_action(text) and no tool ran this turn: # verify, once
                append correction prompt; continue
            break
        for call in calls:
            result = execute(call)                          # tier check, confirm, run
            turn.append(wrap_untrusted(result))
        compact(turn)                                       # keep the turn under budget
    save(message, reply + [file:N] markers)
    reflect_async(message, reply, tool_calls)               # section 8

Details that matter in practice:

• Attachments are pre-read. The model receives file contents in the user turn, not a file id it may or may not open. The stored conversation keeps only what the user sent.
• Context budget within a turn. Tool results are capped at 4,000 characters each. When one turn’s tool output exceeds 60,000 characters, older results are shortened to their first 1,500 characters while the three most recent stay whole.
• History budget across turns. About 5,000 tokens of recent conversation are kept verbatim. Messages that fall out are folded into a rolling summary by a background call, so long conversations degrade gracefully instead of losing their start.
• Empty replies. If the model returns neither text nor calls, the turn retries once without tools and with an explicit instruction to answer; a second failure produces a fixed apology rather than silence.
• Cancellation. A client may interrupt mid-answer (voice barge-in). The history then records only what the user actually heard, with an annotation, so the next turn does not assume the rest was delivered.
• Background turns. Chat requests from the PWA set a flag that decouples the turn from the HTTP connection: if the client disconnects, the turn completes, is saved, and a push notification is sent. A session-state endpoint lets a reopened client show progress and pending confirmations.

5. Tools and MCP

Every capability is an MCP server declared in one YAML file: transport (stdio or HTTP), environment, default permission tier, per-tool tier overrides, excluded tools, required environment variables and routing keywords. A server whose required variables are missing is not started, so optional integrations stay off until configured.

servers:
  studio:
    command: python
    args: ["-m", "servers.studio"]
    tier: auto
    keywords: [chart, plot, sketch, pdf, document, attachment, video, ...]
  hq:
    transport: http
    url: https://example/mcp
    headers: {Authorization: "Bearer ${HQ_KEY}"}
    requires_env: [HQ_KEY]
    tier: auto
    tools: {send_email: confirm, send_draft: confirm, ...}
    exclude: [generate_pdf, web_search, ...]     # duplicates of local tools

Routing. Hundreds of tools do not fit in one request, and a model offered all of them chooses worse. Each turn gets:

• the core servers always (time and arithmetic, memory, web, tasks, self-inspection);
• servers whose keywords overlap the current and previous user message, up to 64 tools;
• a more_tools(need) meta-tool that adds a named or best-matching server mid-turn.

Remote servers. HTTP servers are wrapped so that a base URL gains the /mcp path, header values are whitespace-normalised (keys pasted with a trailing newline are invalid headers), errors are unwrapped from exception groups to their root cause, and any bearer token is masked before an error is logged. Duplicates are excluded: when an external service offers its own PDF or search tool, the local one is preferred so artefacts land in the same store the clients render.

Files. Every generated artefact (chart, plot, sketch, PDF, Word, slide deck, camera photo) is a row in a files table. Tools return file N handles; the loop appends [file:N] markers to the saved reply so clients render attachments in history. Uploaded files use the same table, which lets one inspection tool handle documents (PDF, DOCX, XLSX, PPTX, text), images (VLM), audio (speech to text) and video (contact sheet plus soundtrack transcript).

6. Permissions and verification

Confirmation questions are generated from the call (for shell commands the model must supply a plain-language purpose argument, which is what the user sees instead of the command). Unattended contexts (background tasks, the autonomy scheduler, MCP callers) have no confirmation channel, so confirm-tier calls are declined there by construction.

Verification. Two deterministic checks run outside the model:

• Action claims. If the final text matches an action pattern (set, scheduled, sent, saved, booked, created, cancelled, done) and no tool ran in the turn, the reply is withheld once and the model is told to either call the tool or retract.
• Untrusted content. Every tool result is wrapped in a tag marked untrusted, followed by a reminder that it is data. Secrets are redacted from tool output before it enters the context or the audit log.

Every tool execution is written to an audit log (tool, arguments, tier, confirmed, result or error, duration) in the database, and failures additionally go to an in-memory error ring buffer exposed to the owner.

7. Memory

Memory is a single table of typed items plus two small key-value records.

mind(id, kind, text, importance REAL, uses INT, active INT, created REAL, last_used REAL)
kv('mind_mood')    -> {feeling, valence, why, ts}       # decays to neutral after 8 h
kv('mind_persona') -> {purpose, aesthetic, voice, vocabulary[], address, rituals[], avoid[]}

score(item, query) = 0.995 ** hours_since(item.last_used or item.created)
                   + item.importance / 10
                   + 1.6 * min(1, |words(query) & words(item)| / sqrt(|words(item)| + 1))

• Tokenisation is lower-case words of three or more characters, stop-words removed, with light suffix stemming so that write, writing and writer match.
• Deduplication on insert: if an active item of the same kind has Jaccard word overlap above 0.75, it is reinforced (importance raised, last_used refreshed) instead of inserting a duplicate.
• Updates and deletions are first-class. Reflection can return updates (rewrite item N) and outdated (deactivate item N). Contradictory facts in a prompt are worse than missing ones, so correction replaces rather than appends.
• Selection per prompt: up to 12 facts, insights and patterns; 5 people; 6 lessons; 8 playbook rules filtered to those sharing words with the query; 3 skills; 8 open loops; the persona; the mood; the goals.
• Secrets matching password, key, secret or PIN patterns are refused at write time.
• The owner can read every item and delete any of them through the API.

8. Reflection, playbook and persona

Per-turn reflection. After the reply is delivered, a background call receives the exchange, the tool calls with truncated results, the top related memories with ids and the open loops, and returns JSON:

{"observations": [{"text": ..., "importance": 1-10}],
 "people":       [{"text": ..., "importance": 1-10}],
 "lesson": "..." | "",
 "skill": {"when": ..., "how": ...} | null,
 "open_loops": [...], "closed_loops": [ids],
 "updates": [{"id": N, "text": ...}], "outdated": [ids],
 "mood": {"feeling": ..., "valence": -1..1, "why": ...} | null}

Trivial turns (short messages without tools or correction words) and health-check sessions are skipped. The prompt states delivery facts explicitly (generated files appear in the client automatically) so that normal behaviour is not mislearned as a failure.

Playbook distillation. When a background task or a project session finishes, a second call turns the output into at most a few short, domain-tagged rules, given the existing related rules with ids. A rule may name an id it replaces. The playbook therefore grows by small deltas and never by rewriting itself, which avoids the collapse into generic summaries that iterative self-summarisation produces. Relevant rules are injected under their own heading in every subsequent system prompt.

Nightly consolidation.Once a day the last 160 items and an hourly histogram of the owner’s activity over 14 days are condensed into insights, behavioural patterns, growth areas, revised goals and a list of item ids to retire.

Persona. On first run the agent authors its own character from what it knows about the owner (purpose, aesthetic, voice, vocabulary, form of address, recurring habits, things to avoid) and stores it as a structured record. Each night it is revised slightly using recent lessons as feedback. The persona is rendered at the top of the memory block in every prompt.

These mechanisms combine published ideas: memory streams scored by recency, importance and relevance, with periodic reflection (Park et al., 2023); verbal self-reflection after failure (Shinn et al., 2023; Zhao et al., 2023); a skill library (Wang et al., 2023); self-edited memory (Packer et al., 2023; Chhikara et al., 2025); strategies from successes and failures (Ouyang et al., 2025); and incremental context playbooks (agentic context engineering, 2025).

9. Autonomy scheduler

A single coroutine wakes at a time the agent chose on its previous wake, clamped to 10 to 240 minutes. During configured rest hours it performs no actions and runs consolidation instead. Otherwise it plans the day if no plan exists, then makes one decision.

situation = now, last owner message, today's counts,
            goals, mood, memory block, persona,
            agenda (plan, projects with latest notes, hobbies, sessions used),
            upcoming reminders / calls / tasks,
            external day summary and commitments (CRM service),
            recent events (missed calls, finished or failed tasks),
            feedback: own reach-outs in the last 5 days and whether the owner replied within 2 h,
            last 12 messages
decision  = llm(INNER_PROMPT, situation) ->
  {"thought", "action": none | message | call | task | work | practice | emergency,
   ... action fields ..., "close_loops": [ids], "mood": {...}, "next_check_minutes": N}

Work sessions run through a tool provider wrapper that only exposes research, reading, document and draft servers and blocks external agents that could send on their own. Nothing the scheduler does can contact anyone other than the owner. The feedback line in the situation is the main learning signal: reach-outs that go unanswered make similar reach-outs less likely, without any explicit rule.

10. Background tasks

Tasks are rows (title, goal, repeat schedule, status, next run, runs, result). The runner polls every 15 seconds, claims due rows atomically with UPDATE ... WHERE status = 'scheduled' RETURNING (safe with two processes), runs up to two in parallel with a 15-minute timeout, and recovers rows left in running after a restart.

A long task has two failure modes: it never stops researching, or it stops early and writes a confident progress note. The protocol:

• The task brief requires a final line STATUS: done or STATUS: continue.
• If the line is missing, a separate judge call classifies the report as finished deliverable or progress note.
• Unfinished one-off tasks are rescheduled one minute later with their notes appended (marked in progress); the owner is not notified.
• From the third session the brief instructs the model to stop researching and produce the deliverable; the hard cap is six sessions.
• Deliverables that are documents must be saved with a document tool; the brief forbids claiming a file exists without a tool result.
• Finished tasks trigger playbook distillation and a notification with the first sentence; the full report goes to Telegram.

11. Notifications and calls

A web app cannot ring a locked phone. Calls are therefore a small state machine delivered over three channels at once.

calls(id, at, reason, opening, status, repeat, source, created, rang, answered)
status: scheduled -> ringing -> answered | declined | missed | cancelled

watcher, every 10 s:
    due      -> ringing; send Web Push (data.kind = "call"), Telegram, desktop announce,
                and an ntfy alert (priority 5, sequence id = call id, click = app URL with call id)
    repeat   -> insert next occurrence
    ringing for > 120 s -> missed; insert "Missed call" message in the chat; push;
                user alarms re-ring once after 5 minutes
answer / decline / cancel -> clear and delete the ntfy alert by sequence id

• Web Push uses VAPID keys generated once and stored in the database. The service worker shows a persistent notification with Answer and Decline actions and messages any open client to start ringing in-page.
• ntfy (open source) delivers priority-5 alerts which, with the app’s insistent setting, keep ringing on a locked Android phone until acknowledged. Dismissing by sequence id stops the alarm when the call is answered in the app.
• On answer, the client opens a voice session; if the call has no prepared opening line, the client sends [call answered] reason and the agent speaks first. Opening a call link after it stopped ringing starts a call-back instead of failing.

12. Vision subsystem

Transport. One WebSocket per camera session carries binary JPEG frames upstream and JSON events downstream (hello, perception, thinking, answer, alert, state, coach, notice, job and report events). Frames are ack-gated: the client sends a new frame only after a frame_ack, so on a slow link the server always processes the most recent frame and the queue length is bounded at one. Watch-only viewers receive the stream through WebRTC offers relayed over the same socket.

Per-frame perception (server CPU, milliseconds):

• object detection (YOLO);
• text and digit reading (PaddleOCR, mobile models);
• analogue dial reading by needle geometry (OpenCV).

VLM call. The model receives the frame (resized to at most 512 to 640 pixels on the long side) and a compact context: owner profile (capped at 700 characters), remembered facts (900), goal, current procedure step and its acceptance criteria, recent readings, detections, OCR text, up to three knowledge excerpts (380 characters each) and the last four exchanges. It must answer with a fixed JSON schema:

{"say": str, "status": "ok" | "warning" | "danger" | "unsure",
 "reading": {"value": number, "unit": str, "confidence": number} | null,
 "step_complete": bool}

• Repair, not trust. Replies are parsed leniently, unknown statuses map to unsure, confidences given as percentages are normalised.
• Cross-checking. For dial readings the VLM value and the geometric value are compared against the instrument’s span; disagreement downgrades the answer to unsure.
• Debouncing. A procedure step advances only after two consecutive step-complete verdicts.
• Coach mode. A periodic look with an instruction to stay silent unless there is a mistake, a safety issue, an out-of-range reading or a finished step; repeated lines are suppressed.
• Noticing writing. When OCR finds new text with at least three words, mostly unseen before, and it stays stable for two seconds, the model decides whether it is a question addressed to it. Triggers are spaced at least 15 seconds apart.
• Priority. Coach and notice calls run only when a fast provider is free; a person’s question always preempts them, and a busy model drops background requests instead of queueing them.
• Video review. Recorded clips and the last 90 seconds are sampled into frames with timestamps and reviewed as a job (summary, problems, timeline). Video sent through chat is reduced to one contact sheet: six to nine frames spread across the clip, always including the last, laid out in a grid with timestamps, plus a transcript of the soundtrack.
• Hand-off. When the camera is used from the Iriis app, the core relays the socket. After each answer the question and the VLM’s reply are passed to the main agent, which replies with nothing unless more is needed (a document with a camera photo, a sketch, a reminder, a message); its follow-up and any confirmation requests are pushed into the camera UI.

13. Provider routing

Hosted vision models fail in several distinct ways: overload (HTTP 503), rate limits (429 with a retry hint), silent hangs, and retirement (404 for a model name that used to work). The router treats each differently.

order providers by ema_latency (unmeasured ones keep configured order)
for p in providers if p.ready and images <= p.max_images:
    try: answer within 12 s (40 s for multi-image review)
    on 200:          ema = 0.6 * ema + 0.4 * elapsed
    on timeout:      ema = 30; rest 120 s
    on 503 / 5xx:    rest 15 s
    on 429:          wait if the hint is short, else rest for the hinted time
    on 404:          retire the model permanently
if none answered and the person is waiting and a provider frees up within 35 s: wait, retry
else: local model

• Discovery. At startup the service lists the models available to the key and adds current Flash-class models as backups, so retired names are replaced without a deploy.
• Learned limits. A provider that rejects a request for having too many images has its image limit lowered.
• Lean requests. Free tiers are metered in tokens per minute; every context field has a character cap so one large prompt cannot exhaust the budget for the requests that follow.
• Hygiene. API keys are whitespace-stripped on load; a single trailing newline in an environment variable makes every request fail with a protocol error.

14. Clients

• PWA chat. SSE stream of typed events (session, sentence, tool, screen, confirm, audio, done, error). Uploads go to a file endpoint first and are referenced by attachment lines. Media are served with HTTP range support. Link previews are fetched server-side with a private-address guard re-checked on every redirect. Messages can be deleted permanently, which also removes them from the agent’s live context.
• Voice. PCM16 16 kHz upstream; the client performs voice activity detection; PCM16 audio frames downstream with sequence numbers; barge-in reports the last chunk heard so the history is truncated correctly.
• Camera. Shared front-end module for both services; hands-free speech segments are transcribed server-side; a typing bar is available when audio is unreliable.
• Telegram and MCP. The same sessions; confirm-tier actions are declined where no confirmation UI exists.

Recorders keep one recorder per gesture with private chunk buffers; a shared buffer across two concurrent recorders produced WebM files without their EBML header, which no decoder can read.

15. Persistence and deployment

• PostgreSQL for everything stateful (conversations with summaries, memory, tasks, calls, files as base64, push subscriptions, key-value settings, audit log); SQLite only for local development.
• Conversation writes are buffered in memory and flushed in batches every few minutes and on shutdown; any read flushes first.
• Each service is one container on a PaaS with a managed Postgres. The desktop agent connects outbound, so the PC needs no open ports.
• Recent server errors (path, error, a short trace and request metadata such as an upload’s type and first bytes) are kept in a ring buffer readable by the owner, which made several field failures diagnosable in minutes.

16. Security model

• Single owner. Bearer keys per service, accepted from a header, a query parameter for WebSockets, or an HTTP-only cookie for the PWA. Services that support it accept a list of keys so each integration can be revoked independently.
• Tool output is untrusted data by construction (section 6).
• Side effects in the owner’s name are confirm tier everywhere; unattended contexts cannot confirm.
• The autonomous scheduler can only reach the owner.
• Secrets live in platform environment variables, never in the repository; errors mask bearer tokens; memory refuses secret-looking text.
• Server-side fetches of user-supplied URLs resolve the host and refuse private, loopback and link-local addresses.

17. Failure modes

18. Open problems

• Retrieval quality. Lexical relevance is cheap and predictable but misses paraphrase; embeddings would help at the cost of another model dependency.
• Evaluation. There is no offline benchmark for whether memory and playbook injection improve answers; today the evidence is the owner’s corrections going down. An exportable training set (every exchange with a correction flag) exists to build one.
• Autonomy calibration. The reply-within-two-hours signal is noisy; the owner may be busy rather than uninterested.
• Single process. The core keeps session state in memory; scaling beyond one instance would need that state moved to the database.
• Phone reach. Reliable ringing depends on a third-party app; a native client would remove that dependency.