Auditing every AI inference

The sentence Wend is built around is “every fact has a source, and you approved it.” What that actually requires under the hood is a discipline most AI products skip: treat every inference as an audit event.

We didn’t set out to write a logging system. We started out wanting to answer a different question. When an AI extraction goes wrong (Wend says Kevin is the CTO of the wrong company, a draft references the wrong last meeting), how do we find out what the model actually saw?

The honest answer was that we couldn’t. Most logging frameworks treat AI calls the same as any other side effect: a span in your tracing tool, maybe a request body in your error tracker. That’s enough to debug latency, not enough to debug behavior. So we built an inference-aware audit table from the start, and then realized we could turn the same data into a user-visible dashboard. This is the design of both.

The schema

One Postgres table, audit_log, accumulates a row per AI call. The columns:

create table audit_log (
  id uuid primary key default gen_random_uuid(),
  user_id uuid references auth.users(id) not null,
  event_type text not null,         -- 'ai_extraction_run',
                                    -- 'ai_drafting_run',
                                    -- 'ai_digest_synthesis_run', ...
  model_used text,                  -- the model that ACTUALLY served
                                    -- the call, e.g. 'DeepSeek-V4-Flash'
  provider text,                    -- 'azure', 'voyage', 'anthropic'
  input_tokens int,
  output_tokens int,
  cost_usd numeric(12, 6),
  latency_ms int,
  metadata jsonb not null default '{}'::jsonb,
                                    -- source IDs touched, conflict id,
                                    -- cache token counts, error message
                                    -- when the call threw
  ip_address inet,
  user_agent text,
  country_code text,
  created_at timestamptz default now()
);

create index audit_log_user_recent
  on audit_log (user_id, created_at desc);

A few non-obvious choices worth flagging:

• Token counts, not content. We log how many input and output tokens a call used, never the prompt or completion text. The audit log itself is a privacy surface, and putting full prompts in a row would mean anyone with read access to the table sees the user’s data again, outside its original context. Token counts give us everything we need for cost accounting and for showing usage back to the user. (One narrow exception, disclosed below: an allowlisted deep-trace mode we run on our own QA accounts.)
• The model gets named. This column used to hold an internal tier label (“cheap,” “reasoning,” “vision”). We changed it to record the model that actually served the call, alongside a provider column, for two reasons. Users asking “what read my email?” deserve a specific answer, and the abstraction was hiding a cost bug from us: routed calls were being priced against the fallback model’s rate card, which overstated our own spend by roughly 4x. A label that hides which model ran is a label that eventually lies to you too.
• Source IDs in metadata. Every detail Wend stores has a source_id pointing at a sources row. The audit_log row references those IDs so “why did the model say this?” traces all the way back to the email, file, or chat message that fed the inference.
• Append-only. RLS forbids UPDATE and DELETE from any user session. Even our own staff can’t edit history. If a user purges their account, we delete their rows wholesale rather than mutating them.

The logging pattern

Every AI call in Wend goes through one wrapper. The wrapper’s job is to invoke the model, measure the inference, and write the audit row. It looks like this:

// src/lib/ai/record-call.ts
export async function recordAiCall<T>(
  ctx: RecordContext,                 // supabase, userId, eventType, model
  fn: () => Promise<CallTelemetry<T>>,
): Promise<T> {
  const start = Date.now();
  let result: T;
  let errorMessage: string | undefined;
  try {
    const telem = await fn();
    result = telem.result;
    // telemetry the provider handed back: token counts, cache reads,
    // duration, and served_model (the model that really ran)
    ...
  } catch (err) {
    errorMessage = err instanceof Error ? err.message : "unknown";
    throw err;                        // the caller still sees the failure
  } finally {
    const latencyMs = Date.now() - start;
    const effectiveModel = servedModel || ctx.model;
    const costUsd = estimateCostUsd({ model: effectiveModel, ... });
    try {
      await ctx.supabase.from("audit_log").insert({
        user_id: ctx.userId,
        event_type: ctx.eventType,
        model_used: effectiveModel,
        provider: provider ?? providerFromModel(effectiveModel),
        input_tokens: inputTokens ?? null,
        output_tokens: outputTokens ?? null,
        cost_usd: costUsd > 0 ? costUsd : null,
        latency_ms: latencyMs,
        metadata,                     // includes error when it threw
      });
    } catch {
      // Best-effort. An audit-log failure never breaks the AI call.
    }
  }
  return result!;
}

Calling code looks like:

const entities = await recordAiCall(
  {
    supabase,
    userId: user.id,
    eventType: "ai_extraction_run",
    model,
    metadata: { source_id: emailSource.id },
  },
  async () => {
    const r = await callModel({ ... });
    return {
      result: parseEntities(r),
      input_tokens: r.usage?.input_tokens,
      output_tokens: r.usage?.output_tokens,
      served_model: r.model,          // what actually ran
    };
  },
);

Two details in there are the whole design. The insert happens in a finally block, so a call that throws still leaves a row behind with the error in metadata. Silent failures are the ones that cost you a weekend. And the wrapper is the path of least resistance: reach for recordAiCall and you get cost accounting, latency, provider attribution, and the audit row for free. Skipping it means writing more code, not less.

The stack it records, as of July 2026: extraction and chat run on open-weight models (DeepSeek and Kimi) hosted in a US Azure region, with embeddings, vision and transcription on Azure OpenAI in the same region. None of those providers train on Wend users’ data, and we never will either. Because model_used and provider are per-row rather than per-deploy, a routing change shows up in the audit log the same day it ships, rather than in a documentation update six weeks later.

What the user sees

The same rows power /app/usage, a page every signed-in user can open at any time. It shows:

• A 30-day series of tokens and inference counts per day, charted.
• Month-to-date tokens in and out, and month-to-date cost in dollars.
• A breakdown by event type, so you can see whether the spend went to extraction, drafting, or the digest.
• Research activity. Contact research runs on a research provider account you connect yourself, and the page shows how many lookups have run.
• What Wend is holding for you: people, organizations, relationships, promises made versus kept, and the number of sources behind them. Computed live from your own rows in your own session.

That mock is the shape of a row in audit_log rather than a screenshot of the page. One inference, one row, and the tail of the row is the provenance: the source that fed the call. Its model column is drawn from an earlier version of the table, when we stored a tier label instead of the exact model. In production that cell now reads something like DeepSeek-V4-Flash.

The one exception, stated plainly

There is a second logging mode, and hiding it in a footnote would undercut the whole post. Accounts listed in a WEND_DEBUG_USER_IDS environment variable get verbose debug_trace rows: full prompts, full model output, every tool call and result, the raw captured page text. It exists because diagnosing a bad extraction from token counts alone is guesswork, and because we would rather debug our own graph than ask a user for theirs.

The rules around it: the allowlist holds our own accounts, tracing is a no-op for everyone not named in it, rows are size-capped so a trace can never balloon the table, and failures never propagate into the pipeline. If we ever needed this on a customer account to chase a specific bug, the honest version is asking that customer first. An escape hatch you refuse to describe in public is just a backdoor with better manners.

Check this in the source, not in this post

Everything above is a claim about code you cannot see, which is exactly the problem with trust posts. So we published the layer the audit log hangs on. wend-core is the engine behind Wend, open source under AGPLv3. The parts of this post you can read in the repo:

• schema/20260504_004_graph_instance_layer.sql defines the sources table and, more to the point, declares source_id as not null references public.sources (id) on both node_details and link_details. A fact with no source cannot be inserted. That is the constraint the audit trail resolves against.
• schema/20260505_009_chat_schema.sql defines pending_writes, the queue every AI-proposed fact sits in until a human confirms it.
• src/core/apply.ts is the commit path: what actually happens when you approve a proposal, dedup and all.
• src/core/tool-definitions.ts is the agent tool surface, and the interesting thing about it is an absence. There is no confirm capability in it. An agent can propose; it cannot approve its own proposal.

Two honest caveats. The audit_log DDL and the usage dashboard shown above are part of the hosted product and are not in the public repo today. And a schema file proves the shape of the guarantee, not that our production database is running that exact file. What it does buy you is a much cheaper way to catch us: if the code that governs writes says something different from what this post says, the diff is public.

Why this matters

AI products built on closed models have two trust problems stacked: trust the product, and trust the model behind it. You can’t fully solve the second one, since you don’t get to inspect a hosted model’s weights. You can solve the first one by making everything around the model inspectable.

That’s what an audit log buys us. The user can answer, on their own, without asking us:

• How many AI calls is Wend making on my behalf this week?
• Which features cost the most?
• Did Wend touch this email I’m worried about?
• If a fact is wrong, what specifically was the model looking at?

None of those answers require a support ticket. None require us to grant ourselves read access to a user’s data to investigate. The data the user needs to evaluate us is exposed in the product itself.

The argument got sharper once outside agents started writing to the same graph. Wend ships an MCP server, so Claude, ChatGPT, Gemini, and Cursor can read a user’s memory and propose additions to it. That is several models, owned by several companies, touching one store of who you know. In that world “which model said this, on whose behalf, off which source, and who approved it” is not observability trivia. It is the only thing standing between a relationship memory and a rumor mill. Every agent write lands in the same proposal queue and the same audit trail as our own, because we built the trail before we had guests.

What this doesn’t solve

Audit logging is necessary but not sufficient for trust. Things this design does not give you:

• Model accuracy. The audit log will tell you that a draft was generated. It will not tell you the draft was correct. Output quality is a separate problem.
• Provider trust. The audit log records which provider and model served a call. It cannot prove that provider didn’t retain the payload. For that you rely on the contract. Ours runs open-weight models on our own US Azure deployments under terms that forbid training on the data, and our subprocessor list is published rather than described. What the log gives you is the ability to check the story against the rows: if we claimed one provider and the table says another, you caught us.
• The cold-start problem. A new user with an empty audit log gets no comfort from the page until they’ve actually used the product. The first week’s trust has to be carried by your security page and your reputation.

Closing

The framing I’d push back on: that “AI transparency” is a UX layer you can bolt onto a finished product. It’s an architectural decision. Either every inference flows through a single audited wrapper, or some don’t. The ones that don’t are exactly the ones you’ll wish were logged when something breaks.

For Wend, the audit log started as a debugging tool and turned into our cleanest privacy proof. The most expensive part was deciding what not to log (the prompts and outputs themselves), because the table that records your inferences is itself a record of your data. Build it like you’d want someone else to build it on top of your private graph.

Adjacent posts: we open-sourced the engine · how our region architecture changed · why Wend never auto-updates a fact