Ajiya Shinabaze

A support inbox that answers from sources, and knows when not to

A Gmail agent that filters, classifies, retrieves from a curated knowledge base, drafts, gets judged, and only sends automatically what is safe.

Role
Design, retrieval and prompt engineering, evaluation
Period
2026
Status
In production
Stack
  • n8n
  • Gemini
  • Pinecone
  • Gmail API
  • Supabase
Support agent: retrieve, draft, judge, and only then send Workflow diagram. Steps: New email (Gmail poll + reconcile); Who not to answer (system · suppliers · briefs); Skipped, logged (a human owns it); Classify intent (Gemini); Multi-query retrieval (Pinecone knowledge base); Daily learning loop (corrections → KB); Draft a grounded reply (cites retrieved passages); LLM judge (rubric · revise on fail); Safe to auto-send?; Auto-send (clear, low-risk answers); Draft for a human (buying intent · doubt). Connections: New email to Who not to answer; Who not to answer to Skipped, logged [skip]; Who not to answer to Classify intent; Classify intent to Multi-query retrieval; Daily learning loop to Multi-query retrieval [updates]; Multi-query retrieval to Draft a grounded reply; Draft a grounded reply to LLM judge; LLM judge to Draft a grounded reply [revise]; LLM judge to Safe to auto-send?; Safe to auto-send? to Auto-send [yes]; Safe to auto-send? to Draft for a human [no]. skip updates revise yes no New emailGmail poll + reconcile Who not to answersystem · suppliers · briefs Skipped, loggeda human owns it Classify intentGemini Multi-query retrievalPinecone knowledge base Daily learning loopcorrections → KB Draft a grounded replycites retrieved passages LLM judgerubric · revise on fail Safe to auto-send? Auto-sendclear, low-risk answers Draft for a humanbuying intent · doubt
Filtering happens before any model runs. The judge can send a draft back for revision; only clear, low-risk answers are sent without a human.

The problem

Inbound email mixes prospects, clients, suppliers, bounces and spam. A fast wrong answer (a price that isn’t sold, a reply to a supplier’s invoice, a pitch to a paying client) costs more than a slow right one.

The system

  1. Polls the inbox every 15 minutes with a reconcile pass for missed messages, and filters system mail, suppliers and client-brief threads before any model runs.
  2. Classifies intent and context, then queries the knowledge base from several angles so one phrasing does not decide what is retrieved.
  3. Drafts a reply grounded in the retrieved passages and sends it to an LLM judge, which can return it for revision.
  4. Only clear, low-risk answers are sent automatically; anything with buying intent or doubt stays a draft for a human. A daily loop feeds corrections back into the knowledge base.

Key decisions

Prices come from the storefront’s own source of truth

WhyThe pricing section of the knowledge base is generated from the file the website uses, and a verifier fails when the two disagree.

Trade-offChanging a price means regenerating and re-ingesting, not editing a paragraph.

Re-ingest with a verified purge

WhyVector ids are positional, so a shorter rewrite leaves stale chunks that still get retrieved. Ingestion waits until the index is really empty and reads samples back afterwards.

Trade-offA short window with no knowledge base during each re-ingest.

Decide who not to answer before the model runs

WhySuppliers, and paying clients replying about their briefs, belong to a human.

Trade-offSome messages a model could have handled go to the founder instead.

Results

  • 50+ node workflow with judge-gated auto-send
  • Graded against a golden set of intent scenarios
  • Knowledge base checked against live storefront prices

What I’d do next

Turn the judge rubric into a versioned evaluation suite that runs on every prompt change.