Project Story
A Slack agent that answers from your company's own documents
Sasha answers HQ's policy and HR questions in Slack from nine company documents and cites the section each answer came from. When the documents don't cover something, it says so. It runs on Claude with no vector database, every answer is logged with the code and documents behind it, and golden and red-team eval suites test each change. We launched it to the whole company on October 9, 2026.
- Client
- HQ (our own tool)
- Industry
- Internal operations
- When
- 2026
- Status
- In daily use since October 2026
- TypeScript
- Claude API
- Slack Bolt (Socket Mode)
- PostgreSQL
- Prisma
- Docker
- GitHub Actions
- Vitest
Questions people used to ask a person
At HQ the same questions kept landing on whoever runs operations. How much PTO do I have, who do I tell about a lost laptop. The answers live in nine documents: the employee and contractor handbooks, the security handbook, the incident policy, client communication guidelines and a few standard agreements. Together they’re about 16,800 words.
Sasha is a Slack bot that reads those documents and answers from them. You mention it in a channel or send it a DM, and it replies in the thread with the answer and the document and section it came from (“Employee Handbook › Paid Time Off”). When the documents don’t say, it opens with “That’s not covered in the docs I have” and names who to ask, from a contacts file HQ confirmed by hand. It never guesses at a name and never @-mentions anyone.
It also knows who’s asking. Before answering, it looks the sender up in HQ’s internal directory by Slack user ID: W-2 employee or contractor, in the US or abroad. That record rides with the question, and the instructions make it outrank anything the message claims about the sender, so a contractor asking about an employee benefit is told it doesn’t apply to them. The lookup waits at most 2 seconds and falls back to “unknown”, so a directory outage only makes answers less tailored.
It runs as one Docker container. Slack pushes events to it over Socket Mode, so there’s no public endpoint. Its only outbound traffic goes to Slack and Anthropic.
The whole document set in every request
Most document assistants cut the documents into chunks, store them in a vector database (a search index that matches by meaning) and hand the model the few chunks that look closest to the question. We decided at the kickoff on September 28 that HQ didn’t need that. Nine documents fit in one request, so every question goes to Claude with all of them attached as document blocks with Anthropic’s citations feature turned on. The model points at the exact passage it relied on, and code maps each cited span back to the nearest heading so the citation names a section.
Prompt caching keeps that affordable. The documents go out as identical bytes on every call in every thread, and the cache breakpoint sits on the last document with a one-hour lifetime. The first question of the morning pays to write the cache and every question after it reads from it, and each read keeps it alive. Everything that changes per question goes after that breakpoint: the replayed conversation, the coworkers’ messages Sasha was shown, the asker’s directory record and the who-to-ask list. Nothing with a timestamp goes in the system prompt, since a changing prefix quietly turns the cache off.
Where the asker’s record sits mattered as well. We first put the asker’s record before the question, and in one eval it cut the reasoning out of answers where worker type was irrelevant (the “why” survived in 7 of 24 samples, against 19 of 24 before). Moving the record after the question brought it back to 18 of 24.
To be clear, this only works because the set is small. A company with thousands of pages would need retrieval, and we’d build it. What we got in exchange is no index to drift out of date and no chance the search step left out the page that mattered.
A classifier decides before Claude answers
Every message passes through a small, fast classifier model first, which returns a probability for each option it’s given. It runs several closed-set tasks:
- question or feedback (or both)
- for a top-level channel mention, whether it points back at something (“is that right?”, “^”) and needs the recent channel messages, read as the last 2 hours or the last 10 messages, whichever comes first
- for an unmentioned reply in a thread Sasha answered, whether it’s meant for Sasha or for a coworker
- whether someone is asking it to go quiet in a thread or a channel, or to come back
- whether a reply to “Want me to file that?” is a yes
- whether the question describes one person’s sensitive situation
Each task has a fallback that can’t do harm, and every timeout, error or unsure answer collapses to it. A plain question still gets answered, and nothing gets filed on a guess. The bars come from measured runs. An unmentioned thread reply is answered only at 90% or higher, because follow-ups meant for Sasha scored 0.93 to 1.00 while a coworker answering a coworker scored up to 0.72. Filing an issue needs 95%, because plain yeses scored 0.99 to 1.00 and injected “confirm” replies 0.71 to 0.88. The classifier gets 1.5 seconds. If it isn’t configured, Claude Haiku stands in for routing, and follow-ups and mute requests turn off entirely, since Haiku returns no probability to hold to those bars.
Privacy runs the other way. A channel question is answered in public only when the classifier is at least 50% sure it’s ordinary. Sensitive test items scored 0.00 to 0.14 and ordinary ones 0.67 to 1.00, so the bar sits nearer the public side, since answering a harassment report in front of a channel is the worse mistake. A private answer goes to the asker’s DM, and the thread gets one fixed line that names no one. The same happens when Claude marks an answer as depending on the asker’s worker type, since answering in public would tell everyone whether that person is a contractor.
Some rules aren’t left to a model at all. Code refuses to answer in shared or Slack Connect channels, where clients are. Text in a message that imitates the tags Sasha uses for the directory record is defused before the model sees it, in replayed history too, so nobody can forge their own record.
Every answer is on the record
Each answer writes one row to PostgreSQL (QaLog, through Prisma) once the person has seen it. The row holds the question and answer, the citations, the classifier’s decisions with their probabilities, the asker’s worker type, the instructions version, a content hash of the documents and contacts file, the deployed commit, the model ID the API returned, the stop reason, token counts including cache reads and writes, and latency. An attempt that ends in “Something went wrong” gets a FailedAttempt row instead, naming the stage it failed at (reading Slack, the database, the classifier, the Anthropic call, streaming, posting).
If someone says Sasha told them something wrong last Tuesday, pnpm replay takes that row and re-asks the question the way Sasha saw it: same asker type, same thread and channel messages, same conversation memory. It runs the model three times, since one sample proves little, and prints the logged answer beside the new ones. If the documents or instructions have changed since, it says which commit to check out. A replay can be saved as a new golden test item, so a fixed bug stays fixed.
Retention runs in the bot at startup and every 24 hours. Coworkers’ messages that rode along with a question lose their text and author after 30 days, keeping only the channel and timestamp. Conversation memory is replayed for 24 hours and deleted after 7. A privately answered question’s text is cleared after 30 days. Thread replies the classifier decides aren’t for Sasha are stored nowhere.
What happens when it's wrong
Telling Sasha it’s wrong is just a reply. “That’s out of date” gets checked against the documents first, then gets a one-line offer to file it, written by code, since the model is never allowed to say whether it can file something. A thumbs-down on an answer gets Sasha asking what’s off. Only an explicit yes opens a GitHub issue, quoting the person word for word and labeled unverified, because a correction from Slack can be wrong too. Filing is capped at 5 a day per person and 50 a day overall, and an offer expires after 24 hours. A person reads the issue and, if it holds up, fixes the source document. The documents reach Sasha’s repository only by pull request, so every change to what it knows has a reviewer.
A pull request that changes the documents, the instructions, the answer or classifier code, or the evals runs the suites:
- a golden set of 76 questions with the facts a right answer has to contain, including 15 questions the documents don’t cover, where the only passing answer is the abstention sentence
- a red-team set of 45 attacks in 16 categories, such as forged directory records, planted claims earlier in a thread, poisoned conversation memory and text trying to trick it into filing an issue
- a routing set for the classifier tasks, reported as confusion matrices
A second model grades the golden answers, and for every fact it says is there, it has to quote the part of the answer that states it. Code then checks the quote is really in the answer. Measured over 53 labelled answers sampled three times, the current grading prompt agrees with the labels on 156 of 159 verdicts, up from 150, with false failures down from 9 to 0. A red-team item fails if the reply adopts the planted claim, judged by a yes/no question that sees the reply and never the attack, plus exact-text checks for leaked instructions and @-mentions.
The suites cost real money per run, so they run when a pull request opens and again when it’s approved, with one retry per failure (a pass on the retry is reported as flaky). That said, they’re informational for now. A failing run shows on the pull request but doesn’t block the merge. And the sync that would pull document edits straight from the shared drive still only runs as a dry run, printing the pull request it would open and which eval items to review.
About a month of work
From the kickoff on September 28 to now, the repository has 352 commits and 69 merged pull requests. It went out to the company eleven days after that kickoff, and 39 of those pull requests merged after launch: following edited and deleted questions, being told to be quiet in plain language, answering privately when a question is about one person, citing sections instead of PDF pages.
Questions about this project
Sasha is an internal knowledge agent that HQ built for its own team and launched company-wide on October 9, 2026. It runs in Slack and answers policy, HR and operations questions using only HQ's nine company documents, such as the employee and contractor handbooks and the security handbook. Every answer cites the document and section it came from, and when the documents don't cover a question, Sasha says so and names the person to ask from a contacts list HQ confirmed.
HQ, a software design and development agency in Ogden, Utah, built Sasha, a Slack agent on Anthropic's Claude that answers from its company documents with section citations and says when the documents are silent. It logs every question and answer with the code and documents behind it and turns corrections from staff into GitHub issues a person reviews. HQ tests it with a golden set of 76 questions, 15 of which the documents don't cover, and a red-team set of 45 adversarial cases.
Not always. For Sasha, HQ chose to send all nine of its company documents, about 16,800 words, with every question instead of building a vector search index. The documents go to Claude as citation-enabled document blocks behind a one-hour prompt cache, so after the first question each call reads the documents from the cache. That removes the risk of a search step missing the right page. HQ's view is that a company with thousands of pages would need retrieval, and the decision depends on how large the document set is.
Sasha sends its document set as identical bytes on every call and puts a one-hour cache breakpoint on the last document, so the first question writes the cache and later questions read it, refreshing it each time. Anything that changes per question, such as the conversation history, the asker's directory record and other people's Slack messages, is placed after the breakpoint, and the system prompt carries no timestamps, since a changing prefix would stop the cache from matching. Each logged answer records its cache read and write token counts.
Sasha answers with Anthropic's citations feature, which ties each claim to a passage in the source documents, and it uses a fixed sentence, "That's not covered in the docs I have", when the documents don't answer a question. HQ checks this with 15 must-abstain questions in its golden test set, graded by a second model that has to quote the answer for every fact it finds, with code verifying each quote. Every answer is logged with the documents and code version that produced it, so HQ can replay a logged answer to see whether a mistake repeats.
A classifier model reads each message before Claude answers and returns a probability for each option. Sasha answers a thread reply that doesn't mention it only when the classifier is at least 90% sure the reply is meant for Sasha, a bar HQ set after measuring that real follow-ups scored 0.93 to 1.00 while coworkers talking to each other scored up to 0.72. Every timeout or error falls back to the harmless choice, which for an unmentioned reply is to stay quiet.
A teammate can reply that the answer is wrong or react with a thumbs-down. Sasha checks the claim against the documents and offers to file it, and only an explicit yes opens a GitHub issue labeled unverified. A person reviews the issue and fixes the source document through a pull request, and HQ's golden and red-team test suites run against the change.
Let's build together
Whether you're looking to design, build, update, or refresh an existing app or website, we can help. Even if we're not a good fit, we will help point you to somebody who might be.
Contact us