KB Chatbot · 05 Sep 2026 · Open source

The chatbot was the easy part

Retrieval-augmented generation is a weekend of plumbing. Knowing whether the answers are any good is the actual work. The interesting half of this project is the dashboard that tells me when they are not.

MIT kb-chatbot on GitHub, or run it yourself

I built this for myself, as a proof of concept. I wanted to experiment with RAG a while back, and it turned into a project I kept adding things to, one slice at a time. Nobody else has ever used it. That's why the satisfaction rate further down reads zero.

The Chat tab. A question about hybrid search, a streamed answer describing BM25 plus vector search merged with Reciprocal Rank Fusion, thumbs up and down buttons, and a Sources used table listing the document, match type and similarity score. — The Chat tab. Every answer is followed by the chunks behind it: document, match type, similarity, snippet. The top hit here scored 0.613. Check the score before you blame the model.

The Observability tab. Token and cost tracking showing total cost, tokens and requests, then cost broken down by chatbot and by phase: chat response, ingest embed, chat rewrite and embed query, each with average latency. — Cost per phase. The whole run came to nine cents, of which ingesting all seventeen documents was less than a tenth of one. Answering is the expensive half, and you pay it per question rather than per document.

The Evaluation tab. Total conversations, total messages, no-answer rate, average top similarity and satisfaction rate, above a conversation list with a failures-only filter and a JSON inspector. — The Evaluation tab. Satisfaction reads zero because nobody has rated an answer, not because anyone rated one badly. That metric needs users, and this run had none. These figures include one question I typed by hand after the run finished, so they read 23.1% and 0.521 where the fixed twelve-question set below gives 25.0% and 0.511.

Four formats in

PDF, DOCX, HTML and TXT. A background worker parses, chunks and embeds each one.

Answers with receipts

Every response shows the chunks behind it, and how well each one matched.

Follow-ups that hold

Each follow-up is rewritten into a standalone query before retrieval runs.

One dependency

Docker. Database, queue, object storage and UI all come up together.

How to use it

Five tabs, in the order you need them

From a PDF you have not read to a number telling you whether the answers are any good, the whole loop takes about five minutes. The last tab tells you what it cost.

Create a chatbot
The Chatbots tab. Give it a name; it appears in the list. One chatbot per document set. They do not share an index.
Upload documents
The Documents tab takes PDF, DOCX, HTML and TXT. Each lands as pending while a background worker parses it, splits it into chunks and embeds each one - five to thirty seconds, depending on size - then flips to ready.
Ask it something
The Chat tab streams the answer token by token. Underneath it, a Sources table lists the chunks that were retrieved and how well each matched, so you can see whether a bad answer came from bad retrieval. Follow-ups are rewritten into standalone questions first; you see the original, not the rewrite.
Find out whether it worked
The Evaluation tab, which is the one I actually care about. No-answer rate, mean retrieval similarity, satisfaction from thumbs, and a failures-only filter that takes you straight to the conversations where retrieval found nothing.
See what it cost
The Observability tab totals spend and tokens over a lookback window, then splits them by chatbot and by phase: chat_response, ingest_embed, chat_rewrite and embed_query, each with its average latency. You can see which half of the pipeline your money goes to. Prices come from the vendors' published rates, so the total is an estimate.

Aspect	The usual approach	KB Chatbot
Retrieval	Vector similarity alone	Hybrid: vector plus keyword
Follow-up questions	Sent to retrieval as typed	Rewritten into a self-contained query first
Showing its work	The answer only	Every retrieved chunk, with its similarity score
Knowing if it works	Read a few answers and judge	A dashboard over every conversation

Not a benchmark. Nothing here was measured against another system. Comparing retrieval quality honestly means the same corpus, the same questions and the same judge, and I have not done that.

Check it yourself

The dashboard is the point

Reading a dozen answers tells you how the bot performs on a dozen questions you thought to ask. It tells you nothing about the long tail, which is where the failures live. Every claim below is something the running app will show you in under a minute.

Claim	What verifies it	Where
Retrieval returns chunks that are actually relevant	Sources table, per-chunk similarity	Chat tab
It declines instead of inventing when nothing matches	No-answer rate; failures-only filter	Eval tab
Follow-ups are resolved before retrieval, not after	Ask one, then read the Sources table	Chat tab
Individual answers can be traced end to end	Inspect a conversation by ID	Eval tab
Readers find the answers useful, not just plausible	Satisfaction rate from thumbs up/down	Eval tab
It comes up clean on a machine that is not mine	docker compose up -d; curl :8000/health	Shell

What it found

The dashboard earned its keep in an hour

I pointed it at the seventeen slice specs that describe this app - 157 chunks, sixteen seconds cold - and asked twelve questions: nine answerable from the corpus, three deliberately not. It found two bugs before I'd finished reading the answers.

The first was mine and it was embarrassing: a chunker that looped forever on any sentence it couldn't fit, which is every Markdown code fence in the corpus. Fifteen of the seventeen files hung the worker at 100% CPU with no error and no output. My earlier test files were small and plain, so it had never fired.

The second was more interesting, because nothing was broken. Asked “which authentication modes are supported?”, retrieval returned the right document ranked first, second, third, fifth, seventh and ninth, and then threw all of it away. The top chunk scored 0.343 against a similarity floor of 0.40. The floor was a constant I'd picked by feel, and it sat inside the range of correct answers rather than below it.

Every question in the run, placed by the similarity of its best-matching chunk. The two groups separate cleanly, with nothing at all between 0.284 and 0.343. The floor I had chosen sat well to the right of that gap, discarding correct answers that simply scored low. Only one answerable question sits left of the old floor, though the table below loses two: the second cleared it at 0.462, but the same floor stripped all its supporting chunks bar one, and one chunk wasn't enough to answer from.

Moving the floor into the gap, and turning it into a setting, changed the run:

Measure	Floor 0.40	Floor 0.32
Answerable questions answered	7 of 9	9 of 9
Unanswerable correctly declined	3 of 3	3 of 3
No-answer rate	41.7%	25.0%
Mean top-1 similarity	0.549	0.511

Seventeen documents, one embedding model, and a fixed set of twelve questions: nine answerable from the corpus, three not. Every row is that set, before and after. Enough to fix my own constant; not enough to recommend 0.32 to anyone else, which is why it is now a setting you can tune.

The no-answer rate didn't just improve. It changed meaning. At 41.7% it was mixing two opposite events: three correct refusals on questions with no answer in the corpus, and two documents it simply failed to find. At 25.0% it's exactly the three unanswerable questions and nothing else.

Which should have been the end of it.

Instead I opened the dashboard, read “25% no-answer rate”, and thought for a moment that something was still wrong. Nothing was. Nine of nine answerable questions were answered and three of three unanswerable ones were declined. 25% is the target. A zero there would mean the bot had confidently invented a parental leave policy. I wrote the question set, built the dashboard and named the metric, and the label still caught me out, because no-answer rate reads like a failure rate no matter who is looking at it. It caught me a second time an hour later, when the figure had drifted to 23.1% and I went looking for the regression: I'd simply asked it one more question.

That is a flaw in the instrument. Renaming it wouldn't fix it: messages.no_answer records what the bot did, never whether it was right to do it. One column is carrying two opposite events. Splitting them needs a question set that knows which questions ought to have been answerable in the first place, which is the labelled set and the judge I keep saying I'll build next.

And mean similarity fell, from 0.549 to 0.511, while the system got strictly better. Results that had been discarded for scoring low are now included, which drags the average down.

MIT · Docker

Run it yourself

Everything runs in Docker; the only host dependency is Docker itself. Two API keys and five minutes, and it's answering questions about your own documents, or reproducing the run above against mine.

View the repository

Needs Docker with Compose v2, an OpenAI key for embeddings and an Anthropic key for answers. Demo auth mode accepts any bearer token. Do not expose it as-is.

# clone, configure, start
git clone https://github.com/avestea/kb-chatbot
cd kb-chatbot
cp .env.example .env          # add OPENAI_API_KEY and ANTHROPIC_API_KEY
docker compose up -d
docker compose exec api alembic upgrade head

# confirm, then open the UI
curl localhost:8000/health    # → {"status":"ok","db":"connected","redis":"connected"}
open http://localhost:7860

What it's built on

Postgres holds the documents, chunks and embeddings; Redis is the worker queue; MinIO stands in for S3 so uploads behave identically on a laptop and in a deployment. Gradio is the entire frontend, which is the piece I would replace first for anything real. It bought me a working UI in a day and charges for it in flexibility. The dependency worth abstracting first, though, is the embedding provider: it's one call behind an interface, and it's also the thing that decides how good retrieval is. Per question you pay one embedding call and one completion, plus a small rewrite on follow-ups: Haiku for the rewrite, Sonnet for the answer. That is where nearly all the money goes: ingestion is paid once per document and came to about one percent of the run.

Postgres Redis MinIO / S3 Gradio Alembic Docker Compose text-embedding-3-small Claude Sonnet 4.6 Claude Haiku 4.5

Where the numbers come from

The Evaluation tab counts conversations and messages, the no-answer rate, mean retrieval similarity, and satisfaction from thumbs. What it can't see is the question nobody asked because the previous answer was bad. It also can't tell a confidently wrong answer that earned a thumbs up from a right one. Both need a person reading transcripts, which is what the inspector is for.

localhost:7860 → Evaluation → Refresh Eval tab

What I would do differently

The ordering was wrong. I built retrieval, then the chat UI, then hybrid search, then query rewriting, and only then, at slices fourteen and fifteen, the feedback buttons and the dashboard. Which means every improvement before that point was justified by intuition. Building the measurement first would have told me which of those changes actually mattered. When I finally did measure, it caught a constant I'd chosen by feel and never checked, and then caught me misreading my own dashboard about it.

The other surprise: query rewriting was the change I expected least from and noticed most. Follow-ups are where a naive pipeline quietly falls apart, because “what about the second one?” retrieves nothing at all. Without the Sources table you'd never see why.

If your corpus fits inside a modern model’s context window, retrieval may be pure overhead: paste the documents in, skip the pipeline, get better answers with less machinery. RAG starts earning its complexity when the corpus is too large to fit, changes too often to re-paste, or has to be attributed chunk by chunk. And if you want a knowledge base chatbot rather than an education in building one, buy a hosted product. The distance between a weekend build and a maintained service is a year of operational detail that never reaches anyone’s README.