RETRIEVAL / AI

AI RAG Knowledge Assistant.

A local support-knowledge prototype that retrieves runbook and incident context with sources.

PythonFastAPIMiniLMFAISSLangChainStreamlit Code

0.92retrieval MRR
30-query local benchmark · 18 tests

A REAL EXAMPLE

What happens

Ask a support question such as what to check after an HTTP 503. The local index returns relevant runbook and incident passages with their source names. The benchmark scores that retrieval step.

How I evaluated it 30 questions, scoring and limits Raw retrieval results Hit Rate@5, Recall@5 and MRR Architecture notes From documents to retrieved passages

Why I made it

I wanted to see whether a support assistant could find the right runbook before it tried to answer a question. I treated retrieval as its own problem so I could measure it.

How it works

Runbooks / incidentsChunk / embedFAISS indexTop-k retrievalSource-aware answer

Loaders and chunking for synthetic incidents, runbooks and FAQs.
MiniLM embeddings with a FAISS index that keeps source information with retrieved chunks.
FastAPI and Streamlit application layers for local use.

Choices I made

I kept evaluation questions out of the searchable corpus to avoid measuring leaked answers.
I compared MiniLM with a deterministic local baseline and reported retrieval metrics separately from answer quality.

What I checked

MiniLM: 100% Hit Rate@5, 100% Recall@5 and 0.92 MRR on 30 benchmark queries.
18 automated tests cover ingestion, retrieval, evaluation and application behavior.
The benchmark measures retrieved passages, not whether a final answer is correct.

Where it stands

The pipeline and benchmark were checked locally; there is no public deployment.
The repository still lists end-to-end API/UI validation as remaining work. A small synthetic benchmark does not show performance on real support data.

Next: SmartTask API