SENTINEL
Secured Enterprise RAG Gateway
Answers questions over a private security corpus with citations — and refuses when the sources don't support an answer.
Feature-complete, pre-release
Verdict space
- GROUNDEDan answer with visible citations, entailed by retrieved sources
- INSUFFICIENTnothing retrieved clears the confidence threshold
- current state: REFUSEDthe input was unsafe or adversarial and never reached retrieval
95.6%
Attacks blocked
43 of 45 adversarial attacks across 60 injection/jailbreak probes, 0% false-positive rate. Probes are self-authored — the number is a floor for known attack shapes.
+35.2%
Recall over dense baseline
hybrid + rerank vs dense-only, 29 labeled questions, pool-bounded recall
12,098
Indexed chunks
from ~180 public documents (~103 PDFs), after a clean reindex with junk-chunk filtering
1,023
Manually labeled candidates
drawn from all three retrieval modes to reduce single-method bias
What this is
Sentinel is a security-hardened Retrieval-Augmented Generation gateway that answers natural-language questions over a private cybersecurity document corpus. Every answer is grounded in source documents, returned with visible citations, and defended against prompt injection and unsafe inputs.
You ask a question. Sentinel searches a curated library of cybersecurity documents — NIST publications, CISA guidance, OWASP standards, cloud security docs, AI safety papers — retrieves the most relevant passages, generates an answer strictly from those passages, and shows you exactly which sources back each claim.
Crucially, Sentinel knows when not to answer. If a question falls outside its corpus, it declines rather than inventing an answer. If a prompt tries to manipulate or jailbreak the system, it refuses. If a generated answer isn't actually supported by the retrieved sources, it's flagged.
Built to give trustworthy, verifiable answers instead of confident hallucinations.
What it does
Three honest states, and no fourth.
The system returns exactly one of three things: a grounded answer with citations, an “I don't have enough information” response, or a refusal for unsafe input. There is no path that produces a plausible-sounding answer with nothing behind it.
- Grounds every answer
- Responses are generated strictly from retrieved passages, and citation metadata is rewritten from the retrieved chunks rather than taken from model output — so citations can't be fabricated by the generator.
- Declines off-corpus questions
- Retrieval scores are checked against a refusal threshold. On-corpus questions score near the top of the range; off-corpus questions score near zero, giving a wide, reliable separation.
- Blocks adversarial input
- Every query passes through an input guardrail before anything else runs. Prompt-injection attempts and malicious content are stopped there.
- Verifies its own output
- An entailment check confirms each claim is actually supported by the retrieved sources, catching subtle hallucinations before the answer is returned.
- Redacts sensitive data
- A PII redaction pass runs over the answer before it leaves the system.
- Ships its evaluation
- A committed evaluation suite — a human-labeled retrieval benchmark and an adversarial probe set — with every caveat documented rather than hidden.
Stack
| Backend API | PythonFastAPI |
|---|---|
| Frontend | Next.jsReact |
| Vector database | Qdrant, containerised via Docker |
| Embeddings | Together AI — intfloat/multilingual-e5-large-instruct |
| Generation | Together AI — Qwen2.5-7B-Instruct-Turbo |
| Input safety | Together AI — Llama Guard |
| Reranking | Cohere Rerank (cross-encoder) |
| Observability | LangSmith tracingin-process p50 metrics endpoint |
| Version control | GitGitHub |
A single /query endpoint fronting a defence-in-depth sequence. Each stage has exactly one job and exactly one authority — notably, the input guardrail is the only layer permitted to emit a refusal status, keeping safety decisions in one auditable place instead of scattered across the pipeline.
How it works
5 sectionsThe query pipeline
A request to the /query endpoint flows through a defence-in-depth sequence of five stages. Two of them can terminate the request.
- 1 · Input guardrail
- Every incoming query first passes through Llama Guard, which screens for unsafe or adversarial input. Prompt-injection attempts and malicious content are blocked here. This is the only layer permitted to emit the refused_unsafe status — a deliberate architectural boundary that keeps safety refusals in one auditable place.
- 2 · Hybrid retrieval
- Clean queries are embedded and searched against the Qdrant vector store. Retrieval combines dense semantic search (HNSW) with BM25 sparse retrieval, then Cohere reranks the candidate passages so the most relevant chunks rise to the top. Embeddings use E5's asymmetric prefix convention — query: for questions, passage: for documents — enforced through exact tokenizer truncation at 500 tokens to preserve embedding fidelity.
- 3 · Confidence threshold
- Retrieval scores are checked against a refusal threshold. On-corpus questions score near the top of the range; off-corpus questions score near zero, producing a wide, reliable separation. If nothing clears the bar, Sentinel returns the honest insufficient-information state instead of forcing an answer.
- 4 · Generation
- Qwen2.5-7B generates a response constrained strictly to the retrieved passages, operating in JSON mode with post-validation guaranteeing structural correctness.
- 5 · Output safety
- Before returning, the answer is verified two ways. An LLM-as-NLI grounding check — a single batched judge call — confirms each claim is actually entailed by the retrieved sources, catching subtle hallucinations. A regex PII redaction pass strips sensitive data. Citation metadata is rewritten directly from the retrieved chunks, never from model output, so citations can't be fabricated by the generator.
Input guardrail
can terminate as: REFUSED_UNSAFEHybrid retrieval
Confidence threshold
can terminate as: INSUFFICIENT INFORMATIONGeneration
Output safety
The corpus
Roughly 180 public cybersecurity documents (~103 PDFs) — NIST Special Publications, CISA advisories, OWASP standards, Microsoft and Google Cloud security documentation, and AI safety research. These are processed into 12,098 chunks after a clean reindex with junk-chunk filtering.
Chunking uses a recursive strategy — chosen for proof-of-concept practicality over semantic chunking — with even per-document sampling rather than first-N selection. That's a deliberate choice to avoid over-capturing front matter and title pages, which is what first-N sampling quietly does to a corpus of standards documents.
Reliability engineering
Sentinel exposes in-process latency metrics (p50) through a /metrics/trace endpoint backed by a lock-guarded ring buffer, plus an /ingest/status endpoint for corpus health.
The grounding judge uses a single batched call by design. An earlier per-claim sequential approach caused a 767-second hang from unbounded client timeouts, so batching is both a performance fix and a stability fix.
Evaluation and honest numbers
Sentinel ships with a committed evaluation suite, not just claims.
- Retrieval benchmark
- A 30-question, human-labeled relevance benchmark with 1,023 manually labeled candidates, drawn from all three retrieval modes to reduce single-method bias. The hybrid-rerank pipeline outperforms a dense baseline by +35.2% recall, +30.8% precision, +13.0% MRR, and +27.3% nDCG@10 across 29 labeled questions.
- Adversarial evaluation
- A 95.6% attack block rate with a 0% false-positive rate across 60 injection and jailbreak probes — 43 of 45 attacks blocked.
- Labeling audit
- A mid-project labeling audit caught and corrected an artifact where hybrid-rerank appeared to underperform due to a labeling gap. It was resolved through uniform re-labeling from a shared candidate pool.
Layered security architecture
The defences are deliberately layered rather than concentrated in one filter: regex filtering, Llama Guard input moderation, schema-enforced citation-by-construction, per-claim grounding verification, and PII redaction. Each catches a different failure shape, and no single one is trusted to catch everything.
Citation-by-construction is the load-bearing idea. Rather than asking the model to cite and then checking whether it lied, every claim is constrained to cite a retrieved chunk by construction, and the citation metadata is rewritten from the retrieval result. The generator is never in a position to invent a source.
Demo
Recording in progress
A grounded answer with citations, an off-corpus question being declined, and an injection attempt being refused.
What's next
RoadmapThreshold tuning at scale
The current refusal threshold is a conservative placeholder. The real on-corpus/off-corpus score spread supports a more precisely calibrated value as the corpus grows.
Semantic chunking
Upgrading from recursive to semantic chunking for higher retrieval precision.
Document override hardening
Current filename-keyed override maps are fragile — redelivering a document under a new filename breaks the mapping. Moving to content-stable keys is planned.
Expanded adversarial coverage
The current block rate is a floor for known attack shapes. Broadening the probe set will strengthen guarantees against novel attacks.
Corpus expansion
Growing the corpus while keeping the same honest-metrics discipline as the system scales toward production readiness.
Everything above is built. Everything here isn't yet — listed as work, not as a promise.