Problem
Finding the correct safety recall from a natural-language vehicle complaint is difficult because user queries are short, noisy, and phrased differently from official recall descriptions. Pure keyword search misses semantic matches, while pure dense retrieval can lose exact defect terminology.
What I Built
Built a hybrid retrieval pipeline for automotive recall diagnostics that combines SentenceTransformer dense embeddings, BM25 keyword search, FAISS indexing, candidate pooling, and hybrid ranking. Added a benchmark-driven evaluation harness to measure retrieval quality on paraphrased diagnostic queries.
Architecture
- Dense retrieval with SentenceTransformer embeddings for semantic recall matching
- BM25 keyword search for exact terminology and defect-specific lookup
- FAISS vector indexing for efficient nearest-neighbor retrieval
- Candidate pooling and hybrid ranking to combine dense and lexical signals
- Evaluation harness for Recall@K and MRR on benchmark query sets
Results
Recall@1 (81-query benchmark)
0.94
Recall@10 (81-query benchmark)
1.00
MRR
0.967
The hybrid retrieval pipeline achieved strong first-result accuracy and perfect top-10 coverage on the extended benchmark, showing that combining dense and lexical retrieval substantially improved recall quality for paraphrased automotive diagnostic queries.
Benchmark Snapshot
Small benchmark (eval/recall_queries.jsonl)
| Metric | Value |
|---|---|
| Recall@1 | 0.90 |
| Recall@10 | 1.00 |
| MRR | 0.95 |
| Eval set | 10 queries |
Extended fixed benchmark (eval/recall_queries_100_fixed.jsonl)
| Metric | Value |
|---|---|
| Recall@1 | 0.94 |
| Recall@10 | 1.00 |
| MRR | 0.9671 |
| Eval set | 81 queries |
Key Insights
Why hybrid retrieval helped
Dense retrieval improved semantic matching for paraphrased complaints, while BM25 preserved exact recall-specific terminology such as defect names and component phrases. Pooling both candidate sets before ranking made the retrieval system much more reliable than either method alone.
Why evaluation mattered
The evaluation harness made it possible to test realistic paraphrased diagnostic queries instead of relying only on exact-text matching. Tracking Recall@1, Recall@10, and MRR made performance gains measurable and helped validate the final ranking strategy.
System Components
- Query normalization and structured retrieval inputs
- Dense embedding generation with SentenceTransformers
- BM25 lexical retrieval over recall text
- FAISS-based candidate retrieval
- Candidate pooling and hybrid score combination
- Benchmark evaluation using Recall@1, Recall@10, and MRR