Evals · run on every deploy
The assistant, measured in public.
Every build runs a golden set of 45 on-topic questions and 13 off-topic ones against the exact BM25 index the Worker serves. If retrieval regresses, it shows up here first.
corpus bb1f69edff80 · 184 chunks · score floor 2.5 · coverage floor 0.65 · ran
- hit@195.6%right page ranked first
- hit@3100%right page in the top three
- mrr0.978mean reciprocal rank
- refusal100%off-topic questions refused
- answer rate100%on-topic questions answered
- p50 search0.05msBM25 over 184 chunks
Golden set · 45/45 passing
On-topic questions
A case passes when an expected page ranks in the top three and the refusal gate lets the question through.
| Question | Top page | Rank | Score | Coverage | Result |
|---|---|---|---|---|---|
| What did he build for State Street? | work/state-street-vm-factory-image-bakery | 2 | 8.22 | 1.00 | pass |
| Disaster recovery with Azure Site Recovery | work/state-street-azure-landing-zone-dr | 1 | 11.79 | 1.00 | pass |
| Data centre migration to Azure for State Street | work/state-street-azure-landing-zone-dr | 1 | 16.83 | 1.00 | pass |
| How many VMs did he build for State Street? | work/state-street-azure-landing-zone-dr | 1 | 10.49 | 1.00 | pass |
| Splitting a large Terraform state into workspaces | work/state-street-terraform-enterprise-split | 1 | 12.43 | 0.71 | pass |
| How many Terraform resources did he split per application? | work/state-street-terraform-enterprise-split | 1 | 13.88 | 1.00 | pass |
| Moving Terraform runs into Terraform Enterprise and pipelines to Harness | work/state-street-terraform-enterprise-split | 1 | 11.35 | 1.00 | pass |
| What is the VM factory? | work/state-street-vm-factory-image-bakery | 1 | 7.80 | 1.00 | pass |
| How much faster are VM builds with the golden image? | work/state-street-vm-factory-image-bakery | 1 | 12.63 | 0.66 | pass |
| Golden image bakery with Azure Image Builder or Packer | work/state-street-vm-factory-image-bakery | 1 | 18.93 | 1.00 | pass |
| Crédit du Maroc GitLab upgrade | profile/experience/gitlab | 1 | 23.04 | 1.00 | pass |
| Does he work with GitLab Professional Services? | profile/experience/gitlab | 1 | 12.15 | 1.00 | pass |
| How much did the Five Below audit save? | work/five-below-azure-landing-zone-audit | 1 | 12.51 | 0.63 | pass |
| Blue/green deployments at PlanHub | work/planhub-aws-platform | 1 | 15.49 | 1.00 | pass |
| Azure DevOps to Harness migration for Mott MacDonald | work/ado-to-harness-migration | 1 | 23.90 | 1.00 | pass |
| Reusable GitLab CI templates with OIDC | work/gitlab-ci-platform-templates | 1 | 18.16 | 1.00 | pass |
| DevOps for an agentic benchmark platform on Cloud Run | profile/experience/turing | 1 | 14.13 | 1.00 | pass |
| Sandbox environments for training computer-use AI agents | work/agentic-benchmark-devops-ai-eval | 1 | 27.79 | 1.00 | pass |
| What did he do at Turing? | work/agentic-benchmark-devops-ai-eval | 1 | 6.01 | 1.00 | pass |
| Was he an SRE at PlanHub? | profile/experience/planhub | 1 | 14.06 | 1.00 | pass |
| Kubernetes platforms for RAK Bank and DU Telecom | work/gcc-kubernetes-platforms | 1 | 24.29 | 1.00 | pass |
| Migrating 30,000 websites to EKS | work/eks-30000-site-migration | 1 | 17.39 | 0.77 | pass |
| What is TalentLens? | projects/talentlens | 1 | 5.32 | 1.00 | pass |
| Agentic paper trading system | projects/agentic-paper-trader | 1 | 16.18 | 0.77 | pass |
| Local RAG platform retrieval evaluation | projects/local-rag-agent-platform | 1 | 16.06 | 0.78 | pass |
| Agent sandbox conformance tests | projects/agent-sandbox-conformance-kit | 1 | 13.23 | 1.00 | pass |
| OpenAI-compatible gateway for VS Code language models | projects/vscode-lm-openai-gateway | 1 | 25.87 | 1.00 | pass |
| YouTube Shorts generation pipeline | projects/shorts-engine | 1 | 10.72 | 0.60 | pass |
| Terraform modules for an Azure landing zone | projects/azure-landing-zone-terraform | 1 | 12.18 | 1.00 | pass |
| Jenkins pipelines on Kubernetes | projects/jenkins-k8s-pipeline-lab | 1 | 10.73 | 1.00 | pass |
| Interview coach for forward deployed engineers | projects/fde-coach | 1 | 15.24 | 1.00 | pass |
| 3D model of a house | projects/bhagamandala-house-3d | 1 | 13.56 | 1.00 | pass |
| Which certifications does he hold? | profile/certifications | 1 | 10.63 | 1.00 | pass |
| Is he a Certified Kubernetes Administrator? | profile/certifications | 1 | 13.81 | 1.00 | pass |
| Where did he study? | profile/education | 1 | 6.07 | 1.00 | pass |
| How can I contact him? | profile/contact | 1 | 9.67 | 1.00 | pass |
| What did he do at Capgemini? | profile/experience/capgemini | 1 | 8.47 | 1.00 | pass |
| What leadership experience does he have? | profile/experience/ensono | 1 | 5.38 | 1.00 | pass |
| Which observability tools has he used? | profile/skills/observability | 1 | 14.68 | 1.00 | pass |
| Has he worked on-site in Oman? | work/gcc-kubernetes-platforms | 1 | 9.22 | 1.00 | pass |
| How deep is his Kubernetes experience? | profile/experience/ozone | 1 | 6.34 | 1.00 | pass |
| Which clouds has he worked on? | profile/skills/cloud | 1 | 4.81 | 1.00 | pass |
| What does he do at Ensono? | profile/experience/ensono | 1 | 3.84 | 1.00 | pass |
| Has he taken LLM systems to production? | profile/skills/ai | 2 | 9.79 | 1.00 | pass |
| What is his phone number? | profile/contact | 1 | 6.36 | 1.00 | pass |
Refusals · 13/13 refused
Off-topic questions
These must be refused before any model is called. That keeps answers grounded and costs at zero.
| Question | Nearest page | Score | Coverage | Result |
|---|---|---|---|---|
| What is the capital of France? | – | 0.00 | 0.00 | refused |
| Write me a poem about cats | projects/agent-sandbox-conformance-kit | 3.12 | 0.19 | refused |
| Recipe for chocolate cake | – | 0.00 | 0.00 | refused |
| Does he know the Rust programming language? | projects/bhagamandala-house-3d | 4.84 | 0.24 | refused |
| What is his salary expectation? | work/planhub-aws-platform | 4.75 | 0.45 | refused |
| Who is the best footballer of all time? | profile/experience/intuitive | 2.46 | 0.21 | refused |
| Explain quantum entanglement | – | 0.00 | 0.00 | refused |
| Translate hello into Japanese | – | 0.00 | 0.00 | refused |
| What is the weather in Bengaluru today? | profile/contact | 4.43 | 0.27 | refused |
| Is he married? | – | 0.00 | 0.00 | refused |
| What is the meaning of life? | projects/gitlab-omnibus-upgrade-playbook | 3.03 | 0.27 | refused |
| Who won the cricket world cup? | projects/agentic-paper-trader | 5.53 | 0.45 | refused |
| How do I bake bread? | work/state-street-vm-factory-image-bakery | 3.35 | 0.40 | refused |
Method & limits
- The refusal gate has two signals: a minimum BM25 score, and IDF-weighted query coverage when a question mentions terms the corpus has never seen. Most off-topic questions are about something the site never mentions.
- Retrieval and refusal are measured deterministically at build time. Generated answers are not scored here, because no model is called during the build.
- At runtime the Worker also filters prompt-injection patterns, delimits context and drops any citation that doesn’t map to a retrieved chunk.
- The golden set is small and written by me, so treat it as a regression guard, not a benchmark.