Most "AI workshop" tooling is either a whiteboard with sticky notes or a slide deck. I built something different. Evidence to Decision is a governed, evidence-first pipeline. A facilitator captures what people actually said, the room votes and clusters live on a shared mural, an Azure AI agent drafts a starting problem framing from that evidence alone, a reviewer approves or blocks it, and only then does it become a delivery document. Every step keeps the paper trail. This is why I shaped it that way, and what it took to build on Azure.

I spent the better part of a day convinced that Azure Monitor SLI alerting could not be deployed as infrastructure as code. I wrote a parallel implementation to work around it.

I was wrong, and the way I was wrong is the useful part.

Azure Monitor now does SLIs and SLOs natively, with error budgets and burn rate alerting, which is good news if you have been hand-rolling multi-window burn rate rules out of metric alerts for years. I have been building a demo that pairs Azure Monitor SLIs with the Azure SRE Agent on AKS, all deployed with the Azure Developer CLI, and I wanted every part of it in infrastructure as code.

Getting the infrastructure up was the easy part. Getting a single SLI created against Managed Prometheus metrics took considerably longer, and almost none of the reasons were in the documentation. This post is the list of things I wish I had known before I started, in the order they bit me.

If you only read one line: the metric namespace for Managed Prometheus metrics is customdefault, not prometheus. That one value cost me the most time by a wide margin.

Most Kubernetes monitoring tells you that a pod restarted. It rarely tells you whether a customer noticed, which customers were affected, or whether the business promise is now at risk.

That gap is a product and service-design problem as much as a monitoring problem. The people responsible for a payment journey need to know whether checkout works, how quickly it works, and which customer segment is paying the cost of an incident. The on-call engineer needs enough evidence to choose the right first action. The business needs a trustworthy way to decide when an error budget has been spent.

I wanted to see whether an Azure Monitor SLI could give the Azure SRE Agent enough context to investigate an Azure Kubernetes Service failure, rather than just react to a pod alert. I built a small payment service, chose signals around the customer journey, and deliberately broke it.

The result is an infrastructure-as-code deployment that runs with azd up. It measures availability, latency, the full request journey, the public path, and customer tiers. Azure Monitor alerts on those service-level signals, and the Azure SRE Agent investigates the incident and, when allowed, fixes it.

The important result is not the number of Azure resources. It is the chain from customer need to operational decision:

I got several things wrong. The corrections stay in the post because they expose the difference between a design that looks plausible and one that has been tested with the people, decisions, and failure modes it is meant to serve.

Every codebase that has lasted long enough builds up at least one layer like this. Request stubs call into a wiring dict: a lookup table of small lambda functions, one per route, that exists only to find the right handler. Those lambdas call translation functions that move data between an old dispatch shape and what the framework now wants on its own. It worked when it was written. Every new feature since has paid a cost: it goes through three extra steps to do something the framework could do directly if you let it.