Watching these metrics isn’t much different from caring for an old-school collection of microservices, and often the tools aren’t much different, either. Many of the most popular tools are the same dominant market leaders that have been extended to collect data from the various GPUs, too.

Leading vendors: DataDog, Dynatrace, New Relic, Splunk
Upstarts: Ariz, LangSmith, Braintrust, Galileo
Open-source alternatives: OpenTelemetry, Phoenix, Langfuse

Cost engineering

Many AI pipelines are often more complex than regular microservice architectures, and that means they need someone to carefully watch the bill. Plus, as attention to AI ROI rises, capturing spend is essential — especially given how quickly AI tokens can add up.