It seems like almost every week over the past two years since ChatGPT’s release, new large language models (LLMs) from rival labs or OpenAI itself have been released. Companies are finding it difficult to keep up with the massive pace of change, let alone understand how to adapt to it — which of these new models should they adopt, if any, to power their workflows and the custom AI agents they are building to accomplish them?
Help arrived: AI Application Observability Bootstrapping raindrop launched Experimentsa new analytics feature that the company describes as the first A/B testing suite designed specifically for enterprise AI agents – allowing companies to see and compare how updating agents to new underlying models or changing their instructions and access to tools will affect their performance with real end users.
The release extends Raindrop’s existing observability tools, giving developers and teams a way to see how their agents behave and evolve in real-world conditions.
With Experiments, teams can track how changes (like a new tool, prompt, model update, or complete pipeline refactoring) affect AI performance across millions of user interactions. The new feature is now available to users of Raindrop’s Pro subscription plan ($350 monthly) at raindrop.ai.
A data-driven lens on agent development
Co-Founder and Chief Technology Officer at Raindrop Ben Hylak noted in a product announcement video (above) that Experiments help teams see “how literally everything has changed,” including tool usage, user intent, and issue rates, and explore differences by demographic factors like language. The goal is to make model iteration more transparent and measurable.
The Experiments interface presents results visually, showing when an experiment performs better or worse than its baseline. Increases in negative signals may indicate greater task failure or partial code output, while improvements in positive signals may reflect more complete responses or better user experiences.
By making this data easy to interpret, Raindrop encourages AI teams to approach agent iteration with the same rigor as modern software deployment – tracking results, sharing insights, and addressing regressions before they worsen.
Background: From AI Observability to Experimentation
Raindrop’s release of Experiments builds on the company’s foundation as one of the first Native AI Observability Platformsdesigned to help companies monitor and understand how their generative AI systems behave in production.
As reported by VentureBeat earlier this year, the company – originally known as Dawn AI – emerged to solve what Hylak, a former human interface designer at Apple, called the “black box problem” of AI performance, helping teams detect failures “as they happen and explain to companies what went wrong and why.””
At the time, Hylak described how “AI products constantly fail – in ways both hilarious and terrifying,” noting that unlike traditional software, which throws clear exceptions, “AI products fail silently.” Raindrop’s original platform focused on detecting these silent failures by analyzing signals like user feedback, task failures, declines, and other conversational anomalies across millions of daily events.
The company’s co-founders— Hylak, Alexis Gaubáand Zubin Singh Koticha – built Raindrop after encountering firsthand the difficulty of debugging AI systems in production.
“We start by building AI products, not infrastructure,” said Hylak VentureBeat. “But we quickly realized that to develop something serious, we needed tools to understand AI behavior – and those tools didn’t exist.”
With Experiments, Raindrop extends the same mission as detecting faults to measuring improvements. The new tool turns observability data into actionable comparisons, allowing companies to test whether changes to their models, prompts or pipelines actually make their AI agents better or just different.
Solving the “Evals Pass, Agents Fail” problem
Traditional assessment frameworks, while useful for benchmarking, rarely capture the unpredictable behavior of AI agents operating in dynamic environments.
As co-founder of Raindrop Alexis Gaubá explained in it LinkedIn Ad“Traditional assessments don’t really answer this question. They’re great unit tests, but you can’t predict user actions and your agent is running for hours calling hundreds of tools.”
Gauba said the company consistently heard a common frustration from teams: “Assessments pass, agents fail.”
The experiments aim to fill this gap, showing what really changes when developers push updates to their systems.
The tool allows side-by-side comparisons of models, tools, intents, or properties, revealing measurable differences in behavior and performance.
Designed for real-world AI behavior
In the announcement video, Raindrop described the experiments as a way to “compare anything and measure how your agent’s behavior actually changed in production across millions of real interactions.”
The platform helps users identify issues such as spikes in task failures, forgetfulness, or new tools that trigger unexpected errors.
It can also be used the other way around – starting with a known problem, like an “agent stuck in a loop,” and tracing back to which model, tool, or flag is driving it.
From there, developers can dive into detailed traces to find the root cause and quickly submit a fix.
Each experiment provides a visual breakdown of metrics such as tool usage frequency, error rates, conversation length, and response duration.
Users can click on any comparison to access the underlying event data, providing a clear view of how agent behavior has changed over time. Shared links make it easy to collaborate with teammates or report findings.
Integration, scalability and accuracy
According to Hylak, Experiments integrates directly with “the feature signaling platforms companies know and love (like Statsig!)” and is designed to work seamlessly with existing telemetry and analytics pipelines.
For companies without these integrations, it can still compare performance over time (like yesterday versus today) without additional configuration.
Hylak said teams typically need about 2,000 users per day to produce statistically significant results.
To ensure the accuracy of comparisons, Experiments monitors sample size adequacy and alerts users if a test does not have enough data to draw valid conclusions.
“We are obsessed with ensuring that metrics like task failure and user frustration are metrics you would call an engineer on call for,” Hylak explained. He added that teams can drill down into specific conversations or events that drive these metrics, ensuring transparency behind each aggregated number.
Security and Data Protection
Raindrop operates as a cloud-hosted platform, but also offers on-premises personally identifiable information (PII) redaction for businesses that need additional control.
Hylak said the company is SOC 2 compliant and has released a PII Guard feature that uses AI to automatically remove sensitive information from stored data. “We take the protection of customer data very seriously,” he emphasized.
Prices and Plans
The experiments are part of Raindrop Professional planwhich costs $350 per month or $0.0007 per interaction. The Pro tier also includes deep research tools, topic clustering, custom issue tracking, and semantic search capabilities.
raindrop Initial plan — $65 per month or $0.001 per interaction — Offers basic analytics including issue detection, user feedback signals, Slack alerts, and user tracking. Both plans come with a 14-day free trial.
Larger organizations may opt for a Business plan with custom pricing and advanced features like SSO login, custom alerts, integrations, edge PII redaction, and priority support.
Continuous Improvement for AI Systems
With Experiments, Raindrop positions itself at the intersection of AI analytics and software observability. Its focus on “measuring the truth,” as stated in the product video, reflects a broader push within the industry toward accountability and transparency in AI operations.
Rather than relying solely on offline benchmarks, Raindrop’s approach emphasizes real user data and contextual understanding. The company hopes this will enable AI developers to move faster, identify root causes earlier, and confidently deliver better-performing models.
avots
