While backtesting helps you test a workflow version against historical data, it also has limitations when you don’t have enough historical data or the data doesn’t reflect the latest trends.
Experiments (a.k.a A/B testing) let you test workflow versions on the live traffic to address this problem. In addition, experiments also help you measure the performance accurately with ground truth. For example, a new version of a loan application workflow may approve more users, and you can measure the delinquency rate of this segment through experiments. That won’t be possible in backtesting since those users weren’t approved in the first place.
Starting today, you’ll be able to create and deploy experiments in Sperta. Each experiment uses an input feature such as a user ID to split the traffic into experiment groups. The control group should run the baseline workflow version, and treatment groups should run workflow versions that contain improvements.
Sperta is a no-code platform, so deploying experiments shouldn’t involve code changes either. With a click of a button, you can deploy experiments in Sperta to replace the currently deployed workflow version or experiment.
While the experiment is running, the experiment ID and workflow version ID will be returned in the workflow execution result. You can ingest the results into your data warehouse to join them with labels such as chargebacks and delinquency to calculate metrics such as precision, recall, approval rate, and delinquency rate.
Our next step is to calculate experiment metrics for you and remove one more friction point, stay tuned!
More from Sperta
