In a rules engine, deploying a new workflow version is scary: how do you know the new version performs better than the old one? Rules are so powerful, and mistakes could mean rejecting all of your customers in the worst case. Backtesting solves this problem by letting you test the workflow against historical data to gain more confidence in the new version.
Sperta helps customers make real-time decisions to manage fraud, credit, and compliance risks, and we understand how important the scale of the backtesting dataset is. With more data, you have more confidence in the metrics you calculate from the backtesting results. This is especially true for fraud, which often has an imbalanced dataset. This means if you use 100 samples in a backtest and only one of the samples is labeled fraudulent, you would have a hard time convincing anyone about the precision and recall you calculated from the backtest.
That’s why when we set out to build backtesting in Sperta, we designed it to be a distributed and scalable solution. Currently, a backtest can run tens of thousands of samples, and it can finish in a few minutes. We’re in the process of adding capacity, and we’ll soon support millions of samples in a backtest.
To start a backtest, you just need to choose the workflow version you want to test and upload a CSV file containing the historical data. Each row in the CSV represents a sample, such as an application or transaction. After the backtest finishes, you can download the test result, where the decision and output features are appended as extra columns to the CSV. You can then import the test result into your data warehouse or BI tool to calculate metrics such as precision, recall, and delinquency rate.
Today’s launch adds a key capability for our customers to improve the business metrics of decision workflows. In the future, we’ll make it more scalable, support backtesting against stored workflow execution results, and help users calculate metrics.
More from Sperta
