“If you have a 5% buffer on retention before this price test winner is no longer a winner, maybe don't roll that one out. If retention can go down by 50% and that thing's still a winner — close your eyes and roll it out. Sensitize it to say: if retention comes in 10% worse, does it change my answer?”

Price increases hurt annual retention more than monthly — sensitize tests before rolling out

Falzon describes a recurring mistake at Mosaic: price tests run against monthly subscribers produced misleading signals about annual subscriber impact. Monthly retention data arrived quickly; annual renewal data took 12+ months. Teams would roll out a price increase that looked like a winner on monthly data, then a year later discover annual churn spiked and the test was actually a loss. The fix: before rolling out any price test, model the range of retention impacts and check whether the outcome is still positive under the pessimistic scenario.