“We have a price increase — oh great you know we did amazing, we rolled this — and then six months later you look at the cohort and say oh actually the control group is performing better than the test group because they renew more.”

Don't call a paywall win until the cohort has had time to renew — or not

Conversion lifts are misleading on their own. A price increase that boosts short-term revenue can quietly suppress renewal rates for months. The winning move is to set a calendar reminder — three to six months out — to revisit any experiment that changed price, discount, or plan structure before declaring it permanently rolled out.