Growth through tryouts
Specifics of testing hypotheses in a growing startup
Unreliable data
Metrics are crucial for every company. Nevertheless, their importance changes correspondingly with the amount of users you have.
When there aren't enough users, data is almost random. Out of curiosity, we monitored a conversion funnel and, for the first half-year, mostly saw noise.
That's why the primary strategy at this stage was testing through implementation.
Our primary focus at this point was speed. Every week, ~5 features and updates were implemented.
To keep up such speed, I used a few techniques:
Quick design iterations with feedback. We sought feedback after every design iteration and did so as often as it made sense. Feedback could be collected from the team or random people
Switch to a parallel task when stuck on something. This allowed us to break a loop of thoughts that are formed in such cases. Finding a solution was much easier when I returned to the initial problem
Simplify until making it simpler is impossible. The same feature can be designed in different ways. To move fast, we had to choose the simplest possible UI, layout, and logic (even when it slightly reduced UX)
Thanks to the iterative approach, the product drastically changed in a year. It started generating revenue, and we've got enough users to use A/B tests.
The age of A/B tests
The hypotheses for the tests came from various sources: my own ideas, other team members' ideas, user interviews, Hotjar recordings, and Userbob tests.
The payment modal was an interesting example of A/B test usage.
It consisted of mandatory elements that couldn't be removed, so we could just tweak them a little. Experiments started from changing texts (the title and the description) and visual appearance later. None of them led to a significant improvement.
Meanwhile, moving a few words to a more visible location made a huge difference.
Adding a link to a pricing page was even too successful.
Half of the system functionality was blocked for free users. These pages contained explanations and a link to a payment modal.
We mentioned that there was no easy way for a logged-in user to check prices. One way was opening settings (in 3 unobvious clicks), and another was through the blocked functionality (also in 3 unobvious clicks).
We added the corresponding item to a user menu. Out of curiosity, we added a highlighted variant, which won with an unexpected advantage.
We had a few ideas of making it less extravagant but hadn't the opportunity to test them.
Results
Tests became an essential part of the product evolution. When we finisher our work, there were ~30 simultaneously ongoing A/B tests in production.
Thanks to endless experiments, the product changed drastically. From a confusing, sloppy design at the start, it became a solid product with quickly growing revenue.