The fastest way to run an A/B test for 18 million visitors

Without upgrading an existing system

The problem

The client wanted to launch a new design of their homepage. Before doing so, it was important for them to test the reaction of their UK market users and the viability of the new design.

Their existing CMS didn't have the capabilities to create the new content structure and their existing website wasn't able to render the new layout. Their in-house team of developers weren't able to update their systems within the strict deadline for the test.

The solution

After discussing the problem with key members of the client, we found out that they already had a tried and tested solution for preparing website content. A team of editors and translators would prepare the data in a Google Spreadsheet and this would be input into their CMS by a separate editor.

We felt that the fastest and most optimal solution was to utilise their existing processes and save them the need to manage an additional set of data. This was done by developing a web application that would process and render the Google Spreadsheet data.

The result

After the six week test period, the data was collected, analysed, and compared to the data from their initial homepage design. The results showed them which elements worked and how the layout could be improved.

The success of the project led to the sales team requesting additional builds for two different projects using the techniques used for the homepage test. These new projects required the additional functionality of loading data based on the region selected on the website.

The web application was updated to work with the new updates and requirements. The data was processed for each translation in the Google Spreadsheet and rendered based on the region requested by the user.