Superweek 2022 took place after a year-long pandemic-enforced break and felt more like a school reunion than a conference. Just the chance to see many familiar faces — and a surprising number of new ones — brought a welcome burst of energy and motivation to work. Conversations during breaks or at evening networking events have enormous value in their own right. But Superweek is about more than just chatting in the spa (which remains a pleasant side attraction at the hotels that host the conference).
What value do analysts deliver?
Several talks addressed the topic of the value that analysts and their work provide. I was surprised to find that even today, reporting monkeys can still exist in the market — people whose work ends with producing some kind of overview (especially painful if they are manually copying data from multiple sources into it). I am slightly less surprised that in some organisations an analyst can be more of a status symbol — a demonstration that the company is “data-driven” (whatever management imagines that to mean).
While a developer’s work is visible — a more or less functional application is produced that more or less meets customer requirements, with a long shelf life (in the case of operating systems or specialist software, years or even decades) — the work of an analyst is often more reactive (answering ad hoc questions) and analysts frequently serve as bearers of bad news (unpleasant numbers are easier to blame on a measurement error than on a management failure).
How can an analyst deliver greater value in such conditions?
Automate and build
Inspiration can be found among developers. At Proof & Reason, we automate routine tasks and free up our hands and mental capacity for more strategic thinking. We build tools that speed up the implementation of measurement codes or, for example, the evaluation of tests and surveys. Spotify took a similar approach, creating a neat framework for faster evaluation of dozens of running A/B tests.
Nurture data culture and quality
Company culture also matters a great deal — as does the ability to look at data without needing to submit a request to an analyst. Open access to data fosters a data culture within the company and again frees analysts’ hands for more strategic and valuable work. It does not matter whether this is just a set of auto-updating dashboards for each team, or a detailed data catalogue and open access to the data warehouse.
Equally important is maintaining data quality. If the organisation does not trust its data, there is no reason to look at it. And if there is no reason to look at the data, an analyst is worthless to the organisation — even if it values them as proof of its data-driven approach.
How to work under conditions of uncertainty?
Was the drop in sales caused by a botched campaign? Is there no longer demand for the product? Or was the weather simply too nice and nobody felt like shopping? Could it just be a random fluctuation? And what about a measurement error?
Web analytics has never been able to capture 100% of traffic data. Some is lost to ad blockers, a little more to browsers, measurement services occasionally go down, sometimes a user’s connection is so slow that measurement requests do not get sent in time. Today these losses are compounded by cookie banners and growing user concern about online privacy. Uncertainty in analytical work is only deepening. There are two ways to deal with this:
- Find a different job.
- Learn to live with it.
At Proof & Reason, we love data, and like other agencies worldwide we look for ways to cope with reality as best we can.
“We will never gather enough precise data. Analysts have to learn to live with that.”
Learn to love randomness and experimentation
And statistics.
Cookie banners have caused a drop in measured data on many projects (if you are not seeing a drop in yours, you quite possibly have it implemented in a way that conflicts with the regulator’s guidance and are prioritising your own interests over visitor comfort — short-term functional, long-term short-sighted; or you simply have visitors who genuinely enjoy your cookies). While some performance agencies that rested on their laurels and forgot to innovate are panicking and suggesting clients break the law, we prefer statistical modelling. A 20–30% drop in data can be modelled quite credibly with a bit of statistics and mathematics.
Attribution models are a chapter in themselves. They have been discussed for several years, and Google Analytics 4 has introduced its own data-driven attribution model (available even in the free version). But every attribution model carries a certain bias.
For example, last-click models forget about the sources that brought in the lead and without which customers would never have heard of you (the attention/see stage of the purchase cycle).
Overly aggressive remarketing can also take credit for customers who would have purchased even without the ad — but clicked on it because it was more readily available than organic results.
We run into similar issues when evaluating changes. Was the growth in sales caused by the optimised product page, or by a seasonal uptick in demand? Fortunately, we already know the solution to this type of question: random assignment of visitors to experiments, such as A/B or multivariate tests.
Focus on collecting the right data
We will never have enough data. There is always something else that can be measured. It can always be measured more precisely. But sooner or later we reach a point where the marginal utility of more data decreases — while the cost of more precise, better, and otherwise marvellous measurement rises.
That is why we ask about goals. That is why we create measurement plans. We focus on the areas that deliver the greatest value — and for the rest we settle for approximation or secondary data collection. It is more important to measure orders accurately than, say, category filters, whose usage we can roughly observe from heatmaps.
With shortening cookie lifespans (and increasingly limited options for third-party cookies), accurate user measurement and the ability to keep as much measurement under your own control as possible are becoming more important. This is a textbook use case for server-side tracking. Yes, it is more demanding both technically and financially. But it is also one of the few ways not to fall behind. And the earlier you start down this path, the greater the lead you will gain.
The Google ecosystem is moving in this direction, and Google Analytics 4 together with the server-side version of Google Tag Manager are already at a stage where we can recommend implementing them for all clients (and we would of course be happy to help with that).
“Google Analytics 4 and server-side measurement should feature in your plans for the next year or two.”
Do not forget common sense
Data tells us a lot, but not everything. And we do not always interpret it without our own biases.
Keep context in mind. Walking 78 km over two months is achievable even for an untrained senior citizen. That said, if we are talking about summiting Mt. Everest, it is a challenge even for trained athletes.
Think about what numbers actually mean. Are 1,000 followers a lot or a little? Is it even the right metric? And for what exactly?
Apply common sense. If newsletter sign-up has a low conversion rate, showing a pop-up to a visitor who just clicked through from the newsletter will not fix it. Equally, you can save yourself a lengthy analysis of underperforming product ads by first checking whether the ad’s message matches the website content. Like when an ad claims all sizes are in stock — but only a single item in extra-small is left on the shelf.
Sometimes you simply do not need more data to optimise a website. Sometimes it is enough to ask yourself whether it would work in real life.
Want to know more? Want to discuss the trends?
Our analysts Honza and Ondřej, together with Lukáš Čech from Etnetera Activate, will share further observations at an analytics Google Meet on Monday, 28 February. Check out the event on Facebook and join us. There will be plenty of time for questions and discussion.
If you would like help with your analytics roadmap for 2022, write to us at data@proofreason.com.
Article by: Jan Kadlec, data analyst
