I wanted to share an article that’s been making the rounds of my online circles. It’s Jesse Frederik and Maruits Martijn’s The new dot com bubble is here: it’s called online advertising. The point of the article is exploring whether online advertising even works, and how we know whether it does.
The article goes into several wys that one can test whether a thing has an effect. These naturally get mathematical. Among the tests developed is one that someone who didn’t know mathematics might independently invent. This is called linear regression, or linear correlation. The idea is to run experiments. If you think something causes an effect, try doing a little of that something. Measure how big the effect is. Then try doing more of that something. How big is the effect now? Try a lot. How big is the effect? Do none of it. How big is the effect?
Through calculations that are tedious but not actually hard, you can find a line that “best fits” the data. And it will tell you whether, on average, increasing the something will increase the effect. Or decrease it. There are subsidiary tests that will tell you how strong the fit is. That is, whether the something and the effect match their variations very well, or whether there’s just a loose correspondence. It can easily be that random factors, or factors you aren’t looking at, are more important than the something you’re trying to vary, after all.
In principle, online advertising should be excellent at matching advertising to people. It’s quite easy to test different combinations of sales pitches and measure how much of whatever it is gets bought. In practice?
You have surely heard the aphorism that correlation does not prove causation, usually from someone trying to explain that we can’t really prove that some large industry is doing something murderous and awful. But there are also people who will say this in honest good faith. Showing that, say, placing advertisements in one source correlates with a healthy number of sales does not prove that the advertisements helped any. One needs to design experiments thoughtfully to tease that out. Part of Frederik and Martijn’s essay is about the search for those thoughtful experiments, and what they indicate. There is an old saw that in science what one does not measure one does not understand. But it is also true that measuring a thing does not mean one understands it.
(Linear regression is far from the only tool available, or discussed in the article. It’s one that’s easy to imagine and explain, both in goal and in calculation, however.)