My 2018 Mathematics A To Z: Box-And-Whisker Plot

Today’s A To Z term is another from Iva Sallay, Find The Factors blog creator and, as with asymptote, friend of the blog. Thank you for it.

Cartoon of a thinking coati (it's a raccoon-like animal from Latin America); beside him are spelled out on Scrabble titles, 'MATHEMATICS A TO Z', on a starry background. Various arithmetic symbols are constellations in the background.
Art by Thomas K Dye, creator of the web comics Newshounds, Something Happens, and Infinity Refugees. His current project is Projection Edge. And you can get Projection Edge six months ahead of public publication by subscribing to his Patreon. And he’s on Twitter as @Newshoundscomic.

Box-And-Whisker Plot.

People can’t remember many things at once. This has effects. Some of them are obvious. Like, how a phone number, back in the days you might have to memorize them, wouldn’t be more than about seven or eight digits. Some are subtle, such as that we have descriptive statistics. We have descriptive statistics because we want to understand collections of a lot of data. But we can’t understand all the data. We have to simplify it. From this we get many numbers, based on data, that try to represent it. Means. Medians. Variance. Quartiles. All these.

And it’s not enough. We try to understand data further by visualization. Usually this is literal, making pictures that represent data. Now and then somebody visualizes data by something slick, like turning it into an audio recording. (Somewhere here I have an early-60s album turning 18 months of solar radio measurements into something music-like.) But that’s rare, and usually more of an artistic statement. Mostly it’s pictures. Sighted people learn much of the world from the experience of seeing it and moving around it. Visualization turns arithmetic into geometry. We can support our sense of number with our sense of space.

Many of the ways we visualize data came from the same person. William Playfair set out the rules for line charts and area charts and bar charts and pie charts and circle graphs. Florence Nightingale used many of them in her reports on medical care in the Crimean War. And this made them public and familiar enough that we still use them.

Box-and-whisker plots are not among them. I’m startled too. Playfair had a great talent for these sorts of visualizations. That he missed this is a reminder to us all. There are great, simple ideas still available for us to discover.

At least for the brilliant among us to discover. Box-and-whisker plots were introduced in 1969. I’m surprised it’s that recent. John Tukey developed them. Computer scientists remember Tukey’s name; he coined the term ‘bit’, as in the element of computer memory. They also remember he was an early user, if not the coiner, of the term ‘software’. Mathematicians know Tukey’s name too. He and James Cooley developed the Fast Fourier Transform. The Fast Fourier Transform appears on every list of the Most Important Algorithms of the 20th Century. Sometimes the Most Important Algorithms of All Time. The Fourier Transform is this great thing. It’s a way of finding patterns in messy, complicated data. It’s hard to calculate, though. Cooley and Tukey, though, found that the calculations you have to do can be made simpler, and much quicker. (In certain conditions. Mostly depending on how the data’s gathered. Fortunately, computers encourage gathering data in ways that make the Fast Fourier Transform possible. And then go and calculate it nice and fast.)

Box-and-whisker plots are a way to visualize sets of data. Too many data points to look at all at once, not without getting confused. They extract a couple bits of information about the distribution. Distributions say what ranges a data point, picked at random, are likely to be in, and are unlikely to be in. Distributions can be good things to look at. They let you know what typical experiences of a thing are likely to be. And they’re stable. A handful of weird fluke events don’t change them much. If you have a lot of fluke events, that changes the distribution. But if you have a lot of fluke events, they’re not flukes. They’re just events.

Box-and-whisker plots start from the median. This is the second of the three things commonly called “average”. It’s the data point that half the remaining data is less than, and half the remaining data is greater than. It’s a nice number to know. Start your box-and-whisker plot with a short line, horizontal or vertical as fits your worksheet, and labelled with that median.

Around this line we’ll draw a box. It’ll be as wide as the line you made for the median. But how tall should it be?

That is, normally, based on the first and third quartiles. These are the data points like the median. The first quartile has one-quarter the data points less than it, and three-quarters the data points more than it. The third quartile has three-quarters the data points less than it, and one-quarter the data points more than it. (And now you might ask if we can’t call the median the “second quartile”. We sure can. And will if we want to think about how the quartiles relate to each other.) Between the first and the third quartile are half of all the data points. The first and the third quartiles the boundaries of your box. They’re where the edges of the rectangle are.

That’s the box. What are the whiskers?

Well, they’re vertical lines. Or horizontal lines. Whatever’s perpendicular to how you started. They start at the quartile lines. Should they go to the maximum or minimum data points?

Maybe. Maximum and minimum data are neat, yes. But they’re also suspect. They’re extremes. They’re not quite reliable. If you went back to the same source of data, and collected it again, you’d get about the same median, and the same first and third quartile. You’d get different minimums and maximums, though. Often crazily different. Still, if you want to understand the data you did get, it’s hard to ignore that this is the data you have. So one choice for representing these is to just use the maximum and minimum points. Draw the whiskers out to the maximum and minimum, and then add a little cross bar or a circle at the end. This makes clear you meant the line to end there, rather than that your ink ran out. (Making a figure safe against misprinting is one of the understated essentials of good visualization.)

But again, the very highest and lowest data may be flukes. So we could look at other, more stable endpoints for the whiskers. The point of this is to show the range of what we believe most data points are. There are different ways to do this. There’s not one that’s always right. It’s important, when showing a box-and-whisker plot, to explain how far out the whiskers go.

Tukey’s original idea, for example, was to extend the whiskers based on the interquartile range. This is the difference between the third quartile and the first quartile. Like, just subtraction. Find a number that’s one-and-a-half times the interquartile range above the third quartile. The upper whisker goes to the data point that’s closest to that boundary without going over. This might well be the maximum already. The other number is the one that’s the first quartile minus one-and-a-halt times the interquartile range. The lower whisker goes to the data point that’s closest to that boundary without falling underneath it. And this might be the minimum. It depends how the data’s distributed. The upper whisker and the lower whisker aren’t guaranteed to be the same lengths. If there are data outside these whisker ranges, mark them with dots or x’s or something else easy to spot. There’ll typically be only a few of these.

But you can use other rules too. Again as long as you are clear about what they represent. The whiskers might go out, for example, to particular percentiles. Or might reach out a certain number of standard deviations from the mean.

The point of doing this box-and-whisker plot is to show where half the data are. That’s inside the box. And where the rest of the non-fluke data is. That’s the whiskers. And the flukes, those are the odd little dots left outside the whiskers. And it doesn’t take any deep calculations. You need to sort the data in ascending order. You need to count how many data points there are, to find the median and the first and third quartiles. (You might have to do addition and division. If you have, for example, twelve distinct data points, then the median is the arithmetic mean of the sixth and seventh values. The first quartile is the arithmetic mean of the third and fourth values. The third quartile is the arithmetic mean of the ninth and tenth values.) You (might) need to subtract, to find the interquartile range. And multiply that by one and a half, and add or subtract that from the quartiles.

This shows you what are likely and what are improbable values. They give you a cruder picture than, say, the standard deviation and the coefficients of variance do. But they need no hard calculations. None of what you need for box-and-whisker plots is computationally intensive. Heck, none of what you need is hard. You knew everything you needed to find these numbers by fourth grade. And yet they tell you about the distribution. You can compare whether two sets of data are similar by eye. Telling whether sets of data are similar becomes telling whether two shapes look about the same. It’s brilliant to represent so much from such simple work.

Reading the Comics, July 16, 2016: More To Life Than Mathematics Edition

I know, it’s impolitic for me to say something like my title. But I noticed a particular rerun in this set of mathematically-themed comics. And it left me wondering if I should drop that from my daily routine. There are strips I read more out of a fear of missing out than anything else. Most of them are in perpetual reruns, though some of them are so delightful I wouldn’t dare drop them. (Here I mean Cul de Sac and Peanuts.) An individual comic takes typically little time to read, but add that up and it does take a while, especially on vacation or the like. I won’t actually change anything; I’m too stubborn in lazy ways for that. But it crosses my mind.

Tim Lachowski’s Get A Life for the 14th is what set me off. Lachowski’s rerun this before, and I’ve mentioned it before, back in March of 2015 and back in November 2012. Given this I wonder if there’s a late-2013 or early-2014 reuse of the strip I failed to note around here. Or just missed, possibly because I was on vacation.

Nicholas Gurewitch’s Perry Bible Fellowship reprint for the 14th gives me the title for this edition. It uses symbols and diagrams of mathematics for their graphical artistry, the sort of thing I’m surprised doesn’t get done more. Back in college the creative-writing-and-arts editor for the unread leftist weekly asked me to do a page of physics calculations as an aesthetic composition and I was glad to do it. Good notation has a beauty to it; I wonder if people would like mathematics more if they got to spend time at play with its shapes.

Morrie Turner’s Wee Pals rerun for the 14th name-checks the New Math. The New Math was this attempt to reform mathematics in the 1970s. It was great for me, and my love remembers only liking or understanding mathematics while in New Math-guided classes. But it was an attempt at educational reform that didn’t promise that people at the cash registers would make change fast enough, and so was doomed to failure. (I am being reductive here. Much about the development of New Math went wrong, and it’s unfair to blame it all on the resistance of parents to new teaching methods. But educational reform always crashes hard against parents’ reasonable question, “Why should my child be your test case?”)

Many of the New Math ideas grew out of the work of Nicholas Bourbaki, and the attempt to explain mathematics on completely rigorous logical foundations, as free from intuition as possible to get. That sounds like an odd thing to do; intuition is a guide to useful ways to spend one’s time and energy. But that supposes the intuition is good.

Much of late 19th and early 20th century mathematics was spent discovering cases in which intuitive understandings of things were wrong. Deterministic systems can be unpredictable. A curve can be continuous at a single point and nowhere else in space. Infinitely large sets can be bigger or smaller than other sets. A line can wriggle around so much that it has a volume, it fills space. In that context wanting to ditch intuition as a once-useful but now-unreliable guide is not a bad idea.

I like the New Math. I suppose we always like the way we first learned things. But I still think it’s got a healthy focus. The idea that mathematics is built on rules we agree to use, and that we are free to change if we find they’re not doing things we need, is true. It’s one easy to forget considering mathematics’ primary job, which has always been making trade, accounting, and record-keeping go smoothly. Changing those systems are perilous. But we should know something about how to pick tools to use.

Zoe Piel’s At The Zoo for the 15th uses the blackboard-full-of-mathematics image to suggest deep thinking. (Toby the lion’s infatuated with the vet, which is why he’s thinking how to get her to visit again.) Really there’s a bunch of iconic cartoon images of deep thinking, including a mid-century-esque big-tin-box computer with reel-to-reel memory tape. Modern computers are vastly more powerful than that sort of 50s/60s contraption, but they’re worthless artistically if you want to suggest any deep thinking going on. You need stuff with moving parts for that, even in a still image.

Scott Adams’s Dilbert Classics for the 16th originally ran the 21st of May, 1993. And it comes back to a practical use for mathematics and the sort of thing we do need to know how to calculate. It also uses the image of mathematics as obscurant nonsense.

That tweet’s interesting in itself, although one of the respondents wonders if William meant astrology, often called “mathematics” at the time. That would be a fairer thing to call magic. But it would be only a century after William of Malmesbury’s death that Arabic numerals would become familiar in Europe. They would bring suspicions that merchants and moneylenders were trying to cheat their customers, by using these exotic specialist notations with unrecognizable rules, instead of the traditional and easy-to-follow Roman numerals. If this particular set of mathematics comics were mostly reruns, that’s all right; sometimes life is like that.

A Summer 2015 Mathematics A To Z: well-posed problem

Well-Posed Problem.

This is another mathematical term almost explained by what the words mean in English. Probably you’d guess a well-posed problem to be a question whose answer you can successfully find. This also implies that there is an answer, and that it can be found by some method other than guessing luckily.

Mathematicians demand three things of a problem to call it “well-posed”. The first is that a solution exists. The second is that a solution has to be unique. It’s imaginable there might be several answers that answer a problem. In that case we weren’t specific enough about what we’re looking for. Or we should have been looking for a set of answers instead of a single answer.

The third requirement takes some time to understand. It’s that the solution has to vary continuously with the initial conditions. That is, suppose we started with a slightly different problem. If the answer would look about the same, then the problem was well-posed to begin with. Suppose we’re looking at the problem of how a block of ice gets melted by a heater set in its center. The way that melts won’t change much if the heater is a little bit hotter, or if it’s moved a little bit off center. This heating problem is well-posed.

There are problems that don’t have this continuous variation, though. Typically these are “inverse problems”. That is, they’re problems in which you look at the outcome of something and try to say what caused it. That would be looking at the puddle of melted water and the heater and trying to say what the original block of ice looked like. There are a lot of blocks of ice that all look about the same once melted, and there’s no way of telling which was the one you started with.

You might think of these conditions as “there’s an answer, there’s only one answer, and you can find it”. That’s good enough as a memory aid, but it isn’t quite so. A problem’s solution might have this continuous variation, but still be “numerically unstable”. This is a difficulty you can run across when you try doing calculations on a computer.

You know the thing where on a calculator you type in 1 / 3 and get back 0.333333? And you multiply that by three and get 0.999999 instead of exactly 1? That’s the thing that underlies numerical instability. We want to work with numbers, but the calculator or computer will let us work with only an approximation to them. 0.333333 is close to 1/3, but isn’t exactly that.

For many calculations the difference doesn’t matter. 0.999999 is really quite close to 1. If you lost 0.000001 parts of every dollar you earned there’s a fine chance you’d never even notice. But in some calculations, numerically unstable ones, that difference matters. It gets magnified until the error created by the difference between the number you want and the number you can calculate with is too big to ignore. In that case we call the calculation we’re doing “ill-conditioned”.

And it’s possible for a problem to be well-posed but ill-conditioned. This is annoying and is why numerical mathematicians earn the big money, or will tell you they should. Trying to calculate the answer will be so likely to give something meaningless that we can’t trust the work that’s done. But often it’s possible to rework a calculation into something equivalent but well-conditioned. And a well-posed, well-conditioned problem is great. Not only can we find its solution, but we can usually have a computer do the calculations, and that’s a great breakthrough.

A Summer 2015 Mathematics A To Z: step


On occasion a friend or relative who’s got schoolkids asks me how horrified I am by some bit of Common Core mathematics. This is a good chance for me to disappoint the friend or relative. Usually I’m just sincerely not horrified. Much of what raises horror is students being asked to estimate and approximate answers. This is instead of calculating the answer directly. But I like estimation and approximation. If I want an exact answer I’ll do better to use a calculator. What I need is assurance the thing I’m calculating can sensibly be the thing I want to know. Nearly all my feats of mental arithmetic amount to making an estimate. If I must I improve it until someone’s impressed.

The other horror-raising examples I get amount to “look at how many steps it takes to do this simple problem!” The ones that cross my desk are usually subtraction problems. Someone’s offended the student is told to work out 107 minus 18 (say) by counting by ones from 18 up to 20, then by tens from 20 up to 100, and then by ones again up to 107. And this when they could just write one number above another and do some borrowing and get 89 right away, no steps needed. Assuring my acquaintance that the other method is really just the way you might count change, and that I do subtraction that way much of the time, doesn’t change minds. (More often I do that to double-check my answer. This raises the question of why I don’t do it that way the first time.) Though it does make the acquaintance conclude I’m some crazy person with no idea how to teach kids.

That’s probably fair. I’ve never taught elementary school students, and haven’t any training for it. I’ve only taught college students. For that my entire training consisted of a single one-credit course my first semester as a Teaching Assistant, plus whatever I happened to pick up while TAing for professors who wanted me to sit in on lecture. From the first I learned there is absolutely no point to saying anything while I face the chalkboard because it will be unheard except by the board, which has already been through this class forty times. From the second I learned to toss hard candies as reward to anyone who would say anything, anything, in class. Both are timeless pedagogical truths.

But the worry about the number of steps it takes to do some arithmetic calculation stays with me. After all, what is a step? How much work is it? How hard is a step?

I don’t think there is a concrete measure of hardness. I’m not sure there could be. If I needed to, I’d work out 107 minus 18 by noticing it’s just about 110 minus 20, so it’s got to be about 90, and a 7 minus 8 has to end in a 9 so the answer must be 89. How many steps was that? I guess there are maybe three thoughts involved there. But I don’t do that, at least not deliberately, when I look at the problem. 89 just appears, and if I stay interested in the question, the reasons why that’s right follow in short order. So how many steps did I take? Three? One?

On the other hand, I know that in elementary school I would have had to work it out by looking at 7 minus 8. And then I’d need to borrow from the tens column. And oh dear there’s a 0 to the left of the 7 so I have to borrow from the hundreds and … That’s the procedure as it was taught back then. Now, I liked that. I understood it. And I was taught with appeals to breaking dollars into dimes and pennies, which worked for my imagination. But it’s obviously a bunch of steps. How many? I’m not sure; probably around ten or so. And, if we’re being honest, borrowing from a zero in the tens column is a deeply weird thing to do. I can understand people freezing up rather than do that.

Similarly, I know that if I needed to differentiate the logarithm of the cosine of x, I would have the answer in a flash. It’d be at most one step. If I were still in high school, in my calculus class, I’d need longer. I’d struggle through the chain rule and some simplifications after that. Call it maybe four or five steps. If I were in elementary school I’d need infinitely many steps. I couldn’t even understand the problem except in the most vague, metaphoric way.

This leads me to my suggestion for what a “step” is, at least for problems you work out by hand. (Numerical computing has a more rigorous definition of a step; that’s when you do one of the numerical processing operations.) A step is “the most work you can do in your head without a significant chance of making a mistake”. I think that’s a definition that clarifies the problem of counting steps. It will be different for different people. It will be different for the same person, depending on how experienced she is. The steps a newcomer has to a subject are smaller than the ones an expert has. And it’s not just that newcomer takes more steps to get to the same conclusion than the expert does. The expert might imagine the problem breaks down into different steps from the ones a newcomer can do. Possibly the most important skill a teacher has is being able to work out what the steps the newcomer can take are. These will not always be what the expert thinks the smaller steps would be.

But what to do with problem-solving approaches that require lots of steps? And here I recommend one of the wisest pieces of advice I’ve ever run across. It’s from the 1954 Printer 1 & C United States Navy Training Course manual, NavPers 10458. I apologize if I’m citing it wrong, but I hope people can follow that to the exact document. I have it because I’m interested in Linotype operation is why. From page 308, the section “Don’t Overlook Instructions” in Chapter 7:

When starting on a new piece of copy, or “take” is it is called, be sure to read all instructions, such as the style and size of type, the measure to be set, whether it is to be leaded, indented, and so on.

Then go slowly. Try to develop even, rhythmic strokes, rather than quick, sporadic motions. Strive for accuracy rather than speed. Speed will come with practice.

As with Linotype operations, so it is with arithmetic. Be certain you are doing what you mean to do, and strive to do it accurately. I don’t know how many steps you need, but you probably won’t get a wrong answer if you take more than the minimum number of steps. If you take fewer steps than you need the results will be wretched. Speed will come with practice.