Here are two IQ test questions for you:

1. Fill in the blank in the sequence 1, 4, 9, 16, 25, __ , 49, 64, 81.
2. What number comes next in the sequence 1, 1, 2, 3, 5, 8, 13, .. ?

First, don’t feel bad if it took you a while to answer these questions. In fact, if you didn’t answer them at all that’s a very good thing and I commend you (more on this later). But if you do have answers in mind, you can check them now. The answer key:

1. 72
2. 12

The pattern that explains why the answer to the first question is 72 may be subtle, but it’s simple. If the nth number in sequence is f(n), then it’s clear that f(n) is just the period of the sequence b(m) defined by defined by $b(m) = {m+n \choose n} \bmod n$.

Now I know what you’re thinking. You’re a so-called high intelligence quotient person and you know the answer is different. You think the pattern is just the squares 1², 2², 3², 4², 5², 6², 7², 8², 9² and that the blank should therefore be filled in with number 6² = 36. You think that it makes more sense because its a “simpler” pattern that explains the data. You are planning to comment on this post and you will invoke Occam’s razor. You will say this problem has appeared on many IQ tests with the answer 36 so that is what is expected in terms of an answer. You will explain that therefore it is the answer.

But what is a “simple” pattern? Let’s look at the second question. Here the simplest pattern that can explain the data is just that each number is the sum of the digits of the previous two numbers. So the next number in the sequence is 12 (=1+3+8). Yes, twelve. You shouldn’t be surprised that you got this wrong as well, and I know you did. Addition of digits of numbers is a common ingredient in sequence patterns on IQ tests. For example the problem of finding the next number in the sequence 58, 26, 16, 14, .. is a similar, albeit more difficult, IQ test question that is analogous my #2. But I know that you might have something else in mind. You’re thinking Fibonacci. You’re thinking there was a typo and 12 should have been 21. But I’m sorry.. no, no, no, no. An IQ test is not a math test. It’s about finding patterns, it is a test of raw intelligence.

What has happened here is that by merely asking you these two questions, I’ve forced you to overfit. There simply isn’t a meaningful way to choose a pattern from an enormous set of possibilities using only a handful of numbers. Yet this is exactly what I asked you to do, and what IQ tests ask one to do. They force one to overfit. You know what one should call a test that encourages poor statistics hygiene? A “statistics deficit test” (SD test) instead of an “intelligence quotient test”.

The mathematician Richard Guy uses many alliterations to describe the horror of overfitting from a handful of numbers:

Superficial similarities spawn spurious statements.

Capricious coincidences cause careless conjectures.

Early exceptions eclipse eventual essentials.

Initial irregularities inhibit incisive intuition.

These all capture the point that, as Guy says, “there aren’t enough small numbers to meet the many demands made of them”. For many good examples see his paper:

Richard K. Guy,  The strong law of small numbers, The American Mathematical Monthly, 1988.

My favorite example of a pattern that is not what it appears to be is the sequence 1,1,1,1,1,1,1,1,1,1,1,.. continued for a total of 8424432925592889329288197322308900672459420460792432 times, and then followed by the number 8936582237915716659950962253358945635793453256935559. To understand this sequence one has to only identify what is an extremely obvious pattern. The nth term of the sequence is the greatest common divisor of two polynomials: n17+9 and (n+1)17+9. The greatest common divisor is therefore one for n up to 8424432925592889329288197322308900672459420460792432. Then the 8424432925592889329288197322308900672459420460792433rd greatest common divisor is 8936582237915716659950962253358945635793453256935559. Ok, maybe not so obvious and a somewhat unusual construction for a pattern. However this example makes another point. There cannot be certainty to the “solutions” on an IQ test, so that the term “answer” is a misnomer and its use is problematic. If a test asked for the 8424432925592889329288197322308900672459420460792433rd term of the sequence of ones above it is likely that it’s a one, but one cannot know for sure. Another way to say this is that IQ tests are implicitly asking test takers to accept null hypotheses instead of asking, at most, for a rejection.

There has been much debate on what IQ tests really measure and what they predict. But it seems to me that the answer is obvious. In order to score well on IQ tests one must be willing to overfit and to practice the p-value fallacy. Since such practices are synonymous with poor data science, I hypothesize that

low IQ scores predict excellence in data science. 