Why do scientists get things wrong?

By Dr. Christina Smith

It occurred to me over the last couple of weeks that lots of people have found the changing information and guidelines, recommendations, and findings on COVID-19 to be surprising, sometimes frustrating, and perhaps don’t understand why things can change so rapidly. I’ve heard things like “why should I trust this information if its different to last week’s?” I’m not an epidemiologist or any kind of bio-anything scientist so that’s the only thing I’ll be saying about COVID-19. But, it dawned on me that perhaps it isn’t completely clear that people at the forefront of any field in science are really piecing together bits of information and evidence the best way they can to try and figure out what is happening. With time and enough information the explanations (often – not always) become more and more in agreement but with newer problems and smaller amounts of information, the findings that people come up with can vary wildly.

 

This is quite common in science (in my experience). You’ll have some information, but not everything. An often used phrase is “these results are consistent with...” which translates to “this idea about what we think is happening would expect the measurements that we made to come out how we have seen them to”. It doesn’t translate to “these results prove that our idea is definitely what is happening” - a very important difference.

To put it as a real-world example...

Imagine you have been tasked with finding out what the picture on a jigsaw puzzle is when you don’t have the box and the jigsaw is in bits. The best and most reliable way to do it would be to take the time to put the jigsaw puzzle together and once its together you can say for sure: this is what is on the jigsaw.

Now imagine you don’t have all the puzzle pieces or the box. But you have most of them. It’s frustrating (for anyone who has ever done such a puzzle) to get to the last piece and realize that actually half a dozen of them have been lost at some point in time. But nevertheless, with enough of the pieces, you can still reliably figure out what is on the front of the jigsaw. Sometimes there are some details that you can’t figure out but you have the general idea and can kind of work out the missing pieces.

Now imagine a scenario where you’re given a couple of handfuls of pieces and are asked to figure out what the picture is – and saying “I don’t know” isn’t really an option. You’ll examine every piece carefully, look for clues as to what the picture could possibly be. You see something that has blue and white – could that be a sky piece perhaps? Or the sea? Or just a pattern? You find a corner, that gives you a bit of information. You seem to have some green pieces too. But, curve-ball, some of those pieces aren’t from this jigsaw, so along with figuring out what your puzzle looks like from the pieces you have, you also have to try and figure out which puzzle pieces don’t belong in this puzzle. A very difficult task. If you and your friend are trying to figure out what this jigsaw puzzle looks like separately from only a couple of handfuls of pieces, its likely that you both will have lots of ideas as to what it could look like but they’ll be different ideas. Maybe on a theme, maybe not. That’s ok, the puzzle pieces you have could fit into any of the puzzles you’ve imagined.

So you start to look at what other puzzles with buildings look like and compare them to ones you and your friends have done before. This one looks to be in the same style as another, that’s of a farm and a field – would that fit with the pieces you’re seeing? Not exactly but perhaps its a different view. You start to count the number of blue pieces and the number of green pieces. You’ve got quite a lot of blue in comparison to the green ones if its a farm with fields and some sky. Perhaps it’s just luck that you’ve been given a lot of blue rather than green or maybe this is a blue-heavy picture. You start to compare the amount of blue and green you have to the amounts of blue and green in other similar puzzles. It could be a house and garden by the sea. A farm with a lot of sky. Maybe it’s a blue train in the countryside?

After a while some more pieces come your way, and some of them fit together! You don’t really know where these pieces go in relation to one another, but they’re building up more and more of a picture. You’ve also been told that this puzzle picture includes some kind of building – that really helps. Oh, it seems like the blue bit has a grey patterned edge and some of it kind of looks like water, but that seems odd if it’s the sea... perhaps you need to revise your thinking on that. What else could it be? There also seems to be two colours of blue. Maybe one of them is the sky and the other something else? Maybe a swimming pool? Maybe a decorative pond?

You could simulate each of your ideas for what the puzzle could look like as a whole and see if you can lay your pieces onto it in a way that fits. How well do your pieces match the simulated jigsaw pieces? Do they just not work at all? If so, then those aren’t good candidates for your puzzle picture. Do some of them work and some of them don’t? Maybe you can tweak your ideas see if you can improve them? But you’ve still got to remember that its a got a building in it – keep things consistent with the information you’re pretty sure is reliable. You can’t throw a blue spaceship in the middle for no reason, for example. You have your favourite idea that you’ve simulated and the pieces fit almost perfectly – so you’re pretty confident its probably a picture of a cottage with a decorative pond. Your friend doesn’t agree. They think its a house with a swimming pool and they’ve done their own simulations which match reasonably well too. Another friend disagrees entirely and thinks you’ve put the blue pieces together all wrong and in fact that grey bit is a bit of a pattern path in a garden next to the sea.

As time goes on and you get given more and more pieces and little snippets of information, you all keep updating your ideas with new information, refining what you think the puzzle on the box looks like. Eventually, with enough pieces, you’re in agreement that you’re looking at a big fancy house with a swimming pool in the foreground and a blue sky and mountains in the background. A bit different from what you had originally thought but you got there in the end. You disagree on some of the details, but the general view is the same. Though you have a cluster of pieces that clearly don’t fit in this puzzle, there are a few where its not so cut and dry. You think they all belong to this puzzle, one of your friends think some of them do, and your other friend thinks that none of them do. So you keep working at it.

So now jumping out of the puzzle metaphor...

When things reported on at the forefront of science change, it isn’t necessarily a case of “oh that was wrong” or “they made a mistake” (though it can sometimes be that). It can sometimes be that someone found some new puzzle pieces or an idea that fit better with the puzzle pieces that were already there. You can only work with the information you have at the time. And I personally think that’s something particularly important to keep in mind with everything that’s going on in the world right now.