Would Heu-risk it? Part 29: Tunnel Vision

Getting to the end, a few cards start to blend. Lets see how this trap can.. oh snap, this introduction was not supposed to rhyme! Sorry, the proper verse this time:

”A weathered explorer, you´ve walked this path before
You know every rock, of every danger you are sure
But if you only look for what you know will be there
You might miss the changes that suddenly appear”

So, what does it mean?

Say you are doing something you’ve done a LOT before. You walk or drive a certain road each day to/from work. Have you ever experienced that you get there, and you can’t remember exactly what happened during a stretch of that time? You zoned out, only zoning in again if something unexpected happened or the task (driving/walking) got done.

This happens when something becomes very routine for us and, as so many other things, is a way for us to reserve (cognitive) energy.

This affects us in our testing as well, in combination with a lot of our other energy saving mechanisms, such as confirmation bias (favouring information that confirms our belief) and ambiguity effect (choosing the scenario with the known probability even if the other has the potential to be better).
I spoke about ”the one trick pony” in ”The Shapeshifter” and they overlap somewhat but that card deals more with consciously choosing to try out new things while this card deals more with the dangers of zoning out through repetition. I’ll talk a bit more about confirmation bias as well, since I think it’s a really interesting bias.

When we are doing something very repetitive, say manually testing a certain user flow over and over again with different input, we will start becoming unfocused. We stop actually seeing what is happening on the screen. This is a time when automation can be very handy. Computers only look for what you ask them to look for while humans, amazing as we are, will start taking mental shortcuts to reduce the cognitive load. Our brain adds in information for us, to be helpful.
You know a certain information text always says the same thing so you stop paying attention to it. You know a certain button is always enabled so you stop paying attention to it. Unless something very unexpected happens, you will keep repeating the same thing without paying attention to what happens when you move from A to B to C, all the way up until you come to the thing you have decided is your result.

Another nifty thing it does is that if you tell it to look for something in particular, it will focus on that. And when it does, it becomes practically blind to other things. The more important you feel the thing you are looking for is, the more blind to other things you’ll become. If you’ve watched the ”selective attention test”-video by Daniel Simons and Christopher Chabris you hopefully know what I am talking about.
If you are really focused on how button X behaves, you’ll miss other things. The more important you believe button X to be – the less you’ll see other problems, regardless of how obvious they might be.

We are however, also pretty darn amazing at spotting patterns. Especially breaks in patterns!
I’ve seen people lazily scan through files full of tiny tiny print, comparing two, and flawlessly spotting tiny tiny differences. If you’ve done any ”Can you spot the 8 in the midst of a zillion 0s” – that’s what I’m talking about.
Our brain seems to be really good at spotting those breaks in the patterns – if we tell it to look for it at least.
So ironically, zoning out might at times be a better choice than zoning in (too much). Unfortunately, at other times it’s the other way around so I urge you to do both.

A bit about confirmation bias as well before ending this post.
We are so amazingly good at finding information to prove a point. And so incredibly bad at seeing everything that proves it wrong.
And at least when I was in school, we spent a lot of time learning how to prove things. We had a theory? Defend it. Prove why X + Y = Z.
I don’t think, growing up, anyone ever spoke to me about how to find holes in things by trying to falsify it instead.
So, of course this is not our go-to solution. Instead, we come up with hypotheses and then we take a stand and fight for it.
When in fact, a lot of times, it is way quicker to try to prove it wrong than right.

If I give you the numbers 2, 4, 6, 8 and ask you to explain how that series works, I can almost guarantee that you will either ”know” that the next number in that series is 10 and that the pattern is ”the next even number”. How far would I have to push you to make you come up with alternative answers? Are there alternative explanations to the rule behind the series? No? Are you sure? How would you prove it?
It would be a lot quicker to try to falsify the series by giving an uneven number and expect to be told it is wrong but most of us will just keep trying to prove it right.
Using the rule of three, come up with 3 possible explanations before deciding this is the actual true pattern.

At least, this is my experience! It is based on what I have noticed in other people, and myself. But, thankfully, not everyone agrees.
I’ve included a link to an article that does not agree with me in the list of reading suggestions below: ”The curious case of confirmation bias”.

Story time

At one point in my career a team worked on a web portal. In this web portal there was a field for phone number and you could not continue this particular user flow without entering a phone number. Upon completing the form, a text message would be sent to the phone number to confirm the information had been received.
Let’s just say this turned into a bit of a problem with our test automation.
At some point in time, this had been solved by entering the same number in all automation cases. That number was connected to a phone, stuffed into some cabinet or another, and not used for anything. It probably was done because of things and reasons at some point in time.
Well. Let’s just say we did not make sure that phone number stayed in our possession. I’ll let you imagine the horror scenario that played out.

When this was brought to my attention, the team had already fixed the critical issues by switching to the other (real) phone number we had, on another phone in another cabinet.
To me, this was just replacing a bomb that had exploded with a bomb waiting to explode. Who would make sure the same thing didn’t happen to the new number? What other potential solutions did they discuss?
None actually. They went with the solution they knew. It was fast, they knew it worked (well..) and it required the least amount of cognitive energy. ”This is how it’s always been done”.

I can think of a few other options and in the end we did a combination:
– Don’t send text messages in a test environment. Text messages cost money you know and at some point someone is going to enter someone else’s number and send it.
– Check with the phone companies if there are any test numbers to use. They probably test as well
– Change the form so we don’t have to enter a phone number in test environment
– Whitelist numbers that we know we own and control.

Quote of the day

”When we look at the world around us, we feel that we are seeing it completely and accurately.  With perception, we assume that the world is WYSIWYG (what you see is what you get), but that sense is illusory.  Most of the time, we do see the world accurately, but we do so only because our visual system makes informed guesses about the contents of the world based on the compressed signal projected onto our eyes.  These shortcuts, rules-of-thumb, and heuristics are built into the way we see, and they work so seamlessly that we rarely notice that we’re using them at all.  
Only when we ”break” the system can we reveal these default assumptions.”

Daniel Simons

Reading suggestions

Exploring it – QA Hiccupps
What is a good test – Cem Kaner
Seeing the world as it isn’t – Daniel Simons
The invisible gorilla – Daniel Simons and Christopher Chabris
Confirmation bias in software testing – Gul Calikli, Berna A. Uzundag, Ayse Bener
The Curious case of Confirmation bias – Gary Klein

Previous posts in the series