Testing vs. Checking – separate entities or part of a whole?

There has been a lot of
discussion, heated and calm, about the concept of testing. What it is. What it
is not. Arguments about real testing vs. what a lot of people imagine when they
hear the word. Twitter and blog posts talking
about “real testing”, “the future of test”, “no testing” and so on, seemingly
endlessly.

To be honest, I’ve had a hard
time grasping a lot of it and some distinctions don’t make sense to me. But. I’ve
also had revelations and that is what this blog post will be about.

So, let’s start. What is
testing actually?

To me, testing = Checking + Exploration. Not
one or the other, both. (Shoutout to Elisabeth Hendrickson and her awesome book Explore it!)

Let’s begin with checking. The
stuff that some will say is not ”real testing”.

My opinion? Well. If you _know_ something is
true/false, black/white, start/stop, by all means – check it. Go ahead, it’s
fine. Yes, I promise! If you _know_, from a reliable source, that you are
supposed to be able to login to the site with credentials XYZ from device A
then do it.

• Is it important enough that you want to check it
  all the time? Automate it.
• Is it time consuming enough that you notice
  yourself avoiding checking it manually? Automate it.
• Is it something that would help the team to get
  instant feedback on when it’s not working? Automate it.

Checking, to me, has an
important part to play in what I call testing.

• Important because knowing that I’ve checked the
  “obvious” gives me confidence to go further and deeper.
• Important because knowing stuff works in the
  basic ways frees up headspace and time for more creative work.
• Important because testing things in multiple
  different ways is more valuable than doing one way perfectly.
• And let’s be honest: Important because it still
  find a lot of issues quickly.

Exploration, to me, is all the
other parts. The stuff that might be hard to explain to someone else.

To me, exploration is about
framing a hypothesis, inventing an experiment to try to _falsify_ it (yes,
falsify and not prove), executing the experiment and then observe, analyze,
learn, adapt.

• It can lead to broken assumptions, a feeling of
  knowing less than before, Learning almost nothing useful. Which is ok, it
  will make me a better tester tomorrow.
• It can lead to new stuff to check next time.
  Possibly/probably checks to automate. Which is awesome, it will allow me
  to do even more deep-diving next time.
• It can lead to us having to re-think the entire
  solution. Which is GOOD, it will make the solution better.
• It will find things checks can’t possibly do
  because we didn’t know about it and therefore couldn’t check it. It will
  find risks, loopholes, cracks in the ice and give us information that can
  let us make better decisions.
• Some will say it’s more creative and fun, some
  will say it’s ”real testing”, some will say it’s a waste of
  time.

I say it’s the stuff that
makes me a very valuable asset in any development team. It also makes me worry
too much. Imagine being unable to shut off the ”hm. I wonder what
could possibly go wrong with this scenario” at any point of time in your
life, say for example while trying to look forward to travelling…

So, where am I going with
this. Well I am trying to explain why I think we have such a big gap between
the ”Automate everything”-people and the ”That’s not real
testing”-people.

I confess I struggle with
models like the DevOps infinity-loop. It puts ”TEST” in bold letters
in an isolated part of the loop and if that is your view of the world, how can
you ever take someone like me seriously? I find it seems impossible to find
common ground without a long discussion about ”what is test really?”
and how do I explain that when I say test I don’t mean executing a particular
scenario in an application, I mean basically most of the things I do from the
time I hear a user express a need (yes, I am horrible to talk to) to the time I
stop working on a particular system.

• I test the importance of that need by challenging
  it. Usually by talking to people.
• I test the proposed solution by challenging it.
  Usually by asking questions or making explicit assumptions.
• I test the process by challenging it. Usually by
  observing where people tend to take shortcuts or get flustered from not
  knowing how to proceed.
• I test the solution under creation by challenging
  it. Usually by writing some sort of description of how I will test it and
  making as many people as possible tell me what I did wrong.
• I test the solution when it’s ”done” by
  challenging it. Usually by a combination of checks (the stuff I _know_)
  and exploration (the stuff I _believe_).
• I test the delivery by challenging it. Usually by
  analyzing data, either from an analytics tool or from actually _talking_
  to people. (Did you know customer service tend to categorize calls? Do you
  know how much information you can get by just checking in with them after
  a release? It’s awesome!)

So, if that is what I mean
with testing and the person I talk to see it as writing some sort of test
descriptions, executing them and then report the result… it will clash. And
it does. Repeatedly.

Some will argue exploratory
testing encompasses all the aspects I’ve mentioned and won’t approve of me
separating them. They will argue exploratory testing is more a way of thinking
of testing and I totally get it. I really do!

But. What I’ve come to realize is
that to me, checking and exploration are two totally different mind sets that
make me act and think differently. Yes, call them what you want but I need to
do context-switching to change between them, just as I need to switch to
another way of thinking when building something. And I do. I switch. And I need
to do that switching in order to be the best possible tester I can be. So, to
me they are all tools I use, for solving different tasks.

I’ll stop there for now. In the next blog post I’ll share an example from a
previous assignment.

Cute kitten to calm any upset feelings:

Photo by Romane Van Troost on Unsplash