Exploratory Testing: Lessons from Fact Checking

In 2015, Mike Caulfield popularised the concepts of web content as stream and garden. You can see more in his lecture and post.

His primary work is in collaborative education, and you might also have chanced upon his SIFT approach to sorting truth from fiction. His open-source book Web Literacy for Student Fact-Checkers won a MERLOT award.

In that book, he proposes ‘four moves and a habit’ for people seeking the truth. His instructions connect with me, as an exploratory tester – they remind me of what I do, and exhort me to do it better and to explain it more clearly.

He writes:

Moves accomplish intermediate goals in the fact-checking process. They are [strategies] associated with specific tactics. Here are the four moves this guide will hinge on:
Check for previous work: Look around to see if someone else has already fact-checked the claim or provided a synthesis of research.
Go upstream to the source: Go “upstream” to the source of the claim. Most web content is not original. Get to the original source to understand the trustworthiness of the information.
Read laterally: Read laterally.[1] Once you get to the source of a claim, read what other people say about the source (publication, author, etc.). The truth is in the network.
Circle back: If you get lost, hit dead ends, or find yourself going down an increasingly confusing rabbit hole, back up and start over knowing what you know now. You’re likely to take a more informed path with different search terms and better decisions.

His ‘habit’, since you ask, is to respond to strong emotions with fact-checking.

When we explore, we're trying to learn. We want the truth, we want it fast, and we want it to be relevant. I imagine that fact checkers have similar motivations.

My thoughts are that I have a similar habit, and that I use similar moves. To put my approaches into Caulfield's terms:

• Habit: I look out for emotional reactions: intrigue, bubbling curiosity, amusement, smug satisfaction, surprise, irritation. Those emotions (not an exhaustive set) are telling me something – and I believe that if I listen to them, I'll find out something I don't know. I make this a habit, because if I listen to my emotions and then give my mind something to chew on, I might enhance those deep responses over time – and perhaps that's one way to be a better tester.
• Move: I check for previous work – is this something I already know? Has someone else talked about this? I can check the manual, other parts of the system under test, the bug logs, field reports and customer views. I can consider things I'm not testing, and dig into whether they'd provoke a similar reaction.
• Move: I go upstream – I try to find out about the data, or the code, or the systems involved. I try to think about the edges of what I'm working with, and whether my reaction is to something that is within a specific part, whether it is to do with an interaction. If there's an inconsistency between what I expect and what I perceive, I wonder whether my reaction is to do with my model in my head, or the system – and if outside my head, and outside anything I can influence, what that means for further time spent exploring.
• Move: I work laterally – typically by using diagnostic approaches to judge the causes of what I've found, and also by thinking of the potential impacts and where else similar problems might lurk. I don't follow Caulfield's actions to read what other people say, as I'll have already looked for that earlier and I'm often the first person with specific focus on the problem.
• Move: I circle back – I look out for more emotions, here: confusion, rejection, frustration, ennui. Those tell me that I'm stuck. I might circle back by seeking out a rubber duck, by revisiting what I was doing, by putting a flag in my notes that allows me to forget, by re-assessing where I was, by clearing back to the start. I've written about this in Handholds Framework.

Tags

• Articles
• Exploration