Exploratory Interfaces

This page has temporary content, and acts as a catch-all while I pull these ideas together. We'll be running exercises around this in PlayTime – subscribe to join in.

Software systems have plenty of bits that can be explored. Those bits invite different ways of exploring. The different ways will reveal different things. We can make models* and spot trouble by analysing and comparing parts.

* Depending on the limitations of what we make the models with.

Costs, choices — and opportunities

There are* more ways to explore, and more models to build, than are worthwhile. We need to pick what and how we explore, based on what we might learn (¿get) from the exploration, and on what we need to spend to get it.

An example: imagine we have a range of numbers to explore. We chan choose, in black box style, to experiment with inputs. We might put in an input, see what happens, put in another. We might iterate by going up by one, increasing by factors of 10, random pecking or homing in on something interesting. We might connect a slider and wave it about to eyeball the interesting. We might generate input values to a particular distribution, and analyse the output for changes and patterns. We might use values that are already expected to be interesting. We might throw vast combinations of predictable stuff, and watch for the unexpected.

Imagine that we choose to explore something else, first. We might choose to explore the code that manages the input. We might look at the data that configures that code and which holds the key to where behaviours are expected to change. We might log everything our users put in over a week. We might look in a database at every output we have managed to store. We might look in the logs to see if there are failures related to the component and its input, or related to its output. We might look in change control to see how often it has changed, why it changed. We might ask experts or users or customers or lawyers about what they expect. While there is far more to choose from than can be done, we can still make a choice to explore. We can still learn about the thing we’re exploring – but OMG the cost!

Have a look at Exploring without Interaction.

If we're seeking these kinds of opportunities, we're going to be interested in iterable exploratory surfaces.

`* I believe that this is not worth arguing.

Tooling

Tools can make us faster, more accurate and more consistent — or perhaps we might say that neglecting to use the right tool makes the work larger, and introduces more weirdness.

Putting a tool 0n an exploration can change the cost of iterating by '000s. Tooling doesn't simply make the exploration cheaper, but can make the exploration scalable. Making an experiment scalable may mean that it is worthwhile using an approach that finds something only rarely. The tool opens a door.

Further, ready access to tooling can mean that the cost of making a new custom tool is small, so that we can as testers scale** up the opportunities to make use of a tool. If a tool opens a door, and each door opens onto a different perspective of the same system, and each perspective brings otherwise hard-to-find bugs into view, then tool / approach variety is as important as depth.

If we're looking for trouble then speed, cheapness and variety of tools can make up for poor quality and fit. Generic or flaky tools will still offer useful clues to issues, and we can choose to invest time in chasing down those problems. For example: A blink test as one slides a value through 1000s of experiments, a sort in a spreadsheet, a swift filter with jq can hint to our tester senses.

LLMs make it easy to build ephemeral tools from parts – to my mind, this looks to be one of the most fruitful ways that we might use this new technology in our current work.

`** If we're scaling checks in an exploration, then we need to be cautions of false positives. Perhaps not to avoid them, but the be aware that they may flood our results, and to be able to filter them before putting them in front of us to judge.

Examples

The code has text – screen info, error messages, comments. One way to explore those is to extract them. You don't want to slog through yourself – so you could use grep or a regex pattern or hoik everything out of the language / translation config file. When you've extracted the words, why not check them against some sort of oracle by running them through a spellchecker or grammar checker or sentiment analysis tool?

The software may be configured – in some situations, the configuration is the system. You can look at the configuration, seeing whether you understand what it means, because if you don't then maybe you underestimate what the system is doing. You could use the config as an input to your tests; not only to check the behaviour of the system, but to programatically generate special cases. And while the checks might pass, the system needs your human judgement to analyse what has been produced and to recognise that the configuration is wrong, or that something that appears to be handled with simple configuration actually needs more nuanced treatment. I've explored the config with people who know the goals, regulations or indeed who set the configuration for the business – and found easily-fixable spelling errors, expensive-if-made-real numeric errors, and back-to-the-customisers failures to describe the outside world in a way that met needs.

The system produces logs. One way to explore these is to read through them. You want this to be amenable to tools – you could filter for all logs of a particular type or unify several logs, or pop up an alert when some particular entry is logged. You could make a tool to extract all records around a particular time (perhaps one you'd logged automatically earlier) and to use unique IDs so that you can extract all info related to those IDs to see what was in flight at the time you wanted.

Have a look at Ways to Explore without Requirements for a longer list of shorter things.

Classification

I tend to distinguish between exploratory surfaces that are of the artefact, and those that are not, though the boundaries are fuzzy. Code and transactions are of the artefact. Operating systems and user reviews are not. I'm not sure about integrated libraries and the history of how the system was built. But all are explorable and can tell us interesting things.

I find I want to distinguish between what something is, and what it does. This page is written by me, a ghost blog post, a web page in your browser, an entry in a database. And what it does? It communicates an idea, acts as a space for me to keep stuff, signposts readers elsewhere, lets me empty my mind. Perhaps it's simpler to tool against what it is, but simpler to value what it does. A worthwhile exploration might need to have both relevant.

I find it useful to distinguish between surfaces that invite automation, those that could be automated, and those that don't seem automatable or iterable. I need to have some tools in mind to do this. I do this to understand what might be interesting to automate (click all the links!), not to indicate what should be automated (what does clicking all the links achieve, exactly?).

Definition

oooh, I'm not a man who much likes definitions – but I do need to tune my thoughts, so they're useful for now.

Further reading

Exploring without Requirements

Requirements are helpful rather than crucial

Workroom ProductionsJames Lyndsay

Exploration Without Interaction

We were in the Science Museum, me and my 10-year old, looking at the huge Burnley Mill engine as it chugged. We talked about the parts, saw how they were interconnected, wondered about their purpose and function, listened to the noises, watched people as they oiled and tuned, asked questions,

Workroom ProductionsJames Lyndsay