Exercise – Power of Variety

A 15-20 minute exploration via a simulation

Based on an older exercise, no longer interactive, posts here: An Experiment with Probability, Broken Trucks, Models, lies and approximations, Enumeration hell, Diversity matters, and here's why, Modelling super powers . Go there to read background – I'll bring it here in the next few weeks and update it.

Each of these simulations have a similar core simplifications:

• the simulation knows what can be found, and it knows what's likely to find each thing, but the entities looking don't know either of these things.
• A searchable thing has a limited collection of independent things to be discovered. For testing, think: there are only so many problems.
• Discovery is by chance. Different approaches to searching have different chances. A discoverable thing has a low chance of being discovered by any individual approach – but might have a better chance by another approach. For testing, think: Performance testing will find different problems from usability testing.
• There's a limited budget – the more you spend, the more chances you have to find stuff.
• Something stays found, but only counts the first time it is found.

You, the user, can see and change the number of things that can be discovered, the makeup of the crew that looks for things, and the budget to spend looking. During the simulation, you can see the results of the simulation in terms of the work done, the discoveries found. You can pause and resume the work. You can increase and decrease the budget. Different simulations will throw up different results, and you'll want to keep track of those to see the variation.

Background: a simulation works in chunks. In each chunk, all the explorers have a chance to discover a (random) set of discoverables (of fixed size in the simulation). A discoverable's chance of discovery is set by the explorer's active skill, and the chance of the discoverable being discovered by that skill.

Priming Question

We'll come back to this in the debrief

Simulation 1:

5 minutes

Go to https://exercises.workroomprds.com/discoverysim/1/index.html. Play with "small" or "medium" first to get a feel for it, then switch to "large" to get stats.

Compare one tester, one approach with several testers, also all with a singular approach.

Have a look at how many you're 'found' for the spend – and how long it's taken. You'll want to try this several times to judge the variance you see.

We'll think about the budget – is it "worth" running a second 1000 after the first? How about running half the budget? What does "worth" mean, to you?

Simulation 2:

10 minutes

Go to: https://exercises.workroomprds.com/discoverysim/2/index.html. This simulation goes faster (for you), but the stats remain the same.

Sticking with "large", pick "six testers, ten techniques", put each onto a different skill (click on the approach name), and see how the stats change.

Switch to a tester who chooses to change ("one tester, ten technique"), and switch their skill as they work.

What do you see?

How do different approaches n affect your decisions about budget and stopping?

Explore the other settings – the ones labelled 'switching' will switch techniques at random, some fast, some slow.

Note: For now, there's no sense of a discovery having a value – it's just counted. Also for now. Also for now, there's not sense of better or worse at a technique.

Debrief

5 minutes

How do you feel about spending, and stopping, in a situation where you need to discover new things?