Questions, Questions at EuroSTAR 2022

Individually (1 min) Write down one or more things you want from today.

Collectively (3 mins) I'll write up a diverse set, and we'll gauge interest with hands.

I'll post our choices and will shape my delivery to suit this group.

Individually (3 mins) Write down several questions which you use in your testing work. (Or take ones you use, from my lists below.)

Collectively (10 mins) Put your questions on the Miro board, or on a sticky-note wall. We'll look at what's similar and group those questions together.

Solo or small groups (3 mins) Identify some similarities between questions.

Collectively (10 mins) Suggest groups. We'll take a collection of groups, and see which questions fit, which don't, and which span several.

We may run this several times – for (some of) contexts, topics, purposes, audiences and types

Several Volunteers needed to be in front of the group

Setup: Person A will ask Person B about how they travelled to the event. Person B will avoid telling Person A an important detail (set in advance). Observers will keep track of the questions.

Reporting: Observers will describe the sequence, and infer purpose. Person A and Person B can comment, of course.

Collectively: We'll look at the different ways the questions are asked. We'll use What? So What? Now What? / Huh? Really? So? to do that debrief.

We may run this several times.

Notes: A’s goal is not to find the missing information. It’s to find out as much as she/he can about A’s journey. So if A asks a direct question, and B says “maybe”, or “I’m sorry, I can’t tell you that”, we keep going. The intention of the hidden bit is to get everyone thinking about their model.

Possible things to hide - start / end / start time / end time / one of your means of travel / one of the places you passed through / duration / cost / purpose / travelling companion / travel company / cost / something else important.

Practice (10 mins)

Solo: take a question you know. Find ways to transform it by changing the question word, making a yesno question, using 'Tell me about...'

Pairs or threes: Pick a question each. Transform it, as above - and ask each other the new question.

Collectively (10 mins): What do these tranformations help with?

Pick a model that you use – perhpas a (testing) topic, and your expectations of it. 

Solo (10 mins): Write a short sequence of questions that you might use several times when talking with different people about that topic. Refine those questions – how have you made them better?

Small groups (10 mins): Ask the questions of someone in your group. Listen to thier (brief?) answers. Don't follow your sequence if it doesn't make sense. Where has your sequence been helpful? Where could it be better?

Collectively (7 mins): How do your questions help the people you're asking? 

Advanced: What answers would break your chain of questions?

Imagine that...

• you're meeting a colleague, who needs a new function to be tested
• you've just joined a new testing team
• you're trying to find out what someone needs to see in your test report 

Solo, in pairs or small groups (5 mins): Write down the questions you would ask to find out more information. 

Analysis (5 mins): How would you order these? What patterns and gaps can you see?

We'll take a moment to add any new new questions to the big collection, then we'll try to arrange them for purpose.

We'll use dot voting to identify interesting purposes or questions. I may ask for volunteers to advocate for their interests.

Solo, or in small and large groups (10 mins): Dig into those areas, transforming, expanding, classifying questions. Document those on the Miro board.

Presentation (5 mins): 5 x 1 minute presentations on insights.

Path 1: Take a question, make it worse. Try to make it worse in different ways - reducing the options, imposing a model, re-asking and more. Don't aim only for the spectacularly dreadful examples; include some that are a just a little bit worse

Path 2: Think of examples of awful questions you've been asked (or have asked!).

Share your questions with the room – we'll identify what makes the question bad, and try to guess if this is something you built for today, or experienced in the past. 

Needs two volunteers to be up front. Agree topic, and try to find people in different contexts.

TImeboxed 3 minutes. Person A asks Person B about an aspect of their testing work.

Person A shares their follow-up questions. Person B shares the questions they were waiting to ask. 

The whole group share the questions they wanted to ask next, with the room.

We try to find some principles and meta-questions

• rhetorical questions are used to make a point, not gain information 
• display questions where the questioner knows the answer. Detects knowledge / belonging / group rather than seeks information
• direction questions seek an action to do now rather than information – "when should I do it" / "now" | "where should I put it " / "into the text box"
• converting questions from imperative – give me the data -> would you give me the data?
• loaded question - a question which seeks to embarrass https://en.wikipedia.org/wiki/Loaded_question
• complex questions - trick questions, presupposition question https://en.wikipedia.org/wiki/Complex_question
• leading question - suggests an answer https://en.wikipedia.org/wiki/Leading_question
• Compound question / Double-barrelled question - answer has two parts, question only allows for one https://en.wikipedia.org/wiki/Double-barreled_question
• If the person being questioned wouldn't necessarily consent to the question's constraints, the question is fallacious.
• Display questions, where the questioner is known to know the answer (i.e. quizzes)
• Mirror questions, where the question serves as conversation initiator - A: How are you? B: I'm fine, how are you?

• How would you... (methods and problem solving)
• Tell me about... (most general and open)
• Why do you think... (to ask for inference)
• How should we prepare for... (to ask for speculation)

• (pause) - see what else will come out
• (re-state with ? on end) - double-check / incredulity / ask for more (depending on tone of voice)

• Who am I working with
• Where are the tests kept
• How do I get access to the test environment
• How do I log a bug
• How do I recognise complete
• How do I report progress / share what I’ve learned
• What are the key project deadlines
• What are the next deadlines for me
• What is my budget - and what is it measured in (time, spend etc.)
• What are key constraints (on what? system availability, concurrency, data, more)
• How can I dig deeper (code, config, logs, monitors)
• Wha can I build executable code in
• How do I get access to (things I need to test)
• Who can I ask, when I find an oddness that I can’t judge
• How can I build my judgement
• What regulations apply
• How do we separate work in a shared environment
• HOw much autonomy do I have in executing tests
• Are we looking for trouble, or verifying expectations?
• What do I need, to test (permissions / access, data, purpose, judgement)
• How do I re-create data which my tests have consumed?
• Are we wrangling, debugging or testing?
• Where are my documents – something to set out how testing fits into the rest of the work (a test strategy, a non-testing doc), something to set out what testing will do (а test plan), some record of testing (), some list of lessons learned? A project risk register - a product risk register? Design docs? End-user purpose?
• Tools inventory? Inc. scripts and shared stuff between scripts
• Data inventory?
• How do we share code between tests?
• How do I get to change control for my artefacts - docs, reports, code, data, results, config?
• How do we share information between testers?
• Are we working on a project (with an end product and date) or supplying a service (Business as usual / ongoing budget until something happens)?

• Why didn’t you find that?
• Have you found all the bugs yet?
• Do you really need this much time to test everything?
• Are you actually telling me you’re not going to test everything?
• Can’t you just show me that it works?
• Why should we change?
• But haven’t the coders done all the testing already?
• Why do you think this bug has a high impact?
• Why are you only testing the happy path?
• Can’t the coders stop coding and help you test?
• Can’t the users stop working and help you test?

What processes are mandatory, here?

• Design review
• Code review
• Requirements review
• Test review
• Risk review
• Raising a risk
• Work orders
• Sign-Off (and of what) before release
• Formal acceptance criteria 
• On-boarding
• Off-boarding
• Security clearance
• Setting up as a supplier
• Change control
• Licensing
• Setting up / gaining permissions
• Bringing in a new tool
• Bringing in a new library
• Setting up a new project
• Hiring
• Firing
• Resigning
• Handover
• Invitation to audit / regular audit
• (of people) Annual review / regular review / appraisal
• Signing code of conduct / declaration of secrecy / NDA / Employment contract
• Follow X + checklist to judge compliance with X
• Admission charge
• Minimum qualifications

• what’s changed today?
• How are we doing (this period)?
• How does this compare?
• What happened when?

• A YesNo answer vs detail and context
• Facts vs opinions
• Generalities vs specifics
• Past, current or future
• Advice or evidence
• To verify / validate vs variation
• Reflexive (asking questions of one’s self) vs external
• About a thing, or about a person
• About direct experience, or about something not experienced
• About self, person you’re talking with, or a third party
• Information or instruction
• Act immediately, or building options

These questions are not about testing, but about strategy in general. You can certainly use them for a test strategy. They won’t help you get to the best strategy, but they will help you reveal the actual strategy, and so to see how widely it is shared.

Strategies reflect the values that an organisation holds. Where an organisation has conflicting values, you’ll either find several competing strategies, or you’ll find that you’ve not got to the deepest values.

• How did you arrive at this plan?
• Where does the money get spent?
• What are you paying attention to?
• What risks have you taken recently?
• How do your people spend their time?
• How do you reward 
• those who set direction? 
• those who do the work?
• those who find shortcuts? 
• those who make the work, work better?

Strategies are about what people fear, as well as what they value. People often don’t answer direct questions about fear. I’ve used these as proxies:

• What opportunities have been seen as too risky?
• What have you been relieved about?
• What does this organisation punish people for?
• What are you trying to avoid?

(empty!)

(empty!)

(empty!)

(empty!)

(empty!)

• How do we quantify risk? How would we compare or rank risks?
• Where do we keep risks to remember them? Project risks, product risks, process risks? More?
• How do we prioritise actions to mitigate risks?
• (If doing risk-based testing) how do we allow risks to change our work? What do we do when we find a risk? Mitigate it? Log it?
• Where do risks come from, in this work? 
• Example impacts of product risk - exposure / corruption / loss of data, silent vs public harm, delayed failure, harm to person / property / organisation / wider system, failure to serve, failure to route funds / resources, systematic bias leading to harm
• Example impacts of project risk - late to market, fewer abilities, greater spend? Lessons binned? Miss the boat? Cause customers to leave? Expose to reg failure? Cause staff to leave? Ruin, maim or kill people?
• What systems failures are important? How does our testing address that?

Needs: to get us to our goal

• Requirements
• Examples
• Workflows
• Data transformation and constraints

Needs: to fit with context / avoid hindering others

• Expectations from Audit / Security / Infrastructure
• Outside constraints (time, resource availability & consumption | things, people, intangibles, time)
• Deadlines / env contention / holidays
  

Externally-imposed gateways

• Schemas
• Standards
• Business Rules
• Validations
• Regulations

Agreed Gateways

• Acceptance Criteria
• Quality Criteria
• SLAs

Risks

• Product Risks
• Project Risks

From use

• Examples - Runnable examples, Standard examples, Edge cases / special cases
• User Stories
• User Journeys
• Workflows
• Lifecycles
• Problems to diagnose

Project drivers

• Whatever the PM looks at regularly
• Known bugs
• Time constraints
• Resoource availability / constraint

Driving the individuals in the team

• Big list of tasks imposed by the work
• Skills constraints
• Interest in area
• Opportunity to change
• Power - taking and avoiding
• Responsibility

• Where are your test artefacts kept (and what do you consider a test artefact - code, Scripts, Sessions, charters, results, dataset, test strategy, test plan, test scope, test environment details,)
• Where are your test drivers kept (and what do you consider a test driver - Requirements, acceptance criteria, SLAs, functional constraints, risk register – for the project, and for the product,)
• What work are you responsible for? How do you know when someone gives you work? How can you say no? How do you mark that you’ve taken it on? 
• For a given piece of work, how do you know you’re done? Done well? Done enough? Run out? Overspent?
• How do you get access to an artefact? How do you get access to a resource? Read/change/delete?
• Who Can you ask about xxx? Who should you review your work with? Who do you give it to? Who do you tell you’re done? Who are you working with? Who works for you? Who are your peers? Who should you talk with outside the team?
• Do I (you) log bugs? Where do I log bugs? When? How? 
• What do I need to start running a test? Environment / data / SUT / script(automation/manual) / purpose(req?) / budget (time/money)
• How do you do collective communication, and how do I get on the right email lists, slack channels etc. 
• How do you arrange to spend time together - and how do I get access to key diaries, meeting rooms etc.
• Where do you keep your documents, and how do I search / navigate each of those places?
• Where is the big list of tools, and how do I install / access them?

• Do you have any questions? (YesNo)
• Who has a question? (focus on one person in group and one question)
• What questions have you got? (invitation to ask)
• What else should we cover? (invitation to talk)
• What else shall we think about? (invitation to talk about new stuff)
• What else shall we think about, while we’re together? (time limited)

This is the one I was taught in school. It's memorable, but unclear – the words do double-duty (i.e. when can mean 'at what time' and 'under what circumstances') and it doesn't distinguish between questions that are primarily factual (who, what, where, when), and those rather more opinion-based. It's a good starting point, but (as with everything else you're taught in school) better built on with queries than accepted.

Here's the big page on Wikipedia, which brings it back to Aristiotle. 

Kipling wrote this mnemonic in (189?): 

I have six faithful serving men | They taught me all I knew | Their names are What and Where and When | And Why and How and Who

• http://changingminds.org/techniques/questioning/5w1h.htm

Thomas Wilson (1524-1581) wrote in The Arte of Rhetorique, 1560:

Seuen circumstaunces, which are to bee considered in diuers matters.

• {i. Who did the deede.
• {ii. What was done.
• {iii. Where it was done.
• {iiii. What helpe had he to doe it.
• {v. Wherefore he did it.
• {vi. How he did it.
• {vii. At what time he did it.

The circumstaunces in Meter: Who, what, and where, by what helpe, and by whose: Why, how, and when, doe many things disclose.

From his expanded text, you'll see that he's more specific, so less useful...

• https://scholarsbank.uoregon.edu/xmlui/bitstream/handle/1794/774/arte.pdf

Liberating structures describe What? So What? What now? 

James Bach describes his mnemonic Huh? Really? So?.

You might see parallels with Center, Enter, Turn

• Kickstart: What’s on your mind?
• AWE: And what else?
• Focus: What’s the real challenge here, for you,? Not now, not biggest, not most important .
• Foundation: What do you want?
• Lazy: How can I help?
• Strategy: If You’re Saying Yes to This, What Are You Saying No To?
• Learning: What was most useful for you? Not most important. Not thing. Not learned.

David Grove’s 12 basic Clean Language questions (‘X’ and ‘Y’ denote one or more of the client’s words/ non-verbals)

Developing Questions - Attributes / Location / Metaphor / Relationship

• And is there anything else about X?
• And what kind of X is that X?
• And where is X? / And whereabouts is X?
• And that’s X like what?
• And is there a relationship between X and Y?
• And when X, what happens to Y?

Sequence Questions - Before / After

• And what happens just before X?
• And then what happens/ And what happens next?

Source Question

• And where does/could X come from?

Intention Questions - Desired Outcome / Necessary Conditions

• And what would X like to have happen?
• And what needs to happen for X?
• And can X?

from https://learning.oreilly.com/library/view/peter-druckers-five/9781118979594/

• What is our mission
• who is our customer
• what does the customer value
• what are our resutls
• what is our plan

from https://learning.oreilly.com/library/view/seven-strategy-questions/9781422171639/

• 1. Who Is Your Primary Customer?
• • Does everyone know what your primary customer values?
• • How have you organized to deliver maximum value to your customer?
• • Have you minimized resources devoted to your other constituents?
• 2. How Do Your Core Values Prioritize Shareholders, Employees, and Customers?
• • What tough decisions have been guided by your core values?
• • Do your core values recognize your business’s responsibility to others?
• • Is everyone committed to your core values?
• 3. What Critical Performance Variables Are You Tracking?
• • What is your theory of value creation?
• • What could cause your strategy to fail?
• • How do you create accountability for performance?
• 4. What Strategic Boundaries Have You Set?
• • What are your major reputation risks?
• • Does everyone know what actions are off-limits?
• • What strategic initiatives will you not support?
• 5. How Are You Generating Creative Tension?
• • How are you motivating everyone to think like winning competitors?
• • How do you encourage innovation across units?
• • Have committees and dual reporting made your organization too complex?
• 6. How Committed Are Your Employees to Helping Each Other?
• • What is your theory of motivation?
• • How are you creating shared responsibility for success?
• • How do your compensation policies affect commitment to help others?
• 7. What Strategic Uncertainties Keep You Awake at Night?
• • How do you focus everyone’s attention on these uncertainties?
• • What system do you use interactively to stimulate change?
• • How do you encourage bottom-up information sharing?

A series of questions used by the C.I.A. to help agents view challenges from all possible angles. This list is from From Michalko, Michael. (1991) Thinkertoys, Ten Speed Press, Berkeley, CA. 

Here's a link to a more-recent take by ModelThinkers, which handliy gathers the questions by topic, and points out that it's pretty unaware of bias.

The Problem:

• Why do we need to solve this problem?
• What are the benefits?
• What do we still not understand, or need clarified?
• What isn't the problem?
• Is the existing information enough? Redundant? Contradictory?
• Can we draw a diagram of the problem? Portray it as a model?
• Where are the boundaries of the problem?
• What are the separate parts? How are they related?
• What are the things that can't be changed, and the things that can?
• Have we seen this problem before? Was it in a slightly different form?
• Can you restate the problem in more general terms? More specific? Changing the rules?
• What are the best, worst, and most probable cases you can imagine?

The Plan:

• Can you solve the whole problem? Part of it?
• What would we like the solution to be? Can we picture it?
• How much of the unknown can we determine?
• Have we used all the information? Have we taken into account all essential parts?
• How many different ways have we tried to solve the problem?
• What have others done?
• What/where/how/who should be involved with the solution?
• Can we use this problem to solve other problems?
• What is unique about this problem?
• How will we know when we are successful?

• Am I asking too many questions?
• Do these questions seem relevant? Am I asking the wrong Qs?
• Is there anything else I should be asking? What are the right Qs?
• Are you the right person to answer these questions? Should we be talking?
• Is there anyone else who can provide additional information? Who should I be talking with?
• Is there anything you want to ask me?
• May I return to you with more questions later?

In this tweet, MoT asked: "You only have one question you can ask for the rest of the year to give you the information you need in any situation. What do you ask?". I've picked out some of the answers (and I'll attribute soon!)

• Please give me an example
• What else?
• WHat are we trying to solve?
• How can this go wrong?
• What's the value in this?
• How can I help?
• What do I need to know?