Questions about Metrics

Background materials to the Questions Workshop

Basics

• What is this a proxy for?
• Why is this a good proxy?
• What does this  measurement tell us?
• What situations suit this metric?
• Does this metric lead you to enquiries, or offer answers?

Alerts

• Under what circumstances might it stop being a good proxy?
• What does this metric leave out?
• Under what circumstances might it decieve you?
• How might someone else use it to to game the system, or to deceive you?

Practicalities

• How do you measure this metric?
• Does it need to be measured regularly to make sense?
• How frequently should the measure be taken?
• Is it noisy, and how do we deal with the noise?
• Does it aggregate several other metrics?
• Is it delayed, and what it the lag?
• Does the metric very by who is measuring it?
• is this a relative metric, or absolute?
• What are the units?

Interpretation

• What does “going up” mean for this metric?
• What does “going down” mean for this metric?
• Can it go up and down?
• What does 0 mean?
• Can it go negative? What does that mean?
• Do non-integer values make sense? What do they mean?
• Is everything it counts as a unit the same size?
• Is there a minimum? What does minimum mean?
• Is there a maximum? What does maximum mean?
• What does no change mean?
• What does slow change mean? What is slow for this measure?
• What does fast change mean? What is fast, for this measure?
• What might it mean if  doing work, over a given period, makes no change to the metric?
• Can the measure change little, but what is measured change a lot (i.e. there are still 10 failures, but they're all new)
• What trends are interesting? Over what periods?
• Is the change, or rate of change, more interesteing than the actual value?
• What other measures generally change in the same way, or the opposite way? What might be happening in circumstances where this corrleation breaks down?
• Do you have a model which attempts to explain this metric? WHere does that model break down?