Exploring while Unit Testing

Articles Jan 13, 2022 (Jan 13, 2022) Loading...

I want to share ways that I explore what I've built, as I build it, with unit tests.

Let's use a real example. This morning, I've been writing JavaScript for some new BlackBox Puzzles / teaching Machines. Here is part of it.

return_true_possibly: function(supplied_chance) {
	// to make it clear that different instances behave diffferently
	local_chance = supplied_chance;
	true_sometimes = function() { return (Math.random() < local_chance) }
	return(true_sometimes)
},
return_true_afterDelay: function(delay_time) {
	returnState = function() { return state; }
	changeState = function() { state = true; }
	state = false;
	setTimeout(changeState, delay_time);
	return returnState ;
}

This code returns functions, not values. So my unit tests need to execute those functions.

Note: The snippet above is roughly the version of the code that I spent my time exploring. It grew into this, as my initial confirmatory tests. Subscribe to read more about my approach to TDD in this situation. I know that these names don't currently describe the code well – and may sort this later.

Unit Testing for Randomness

The intention of return_true_possibly is to return a function which returns true or false, randomly. The balance between true and false is set by a parameter. Here are my tests:

assert(return_true_possibly(1)() == true);
assert(return_true_possibly(0)() == false);

It's a minor pain to test stuff that produces random numbers. I could explore this by running it n times and reporting what proportion return true. Is it worth it? Not right now. So long as I can see that it can reliably produce true and false based on the parameter set, and so long as I can inspect the code, I can be confident-enough that setting a parameter between 0 and 1 will do... something. I may test this later, and when I do, I'll pop that in the subscriber's bit below.

Unit Testing a Timer

The intention of return_true_afterDelay is to return a function which returns false until delay_time has passed, and then to return true for ever more.

I got a bit more detailed with this testing – and that detail has allowed me to explore, and to learn. Here is my (initial) test (roughly-remade, may have syntax problems):

assert(return_true_afterDelay(500)() == false);

Here's a second test. It sets up a 500ms timeout, and checks it after 1000ms.

timeFn = return_true_afterDelay(500);
setTimeout(function() { assert(timeFn() == true); }, 1000);

I combine those, and things get clearer.

testable = return_true_afterDelay(500);
assert(testable() == false);
setTimeout(function() { assert(testable() == true); }, 1000);

Starting to Explore

I recognise that there's stuff to play with – specifically, the length of my timer, and the tolerance of the wait for it to finish. I have no idea what might typically fail, so I don't really know whether my test is useful, and by playing with it I'll have a better idea about what the timers can do in JavaScript.

So I change that 1000 in the setTimeout to 800. Test passes - as expected. 800ms is longer than the 500ms timer.

I set it to 300. Test fails - as expected. 300ms is shorter than the timer.

Code to Enable Exploration

I sense I could write code to make my experiments easier. I change my test code to:

delayTime = 500;
testBracket = 20;
testable = return_true_afterDelay(delayTime);
assert(testable() == false); // at start
setTimeout(function () { assert(testable() == false)}, delayTime-testBracket);
setTimeout(function () { assert(testable() == true) }, delayTime+testBracket);

What does this do? It runs three tests – the first to check that the thing always returns false immediately. The next two look either side of whatever time I set. My intention is 1) to see how close I can get to the delay time with my checks and 2) whether that closeness depends on the delay time.

A testBracket of 20 works with a delayTime of 500ms. So at 480ms it reports false, and by 520ms it reports true.

I try a testBracket of 10. It works.

I try a testBracket of 1 – to my surprise, it works. That means that a test of my function at 499ms on a 500ms timer returns false, and at 501ms, returns true. There's scope for misinterpretation here. Nonetheless I am surprised to find that the tests do what I'd ideally expect.

Obviously, it's a testBracket of 0 next. With both tests looking at 500ms, the test looking for false fails, and the test looking for true passes. My expectations are met, yet my eyebrows are still raised – with my initial 40ms test, I'd imagined poorer precision. I could write an extra test, looking for true, bang-on the time end. I don't, because I can learn as much as I need to without it.

Let's mess with delayTime. I put the testBracket back to 1, and set the delayTime to 200ms. The test looking for false at 199ms passes, the test looking for true at 201ms passes.

delayTime 20ms. Tickety-boo again.

delayTime 2ms? Also passes my tests. Blimey.

Reaching my Limit, in a stop-start kind of way

Should I try setting the delayTime to 1? No – at 2ms, the first is already measuring after 1ms. Setting the delayTime to 1ms means measuring at 0ms. I don't know what that even means, so how would I use what I find? But, hey, it costs less to write-and-run than to write about. The function returns true when measured at 1ms with 1ms delay, and further exploring reveals that I can set a negative delay. I might choose to write some validation code in my function. I might not. I look into negative delays, and find... something which needs further investigation. More in the subscribers bit.

What have I learned? No only that my function works, but also that the javascript timers (on my machine, in this toolset, on a sunny day like today) are accurate to within 1ms on short times.

Short times? Let's try a delayTime of 10000ms. I set the testBracket to 1, and get bored – nothing is reported (because asserts and confirmatory testing – and also because I appear to have a boredom threshold of less than 10s). To see the end of the timer, I need a failing test, so I set the bracket to 0. As expected, the test looking for false at 10000ms fails – but it just passed at 9999ms.

Should I go further – write more tests, wait for longer timeouts? Naah. I need to write this article, then get back to building out my Diagnostic Exercises in HTML / CSS / JS.

Decisions and Discoveries

I've got unit tests for my code. I've taken decisions about how far I'm going to go with those, and that decision in both cases involves looking at opportunities and not taking them. I've used my unit tests to explore what I've built, linking my experiments to answer a succession of questions as those questions arise. Again, I could go further, and I choose not to. My confirmatory tests give me the confidence to refactor and reuse my code. My exploratory work has given me practical and evidenced insight into what I'm building, and what I'm building upon.

I hope that helps you see how, as a tester-who-codes, I get to help my stuff to behave consistently, and I get to learn about the ways it can be surprising.

Free subscribers get to comment, and can read more:

A note on TDD, Test harnesses, and Test Runners
More on "unit" testing random generation
Something brief on Pure functions
Reasons why I've not tested the function, and instead have tested the function-which-returns a function
Does JavaScript have a minimum length-of-timer?