Learning with an LLM

tl;dr: use examples, get chatting, don't take the first answer, work in areas you know.

Asking an LLM to explain stuff is a new learning technique for me. An LLM is a fine rubber duck: setting out a clear question requires me to engage with the subject carefully and precisely. Reading my questions helps my own imagination to pop up insightful answers. Critically parsing any response makes me think about what is plausible and consistent, which makes me think about the underlying models that I'm using, and how they might change.

However, the quality of the LLM's responses is... unreliable. The words generally fit together into a clear explanation, but the information explained can be false. Key points might be left out, irrelevancies included. I've seen answers that purport to be about shell but include syntax for python, legal advice with no legislative support, opinions offered as gospel but as reliable as gossip. An LLM is very polished random walk through a lossily-compressed and limited repository. If you're planning on learning with it, you need to have ways to cope.

To limit the loopiness, I find it useful to use an example. LLMs are, by training, plausible and confident – in the necessarily shifting sands of new knowledge, an example is more solid than the LLM's ramblings. On the one hand, I would ask for an example of executable code or tests. These are easy examples to work with, as running them will give you immediate definitive feedback. In other areas, I've found it helpful to ask for relevant legislation when learning about legal stuff, or for specific quotes when asking about books. Most of the time, requests for specifics will either give you a source for what you need, or indicate that the LLM's response is down to the opinions in the work it has ingested, and not necessarily supported. On the other, I also found it helpful to ask it to explain an example – I used it to pick apart the output of a linter, test and compiler failure messages, and examples of grammar.

It helps to get chatting, primarily because a conversation engages me more with the topic: An LLM has no beliefs, so feels refreshingly open to changing its min, particularly if there's more in the training data to (stochastically) parrot. I've recently heard that's a great discovery!, thank you for that hint!  and You're right, as well as the regular I apologise for the confusion as the LLM spins on its metaphorical heels. What an encouraging rubber parrot it is.

A less wooly reason to converse is that your corrections and hints become part of the prompt, shifting the perspective for the rest of the conversation. This is a subpattern of something that we should be familiar with, as testers: don't take the first answer. Following that principle, I've found it useful when learning to:

• ask the same thing in a new conversation, or with a different system prompt
• ask for alternatives
• ask several LLMs
• ask the LLM to take apart its own answer.

I find that learning with an LLM is best when I already have insight: I learn better on subjects where I already know enough to tell good ideas from dumb. I'm pretty happy learning from an LLM about code patterns and syntax, or around application of a tiny and specific subset of UK education law. I'd not want to dig into Nelson's naval tactics, or the biochemistry of pollens. And there are some things where written language means not an awful lot, so I think I'd avoid using an LLM to discover Gabber or Hyperrealism. I imagine that most of us pick our learning materials to suit our expertise and learning style, so this is not a novel principle. Nonetheless, an LLM's fluency and speed makes can make it appear temptingly useful as a start point or as an expert, and it can be misleading in both circumstances.

In the right circumstances, an LLM is a delight to learn with. If you can stand to personify your tools, it's easy to cast it in the role of an open, informed, encouraging, helpful, endlessly available interlocutor. I've never had a study buddy like it.

As an example, here is what I learned in a recent single conversational thread with Claude. I was requesting information around shell scripts (whether built, suggested or generated). Often, I had run ShellCheck and was wondering about ShellCheck's reasons and about my alternatives. I learnt masses (particularly where the scripts themselves were obscure). The LLM gave me relevant information in moments, explained it well, and I could immediately try it out.

Claude reminded me of:

• the difference between '' and "",
• spaces around = ,
• the absence of booleans,
• [ as a 'test' command.

Claude helped me to use approaches which were new to me, including:

• functions in scripts,
• here-documents and here-strings,
• moving parameters around,
• do-nothing with :,
• set -x and set -e,
• comparing [ with [[,
• expanding with ${llmCodeContentParameter[@]},
• how to set up a config file,
• running executables with ./«name» compared with sh ./«name» or just «name».

And, aside from explaining, Claude could

• plausibly pick out idiomatic oddnesses and root cause of syntax errors,
• offer options for how to do things more clearly,
• unpick regex,
• spot antipatterns,
• give plausible pros and cons, and
• propose ways to strengthen to code against common failures.

The LLM put forward bad ideas throughout the learning process. If I'd not been able to temper its dreaming from the deterministic side by executing the code, and from the thoughtful side by my own insights into code, I'd have been conned.

A little voice wonders if I have been conned anyway, in some way that I don't see right now. I pay attention to that feeling, but can't address it myself: if you can help me work out what I've missed, let me know below.