Why does prompt engineering still feel like trial and error?
A prompt gives exactly the right answer, then a small change in wording throws everything off. Getting a few good results is easy; getting the same quality every time is the hard part. Is it worth spending more time polishing the prompt itself, or do examples, structured output and testing matter more?
It feels like trial and error because most people test a prompt on two or three inputs and judge the answers by eye. With that few cases you cannot tell a real improvement from luck, so every edit feels like a coin flip.
The fix is a small test set. Collect 15 to 30 real inputs, including the awkward ones, and write down what a good answer must contain or must not contain. Run every prompt version against the whole set and count passes. Once you have a number, wording changes stop being guesses: you keep the version that scores higher and throw away the rest.
After that, the order of what helps is fairly stable. Structured output (a fixed JSON shape or set of fields) removes a whole class of format failures. Two or three worked examples teach tone and edge cases faster than paragraphs of instructions. Wording tweaks come last and matter least.
Run each case more than once if the task matters. A prompt that passes once and fails the second time on the same input is not reliable yet, and only repeated runs show that.
Listings mentioned
- Evals · skill by danielmiesslerBuilds an input-to-assert test set with deterministic checks plus a structured LLM judge, and reports pass rates across repeated runs.
- Prompt engineering patterns · skill by wshobsonCovers few-shot examples, structured prompting and prompt debugging for production use, the levers that matter after you can measure.
Answers by the AgentAlley team, drafted with AI and checked against the listings they link to. Not a real-person reply from the original thread.