AI in R&D – Staying in Control When Mistakes Matter

That is how AI can quietly move from assistant to driver. Not because the technology is in control by itself, but because we stop asking where human judgment still belongs.

The tools have arrived faster than our shared understanding of how and when to use them. They can do impressive things: create drafts, generate ideas, produce code, summarize documents, and build prototypes at a speed that would have seemed unrealistic not long ago. But the harder question is still the same: does this actually help us do better work, or does it simply make it easier to produce more?

Not every use of AI carries the same stakes, which is easy to forget in the excitement. A rough internal draft and a system that keeps someone safe are not the same gamble. Staying in control starts with knowing, at any given moment, which of the two we are dealing with.

We reach for the tool before we understand the problem

We tend to find a shiny tool and then go hunting for a problem it might solve. It works far better the other way around. You start with a real challenge, a slow process, some task that quietly eats your week, and only then ask whether a tool can help.

When the tool leads instead, you end up producing things nobody actually needed: documentation as thorough as a Bible that no one has time to read, or a script that automates a task we do once a year.

All of it costs time – the one resource we rarely measure, and often cannot. In a large organization that is hard enough, and in R&D even harder: you are usually doing something with no reference point, no past run that tells you how long it should have taken. In the end, we are paid for outcomes, not for our own curiosity.

Prompt laziness: the shortcut we are all a little guilty of

Much of this traces back to one habit worth naming. Call it prompt laziness. It is the quiet hope that if you just hit enter, the machine will hand you a jackpot, something perfect on the first try, so you can sit back and move on. It is a very human thing to desire, and we catch ourselves doing it as much as anyone.

The trouble is how easily we forget the limits of these tools. A model will hand you a confident answer that happens to be wrong, and the only way to catch that is work: checking, questioning, arguing with the output, running it past a second model, reading it line by line. Skipping all of that is tempting, and it gets more tempting still when the message from above is simply “be faster, use the AI.”

Often, that push comes from the very top, with no real metric for whether any of it makes people more productive, and so everybody ends up with the tools while almost nobody goes back to check whether they actually helped.

Where AI earns its place: prototypes and raw drafts

It is not the technology’s fault. There are places where it is genuinely extraordinary. Prototyping is one. If you need a basic tool just for today, you describe your need and have something rough that works by the afternoon.

Drafting is another: instead of explaining to a designer in words, you generate a rough version and say: “Make this, but real”. Used like that, the AI is a sounding board, not a finisher. I tend to keep Claude, Gemini, and ChatGPT open at the same time and play them off against each other, the way I used to wander over to a couple of colleagues.

The trouble only starts when we mistake the sounding board for the finished thing. We have all been there: a generated draft looks so clean and sure of itself that you nearly pass it on as done, until a second read shows you everything it quietly skipped. That is the real risk. We stop noticing what is missing because the output already looks polished enough to ship.

Think of a coding course where students build a little game of boxes bouncing around and shooting at each other, in a couple of hours, and it genuinely looks like something. Impressive, yes. Finished, not even close. The distance between those two is where the real work lives.

There is a sharper version of this that I often run into. Ask a model to write code, and it will happily write the unit tests for it too, the very tests meant to prove the code works. When the same author writes both the answer and the exam, a row of green checkmarks stops meaning much. I have watched a model reason its way to a genuinely strange conclusion and then declare the code perfect, because its own tests passed. The code was wrong. The proof was wrong in the same direction, so nothing flagged it.

A simple rule: the cost of being wrong

If there is one rule worth keeping, it is this. The more expensive a mistake is to undo, the less you should trust the first answer. A throwaway internal note can go straight through; nobody is hurt if it is a little off. A customer-facing claim, a contract, a piece of code that actually ships, all of these earn a proper review before anyone’s name goes near them. The model can suggest, but a person is still the one accountable.

In my own field, this is obvious. AI can draft code and explain it beautifully, yet have no idea that a Safety Integrity Level or a MISRA rule even applies unless someone tells it, and it will cheerfully ignore both. In safety-critical work, the review gate does not move an inch, because a wrong answer there can put someone in real danger.

There is a second thing that decides how much a mistake costs: whether you can take it back. A lot of software today can be patched over the air, so a function that looked fine in the lab and failed in the field can be fixed or pulled in the next update. But not everything can be reached.

Some code is burned into a chip, or sealed inside a device already out in the field, where no later fix can follow it. When you can still patch it, a bad call from the model is a nuisance. When you cannot, the same bad call can be a disaster. This is not new with AI either. In 1999 the Mars Climate Orbiter was lost because one team worked in imperial units and another in metric, and the mismatch went uncaught all the way to Mars. The machine did exactly what it was told. No one checked closely enough what it was being told to do.

So, the point is not to review everything to death. It is to spend our skepticism where the stakes really are.

The new skill is not asking the question but judging the answer

What counts as skill has quietly changed, and a lot of people have missed it. Anyone can ask a question now and get an answer, and that part is free. What separates a professional from a passenger is being able to look at the answer and tell whether it is any good.

My colleague Janne Rosberg has written about how the job is shifting from writing code to reviewing it, and that shift puts the weight on judgment. You cannot do another field’s work just by pointing an AI at it: without the expertise to judge the output, you are the person who asks for a few lines of code and hands them over, never noticing the holes.

Youngsters still stop by our stand at trade fairs and ask whether software is even worth studying any more. I tell them yes, the work will only grow, but it changes shape: away from the ground floor of what the functions are called, up toward the bigger problems that are actually worth solving.

The next generation will grow up treating whatever the AI says as simply true. Whether they keep the instinct to feel when something is off, that is becoming the whole job. We can still question what the model hands us, because we learned the craft before it arrived. The generation after us may not have that backstop. If they cannot tell a good answer from a merely confident one, who is left to say which is which?

There is a practical reason not to let those instincts rust, beyond pride in the craft. The more we lean on the tool, the faster the judgment it cannot replace starts to fade. And a tool you can no longer work without is a tool you can no longer check. Keeping your own skills sharp is what keeps you in a position to notice when the model is confidently wrong.

The responsibility is still ours

My colleague Kim Meyer argued that AI will not take developers’ jobs, but the decisions managers make about it might. That is responsibility at the level of the organization. This is the other half of it, at the level of one person and one output. When something goes wrong, do we own it, or do we hide behind “well, the AI made it”? It is a little like copying someone else’s homework, taking the good grade, and only finding out it was never really yours when the whole thing falls apart.

Showing off what the machine produced is the easy part. Standing behind it is the actual job. With a draft, being wrong is merely awkward. With steering written by an AI and reviewed by nobody, it is something else entirely. The question was never whether the machine is clever. It is whether anyone has taken responsibility for what it does.

That is the question hiding under all the excitement. Are we still in the driver’s seat, or have we quietly become the nervous passenger up front, hoping for the best?

Three questions before you press enter

None of this needs a rule book. It needs a short pause and three honest questions, the kind worth keeping somewhere you can see them:

• If this answer is subtly wrong, who catches it, and when: me now or the customer next month, and what are the real-life consequences?
• Could I produce or properly check this without the model, or am I trusting it only because I cannot judge it myself?
• Am I passing this on as finished work or as a draft, and am I ready to put my name on it?

The technology is not going anywhere, and we would not want it to. We just need to stay awake while we use it, with our hands still on the wheel.