AI development: the good and bad

• AI-assisted development accelerates both coding speed and error propagation, with risks like cascading mistakes and design flaws magnified by AI’s context limitations, akin to Knight Capital’s $500M human-error-driven collapse.
• Real-world failures (e.g., Knight Capital) stem from process gaps, not technology; AI intensifies this by compressing multi-year human error patterns - like developers Alice, Bob, and Charlie inheriting undocumented code under time pressure - into hours or days.
• AI lacks persistent context, acting as a “new developer” per request, which forgets safeguards, amplifies poor documentation, and misses architectural pitfalls unless explicitly guided by experienced oversight.
• Defensive coding practices (e.g., explicit error checks, copy-before-move for files) are critical since AI won’t self-advocate for safeguards - users must enforce quality control as architects and reviewers.
• While AI democratizes development, inexperienced users often overlook foundational pitfalls, making seasoned judgment essential to prevent small oversights from escalating at machine speed.

Introduction

There is a lot of discussion about using AI for development. Much of it is over-stated hype, either for or against.

“AI only turns out garbage” is what some people will tell you.

“AI will replace developers” is what others will say.

Like most things, the truth is somewhere in-between.

Summary

AI-assisted development can help create code faster, but pitfalls like amplified design flaws and cascading errors demand careful handling

This article cuts through the hype by showing how unchecked AI use mirrors real-world failures like Knight Capital’s $500M loss (caused by human process gaps, not technology).

Even skilled developers like “Alice, Bob, and Charlie” introduce subtle mistakes when inheriting code under time pressure, and AI accelerates these risks by acting like a new developer every time you hit “enter” - forgetting context, skipping safeguards, and magnifying poor documentation or rushed decisions.

It is important that you need to enforce defensive practices (like file operation safeguards and explicit error checks) while juggling roles as architect, reviewer, and quality gatekeeper to prevent small oversights from exploding at machine speed.

Knight Capital Group

In 2012, due to a poor code rollout, Knight Capital Group lost almost a half-billion dollars in less than an hour. This ultimately led to the company folding.

There was no AI involved, it was strictly a human error. They lacked proper controls and testing.

This story is used to illustrate a point: bad code and practices aren’t exclusive to AI. Many of the horror stories you read about in the news have been happening - are still happening - in situations where AI isn’t involved, at all.

AI accelerates development flow, including mistakes. Knight Capital didn’t fail because of a single mistake. It was a series of mistakes and omissions in technology and processes which led to that final mistake. The same thing can happen when using AI.

Human errors cascade in slow motion, now let’s see how AI accelerates that pattern.

A tale of three developers

Let’s dive a little deeper into a normal development flow to give us a framework for discussing AI-driven development.

Consider three developers: Alice, Bob, and Charlie.

All three are experienced senior developers. They’re all good with algorithms, know how to structure code properly, understand the company’s problem domain, and just generally all-around experts. They’re the kind of developers every company wants to hire.

They’re also people. They have different specialties, with some overlap. For example, Alice may be great with Airflow, while Bob and Charlie have only some experience with it. Bob and Charlie also have their own specialties.

They all have their strengths and weaknesses, with some overlap. They all have their own coding styles, and their own approaches to solving problems.

All three are also under time pressures to produce code quickly. They wouldn’t necessarily review the entire codebase, formulate a comprehensive plan, and then make changes. They’d use what is readily and easily available.

Alice writes some code

Near the start of a project, Alice is asked to write a large amount of code to satisfy a major requirement. It involves Airflow, which is her specialty, so the code is written quickly and efficiently. It is rolled out into production, and everyone is happy.

Bob writes some code

About six months pass, and the customer has a new feature request. Alice isn’t available. Maybe she’s on another project, or got another job…it doesn’t matter, Bob is assigned the task of implementing the feature. He’s not very familiar with the libraries being used, but he’s able to look up some examples. He checks in some code that works, and everyone is happy.

Charlie writes some code

About six months pass, and the customer has a yet another new feature request. Bob isn’t available. That’s just something that happens during ongoing development. Charlie is given the task. He doesn’t have much time, but he’s good at his job and he checks in code that works, and everyone is happy.

What is hidden in this example

All three developers may be experts, but there is a lot of room for mistakes in that example.

When Alice wrote the original code, would her design accommodate the features being requested in the future? Was her documentation so comprehensive that a future developer could pick up where she left off?

Bob isn’t an expert in the specific technology. Did Alice use some advanced techniques? Are there pitfalls in the approach that Bob wouldn’t know to consider?

Charlie doesn’t have much time. Maybe he was assigned the task late on Friday, and it’s due early the next week. He has other obligations, professional and personal, to consider. Would he look for and read related material? Would he review all of the associated code and tests?

AI accelerates the development process

What was just described is pretty typical of any application developed over time. Using AI compresses the amount of time taken. What may have occurred over a year with human developers can happen in weeks, days, or even hours using AI.

When you’re using AI for development, every time you hit the “Enter” key it has been six months, and it is a new developer. Because AI “forgets” between requests, it’s like handing off to a new developer with incomplete docs.

The AI has only the code, the tests, the logs, change sets, any supporting documentation, and whatever is left in the context. If those things aren’t provided, or are insufficient, then the AI will tend to make more mistakes.

Much like a different developer working on existing code written by someone else, it doesn’t have an internal “picture” of what it was “thinking” before. It doesn’t have a style or approach that it uses consistently.

Mistakes in design and code and omissions in process have a cumulative effect, whether it’s the humans at Knight Capital, or your own AI-driven projects.

AI amplifies mistakes

The development process can be greatly accelerated by using AI. That includes the good and bad.

If the documentation is poor, or the API designed haphazardly, or there is insufficient logging for good troubleshooting - any of the things that good development process follows - then those omissions and failure accumulate, as well.

Poor design or architecture choices early in the development process can result in the need for a major code refactor. An application which was started as a simple command-line application may not be structured to be a full-fledged web backend.

In a “normal” development process, these mistakes may not be evident or noteworthy for weeks or months at a time. With an AI, these mistakes show up in minutes, hours, or days. A poor design choice can lead to refactoring major pieces of code in a matter of weeks.

Defensive AI coding

A recent news story was about someone who asked the AI to sort and move a bunch of files. This ended up failing and destroying the user’s data.

Just reading that scenario, there are at least a couple of problems.

An obvious one is “move files”. Someone with more experience seldom moves files. They copy files, check the output, and then delete the source.

Being a little more sophisticated is adding code which checks for success of operations before continuing. In other words, there would be a line or two of code which checked to see if the file was moved successfully, and exiting with an error if it wasn’t.

It is unlikely either of those things were done, or the AI was told to do them. The users wouldn’t have the experience to know to ask. It’s fine to think that the AI should have thought of it, but there’s also people out there which would be frustrated with emitted code which seemed overly-complex.

An AI won’t advocate for itself. It is up to the user to ask for checks and safeguards, or provide their own.

Experience matters

It’s great that AI is expanding development to more people. There is a lot of code that isn’t written right now because it’s too much effort, whether it’s learning to code or finding someone who can do it.

These new developers are going to end up learning about the pitfalls and problems of the development process, often the hard way.

Someone with more experience, no matter how they gained it, will make fewer mistakes.