Category: Programming

We’ve had full houses at tech meetups at KForce heaquarters before, but never this full! It was a packed room last Wednesday, with everyone there to see Dr. Venkat Subramaniam give his talk, Power and Perils of AI Aided Coding.

For those of you who couldn’t make it, as well as for those of you who did, and want a recap, here are my notes.

What AI is and isn’t

“It is not intelligent,” said Venkat as he started his talk. “This is one of the myths we carry around with us.”

He’d recently been in Switzerland, where a colleague told him about the AI tool she’d been using. She asked whether she should call the AI “he” or “she”, and Venkat’s  was the same as the one I would give:

“It’s called it. When I use a hammer, I don’t say ‘I like him.’”

His own expansion of the acronym “AI”, which he told us at his appearance in Tampa, is Accelerated Inference. AI as we know it today an extraordinarily fast inference engine. Inference is something we do too, albeit more slowly, and with a hard ceiling on how much information we can hold at once. Watching a machine do inference tricks us into believing something beyond logical interence is happening, but in truth, it’s doing pattern matching at machine speed and machine scale.

This was the spirit of the whole talk. Everything else Venkat said, which covered testing, conveying intent to an AI coding tool, to the cost of failure, to skill files… they all come from the general idea of accelerated inference.

Why on Earth would you build a machine that eats your ice cream?

If AI isn’t intelligence, what’s it actually good for? Venkat provided an answer that was specific (and refreshingly so), and he opened it with a thought experiment.

“Raise your hand if you’d like a machine that washes your clothes, folds them, and puts them away”, he said, and pretty much every hand in the room went up.

“Now raise your hand if you’d like a machine that eats your ice cream for you.” The hands all went down, and there was some chuckling in the office.

“You don’t want the machine to do what humans are good at. You want the machine to do what we suck at.”

If you’re in a line of work where your output is a direct byproduct of thinking, you know that we suck at dealing with cognitive load. Venkat, who does consulting work, described the recurring experience of being parachuted into a client’s codebase. He’s had to stare down functions that are thousands of lines long, and he’s asked “Why did they create the function this way?”

Pasting that massive function an AI is useful. In moments,  he can an explanation of what that function does and betteer still, a list of what’s probably wrong with it. That’s the superpower: the machine has no cognition to overload. It tirelessly compares the code against patterns that it’s been programmed with.

He named two more AI superpowers that he harnesses as a consultant:

• Typing. Venkat has been programming for 40 years and types fast; in fact, his presentations don’t feature slides, but a vim screen where he types the statements he wants the audience to remember. But as fast as he is, he’ll never type at the speed of thought. The gap between “wouldn’t it be cool to try this” and “ugh, that’s twenty files” is where good ideas quietly die. AI doing the typing can close that gap.
• Prototyping. With AI, it’s now effectively free. This was his best story of the night:Three weeks ago a user came to him with a feature request, and his instinct was the smart consultant’s natural first instinct: to find a nice was to say “no”. But then he caught himself, and instead of rejecting it, he built the thing in about ten minutes with AI assistance and handed it back.The user tried it and said it was exactly what they wanted. A few minutes later: “that’s terrible.”A few minutes after that: πI thought this would be a great idea, but now I see why it isn’t. Thanks for trying. We don’t want this.”“That’s another power of AI: it reduces the cost of prototyping to near zero.” Prototyping is how you find out whether you’re pointed in the right direction, and if it costs nothing (or at most, a couple of hours instead of a couple of weeks or months), you can afford to be wrong out loud.

What’s the cost of failure?

The theme for the talk from this point on was simple: What’s the cost of failure?

Venkat recounted that he’d recently run into an old friend who was thrilled about shipping software without developers (that friend is a demented and sad individual).

Venkat asked him, directly: “If what you’re building fails in production, what’s the loss?”

“Oh, it’s nothing,” replied the friend. “What I’m building doesn’t affect anybody.”

“That’s the problem,” Venkat told the room. Because when people chant loudly that AI produces gold quickly, that’s the only part anyone hears. Nobody gets the memo with the caveats.

He told two stories to illustrate the point…

Cost of failure story #1: The vanishing flight segment

Venkat booked a September trip back in June (he likes to book distant trips well in advance). The “there” route is Denver to Chicago, Chicago to Zurich, and then Zurich to Oslo, where he’ll stay for two weeks. The “back again” route starts in Oslo, then goes to Brussels, then Washington Dulles, and finally Denver.

Two Mondays ago at 6:30 a.m., he got an email from the airline with a receipt for travel. He nearly deleted it, but had a nagging feeling that made him wonder: “What travel? I haven’t touched anything.”

(At this point in the story, I started getting paranoid about a mid-October/early November trip I booked back in July.)

Upon closer examination, it turned out that his instinct was right. The Brussels-to-Dulles segment that would get him home from Europe was simply gone. It took a two and a half hour phone session with the airline to get it restored, and when he asked what had happened, the answer was “We have absolutely no record.”

“Now think about that for a minute,” Venkat said. Scalability, performance, traceability, observability, security are the things that keep him awake at night, and he’d nearly become the victim of system that could silently mutate a customer’s itinerary and leave no trace.

Cost of failure story #2: The Mother’s Day problem

A developer took his wife to lunch for Mother’s Day. At the end of the meal, his card was declined because his limit had been exceeded. He knew that was impossible, and  he was also the implementor of that system’s code.

He apologized to his wife, told her she was on her own getting home, and went to the office to find out what was causing the problem. That’s where they discovered the system had been duplicating charges on everyone’s cards for several hours. Venkat asked him afterward what the bank did about it. “I can’t tell you,” he replied, “but in short, we paid them all off.”

Venkat ended story time with this observation: speed in and of itself is useless; it’s sustainable speed that’s valuable.

When someone above you is chanting “Go faster, go faster!”, ask them two things:

1. What’s the cost of failure?
2. Would you please sign a document saying you’ll be responsible when it fails?

“You can delegate your coding. You cannot delegate your responsibility or your reputation.” He pointed to Air Canada’s chatbot lawsuit as the precedent for how “The AI did it!” holds up in the real world.

Demo one: The repeated-letter function, and a karmic reckoning

Venkat then did what makes his talks worth attending: he opened a laptop and did some live coding, where anything can (and often does) happen.

He gave Claude a prompt: Write a Java function that returns the first letter repeated in a string; return an empty character if the string is empty or nothing repeats; ignore spaces.

First, a note on ambiguity that I thought was the sharpest teaching moment of the evening. In “hello”, the first repeated letter is l. Easy.

In “hello there”, most people say teh first repeated letter is l. But it’s actually h! The spec says to return the first letter that is repeated, not the first letter that’s repeated consecutively. English is ambiguous.

“This is why we write unit tests!” said Venkat. Tests provide the redundancy that removes natural language’s ambiguity.

Claude produced working Java in an imperative style, with variables named s and c. Venkat pointed out the reasons why:

• Because that’s what’s in the training data.
• Because that’s what most programmers write.
• Because most people use single-letter variable names.

Then he asked the room to raise a hand if they agreed that, collectively, humans write fantastic code. Not one hand went up

“So here’s the irony. Humans write bad code. AI was trained on code written by humans. Humans complain that AI writes bad code. This is called karma.”

“It’s an imitating engine, not an intelligent being.”

He prompted for a rewrite in functional style and got one. He polled the room: “Thumbs up if this is beautiful enough to frame.”

The code got some thumbs sideways and some thumbs down. The code was clunky with two passes over the data (a code smell in the functional world), a map built up top and consumed below, and overly complex for the problem at hand.

It wasn’t as bad as a previous run of this same demo, which happened in front of a thousand people and on camera. That one produced code that mutated a variable inside a filter. He declined to describe what he’d do to a colleague who did that, other than to note it would happen after invited that colleague for a chat in the parking lot.

At that recorded talk, he’d asked the model what it thought of the code it had just written. It agreed, in that LLM way we’ve grown sadly accustomed to, that the code was bad. He asked the model to fix it, but the rewrite turned out worse. He pushed harder, and the model blamed the language, telling him Java wasn’t a good choice (and this once, I agree with the model). When he named a specific method it should have used, it agreed instantly and produced the right implementation.

An attendee offered the next-token-prediction explanation: nobody asks “What do you think of your work?” when they’re happy with the outcome, so naturally the completion that follows that question skews negative.

Venkat liked that explanation, but offered an alternative rooted in his own framing. When you ask an LLM to write code, it infers from code, with all the bad habits and practices included. When you ask it to evaluate code, it infers from blog posts, tutorials, and books, where authors show bad examples and explain at length why they’re bad. As a book author himself, he noted with some feeling exactly whose work that training corpus is made of.

For the record, his own solution:

// Java
Stream.of(word. split(""))
filter(letter -> word.LastIndexof(letter) > word. indexof(letter); 
    findFirst()
    orElse("");

Much fewer lines than what Claude generated, and so much simpler.

“AI tends toward more complexity. Why? Because collectively, we love complexity. Complexity lets us hide behind things.”

AI is an amplifier

Between demos, there are two notable lines:

The first line: A project manager in the Netherlands had told Venkat: “If I don’t have the fundamentals, AI will cover my back.” Venkat’s reply: “With all respect, if you don’t have the fundamentals, AI will expose your back in public, in the most humiliating way you can imagine.”

AI doesn’t do things differently; it amplifies what you already know. The smart get smarter and the rest get worse. If you know the fundamentals, it’s a good tool to have in your “belt”. If you don’t, you have no ability to evaluate what it just handed you. As he put it when an attendee raised exactly this point: people who don’t know functional programming can’t tell whether that rewrite was good or bad. People who do can smell it immediately.

Here’s the second one, from his boss of 40 years ago, which he was careful to credit: “A fool with a tool is a dangerous fool.”

Hence his revision of the old maxim. Trust but verify is out. The new norm: don’t trust, and verify the heck out of it.

Demo two: Wordle from a spec, and the missing test directory

The second demo had a bigger scope. Venkat wrote a plain-English spec file for the game Wordle…

• 6 rows of 5 boxes
• Green for “right letter, right position”
• Yellow for “right letter, wrong position”
• Gray otherwise
• Spell-check each guess against a web service
• Pick from a list of about a hundred words
• Build it in Java and JavaFX

…and turned Claude loose on it.

It worked. We played a round as a room, shouting guesses at the screen.

He turned off his Wi-Fi mid-demo to see what would happen with no connectivity. Previous runs had either assumed unverifiable words were valid or written a graceful message; this run produced a message in red text: “Could not reach the spell checking service. Please try again”. This was a good outcome, but it happened because of luck instead of any spec.

Venkat ran tree on the egenrated project and asked the room what we noticed before looking at a single line of code.

The project used Maven as a build tool, and its layout was reasonable. But there were no interfaces anywhere. The spellcheck had no abstraction behind it, breaking the Open/Closed principle, which would make swapping implementations more difficult than necessary. There were redundant-looking classes.

One sharp attendee pointed out, there was no test directory. “Can I offer you a hug?” Venkat asked in reply.

Venkat tied the subject back to process. Agile development, when you take away all the ceremony and everything you sat through in the two-day course, is simply feedback-driven development, where you know that what worked still works, and that a change moved you forward rather than seven steps back. If you ask AI to change your code, what tells you it didn’t break the things that already worked? If there’s no test suite, there’s no feedback, and you don’t know if something that worked before is now broken as a result of the last change.

The story was the same inside the code. A function in the GuessEvaluator function long enough to fail SLAP (Single Level of Abstraction Principle), broken up every few lines by comments explaining what the next chunk does. Venkat’s read on that habit is oddly sympathetic: programmers who write long functions and litter them with comments are bad programmers but not bad humans. The comments are an act of empathy toward whoever comes next. The AI reproduced the pattern exactly, because that’s what it learned from us. He reached for a country song he couldn’t quite name, about a kid mirroring his father: Dad, I’ve been watching you. (I suppose Harry Chapin’s Cats in the Cradle is also applicable.)

There was also too much code. “Two characteristics of humans: we write way more code than we should, and we eat more food than we should. I’m succeeding a little better on one of those.”

Credit where due, though: the generated code used correct, present, and absent rather than green, yellow, and gray. That’s a better vocabulary than the one in the spec.

The actual lever: skill files

At the end of the session, Venkat switched gears and went from descriptive to prescriptive.

“It does not make sense to use AI without proper skills, both that of the human and the ones we provide to AI.”

Venkat deleted the whole project, added a .claude directory, and added a few lines to the spec asking for JUnit, Maven, and a combination of unit, acceptance, functional, and integration tests. “Be generous in asking,” he remarked.

He also said something I suspect is the real reason he’s optimistic: we have never managed to bring as much discipline into our practice as we wanted, and if the machine can write the tests, “I don’t have time” is no longer available as an excuse.

The .claude directory held three small files:

1. Clean code practices
2. Java coding practices
3. JUnit practices

The clean code one is deliberately language-agnostic. Anyone who decides that they want a project implemented in C#, Python, JavaScript, or PHP can use it unchanged. A sample of what’s in it:

• Avoid single-letter variable names, including lambda parameters. Use short but meaningful names.
• Always use curly braces, even for a one-statement if or for. (Python folks excepted, obviously.)
• Do not write comments that tell me what the code is doing. Write self-describing code. Tell me why.
• Avoid long functions. Apply SLAP and SOLID where it makes sense. Keep code cohesive and dependencies decoupled. Follow DRY and SRP. Keep UI and logic separated.

The Java file gets specific:

• Prefer JDK 25
• Choose the latest language features where possible
• Prefer newer methods
• Avoid deprecated methods
• Avoid preview features, which he added after the model kept generating code that wouldn’t run
• Prefer immutability
• Prefer functional style over imperative

A separate functional-idioms file: avoid mutating external data, avoid side effects, avoid multi-line lambdas (extract them into a private method), prefer method references, create immutable results, keep streams simple, don’t pile multiple dots on one line, align dots vertically.

Here are four things Venkat said about skill files that I’ve decided to internalize:

1. Skill files are dynamic. They’re not static. They should live in version control and evolve through pull requests like anything else. Anyone on the team can propose a change; others review it. Your most junior teammate may be the one closest to the code that matters.
2. Cohesion applies to skill files too. Someone told him they’d written one enormous skill file, and the model ignored most of it while burning tokens. Prefer many small files, each targeting one area, over a few giant ones.
3. An older model with good skill files beats a newer model with bad ones, and at lower cost, too. And even where the newest model with the same skills does somewhat better, the older one may still be the better deal. “It literally comes down to how good your skills are.”
4. When the AI misbehaves, don’t complain; edit your skill file. “You may curse at it first. That’s one advantage over working with humans.”

He mentioned that community skill-file repositories are emerging, with quality metrics and security analysis attached. The security part matters because prompt injection via context and skill files is one of the things he’s genuinely worried about. The same idea works as a private repository inside an organization, giving you three tiers to draw from: community skills, company skills, and team skills.

The rerun, with skill files in place, produced the following:

• A test directory (finally)
• Variable names like rawGuess instead of s
• Small functions
• A record for the letter result
• A switch with pattern matching
• An interface for the spell checker

There was still more code than the problem called for, and that’s exactly what goes into the skill file next.

His caveat was unsentimental. Even with skills, LLMs remain inconsistent and unpredictable. Skill files don’t get you the beautiful code you imagine; they move you along a spectrum from terrible toward acceptable and evolvable. That’s still a large win.

And one sequencing tip worth pulling out: review the tests first. “I don’t care as much about reviewing the code before reviewing the tests, because if the tests are not good, it doesn’t matter what we do after that.”

Q&A

In the post-talk Q&A session, the room pushed back, which is how you know it was a good crowd.

Q: “You have the fundamentals, and AI still did a bad job, and you had to review and correct it repeatedly. I’ve heard senior developers spend more total time working this way than just writing it themselves. Is that true?”

A: “With a good set of skill files it converges much better. That’s the difference. But the failure mode he sees most often isn’t senior developers generating code. It’s junior developers generating code and senior developers drowning in review. The answer is incremental development and feedback loops small enough to actually manage.”

On being told to go faster regardless: “If you mess with physics, you know who’s going to lose.”

On running multiple agents in parallel: Your mileage may vary, but Venkat says he’s not good at it. He’s still the bottleneck at the end of the pipe, and the quality of what he produces is affected by the volume coming at him.

On cost: An attendee did the math on tokens burned during a single live demo and asked whether this is survivable. Venkat’s honest answer was hope rather than analysis. He’d read that day about a 9GB hard drive once costing four thousand dollars. He also talked about a genuine concern that current pricing is heavily subsidized and not sustainable for the vendors either. Someone else volunteered that GitHub Actions had cost them $600 in five days last month before they migrated to a flat-priced VPS. Local models came up too, with the obvious trade-off: you need real compute at your desk.

On automatic approval: An attendee noted Anthropic had turned auto-approval of code off by default, reportedly because users approving under cognitive load had a worse error rate than the model did. They asked whether verification will eventually improve enough that we review only what needs reviewing. Venkat said he’d want to address the “if” before the “then,” but that his hope for the distant future is that we’ll treat AI-generated code the way we treat compiler output. “That’s a desire. I don’t have any proof it’s going to happen. We’re nowhere close to it yet.”

The closing checklist

Venkat wrapped with recommendations for minimizing risk, which I’ll reproduce more or less as delivered:

• Have an extensive set of tests.
• Review every piece of code that gets created.
• Use different models to review the code — in addition to human review, never instead of it. If AI finds problems, great. If it finds nothing, you still look.
• Consider the cost of failure. It helps to imagine the user knows your home address, because if it goes wrong, they’ll come knocking. Your reputation is the collateral.
• Run code quality tools. Run the CI/CD pipeline. Run the metrics tools. Have it evaluate dependencies, licenses, and keep them up to date.
• Keep an eye on what it’s actually doing. We get complacent and check only the results. Read the logs.

The evening closed with the usual excellent Tampa JUG happenings: a JetBrains license raffled off (and a dad joke from Venkat: “There are only two kinds of programmers, those with IDEA and those with no idea”), a stack of O’Reilly books including a signed copy of AI Engineering, which Venkat handed over with a straight face and the observation that nobody needs to learn the fundamentals anymore, and a conference ticket for October.

Big thanks to Kforce for the space and Kong for the food, and to the Professor Andy Seely who brought his class from Hillsborough Community College. (The next event is at HCC, by the way).