AI engineering

Beyond Vibe Coding

I was incredibly skeptical when ChatGPT 3.5 burst onto the scene in late 2022, especially because I heard more hype from my non-technical friends than my technical friends. As they say in finance, when grandma is buying stocks, it’s a bubble, and LLMs in 2023 definitely seemed to pass the bubble test.

Come 2024, after an 85% reduction and the slaughter of the engineering market from 2023, I went from more of an architecture and mentoring role, back to an IC and it was strongly encouraged to use LLMs. I was extremely apprehensive to adopt such a technology, however, due to the fact that we changed to the full-stack paradigm from a split backend and frontend team, and because I hadn’t touched React for 6+ years, LLMs seemed to be the answer. So now, for the past 10 months, tools like Cursor, MCP servers, Claude Code have been my life and building RAG pipelines, LLM document generators, and vector search pipelines have been my projects. I think it’s safe to say I know a thing or two about AI.

LLM Basics

LLMs are next token prediction machines. They start off with a bunch of randomly initialized weights and as the neural network trains it, it processes batches of text sequences during each training round called an epoch. The model tries to predict the next token in each sequence, compares its prediction to what the actual next token should be, calculates the prediction error, then propagates that error back through all the layers to update the weights. It moves through massive datasets scraped from the web, scanned in from libraries, or found on Github.

The end result is a model that exists as a bunch of individual scalar weights of how tokens in different contexts relate to one another. When you see “100b parameter model” that’s 100 billion individual numbers scattered throughout the network.

When we query an LLM, it uses these weights and runs your input through multiple transformer layers where each token computes attention patterns with every other token in the sequence so a 100-token input requires 10,000 attention calculations per layer since every token compares itself to all 100 tokens. This quadratic scaling is why longer contexts get expensive fast. The model applies learned transformations to build up representations and output a probability distribution over the vocabulary for the next token.

Why “Vibe Coding” is Dangerous

Garbage In, Garbage Out

Models are mostly trained on public data, not high quality editorial. While there seems to be a few exceptions to this, this means that the token prediction to produce the final output, can in fact be incorrect garbage.

While Anthropic and OpenAI have both recently seemed to tackle this issue with specialized training on various domains such as programming and health to reduce hallucinations where it matters, this domain training is still generic, trained on all public programming texts, not your specific codebase, frameworks, or business logic.

Tangibly this means that most LLMs are a jack of all trades and a master of none. Ultimately, to properly use the LLM an expert needs to be behind the wheel, driving in domain context in every prompt.

Context Dilution or Contextually Necessary?

Remember our explanation of how LLMs assigns attention to our context? Well the distribution is put through a function called softmax. What this means is that the probabilities are normalized to all add up to one. Mathematically, as the context grows, attention elements become diluted and lose importance.

In practice this means context vs quality of output in most LLMs looks like a standard distribution. Too little and you get quite a few hallucinations, too much and you get gibberish.

Unfortunately, this means managing context is more of an art than a science and a constant and knowledgable eye is required to ensure that the LLM is consistently doing what’s asked of it properly.

Hallucinations and Ambiguity

LLMs are famous for hallucinating, and of course they are. If the final mechanism is a prediction, then sometimes we will get those predictions wrong. That’s the nature of statistics. LLMs have improved to the point where hallucinations mostly correlate with context quality — the more ambiguous your instructions, the more likely you’ll get hallucinations. But even with crystal-clear context, the LLM’s domain knowledge is only as good as the user’s knowledge while prompting.

While fine-tuning and RAG (retrieval augmented generation; is exactly as it sounds — database augmented context) are the only two methods currently of providing domain knowledge they still have issues. Fine-tuning offers no guarantee against hallucinating and RAG is only as good as the data, data chunking, and search methods used to retrieve the data and is somewhat unpredictable until you have solid usage analytics which is needed to fully account for what data will be needed to augment the LLM.

Even if these were valid methods for “vibe coding” the tooling right now for it is just in its infancy. MCP servers offer some augmentation for vibe coding but it doesn’t even come close to alleviate the need for an expert using the LLM. The current MCP-based RAG methods are more documentation search, or refined web-search than something that would help someone who isn’t a Senior/Staff level engineer to become a vibe coder.

The Skills Required

Now hiring

Senior Software Engineer

Full time, remote

We’re looking for a passionate engineer to join our team! We’re looking for someone who understands the following technologies with at least 6 years of experience in each:

Solid understanding of SOLID principles
Regularly applies Extreme Programming principles
HTTP/1, HTTP/2, QUIC (HTTP/3)
MongoDB, SQL aggregations and all
Blockchain technology, especially Solidity and Ethereum contracts
Proficient in C++, Rust, Go Ruby, JavaScript, Python
Understanding of C# with Unreal Engine 3
Big data engineering with Kafka and Apache Spark / Hadoop
Experience with both TDD andBDD
Leadership mentality; mentors other engineers
8 years experience with LLMs; negotiable
Able to plan and implement features from beginning to end
Expert in application security

Apply now

Have you ever seen a job posting where they wanted the kitchen sink with years of experience required that seem impossible? C# and Solidity? What an odd skillset. 8 years of LLMs? Impossible.

Unfortunately this is the reality with LLMs. If you’re building smart-contracts in Ethereum, you should know Solidity to effectively prompt and manage the context of your blockchain contracts. If you’re building a Unity based, photorealistic simulation engine for training Teslas and other CV-powered robots, you should probably know the basics, or at least know how to find out the basics of both Unity and whatever language and frameworks you’re using.

At the time of this writing, there is no AI enabled IDE, CLI programming tool, or AIaaS (like Lovable) that will give you a finished product that is readable and maintainable, secure, with a good user experience without constant monitoring of context with constant interruption and supplemental technical redirection when the LLM has gotten off track. Period. Full stop.

Putting it together

Plan

To get the most out of vibe coding, planning needs to be done. Functional requirements should be laid out for the LLM in a product definitions (or feature definition) and non-functional requirements need to be low-enough level to be considered more implementation details than non-functional requirements.

Product definitions should have everything from a domain context summary, to the folder structure used, the technologies used, your brand guide (and PSDs if you have them).

Feature definitions should dive deep into libraries, design patterns, algorithms and structure with instructions outlining the existing packages in which you want used.

While we don’t always have the answers, it’s okay to ask the LLM once you have build much of your logic to tell you the best way to do something, given the current implementation once you have the baseline established.

Analyze & Ask

Before each feature, or even each part of a feature, ask the LLM to analyze your product definition, the feature definition and the current implementation before starting. Ask the LLM for suggestions on implementation given these things if you’re unsure of the best way to implement the feature yourself.

Interrupt

Interrupt the LLM frequently. As context grows, “attention” can lose meaning. LLMs easily get off track because quite often when the LLM is searching your codebase, it doesn’t know exactly what to search and brings in a ton of irrelevant code. If you’re vibe coding, you’re speed-reading every line of code it’s writing and redirecting it. In my experience interruption is required quite often. Sometimes once every 5 minutes, other times once every 2 minutes.

Confirm

Confirm ALL of the output of the LLM. You’ll frequently find misnamed files, code that isn’t wired up, random // TODO, or // in a real implementation, that skips critical functionality, rewriting methods that exist in libraries, random reports and a host of other issues. You’ll need to periodically go through the feature you’ve been writing with the LLM and audit it.

Revise

Once you’ve confirmed what the LLM did and did not do correctly or at all, you’ll need to concisely and eloquently write out a task list of things to fix, again reminding it to look at the Product and Feature defintion. Look for common mistakes and update the product and feature definitions as needed.

Conclusion

LLMs are powerful tools, but that’s all they are. Tools. You wouldn’t hand a junior developer a production codebase and expect magic, and you shouldn’t expect it from AI either.

The promise that ‘anyone can code with AI’ is basically the same promise as ‘anyone can learn to code at a bootcamp in 12 weeks’, and we all see how that worked out for most people. Effective vibe coding actually requires more expertise, not less. You need to understand your domain well enough to plan properly, catch when the LLM goes off the rails, and fix the inevitable mistakes it makes.

PAICR isn’t some revolutionary framework. It’s just what happens when you actually try to build production-ready software with these things instead of something that will be hacked into within the first week of operation. You plan because you have to. You analyze because context matters. You interrupt because LLMs lose focus. You confirm because they make mistakes. You revise because nothing works perfectly the first time.

After 10 months of strategically using LLMs in my daily work building RAG pipelines and dealing with this stuff daily, here’s what I know: LLMs don’t replace engineering knowledge. They amplify it. If you don’t know what you’re doing, the LLM won’t either. If you do know what you’re doing, these tools can make you incredibly productive.

Vibe coding works, but only if you’re not actually vibing. You need to know what you’re doing or have the humility, drive and curiosity to figure out what you’re doing quickly.