Call Join free
Tech Lab Miami
Tech Lab MiamiTECH LAB MIAMI
← Back to Software & Tools

AI Wrote Half Your Codebase. Here’s What to Review Before It Ships

A working demo is not release evidence. Review authorization, secrets, dependencies, tests and rollback plans before shipping AI-generated software.
AI Wrote Half Your Codebase. Here’s What to Review Before It Ships

Hypothetical scenario: It's Thursday afternoon in Doral. The owner of a growing logistics company has a pull request waiting for approval. Her two-person dev team used an AI coding assistant to build a customer portal in three weeks instead of three months. The demo looked great. Customers can log in, see their shipments, and download invoices. The team wants to ship Monday. She doesn't read code, and she has to decide whether to say yes.

The title's "half your codebase" describes a situation like hers. It is not a statistic. Whether an AI assistant wrote one function or an entire feature, the decision that matters is what evidence you require before that code reaches customers. Syntax rarely sinks a release. Wrong assumptions do: who can see what, what happens when input is hostile, and how you back out when something breaks.

Why "it works in the demo" isn't release evidence

A demo proves the happy path. A logged-in user with good data clicks the expected buttons. AI-generated code is often very good at the happy path, because that is what the prompt described. The risk sits in everything nobody described: the second customer, the expired session, the malformed upload, the database change that can't be undone.

The tool vendors say the same. GitHub's responsible-use documentation for Copilot agents warns that generated code can look valid while being incorrect or failing to reflect what the developer intended, and that it may contain security vulnerabilities. GitHub recommends reviewing and testing agent output before merging and treating the agent as a tool, not a replacement for human review. Its cloud agent also has built-in limits: it pushes only to its own branch, and workflows on its pull requests need approval from someone with write access. Those limits only help if a person actually does the review.

For a broader frame, NIST's Secure Software Development Framework (SP 800-218) sets out high-level practices meant to reduce vulnerabilities in released software. NIST also notes it gives buyers and suppliers a shared vocabulary. That is useful for an owner: you can ask your team or vendor for evidence in recognized terms, not on instinct. SSDF is a set of recommendations, not a checklist that certifies your app. The questions below are our own working framework, informed by those sources.

Seven areas where you should ask for evidence, not reassurance

1. Authentication vs. authorization

Authentication answers "who are you?" Authorization answers "are you allowed to do this?" The OWASP Authorization Cheat Sheet keeps the two apart. It points out that a logged-in user is often not entitled to every action the system technically allows. It recommends denying access by default, checking permissions on every request, and enforcing checks server-side rather than relying on hidden buttons in the browser. Generated code often gets login right and skips the per-record check. Ask your team to show you where the "is this user allowed to see this record?" check lives.

2. Negative and tenant-isolation tests

Most generated tests confirm that things work. You also need tests proving that things fail when they should. OWASP specifically flags users reaching other users' data by changing an ID in a URL, a flaw class known as insecure direct object reference. If your app serves multiple customers, ask for a test where Customer A's account tries to open Customer B's invoice and is refused. OWASP also maintains a Multi-Tenant Security Cheat Sheet for teams building this kind of separation.

3. Secrets and dependencies

AI assistants will happily paste an API key into a config file or pull in a package nobody evaluated. Ask where credentials are stored and how they're kept out of the repository, and ask for the list of new third-party packages this release adds. OWASP's Secrets Management Cheat Sheet covers the first. Its Authorization Cheat Sheet recommends having a process for detecting and responding to vulnerable components.

4. Input and output trust

Every form field, file upload, webhook and API response is untrusted until validated. That includes output from an AI feature inside your product. Ask which inputs are validated, which outputs are encoded before display, and what happens when a field contains something unexpected.

5. Tests against requirements, not the same assumptions

This one is easy to miss. If the same AI session wrote both the code and its tests, the tests can repeat the code's misunderstanding and still pass. Ask for a short list of business requirements in plain language, written by a person, and a mapping from each one to a test. "Drivers can only see their own routes" should point to a specific test, not to a coverage percentage.

6. Migration and rollback

Database changes are where "we'll fix it Monday" stops being an option. Ask whether this release changes the data structure, whether the change can be reversed, and whether anyone has actually run the rollback on a representative, sanitized staging dataset.

7. Observation and maintainer handoff

After launch, someone has to notice problems and understand the code well enough to fix them. Ask what gets logged, especially failed access attempts. Ask who gets alerted and whether a developer who didn't write the code can explain how it works. If the honest answer is "only the AI session knew," you have an ownership problem as well as a security one. We covered the access side of that in Your Developer Disappeared: Do You Own Your Code, Domain and Hosting?

The owner-friendly request-for-evidence checklist

You don't need to read code to use this. Send it to your team or vendor before release and ask them to answer each row with something you can look at.

Ask for Good evidence looks like Red flag
Where authorization is enforced A named server-side check, with deny-by-default behavior "The button is hidden for non-admins"
Cross-customer access test A test where one account is refused another account's records Only happy-path tests exist
Secrets and new packages Credentials outside the repo; a short list of added dependencies, each reviewed Keys in config files; "the AI added what it needed"
Input and output handling Named validation for forms, uploads and webhooks "Users won't enter that"
Requirements-to-tests map Plain-language rules written by a person, each linked to a test Tests written in the same AI session as the code, with no human-written requirements
Rollback plan Rollback rehearsed on staging with realistic data "We'll restore from backup if needed," untested
Monitoring and handoff Failed-access logging, a named alert owner, a short maintainer README No one but the original session can explain the code

Run a staging release drill before the real one

This drill is our recommendation, not an official standard. It turns the checklist into a supervised exercise whose duration depends on the release.

  1. Deploy to staging exactly the way production will be deployed, using the same scripts and no manual shortcuts.
  2. Log in as two different customers and try to open each other's records by changing IDs in the address bar. You should see a refusal, not data.
  3. Log in as the lowest-privilege role and attempt one admin action directly.
  4. Submit bad input on purpose: an oversized file, an empty required field, odd characters.
  5. Run the database migration, then roll it back, and confirm the app still works on the previous version.
  6. Check the logs and confirm the failed attempts from steps 2–4 were recorded and someone would have been alerted.
  7. Ask a developer who didn't build the feature to explain one critical flow back to you.

Hypothetical example: Back in Doral, the owner runs the drill. Step 2 fails: changing an invoice number in the URL shows another customer's invoice. The generated code checked that the user was logged in but never checked who owned the invoice, and the generated tests only covered single-customer scenarios. The fix is small. Finding it after Monday's launch would have meant a disclosure conversation with a client. She ships the following Thursday instead.

Where we fit in

Tech Lab Miami is a South Florida community built by operators, focused on the practical decisions behind technology adoption. AI makes building faster, but the release decision still belongs to a person who has to live with the outcome. Our AI consulting work focuses on that responsible-adoption layer: assessing where AI-assisted development fits, designing review workflows, and helping teams build habits like the drill above. When the risk lives in how systems connect, such as webhooks, integrations and data handoffs, our business automation practice starts with a workflow audit and a measurement plan.

If you're deciding how much to build versus buy, our software strategy overview is a reasonable starting point. If your team wants to build the skills itself, look through what we cover on our learning hub. And if you have an AI-assisted release on your desk right now, tell us about the workflow behind it. We're glad to talk it through with fellow South Florida operators.

FAQs

Is AI-generated code less secure than human-written code?

Not by definition. The risk is that it looks finished before it is. GitHub's own documentation says agent-generated code can contain vulnerabilities and should be reviewed and tested before merging, which is the same standard you'd apply to any code your team didn't independently write.

Do automated security scans replace this checklist?

No. Scanners catch known patterns such as leaked secrets and vulnerable packages. They usually can't tell whether Customer A should see Customer B's invoice, because that depends on your business rules.

I'm not technical. Can I really run the staging drill?

Yes. Most steps are clicking, changing a URL and reading a result. Your developer handles the deployment and rollback. You watch and ask the questions.

Does following NIST or OWASP guidance mean we're compliant?

These are widely used recommendations and reference guides, not a certification. Whether specific regulations apply to your business is a separate question for qualified counsel.

Release review, illustrated

Illustrative AI-generated image of a software developer reviewing code on a laptop in a naturally lit office
Illustrative AI-generated image: reviewing evidence, not just syntax, before an AI-assisted release ships. Not a client or case photo.
Illustrative AI-generated image of a team discussing a software deployment plan in a bright collaborative workspace
Illustrative AI-generated image: a team walking through rollback and monitoring before go-live. Not a client or case photo.
← Back to Software & Tools
Chat with us