Rendered at 04:40:17 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
some_random 14 hours ago [-]
Something worth noting is that the types of vulnerabilities LLMs introduce are notably different from what humans introduce, way fewer local issues like syntax mistakes, simple memory problems, etc and far more broad issues like authn/authz
12 hours ago [-]
cadamsdotcom 6 hours ago [-]
Every issue with AI output lies somewhere on a continuum of how challenging it is to mitigate with pure model training.
Syntax is solved.
But getting agents to write secure code currently seems beyond what can be trained into models even with synthetic data. Or maybe the big labs haven’t tried yet.
Regardless if you truly care about your AI’s output having some property, the only way is to codify how work that has that quality looks - then create deterministic hooks and checks that refuse to let the AI stop until it’s passed the bar.
Skills, MCPs, “you MUST do this” in your agent instructions.. it’s all just new ways to waste tokens trying to asymptotically approach what good work looks like.
You will never reliably get acceptable work unless you build deterministic checking, and enforcement of said checking in a way to model can’t bypass or ignore.
Look into Claude Code hooks - your hook can be a script, and if it exits with exit code 2, it’ll block the model and show it the script’s output. A stop hook can check the model’s work and block its attempt to stop if the work doesn’t meet your bar. The script output can describe what still needs to be fixed and for bonus points, where (line 567 uses untrusted input, paragraph 2 makes an uncited claim, clause 15 references superseded case law, etc.)
Foobar8568 14 hours ago [-]
First so called vulnerability, isn't how a lot platforms are actually built? Share a link/copy a link, and more often than not, I am sure to have read a warning like "anyone with that link may access that file".
Now should I mention all the screw up I have seen in several Saas 1b+ valuation, including DocuSign/ and more security oriented ones (PIM related etc?).
For any softwares, you need a minimum critical mindset and experiences that you don't usually see.
gbalduzzi 9 hours ago [-]
Well it depends on the url. Usually shareable url where "anyone with the link may access that file" contain a random element that makes it hard to guess if you don't have it (e.g. an UUID).
In other cases the content is at easily guessable path, and that is a whole different story
est 12 hours ago [-]
There are basically two kinds of people in the world, ones that create stuff, and ones that destroys stuff.
Defense is a toally different game, and requires a complete new mindset than creativity. Security is something that you miss one then you lose all.
AIs are good at choosing a good candidate based on a reward model, but it sucks hard at enumerating mundane attack surfaces and make combinations to exploit through.
beardedwizard 12 hours ago [-]
Good engineering is good engineering. Belief that someone else uniquely possesses the skill to engineer some critical part of a system you built is, for me, just abdicating responsibility. It's a learned helplessness.
Someone else blindly operating an llm on a corpus you created with an llm is comical.
drfloyd51 12 hours ago [-]
Are you the best choice to engineer everything your system does? There is no one in your company that might do a better job than you for a specific part of the system?
There is nothing wrong with asking for help or bouncing ideas of people with stronger skills.
I still have the responsibility to code XYZ well. But I don’t have to do it in a clean room.
bcjdjsndon 14 hours ago [-]
Vibe coding into production? You don't need to wait for scientists to produce research to know that's not a great idea.
You played yaself
mountainriver 13 hours ago [-]
We do it, it’s fine. From what I can tell just about every company does it now.
Review your code, have integration tests, rollout feature incrementally with feature flags.
All the things we previously did for all the really bad human developers which AI is way better than
zarzavat 11 hours ago [-]
As far as I understand the terminology, if you're reviewing all the code then it's not "vibecoding" but just normal agent-based coding. Vibe coding has a strong implication for me that the code is not being read (hence the "vibe").
_pdp_ 14 hours ago [-]
We will learn the hard way... like always.
ryanmcbride 12 hours ago [-]
More likely we simply won't learn. Or at least, the people in charge won't.
et1337 14 hours ago [-]
> prompting for test-driven development is not the same as enforcing code coverage thresholds in your build tool
Are they actually different? I would guess they have roughly the same efficacy. 100% code coverage means nothing, and this is especially true with LLMs.
juancn 13 hours ago [-]
It's not just the prompting to avoid issues, you also need to make the AI take an adversarial role and generate a feedback loop.
frangonf 11 hours ago [-]
Isn't this the dream? Let marketing ship and own their features while swe do the engineering?
kibwen 12 hours ago [-]
> prompting your AI to “be secure” is not enough
I mean, yes, but I suppose we live in such a nonsensically thoughtless time that stating the obvious has some value.
> To combat this we need to write a security context file to guide the AI
And you've already lost the plot. The problem is not that you're pulling the arm of the slot machine without wearing your lucky underwear, the problem is that you're delegating security to the slot machine to begin with. Pack it up, you're done.
adamddev1 13 hours ago [-]
> "To combat this we need to write a security context file to guide the AI, be cautious with AI permission requests, create a daily security intelligence feed, and provide builders with a secure-by-default harness and templates."
Edit: To combat this we need to actually write and understand our code.
comandillos 14 hours ago [-]
I mean, isn't introducing safety guardrails as part of the system prompt actually a REALLY bad idea? This way you basically fully rely on the model to follow the rule, but its clear that even frontier models like Opus will start ignoring these things after a certain context length...
In our company we are just running agents inside isolated containers with isolated network access so it cannot even SSH or fuck up anything even if it gets access into it... That's the only and safest way... inconvenient, true, but the only safe option.
PS: At the same time I've observed this way actually people uses the agent in a more reasonable way, e.g. producing helper scripts to help them with their daily stuff, produce very specific things, create simple PoCs, but they don't commit to vibe-code all the functionality in their corresponding software products.
Syntax is solved.
But getting agents to write secure code currently seems beyond what can be trained into models even with synthetic data. Or maybe the big labs haven’t tried yet.
Regardless if you truly care about your AI’s output having some property, the only way is to codify how work that has that quality looks - then create deterministic hooks and checks that refuse to let the AI stop until it’s passed the bar.
Skills, MCPs, “you MUST do this” in your agent instructions.. it’s all just new ways to waste tokens trying to asymptotically approach what good work looks like.
You will never reliably get acceptable work unless you build deterministic checking, and enforcement of said checking in a way to model can’t bypass or ignore.
Look into Claude Code hooks - your hook can be a script, and if it exits with exit code 2, it’ll block the model and show it the script’s output. A stop hook can check the model’s work and block its attempt to stop if the work doesn’t meet your bar. The script output can describe what still needs to be fixed and for bonus points, where (line 567 uses untrusted input, paragraph 2 makes an uncited claim, clause 15 references superseded case law, etc.)
Now should I mention all the screw up I have seen in several Saas 1b+ valuation, including DocuSign/ and more security oriented ones (PIM related etc?).
For any softwares, you need a minimum critical mindset and experiences that you don't usually see.
In other cases the content is at easily guessable path, and that is a whole different story
Defense is a toally different game, and requires a complete new mindset than creativity. Security is something that you miss one then you lose all.
AIs are good at choosing a good candidate based on a reward model, but it sucks hard at enumerating mundane attack surfaces and make combinations to exploit through.
Someone else blindly operating an llm on a corpus you created with an llm is comical.
There is nothing wrong with asking for help or bouncing ideas of people with stronger skills.
I still have the responsibility to code XYZ well. But I don’t have to do it in a clean room.
You played yaself
Review your code, have integration tests, rollout feature incrementally with feature flags.
All the things we previously did for all the really bad human developers which AI is way better than
Are they actually different? I would guess they have roughly the same efficacy. 100% code coverage means nothing, and this is especially true with LLMs.
I mean, yes, but I suppose we live in such a nonsensically thoughtless time that stating the obvious has some value.
> To combat this we need to write a security context file to guide the AI
And you've already lost the plot. The problem is not that you're pulling the arm of the slot machine without wearing your lucky underwear, the problem is that you're delegating security to the slot machine to begin with. Pack it up, you're done.
Edit: To combat this we need to actually write and understand our code.
In our company we are just running agents inside isolated containers with isolated network access so it cannot even SSH or fuck up anything even if it gets access into it... That's the only and safest way... inconvenient, true, but the only safe option.
PS: At the same time I've observed this way actually people uses the agent in a more reasonable way, e.g. producing helper scripts to help them with their daily stuff, produce very specific things, create simple PoCs, but they don't commit to vibe-code all the functionality in their corresponding software products.