The problem isn’t “vibe coding” (anyone who has ever managed early career staff will be able to attest that… the bar is REAL fucking low). The problem is a complete lack of testing or any sort of “investment” in caring if production breaks.
A lot of it is general apathy induced by… gestures around. But it very much goes beyond just the obnoxious rise in brain drains over “vibe coding”. Personally speaking, I am THIS fucking close to driving over to my company’s head of IT’s house and burning it down with him in it (For legal purposes, this is a joke) as that entire team continues to think “We’ll just wait until people tell us what is broken” is at all fucking acceptable.
But pretty much any SDLC is going to be built around code review. And code review is how you handle developers of different skill and sanity levels. Whether they are old hats who have been in the basement since before you were born, youngins who can’t stop talking about Rust, or chatbots.
Unfortunately a lot of people are trying to outsource code review to LLMs as well. Also, LLM generated code is more likely to have subtle errors that a human would be very unlikely to make in otherwise mundane code. Errors that are easy to gloss over if you don’t take a magnifying glass to it. My current least favorite thing is LLM generated unit tests that don’t actually test what they say they do.
Shit code review is not code review. If you just rubber stamp everything or outsource it to someone who will, you aren’t doing code review.
Aside from that:
LLM generated code is more likely to have subtle errors that a human would be very unlikely to make in otherwise mundane code.
Citation requested
My current least favorite thing is LLM generated unit tests that don’t actually test what they say they do.
If I had a nickle for every single time I had to explain to someone that their unit test doesn’t do anything or that they literally just copied the output and checked against it (and that they are dealing with floating points so that is actually really stupid)… I’d probably go buy some Five Guys for lunch.
Its like saying that the problem is that you are using robots to assemble cybertrucks rather than people. The problem isn’t who is super glueing sharp jagged metal together. The problem is that your product is fundamentally shite and should never have reached production in the first place. And you need to REALLY work through your design flows and so forth.
I keep seeing it over and over again. Anyone that actually has to deal with coworkers using this bullshit that isn’t also in the cult is going to recognize it.
If I had a nickle for every singl yada yada yada
Sure, there have always been better and worse developers. LLMs are making developers that used to be better, worse.
Bad developers just do whatever. It doesn’t matter if they wrote the code themselves or if a tool wrote it for them. They aren’t going to be more or less detail oriented whether it is an LLM, a doxygen plugin, or their own fingers that made the code.
Which is the problem when people make claims like that. It is nonsense and anyone who has ACTUALLY worked with early career staff can tell you… those kids aren’t writing much better code than chatgpt and there is a reason so many of them have embraced it.
But it also fundamentally changes the conversation. It stops being “We should heavily limit the use of generative AI in coding because it prevents people from developing the skills they need to evaluate code” and instead “We need generative AI to be better”.
It was the exact same thing with “AI can’t draw hands”. Everyone and their mother insisted on that. Most people never thought about why basically all cartoons are four fingered hands and so forth. So, when the “studio ghibli filter” was made? It took off like hotcakes because “Now AI can can do hands!” and there was no thought towards the actual implications of generative AI.
Nothing outside of the first paragraph here is terribly meaningful, and the first paragraph is just trying to talk past what I said before. I’ll reiterate, very clearly.
I have observed several of my coworkers that used to be really good at their jobs, get worse at their jobs (and make me spend more ensuring code quality) since they started using using LLM tools. That’s it. That’s all I care about. Maybe they’ll get better. Maybe they won’t. But right now I’d strongly prefer people not use them, because people using them has made my experience worse.
I have seen at least 1 out of every 5 comments from coderabbitai that lead me down a rabbit hole looking to see if the suggestion is correct. It can waste so much time trying to validate their suggestions only to find out it’s complete BS.
… Basically nobody in software development has done QA in about 15 years, aside of basically the last ol timer server admin types, who are now being replaced with … whatever Microsoft is calling Tay or Cortana now.
There’s always a few who try, and then corporate beats that out of them because it slows down ‘productivity’, you can basically only resist this for more than a year, two tops, if you are literally the only person on the planet who knows how to do what you do.
… which is also why those same old timers tend to … seemingly intentionally not document anything usefully: job security.
You will never have resources to “test absolutely everything”. It is ALWAYS about building out personas and deriving tests from those.
What this tells us is that one of two things happened:
This was not tested at all
The testing harness resets the environment after every check (e.g. “does process close when killed”) rather than involving a manual reset (i.e. “close and re-open task manager”)
The latter is a lot more common than you would think since it makes it much easier to automate these harnesses rather than having a human at a VM. But… this is what happens when you don’t step through the entire workflow.
They used to have very comprehensive automated testing processes to exercise all sorts of things. Unfortunately, like many tech companies these days like Apple, Google, etc., they’re all punting QA as a concept because they just don’t care - what are you going to do, go use another oligopoly platform?
Well, there may have been a period when MS was trying to improve product quality, and in that time, yes maybe they did have very comprehensive automated testing processes. But before that, up to the time of Windows 7 I guess, their quality was dog shit.
In the early days, MS was an undisputed monopoly though, and not only did they not test thoroughly, they hardly even tried to fix bugs - the userbase had to take care of that too. Earlier versions of Windows had all sorts of workarounds and 3rd party tools to try and get things to work properly.
I suspect that once they’d achieved their objective of improving quality, there just weren’t the incentives there any more for middle management to allocate resources to things like comprehensive tests.
For people to test, you need management that is willing to invest in QA. But that incentive disappears for a corporation when there’s no free market of competitors who can poach your customers by making a better quality product or service.
Vibe coding or no, this is a massive QA failure.
That is the reality.
The problem isn’t “vibe coding” (anyone who has ever managed early career staff will be able to attest that… the bar is REAL fucking low). The problem is a complete lack of testing or any sort of “investment” in caring if production breaks.
A lot of it is general apathy induced by… gestures around. But it very much goes beyond just the obnoxious rise in brain drains over “vibe coding”. Personally speaking, I am THIS fucking close to driving over to my company’s head of IT’s house and burning it down with him in it (For legal purposes, this is a joke) as that entire team continues to think “We’ll just wait until people tell us what is broken” is at all fucking acceptable.
But pretty much any SDLC is going to be built around code review. And code review is how you handle developers of different skill and sanity levels. Whether they are old hats who have been in the basement since before you were born, youngins who can’t stop talking about Rust, or chatbots.
Unfortunately a lot of people are trying to outsource code review to LLMs as well. Also, LLM generated code is more likely to have subtle errors that a human would be very unlikely to make in otherwise mundane code. Errors that are easy to gloss over if you don’t take a magnifying glass to it. My current least favorite thing is LLM generated unit tests that don’t actually test what they say they do.
Shit code review is not code review. If you just rubber stamp everything or outsource it to someone who will, you aren’t doing code review.
Aside from that:
Citation requested
If I had a nickle for every single time I had to explain to someone that their unit test doesn’t do anything or that they literally just copied the output and checked against it (and that they are dealing with floating points so that is actually really stupid)… I’d probably go buy some Five Guys for lunch.
Its like saying that the problem is that you are using robots to assemble cybertrucks rather than people. The problem isn’t who is super glueing sharp jagged metal together. The problem is that your product is fundamentally shite and should never have reached production in the first place. And you need to REALLY work through your design flows and so forth.
I keep seeing it over and over again. Anyone that actually has to deal with coworkers using this bullshit that isn’t also in the cult is going to recognize it.
Sure, there have always been better and worse developers. LLMs are making developers that used to be better, worse.
Bad developers just do whatever. It doesn’t matter if they wrote the code themselves or if a tool wrote it for them. They aren’t going to be more or less detail oriented whether it is an LLM, a doxygen plugin, or their own fingers that made the code.
Which is the problem when people make claims like that. It is nonsense and anyone who has ACTUALLY worked with early career staff can tell you… those kids aren’t writing much better code than chatgpt and there is a reason so many of them have embraced it.
But it also fundamentally changes the conversation. It stops being “We should heavily limit the use of generative AI in coding because it prevents people from developing the skills they need to evaluate code” and instead “We need generative AI to be better”.
It was the exact same thing with “AI can’t draw hands”. Everyone and their mother insisted on that. Most people never thought about why basically all cartoons are four fingered hands and so forth. So, when the “studio ghibli filter” was made? It took off like hotcakes because “Now AI can can do hands!” and there was no thought towards the actual implications of generative AI.
Nothing outside of the first paragraph here is terribly meaningful, and the first paragraph is just trying to talk past what I said before. I’ll reiterate, very clearly.
I have observed several of my coworkers that used to be really good at their jobs, get worse at their jobs (and make me spend more ensuring code quality) since they started using using LLM tools. That’s it. That’s all I care about. Maybe they’ll get better. Maybe they won’t. But right now I’d strongly prefer people not use them, because people using them has made my experience worse.
I know it’s not related, curious about this part.
I know it has an aluminum based frame which should inhibit it’s use to haul heavy loads, but what else?
I have seen at least 1 out of every 5 comments from coderabbitai that lead me down a rabbit hole looking to see if the suggestion is correct. It can waste so much time trying to validate their suggestions only to find out it’s complete BS.
And automatic test systems. It’s not that hard.
You think the QA was done by a person? I’ll bet this was Vibe QA’d.
Can’t fail QA if there is no QA.
… Basically nobody in software development has done QA in about 15 years, aside of basically the last ol timer server admin types, who are now being replaced with … whatever Microsoft is calling Tay or Cortana now.
There’s always a few who try, and then corporate beats that out of them because it slows down ‘productivity’, you can basically only resist this for more than a year, two tops, if you are literally the only person on the planet who knows how to do what you do.
… which is also why those same old timers tend to … seemingly intentionally not document anything usefully: job security.
The public, aka Microsoft QA Team, found the bug. It’s a QA success!
Well, who’d ever think of testing that closing the app works. Some things you have to take for granted or you’ll never test anything meaningful.
If you are a company the size of Microsoft, you have more than enough resources to test absolutely everything.
You will never have resources to “test absolutely everything”. It is ALWAYS about building out personas and deriving tests from those.
What this tells us is that one of two things happened:
The latter is a lot more common than you would think since it makes it much easier to automate these harnesses rather than having a human at a VM. But… this is what happens when you don’t step through the entire workflow.
Didn’t Microsoft fire their dedicated human testing team in about 2014?
Probably?
This is the kind of thing that a LOT of companies outsource. Mostly for ill.
They used to have very comprehensive automated testing processes to exercise all sorts of things. Unfortunately, like many tech companies these days like Apple, Google, etc., they’re all punting QA as a concept because they just don’t care - what are you going to do, go use another oligopoly platform?
Well, there may have been a period when MS was trying to improve product quality, and in that time, yes maybe they did have very comprehensive automated testing processes. But before that, up to the time of Windows 7 I guess, their quality was dog shit.
In the early days, MS was an undisputed monopoly though, and not only did they not test thoroughly, they hardly even tried to fix bugs - the userbase had to take care of that too. Earlier versions of Windows had all sorts of workarounds and 3rd party tools to try and get things to work properly.
I suspect that once they’d achieved their objective of improving quality, there just weren’t the incentives there any more for middle management to allocate resources to things like comprehensive tests.
Latest news is that Xbox Games Division had been required to aim for a 30% profit margin for the last several years.
Thats why everything sucks and blew up.
Thats a fucking insane baseline target.
Line for next quarter profit must go up, therefore, cut costs.
This is obvious self destructive in the long run, but that doesn’t matter, what matters is C Suite’s golden parachutes.
Isn’t that the final step in the testing plan for every app though? The first step is always opening it.
A bug like this means literally nobody tested it at all on this build, or was so apathetic they didn’t file an obvious issue.
For people to test, you need management that is willing to invest in QA. But that incentive disappears for a corporation when there’s no free market of competitors who can poach your customers by making a better quality product or service.
The person who fiddles with the always on top option in this monstrosity of a compositor maybe?