True! I’m an AI researcher and using an AI agent to check the work of another agent does improve accuracy! I could see things becoming more and more like this, with teams of agents creating, reviewing, and approving. If you use GitHub copilot agent mode though, it involves constant user interaction before anything is actually run. And I imagine (and can testify as someone that has installed different ML algorithms/tools on government hardware) that the operators/decision makers want to check the work, or understand the “thought process” before committing to an action.
Will this be true forever as people become more used to AI as a tool? Probably not.
Ok, I give up, where’s loss?
The loss is the jobs we lost along the way.
Booring. Find a new joke.
They can’t possibly train for every possible scenario.
AI: “Pregnant, 94% confidence”
Patient: “I confess, I shoved an umbrella up my asshole. Don’t send me to a gynecologist please!”I want to see Dr House make a rude comment to the chatbot that replaced all of his medical staff
Imagine an episode of House, but everyone except House is an AI. And he’s getting more and more frustrated by them spewing nonsense after nonsense, while they get more and more appeasing.
“You idiot AI, it is not lupus! It is never lupus!”
“I am very sorry, you are right. The condition referred to Lupus does obviously not exist, and I am sorry that I wasted your time with this incorrect suggestion. Further analysis of the patient’s condition leads me to suspect it is lupus.”
My knowledge on this is several years old, but back then, there were some types of medical imaging where AI consistently outperformed all humans at diagnosis. They used existing data to give both humans and AI the same images and asked them to make a diagnosis, already knowing the correct answer. Sometimes, even when humans reviewed the image after knowing the answer, they couldn’t figure out why the AI was right. It would be hard to imagine that AI has gotten worse in the following years.
When it comes to my health, I simply want the best outcomes possible, so whatever method gets the best outcomes, I want to use that method. If humans are better than AI, then I want humans. If AI is better, then I want AI. I think this sentiment will not be uncommon, but I’m not going to sacrifice my health so that somebody else can keep their job. There’s a lot of other things that I would sacrifice, but not my health.
My favourite story about it was that one time when neural network trained on x-rays to recognise tumors I think, was performing amazingly at study, better than any human could.
Later it turned out that the network trained on real life x-rays with confirmed cases, and it was looking for penmarks. Penmarks mean the photo was studied by several doctors, which mean it’s more likely to be the case that needed second opinion, which more often than not means there is a tumour. Which obviously means that if the case wasn’t studied by humans before, the machine performed worse than random chance.
That’s the problem with neural networks, it’s incredibly hard to figure out what exactly is happening under the hood, and you can never be sure about anything.
And I’m not even talking about LLM, those are completely different level of bullshitNeural networks work very similarly to human brains, so when somebody points out a problem with a NN, I immediately think about whether a human would do the same thing. A human could also easily fake expertise by looking at pen marks, for example.
And human brains themselves are also usually inscrutable. People generally come to conclusions without much conscious effort first. We call it “intuition”, but it’s really the brain subconsciously looking at the evidence and coming to a conclusion. Because it’s subconscious, even the person who made the conclusion often can’t truly explain themselves, and if they’re forced to explain, they’ll suddenly use their conscious mind with different criteria, but they’ll basically always come to the same conclusion as their intuition due to confirmation bias.
But the point is that all of your listed complaints about neural networks are not exclusively problems of neural networks. They are also problems of human brains. And not just rare problems, but common problems.
Only a human who is very deliberate and conscious about their work doesn’t fall into that category, but that limits the parts of your brain that you can use. And it also takes a lot longer and a lot of very deliberate training to be able to do that. Intuition is a very important part of our minds, and can be especially useful for very high level performance.
Modern neural networks have their training data manipulated and scrubbed to avoid issues like you brought up. It can be done by hand, for additional assurance, but it is also automatically done by the training software. If your training data is an image, the same image will be used repeatedly. For example, it will be used in its original format. It can be rotated and used. Cropped and used. Manipulated using standard algorithms and used. Or combinations of those things.
Pen marks wouldn’t even be an issue today, because images generally start off digital, and those raw digital images can be used. Just like any other medical tool, it wouldn’t be used unless it could be trusted. It will be trained and validated like any NN, and then random radiologists aren’t just relying on it right after that. It is first used by expert radiologists simulating actual diagnosis who understand the system enough to report problems. There is no technological or practical reason to think that humans will always have better outcomes than even today’s AI technology.
That’s why too high a level of accuracy in ML is always something that makes me squint… I don’t trust it, as an AI researcher and engineer, you have to do the due diligence in understanding your data well before you start training.
When it comes to ai I want it to assist. Like I prefer the robotic surgery where the surgeon controls the robot but I would likely skip a fully automated one.
I think that’s the same point the comic is making, which is why it’s called “The four eyes principle,” meaning two different people look at it.
I understand the sentiment, but I will maintain that I would choose anything that has the better health outcome.
Except we didn’t call all of that AI then, and it’s silly to call it AI now. In chess, they’re called “chess engines”. They are highly specialized tools for analyzing chess positions. In medical imaging, that’s called computer vision, which is a specific, well-studied field of computer science.
The problem with using the same meaningless term for everything is the precise issue you’re describing: associating specialized computer programs for solving specific tasks with the misapplication of the generative capabilities of LLMs to areas in which it has no business being applied.
We absolutely did call it “AI” then. The same applies to chess engines when they were being researched.
more like “chess computer” and “computer analysis”
No-one thought of them as intelligences
Machine Learning is the general field, and I think if we weren’t wrapped up in the AI hype we could be training models to do important things like diagnosing disease and not writing shitty code or creating fantasy art work.
We are. Why do you think we stopped?
chess engines are, and always have been called, AI. computer vision is and always has been AI.
the only reason you might think they’re not is because in the most recent AI winter in which those technologies experienced a boom they avoided terminology like “AI” when requesting funding and advertising their work because people like you who had recently decided that they’re the arbiters of what is and isn’t intelligence.
turing once said if we were to gather the meaning of intelligence from a gallup poll it would be patently absurd, and i agree.
but sure, computer vision and chess engines, the two most prominent use cases for AI and ML technologies - aren’t actual artificial intelligence, because you said so. why? idk. i guess because we can do those things well and the moment we understand something well as a society people start getting offended if you call it intelligence rather than computation. can’t break the “i’m a special and unique snowflake” spell for people, god forbid…
There’s a whole history of people, both inside and outside the field, shifting the definition of AI to exclude any problem that had been the focus of AI research as soon as it’s solved.
Bertram Raphael said “AI is a collective name for problems which we do not yet know how to solve properly by computer.”
Pamela McCorduck wrote “it’s part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, but that’s not thinking” (Page 204 in Machines Who Think).
In Gödel, Escher, Bach: An Eternal Golden Braid, Douglas Hofstadter named “AI is whatever hasn’t been done yet” Tesler’s Theorem (crediting Larry Tesler).
https://praxtime.com/2016/06/09/agi-means-talking-computers/ reiterates the “AI is anything we don’t yet understand” point, but also touches on one reason why LLMs are still considered AI - because in fiction, talking computers were AI.
The author also quotes Jeff Hawkins’ book On Intelligence:
Now we can see the entire picture. Nature first created animals such as reptiles with sophisticated senses and sophisticated but relatively rigid behaviors. It then discovered that by adding a memory system and feeding the sensory stream into it, the animal could remember past experiences. When the animal found itself in the same or a similar situation, the memory would be recalled, leading to a prediction of what was likely to happen next. Thus, intelligence and understanding started as a memory system that fed predictions into the sensory stream. These predictions are the essence of understanding. To know something means that you can make predictions about it. …
The human cortex is particularly large and therefore has a massive memory capacity. It is constantly predicting what you will see, hear, and feel, mostly in ways you are unconscious of. These predictions are our thoughts, and, when combined with sensory input, they are our perceptions. I call this view of the brain the memory-prediction framework of intelligence.
If Searle’s Chinese Room contained a similar memory system that could make predictions about what Chinese characters would appear next and what would happen next in the story, we could say with confidence that the room understood Chinese and understood the story. We can now see where Alan Turing went wrong. Prediction, not behavior, is the proof of intelligence.
Another reason why LLMs are still considered AI, in my opinion, is that we still don’t understand how they work - and by that, I of course mean that LLMs have emergent capabilities that we don’t understand, not that we don’t understand how the technology itself works.
To expand on this a bit AI in medicine is getting super good at cancer screening in specific use cases.
People now heavily associate it with LLMs hallucinating and speaking out of their ass but forget about how AI completely destroys people at chess. AI is already getting better than top physics models at weather predicting, hurricane paths, protein folding and a lot of other use cases.
AI’s uses in specific well defined problems with a specific outcome can potentially become way more accurate than any human can. It’s not so much about removing humans but handing humans tools to make medicine both more effective and efficient at the same time.
The problem is the use of ai in everything as a generic term. Algorithms have been around for awhile and im pretty sure the ai cancer detections are machine learning that are not at all related to LLMs.
Yeah absolutely, I’m specifically talking about AI as a neural network/reinforcement learning/machine learning and whatnot. Top of the line weather algorithms are now less accurate than neural networks.
LLMs as doctors are pretty garbage since they’re predicting words instead of classifying a photo into yes/no or detecting which part of the sleep cycle a sleeping patient is in.
Fun fact, the closer you get the actual math the less magical the words become. Marketing says “AI”, programming says “machine learning” or “neural network”, mathematicians say “reinforcement learning”.
I guess I worked with a guy working with algorithms and neural networks so I sorta just equated them. I was very obviously not a CS major.
One of the large issues was while they had very good rates of correct diagnosis, they also had higher false positive rates. A false cancer diagnosis can seriously hurt people for example
Iirc the issue was that the researchers left the manufacturer’s logo on the scans.
All of the negative scans were done by the researchers on the same equipment while the positive scans were pulled from various sources. So the AI only learned to identify which scans had the logo.
iirc the reason it isn’t used still is because even with it being trained by highly skilled professionals, it had some pretty bad biases with race and gender, and was only as accurate as it was with white, male patients.
Plus the publicly released results were fairly cherry picked for their quality.
Yeah, there were also several stories where the AI just detected that all the pictures of the illness had e.g. a ruler in them, whereas the control pictures did not. It’s easy to produce impressive results when your methodology sucks. And unfortunately, those results will get reported on before peer reviews are in and before others have attempted to reproduce the results.
That reminds me, pretty sure at least one of these ai medical tests it was reading metadata that included the diagnosis on the input image.
Medical sciences in general have terrible gender and racial biases. My basic understanding is that it has got better in the past 10 years or so, but past scientific literature is littered with inaccuracies that we are still going along with. I’m thinking drugs specifically, but I suspect it generalizes.
That’s because the medical one (particularly good ar spotti g cancerous cell clusters) was a pattern and image recognition ai not a plagiarism machine spewing out fresh word salad.
LLMs are not AI
They are AI, but to be fair, it’s an extraordinarily broad field. Even the venerable A* Pathfinding algorithm technically counts as AI.
When I was in college, expert systems were considered AI. Expert systems can be 100% programmed by a human. As long as they’re making decisions that appear intelligent, they’re AI.
One example of an expert system “AI” is called “game AI.” If a bot in a game appears to be acting similar to a real human, that’s considered AI. Or at least it was when I went to college.
The important thing to know here is that those AI were trained by very experienced radiologists who are physicians that specialize in reading imaging. The AI’s wouldn’t have this capability if the humans didn’t train them.
Also, the imaging that AI performs well with is fairly specific, and there are many kinds of imaging techniques and diagnostic applications that the AI is still very bad at.
Yeah this is one of the few tasks that AI is really good at. It’s not perfect and it should always have a human doctor to double check the findings, but diagnostics is something AI can greatly assist with.
It’s called progress because the cost in frame 4 is just a tenth what it was in frame 1.
Of course prices will still increase, but think of the PROFITS!Also, there’ll be no one to blame for mistakes! Failures are just software errors and can be shrugged off! Increase profits and pay less for insurance! What’s not to like?
deleted by creator
I hate AI slop as much as the next guy but aren’t medical diagnoses and detecting abnormalities in scans/x-rays something that generative models are actually good at?
They don’t use the generative models for this. The AI’s that do this kind of work are trained on carefully curated data and have a very narrow scope that they are good at.
Yeah, those models are referred to as “discriminative AI”. Basically, if you heard about “AI” from around 2018 until 2022, that’s what was meant.
The discriminative AI’s are just really complex algorithms, and to my understanding, are not complete black-boxes. As someone who has a lot of medical problems I receive care for as well as being someone who will be a physician in about 10 months, I refuse to trust any black-box programming with my health or anyone else’s.
Right now, the only legitimate use generative AI has in medicine is as a note-taker to ease the burden of documentation on providers. Their work is easily checked and corrected, and if your note-taking robot develops weird biases, you can delete it and start over. I don’t trust non-human things to actually make decisions.
That brings up a significant problem - there are widely different things that are called AI. My company’s customers are using AI for biochem and pharm research, protein folding, and other science stuff.
I do have a tech background in addition to being a medical student and it really drives me bonkers that we’re calling these overgrown algorithms “AI”. The generative AI models I suppose are a little closer to earning the definition as they are black-box programs that develop themselves to a certain extent, but all of the reputable “AI” programs used in science and medicine are very carefully curated algorithms with specific rules and parameters that they follow.
My company cut funding for traditional projects and has prioritized funding for AI projects. So now anything that involves any form of automation is “AI”.
Image categorisation AI, or convolutional neural networks, have been in use since well before LLMs and other generative AI. Some medical imaging machines use this technology to highlight features such as specific organs in a scan. CNNs could likely be trained to be extremely proficient and reading X-rays, CT, MRI scans, but these are generally the less operator dependant types of scan, though they can get complicated. An ultrasound for example is highly dependent on the skill of the operator and in certain circumstances things can be made to look worse or better than they are.
I don’t know why the technology hasn’t become more widespread in the domain. Probably because radiologists are paid really well and have a vested interest in preventing it… they’re not going to want to tag the images for their replacement. It’s probably also because medical data is hard to get permission for, to ethically train such a model you would need to ask every patient in for every type of scan it their images can be used for medical research which is just another form/hurdle to jump over for everyone.
It’s certainly not as bad as the problems generative AI tend to have, but it’s still difficult to avoid strange and/or subtle biases.
Very promising technology, but likely to be good at diagnosing problems in Californian students and very hit-and-miss with demographics which don’t tend to sign up for studies in silicon valley
Basically AI is generally a decent answer to the needle in a haystack problem. Sure, a human with infinite time and attention can find the needle and perhaps more accurately than an AI could, but practically speaking if there’s just 10 needles in a haystack it’s considered a lost cause to find any of them.
With AI it might find in that same stack 30 needles, of which only 7 of them are the needles, which means the AI finds more wrong answers than right, but ultimately you do end up finding 7 needles when you would have missed all 10 before, coming out ahead.
So long as you don’t let an AI rule out review of a scan that a human really would have reviewed, it seems a win to potentially have more overall scans get a decent review and maybe catch things earlier in otherwise impractical preventative scans
Despite what the luddites would have you believe, AI is an amazing assistive tool when paired with a human reviewing the results.
The “luddite” reaction is largely a reaction to the overhype applied by the industry that pretends the current wave of text/image generators is general intelligence and in conjunction with robotics can replace every job and allow the upper class folks to live a full life without that pesky labor class.
So it’s naturally to expect a wave of such hype pretending it’s unambiguously amazing and perfect to get hit with a counter that’s overly dismissive and treats AI as a very bad brand. Also, in some contexts even if it is a net win, it’s still kind of annoying. In my haystack example, a human would have reviewed 23 things confidently declared by the AI to be needles and said no to them. Practically speaking, that’s unimaginably better than reviewing millions of not-needles to get to some needles, but we are more annoyed because in our mind the things presented were supposed to be needles. Same applies to a lot of generative AI use, it might provide a decent chunk of content that’s nearly usable 20% of the time so quick as to be worth it, but it’s hard to ignore the 80% of suggestions that it throws at you that are unusably bad. Depends on your job and your niche as to what the percentage will be. From a creative perspective, it generates milquetoast stuff, which may suffice for backgrounds and stuff that doesn’t matter, but is a waste of time when attempted as the key creative elements.
Broadly society has to navigate the nuanced middle ground, where it can be pretty good assistive technology but not go all out on it. Except of course there are areas likely to be significantly fully automated, like customer support or food order taking (though I prefer kiosks/apps for more precise ordering through tapping my way through, but either way not a human).
The nuanced middle ground is what I said, treating it as an assistive tool with human review.
That’s fine, just saying so long as there are people pumping ridiculous amounts of money into the fiction that it can do anything and everything I won’t fault folks for having the counter reaction of being overly dismissive/repulsed by mentions of it.
I’m hopeful for the day when the hype subsides and it settles into the appropriate level of usefulness and expectations, complete with perhaps less ludicrous overspend on the infrastructure.
Simp all you want.
You’ll also be shivering in the streets before long.
I always was going to under a capitalist system.
At first i thought this was an open house where the visitors slowly became relplaced by AI, honestly i thought this was speaking upon the fact that AI would be able to replace even the housing industry, imagine the amount of land that would be bought up if it were given the resources to generate wealth off of unused land, imagine this scenario but replace the “crime” with anything else.
This IS our future if we let it be.
imagine the amount of land that would be bought up if it were given the resources to generate wealth off of unused land,
It’s funny, if I take it the parts that reference AI, I just see a description of today
yeah but increase that outcome by a multiple of 10, and speed it up the process it takes to do so just as much.
They skipped the phase where all the doctors were replaced by NPs and PAs.
My current “provider” is an NP. I like her, she’s personable and does the basic stuff well enough. I can understand having her do the basic annual physical type stuff for relatively young and healthy people.
But, for one of my recent visits, they scheduled me with a doctor instead (dunno why), and the experience was honestly almost night and day for the better. Granted, the way my health insurance works (ugh USA), the NP visits only ever cost me a flat amount, perhaps $45 for the copay. The doctor’s visit cost me the $45 copay, plus additional coinsurance down the line that I got billed a couple of months later because the clinic apparently charges two different rates depending on whether you see a doctor or not, I guess?
