Workarounds for those sorts of limitations have been developed, though. Chain-of-thought prompting has been around for a while now, and I recall recently seeing an article about a model that had that built right into it; it had been trained to use <thought></thought> tags to enclose invisible chunks of its output that would be hidden from the end user but would be used by the AI to work its way through a problem. So if you asked it whether cats had feathers it might respond “<thought>Feathers only grow on birds and dinosaurs. Cats are mammals.</thought> No, cats don’t have feathers.” And you’d only see the latter bit. It was a pretty neat approach to improving LLM reasoning.
Workarounds for those sorts of limitations have been developed, though. Chain-of-thought prompting has been around for a while now, and I recall recently seeing an article about a model that had that built right into it; it had been trained to use <thought></thought> tags to enclose invisible chunks of its output that would be hidden from the end user but would be used by the AI to work its way through a problem. So if you asked it whether cats had feathers it might respond “<thought>Feathers only grow on birds and dinosaurs. Cats are mammals.</thought> No, cats don’t have feathers.” And you’d only see the latter bit. It was a pretty neat approach to improving LLM reasoning.