I think “pure” intelligence by itself is an algorithm. And as such an algorithm has no desire to do anything. Just like a web server has no desire to output web pages - it just does because it’s what the logic happens to make. It’s possible we need some sort of a quantum computer to achieve brain-like performance of intelligence - but I think intelligence is an algorithm specifically because LLMs exist.

Being able to pick an option is not necessarily a hallmark of cognition either - again a web server does that all the time. So I have a desire to think because it will end up getting me food and shelter and all the things I want. If we give a computer - a capability to think - it would have no desire to do so unless we executed the thinking algorithm.

Based on my research the intelligence by itself has no wants. Our wants come from being biological and evolving basic instincts and feelings like pain is bad, hunger, want to reproduce. All our other wants come from intelligence applied and predicting outcomes that maximize the biological wants. Without biological wants there’s not much to build off of.

I like this path of reasoning. Let’s assume whoever invents the AGI wants “greater good”. However malicious humans exist so how does that interact with AGI technology. Let’s say the governments started implementing some sort of universal safety net, but I am malicious and want to be have better status. What can I do?

I can start stockpiling still-scarce resources. With AGI that’s land and natural resources. I can made my own AGI that aligns with my own values and start a smearing campaign that degrades values I don’t like… but if everyone has AGI access the latter is probably not very effective. I can set up traps and bully others to show how much control I have to get people to fear me… I suppose AGI can amplify this but hopefully it amplifies the “good guys” as well so it cancels out at a societal scales but local cliques will still get hurt.

I guess land and natural resource hoarding become most viable forms of greed… I wonder if this wood make feudal societies a thing again. Did modern governments happen because intellectual labor was more profitable and so it was more profitable to share land and natural resources in order to have intellectuals work together?

Your statement implies AGI is good only for bad things (things I view as bad). That’s not the case as I see AGI being good for just doing things I don’t like doing. (AGI I’m asking about are not LLMs that we have now)

This is along the same lines I imagined it as well. And yes modern LLMs are nothing like what I’ve described. We’d need kilowatts of power per every single LLM conversation if we were to gradient descend them per conversation…. Funny enough human brain runs at like 10 watts. So I imagine there’s some huge paradigm shift from what LLMs do to what human brain does. You don’t get 1000 time efficiency boost from just putting all the world’s ram into it.

Hmm I see your points and I generally agree with them, except I don’t think we will ever collectively agree to ban AGI research before we invented it. There’s just too much juicy power for the governments to agree to ban it. As a result, it will happen eventually - maybe not in our lifetime but it will happen.

Oh I agree that “if these guys make agi” then we’re screwed. But I also don’t think Altmans and Amodeos of the world will invent AGI. However in that case the road isn’t smooth either. I dream of Star Trek like society but to get there is still going to be hard because it’s a disruption.

What about the benefits like having super cheap doctors available at your home at your schedule? You doing think the benefit is worth the change?

You just… invented dependency management.

It doesn’t run doom, it plays doom. Very different. We don’t understand the neurons well enough to translate doom to neural networks, but we can train them to play doom, due some definition of playing.

No in the matrix, smart machines captured humans and used their body heat to work.

Here dumb humans captured useless human tissue in hopes to run dumb machines.

Its an interesting perspective, except… that’s not how AI works (even if it’s advertised that way). Even the latest approach for ChatGPT is not perfect memory. It’s a glorified search functionality. When you type a prompt the system can choose to search your older chats for related information and pull it into context… what makes that information related is the big question here - it uses an embedding model to index and compare your chats. You can imagine it as a fuzzy paragraph search - not exact paragraphs, but paragraphs that roughly talk about the same topic…

it’s not a guarantee that if you mention not liking sushi in one chat - talking about restaurant of choice will pull in the sushi chat. And even if it does pull that in, the model may choose to ignore that. And even if it doesn’t ignore that - You can choose to ignore that. Of course the article talks about healing so I imagine instead of sushi we’re talking about some trauma…. Ok so you can choose not to reveal details of your trauma to AI(that’s an overall good idea right now anyway). Or you can choose to delete the chat - it won’t index deleted chats.

At the same time - there are just about as many benefits of the model remembering something you didn’t. You can imagine a scenario where you mentioned your friend being mean to you and later they are manipulating you again. Maybe having the model remind you of the last bad encounter is good here? Just remember - AI is a machine and you control both its inputs and what you’re to do with its outputs.

Apparently Nintendo switch 2 is using the standard already, so it might go over better than Sony.

Global interpreter lock

Good point! I wonder if we’re spoiled by computer invention though. Would be interesting to compare preWW2 invention rates and now. I suspect computers just made everything else easier, but now we’re back to hard problems

To be fair, there’s only been 24 year’s of 21 century. Most things you gave listed happened at the end of the 20th century. But also the question is somewhat self negating - we won’t know what’s the greatest invention until we see it working great, but it takes much more than 24 years to take an invention from concept to consumption. For example computational biology is kicking off. Computer aided dna generation started in the past 24 years. But it’s so new few people think about it. Just like no one thought of internet as the greatest invention in the 70s… it was just too new

Gentoo. It makes me feel like I’m in full control of my system.

My friend, let me tell you a story during my studies when I had to help someone find a bug in their 1383-line long main() in C… on the other hand I think Ill spare you from the gruesome details, but it took me 30 hours.

The Test part of TDD isn’t meant to encompass your whole need before developing the application. It’s function-by function based. It also forces you to not have giant functions. Let’s say you’re making a compiler. First you need to parse text. Idk what language structure we are doing yet but first we need to tokenize our steam. You write a test that inputs hello world into your tokenizer then expects two tokens back. You start implementing your tokenizer. Repeat for parser. Then you realize you need to tokenize numbers too. So you go back and make a token test for numbers.

So you don’t need to make all the tests ahead of time. You just expand at the smallest test possible.

What are you talking about? Modern phones are all just slightly different to be incompatible with what you’re talking about.