This is a natural phenomenon called the Pareto principle. Roughly 80% of the users will sign up for the top 20% of instances. It happens everywhere in nature and it’s unavoidable. But I don’t think it will be a problem, federation is designed to work like this. You’re not forced to stay on lemmy.world, you can move whenever you want.
The drivers were removed in 6.3. Debian 12 is still running on 6.1. Debian 12 just came out and still has many years of support ahead of it (at least 5). You can get plenty of use out of these cards before they stop working.