It seems to me that by speaking up when you’re still calm, you can avoid erupting at an inopportune moment and causing the huge scene that might worry you.
I used to feel terrified about how people judged me–for good reason, based on how people treated me when I was young. Eventually, I grew utterly exhausted from trying to please everyone, after which it became much easier to speak up for myself.
Now I find it easy to offer a cheery “No, thanks” while acting like it’s perfectly normal and leaving the other person to be confused and to deal with it.
I wish you peace as you work towards finding your voice.
“I’m not looking for a family here. I never was. Thank you for the invitation, but no.”
It could be that simple.