They’re concerned about “their” people, because it’s declining only in rich countries and those tend to see themselves as “better” and don"t like “unregulated immigration” (while the regulated one costs shit tons of money). Also those who bring thst up are usually right-wingers.

Or to say it bluntly: Xenophobia and racism.