It's not just users to users.
Third party app users are likely some of the most engaged users Reddit has, so if or as those users find a new home, the overall content quality on Reddit declines and lurkers shift their attention elsewhere.
And keep in mind, it's not Reddit v. Lemmy. It's Reddit v. Netflix, Instagram, TikTok, etc because Reddit is competing for attention.
I think that's a portion but I know a few people that just take the path of least resistance. Right now, that is absolutely not the fediverse. In a few months with all the apps already in development, it might be a better experience with better content.