Main issue is that in a large enough group of people, there are bound to be a few Nazis that slip in and start causing shit.

Going nuclear (defederating) on the whole group before they are aware or can react to this revelation is completely unreasonable. Even more so when the instance owners, moderators and the community are aware and actively down-voting and removing comments and communities.