A moderation bot that collated reports against rules and selected active users to vote on if the rule was broken or not seems like a useful thing to create to me.
Batching these into small groups and assembling a jury to decide if a particular rule was broken by each of the 1-7 reported posts should be a short task for the people involved. As long as the same people don't always end up being selected for the jury the results should be mostly good too.
I think this could be a pretty good system, but it would require a far more robust moderation framework than we have right now, as well as a lot of thought. I'm just brainstorming here, but I think at a minimum we would need:
save