In my experience, most of those are bots or people targeting you (or the people around you) because you post good content. Lemmy has gotten big enough to attract those kinds of people, that's the issue. My instance found over 100 bots that downvote.

This is from 2016 and I've seen these tactics used a lot on Lemmy.

“Once we isolate key people, we look for people we know are in their upstream – people that they read posts from, but who themselves are less influential. We then either start flame wars with bots to derail the conversations that are influencing influential people, or else send off specific tasks for sockpuppets (changing this wording of an idea here; cause an ideological split there; etc).”

The goal is to keep opinions we don't want fragmented and from coalescing in to a single voice for long enough that the memes we do want can,...

https://archive.is/PoUMo

Maybe take it as a compliment? What I have noticed that's positive, is that most people don't interact with the trolls, agents or bots. They leave a comment that counteracts them or just downvotes and moves on. This is the best version of a somewhat popular internet site I've seen.