Check permission on paginated overview per entity
Ah, same old "pagination with conditions" problem, now for the new age :)
(I don't think that even google have solved it, btw. Sometimes you see more pages in their pagination output, then you actually get).
Now, of course, the problem in your case is that Voter will not help you in any way, because it's not for your issue - it's used to check access on the object level, but you need to perform it on the sql/dql/query level. And the biggest difference is that, instead of returning abstain, denied or granted, your method will need to return dql conditions. So the task is separate.
What you can do, though, is to add function that will return those conditions into your voter class, and inject it into your repository. At least this way - your access logic for the same object will be in the same class.