You're right that there's no middle ground if you set it up that way, but the trick is to look for ways of framing an issue that do allow for middle ground. For example, we could ask which specific political consequences ought to follow from the fact that Black lives matter, and in which order. There's lots of room for middle ground there. It's a question of whether we want to find it or not.
Certainly, there's middle ground (concessions) in several issues like tax rates, zoning, visa policy.
Rights-based positions, on the other hand, are no-compromise by their premise. They're binary issues. "Women/BlackPeople/Felons/OtherGroup should be allowed to vote." "Women/BlackPeople/Felons/OtherGroup should not be allowed to vote." Sometimes you have what looks like a concession in these issues but it's really just one side masquerading like a concession. Example: voter disenfranchisement disguised as implementing voter literacy tests or "you must pay your court debt". Straight up abortion denial disguised as good faith "limitations" (limit after x weeks, insurance requirements, clinic closings, heartbeat, sleep on the issue, fake clinics, etc.).
The policy that follows isn't the thing that most people find appalling. It's the premise. You're absolutely right that we should be focused on policy, but only if people are on board with the premise. Check out the top ranked articles shared on Facebook and you'll agree.
If you take your comment and apply it to moments that sparked huge civil rights changes, I don't think minority groups would have gotten this far. It's the "no compromises" attitude that leads to great social change. Not half measures.
Edit: On further thought, it's arguable that segregation was the "middle ground" between slavery and equal rights. The middle ground just doesn't sound too appealing when it comes to equal rights. Minorities are right to instinctively distrust invitations towards "compromise".
