They can't file a DMCA counterclaim because they never received a DMCA claim in the first place. Youtube simply blocks the content, you don't even get a proper claim with the contact info.
"They would then be required by law to reinstate the content. The step after that would be that the other party now has to go to court, to take down your content." is the followup process for a DMCA claim - but that's not what's happening here. Youtube is not requiring claimants to submit a DMCA claim (with all the associated process) and they are not submitting DMCA claims. Youtube could make their copyright claim process following these rules if they wanted to be on the side of users, but they have not. They can also choose to have a process where any complaint simply causes content to get removed with no recourse, and this is pretty much what they have done.
"They would then be required by law to reinstate the content. The step after that would be that the other party now has to go to court, to take down your content." is the followup process for a DMCA claim - but that's not what's happening here. Youtube is not requiring claimants to submit a DMCA claim (with all the associated process) and they are not submitting DMCA claims. Youtube could make their copyright claim process following these rules if they wanted to be on the side of users, but they have not. They can also choose to have a process where any complaint simply causes content to get removed with no recourse, and this is pretty much what they have done.