I believe that's not correct. First a nitpick, I think it's only software that can be open or closed source. Protocol specification might be kept secret, though.

Regarding specifically BitTorrent protocol specification[0], it's free to use. Anyone can build their open or closed source BitTorrent client without reverse engineering.

[0] http://www.bittorrent.org/beps/bep_0003.html