But if you do that, you're testing your client rather than your API. That's bound to matter: either your client is abstracting behavior and using only a specific subset of how your API could be used, or your client is not really useful.
For example, if your client implements fallback handling and caching behavior "the right way" for your API, and you only test using your client, then you're not testing your API when it is used "the wrong way" by developers who bypass your client. Those are important test cases too, unless you treat the wire protocol as an internal implementation detail and use your client's API as "the API". But if you do that, don't call it RESTful. (Technically it might still be RESTful, but that's a loaded term now and it will confuse developers who expect raw HTTP.)
For example, if your client implements fallback handling and caching behavior "the right way" for your API, and you only test using your client, then you're not testing your API when it is used "the wrong way" by developers who bypass your client. Those are important test cases too, unless you treat the wire protocol as an internal implementation detail and use your client's API as "the API". But if you do that, don't call it RESTful. (Technically it might still be RESTful, but that's a loaded term now and it will confuse developers who expect raw HTTP.)