Yes, I have politely ended a couple of interviews and the interviewer is always shocked, because I think most people think the interviewer is in the power position.
They may be a gatekeeper for the job, but you are the gatekeeper for your time. Hiring a person is a mutual decision. Since job descriptions are usually quite vague on some important details about the team and company, applying for the job is simply expressing interest in the job, not a confirmation that you would take it if offered.
My experiences are similar. And I always end nicely regardless of what transpired. I like to find and maintain high ground in these kinds of interactions.
You just never know who you may find on the other side of the table!
Overall, people seeking work have less overall power and a generally poor position, unless they have saved or earned enough to not require the work.