Who's "we"? The good way to do this would be to have someone who was knowledgeable, but had nothing to do with the rest of the hiring process, take the candidate to lunch. Even with the best intentions, it's hard to grill someone for an hour on the record, chat with them for another hour off the record, then not let the latter influence your judgement when your manager asks if you should hire them.
The "good way", as you put it, is exactly how engineering interviews work at Google. The person assigned to take the candidate to lunch is completely separate from the actual slate of interviewers, and does not submit written (or any other kind of) feedback into the hiring process.
In my team's case, it's generally the person who did the candidate's phone interview who takes them to lunch when they come on-site, but I don't know if that's a common thing.
