I got an Apple Watch for this reason when they first came out and found out that people constantly thought I was bored or had to leave because I kept glancing at my watch. Looking at the phone never got a negative response.