Guys/men, am I reading too much into his actions?

this guy at work when he's with me, he's very attentive. he always notice everything I do. he seems bothered whenever he thinks I'm pissed, or annoyed, or impatient. he jokes around a lot even with me, which made me feel comfortable with him. but people have been noticing something that I refuse to notice. this guy's attentiveness towards me seemed to have triggered people's suspicion that something's going on between us. this guy always makes sure I'm comfortable, like, he brushes my hair away from my face with his fingers whenever my hair strays there. I really just want to think he's just being a gentleman, but my co-workers are giving me something to think about. this guy is not even saying anything to me, like he's interested in me. I just want to continue feeling comfortable with him, but with people glancing at us as if something is going on between us makes me feel uncomfortable. I just can't dismiss the thought that he might be interested in me because of what my co-workers are saying. if you're in the same situation, what are you going to think?