you would be better to worry if he stops looking.

No. I would worry if he started looking at men.

Have you ever flipped the coin on this one?
Do you know that women do the same thing to other guys?
It is only a natural thing that we look at other people. It's what you do with those thoughts/actions that count.
I am not saying its o.k. to purposely stare at someone, and wonder what if? but you it is normal for people to look at other people. As long as it is not done purposely then I thin you are going to be o.k.

The question arises though...Do you not trust him? You have known him for long enough. Perhaps there are deeper issues here than you think...

I agree with the guy above me. Woman do the same thing but I think girls are able to hide it when they look. Guys do it and everyone in the room knows we are looking at the girl that just walked past us. Even when people fall in love they can still think someone else is pretty or they are atractive. I think I would be more worried if the person you are with starts lying or hiding that they think someone else is pretty or attractive. If they hide an opinion then what else are they hiding. My girlfriend and I are able to tell each other if we think someone else is attractive and not worry if they are going run away with that person. I would relax if I was you and not bring up the issue.