Let's take a look at the definition of "Jealousy;"
1 - jealous resentment against a rival, a person enjoying success or advantage, etc., or against another's success or advantage itself.
2 - mental uneasiness from suspicion or fear of rivalry, unfaithfulness, etc., as in love or aims.
3 - vigilance in maintaining or guarding something.
.... So do you feel this way when he spends time with other guys? If not, the issue clearly isn't that you resent that his attention is elsewhere. With this in mind, if you only mind him being with girls, not guys, your probably do not REALLY trust him.
It's entirely likely that your conscious, rational mind knows that he won't do anything wrong. The unconscious mind, the part more directly controlled by emotions and driven by instinct, is probably much less trusting.