You're physical body is more vulnerable while you astral project, which is why I always recommend that people have a spirit guide with them (when they are starting out) and protect their physical body with a circle or amulet.
You are not leaving your body entirely when you project, if your spirit did so, your body would die, and it doesn't. What travels is your consciousness and with it a bit of your energy, or what new agers like to call your "astral body" which is pretty much nothing more than your mind forming your energy into your likeness. I have projected plenty of times without needing a form, although its nice to have one if you wish to interact.
When you travel, you're not aware of your physical body, so just as when you are sleeping/dreaming, you are more vulnerable. This doesn't mean there is a hollow space that a spirit can jump into, but there is a possibility that a low spirit could try to latch on to you. I like to call these spiritual pests or vermin. They generally do nothing more than leech your energy, and simple cleansing and banishments are generally enough to get rid of them.
The other less likely possibility is that you may have mediumship capabilities and a spirit has attempted to use your moment of distraction to attach and ride you. There are a few things about such things on the board, if you feel you are a natural medium. But, I've found it very rare for men to be mediums, and far more common for women. My theory is that it is due to our natural born advantage in communication which is supported by science.
So banish! And make sure to protect yourself next time ;)