As Eissy said, if you have yourself grounded well enough and clear from any negative thought, you'll be fine.
It's the astral, where the thoughtforms rule :) Believing you'll be hurt can really bring this to you in the astral. Though by having a strong connection with your corporal body, within a split second you can be back.
Anxiety, shock, fear and etc are like a magnet to pull you back to your body again. However, you would also like to have your chakras cleaned in order to have a safe body to return rather than attracting negativity. Cleanse the area where you want to perform Astral Projection and don't be scared. Fear will only ruin your moments there instead of enjoying the pleasurable experiences.