Well, from my experience, astral projection is an interpretation. Some people dream and think that they're astral projecting. Some people use their imagination to go someplace and they think they're astral projecting. I've had lucid dreams, and out-of-body experiences through meditation that weren't dreams, and lucid dreams where I meditated to go out of the body. That last one isn't necessarily astral projection.
But, whether that fear and coming back came from you or what some onieronauts call the "dream ego", it's worth writing down to consider and interpret...
Is your goal for astral realm travel stronger than the cause for your fear?