During astral projection, you're pretty much in a sleep state. While you sleep your blood pressure drops and body functions slow down. The deference is that astral projection is like... Conscious sleeping. You're more aware of things, which is how you travel. So yes, it should be normal to feel cold. This is why some recommend wearing blankets or warmer clothing while projecting.
However, your brain does not disconnect from your body. That would kill you. Your consciousness moves from your physical body to your astral body (a body less solid than your physical, but more condensed than your soul). Your blood flow also has to continue. If that stopped, that would also kill you. some people believe that your astral self literally leaves your body. Others believe that astral projection is just lucid dreaming. Either way, your body enters almost a sleep state. This explains the cold feeling.
The cold might affect your projecting negatively. Maybe turn up the heart before projecting. It would take some time to heat up the room. By the time the room is heated, you will have already been projected out of body. That way by the time you return your physical body will be completely comfortable.