Similar instances have happened to me before, as many people don't realize that:
Witches/Wiccans/Pagans actually exist
They are different from how Hollywood portrays them
It's good that he was generally accepting of you (from how you explained it) but I would advise you to share more about it with him. It will prevent him from making absurd conclusions such as you flying on a broom.
You could try giving him a pamphlet on your beliefs, some of which may be found here for example: