It may or may not. A Ouija board itself is just a tool to communicate with spirits around you. People often make the mistake of thinking the spirit is either in the board (rofl) or that the board summons one. It is simply a way of communicating with them more effectively. If you feel there is a presence in your home, you could try communicating with it again.
If the idea of it being there upsets you, try a cleansing. This will get rid of many negative energies/entities. If the spirit is harmless it will most likely stay until it wishes to leave.
You can cleanse by doing a simple sage smudging. Burn a sage smudge stick (or loose sage of any variety) and walk from room to room. Since you feel it in your closet, I would start there. Fan the smoke so that it reaches every corner of your home. If you worship any deities, you can ask for their blessings. For an added touch of protection I add cedar, pine or willow to my smudging at the end. You can also sprinkle sea salt or pine needles in the windowsills and doors to ward off negativity. Hope that helps :)