You are talking about the brain. Remember that if you have one scene and five persons who witness it, none of the accounts will be exactly the same. Just because your elders don't remember this incident you recall at the age of four, does not mean that it didn't happen. It means that the event wasn't meaningful enough for them to register into their long term memory.
I have always been taught that magic and the craft are about perspective. What is a more important question might be, why would this scene be meaningful for you and what importance and lessons does this memory have now? Just because your perspective of an event is different does not mean that it is incorrect or even inaccurate.