It wouldn't surprise me that you don't remember much if anything before that. Ignoring the fact that you were three and assuming your memory was recalling that far back (mine certainly doesn't but mine barely recalls five minutes ago) you were "given" to your grandparents suggesting a traumatic few years with your parents or a traumatic event, the memories of which are often suppressed particularly by young children.
You may find though that the dream is not your earliest memory, it can be difficult to put early memories into order and you can often find that you get things a bit jumbled, not that it's a particularly important point mind.
I doubt the dream contained Latin personally, or at least perfect Latin as the odds of you knowing that language at three are rather slim so if the dream came from your own mind then it'd likely just be a coalescence of words and sounds (possibly of Latin origin). If it came from the other possibility (some form of spirit) then why talk in a language you wouldn't understand, it strikes me as being a little pointless and so I find the idea of a spirit being involved a little unlikely personally.