Magic follows intention, which can be clearer when it's aligned with your emotions (Although not always, because with some people emotions can actually get in the way of casting a spell, if--this is just my belief--perhaps because their emotions are generated by an illusionary bias for which the person is just a focus but they do not have a true, real, clear idea of the targeted other person.)
Perhaps your hatred is stronger than your neutrality or ecstasy/joy in life.
Or perhaps your hatred has seeped into some subconscious confirmation bias so if anything bad happens to somebody you don't think, you automatically think, "Oh, I did that!"
It can be for all sorts of reasons. If it's for the second reason, though, it would be a good idea to try to find what about life can make you really happy, so you can at least have some balance.