There are many possible options. To rid whatever 'residue' he leaves behind, a banishing may be in order. There are many to choose from, ranging from visualizations, to elaborate ritual.
A space cleansing may be another option. Basically, it is meant to purge all negative energy from a space. These can range from spiritual-based rituals, to non-spiritual chants, smudging with sage or incense (and following up with other smudge sticks of certain types, depending on belief and method). Sometimes a light-bringing ritual is used.
Some people may recommend a binding, to bind him away from your family (at least I have heard the term used as such; binding usually is for keeping things or people together, but some people will bind to separate), or even banish him. Cutting ties symbolically may also be a good option. There are some symbolic rituals using strings or ropes, whether by cutting or tying knots.
There are also plenty of mundane options. Telling him to leave your family alone may be the best and most direct option. If he persists, there may be recourse through legal charges.