Keeping those people away is a choice that needs to be made by your husband and you. Or put up a ward to nil the energy.
As far as lust, most men think of sex all the time. You just can't fight hormones. But the actions taken are the most important. Play some games, make it interesting for him. This will keep him to you, hopefully. (Faithfulness spell wouldn't hurt.) BB