Though there may be spells which assist, and perhaps even possibly cause auto-suggestion deep enough to be used as self-hypnosis, more often than not breaking a bad habit is a process.
Magic can be used nonetheless! Remember: magic is change in accordance with will! Many traditions consider inner magic the ultimate goal: enlightenment, becoming a better person, positive change.
Mantras, meditations, visualixztions, talismans charged with intent, what have you! Yes, even spells to make progress. All of these things can aid your goal to stop biting your nails!