I assume it's someone ELSE who's trying to destroy your marriage. Having a piece of their clothing is an excellent tie to work through, the more recently they've worn it, the better. If it has sentimental value to them, all the better, and the time period doesn't matter.
What exactly do you want to accomplish with the spell you're looking to cast? That's the first step.
Do you want them to come to harm, or to leave your life?
Need a little more to work with, here.