If they should continue to inquire as to why you choose not to share their worship (which would be impolite of them, incidentally), you need simply be honest and tell them that you have your own beliefs which you hold sacred. Since they (presumably) are in the same situation, they should understand and let it go at that.

In my experience, Mormons (missionaries from the Church of Jesus Christ of Latter Day Saints, I think?) tend to be the most reasonable when told that you already have a spiritual belief which works for you. Mind you, they'll still be more than happy to discuss matters of faith with you until you're dead and buried, but they'll typically accept a polite "no, thank you" as the end of that conversation topic. J/W's, BAC's - now they, tend to get pushy, requiring firmer measures.