I feel if they can't accept you for who you are, as the person that your wife married and you only talk to them a few times a year don't worry about what they feel about it.
If you want do to something and can legally do it and are constantly around people that shun and point fingers at you, it might be time to find different people to associate with. You might like some of the people but if they can't accept you then I don't see that they see you as a very good friend/acquaintance.