Is my marriage in trouble?

I would very very bluntly tell him, "Your family are a bunch of damn idiots. You've lost your damn mind if you think I'm going to take the blame for that! I'm warning you, you better think twice before you open your mouth because right now, you act just like those idiots. And you best consider yourself damn lucky that I haven't sent you back to your family."