"People couldn’t believe that I was saying no for any other reason, that they had to assume that my marriage was falling ...