Any Walking Dead fan will tell you, if there is one guy you don’t want to piss off, it is the dude known for carrying around the barbed wire-covered baseball bat. But that's exactly what the cast of Destination X did when they got on host Jeffrey Dean Morgan ’s bad side at one point during filming.

I said, 'I think you have the wrong person,'" Morgan tells PEOPLE exclusively. "I live on a farm, and I kill zombies, and I'm a dad, and I'm kind of a pain in the a--."Throughout his decades-long career as a successful actor,

The Walking Dead star had reservations about hosting NBC's new competition series, but he had a little help from someone close to him.