Who was the physically toughest U.S. President? There are many contenders for the title. In addition to his combat experience in the War of 1812, Andrew Jackson fought numerous duels. Ulysses S.