The 80 year-old US Commander and Chief annoyed many Unionists with his comments