The comedian, 41, was banished from the roundtable during Thursday’s semi-final after he came clean and admitted he was a traitor to his fellow players