Prime Minister Mark Carney argued the vote had been held “at the worst possible time”.