The PM said he wants to discuss the idea - but the Conservatives and Reform have slammed the 'Brexit betrayal'