Downing Street has repeatedly said the government will not rejoin the single market or customs union.