Port plan to go ahead after row over minister's comments