The Prince and Princess of Wales "have worries" about what the Duke is planning next, following his recent return to the UK.