The Royal Family will be "worried", claims an insider.