Not everybody has the privilege of owning another house specifically to get away from it all, so it is no surprise that those who do can find themselves becoming more popular with their friends and family. They want the benefit of a free vacation and are more than happy to cozy up to somebody if they think that they will give it to them. What with the price of traveling these days, it's hard to blame them.
As the homeowner, it is only fair that you get to be a little selective about who gets to use these facilities and when. It is unlikely that you would want to hand over a set of keys to just anybody. However, unless they really can't be trusted or you can't get along with them, it isn't too much of a stretch for close relatives to expect that they might get some occasional use out of the place.
Naturally, it is important to have some ground rules for anybody who visits your property, and it is always going to be the owner who has the final say. It's not fair for a potential guest to try to override that.