Can my landlord do this?

Two years ago my husband and I began renting the 3 bedroom house and 4 acres of land we are currently living on. When we moved in the in-ground pool in the backyard was dry, covered and unused. The lanlord informed us that if we wanted to use the pool we could feel free to have it filled and use it, but that we would have to pay for the water, chemicals and maintenence. We decided to have it filled last year. My problem is that about 3 months after we had it filled the landlord and his family began showing up unannounced to use the pool. They wouldn't even knock on the door or anything they'd just go right in the back yard and hop in. I didn't want to make an issue of it because we'd had a wonderful relationship with LL and it's not as if they were walking inside the house or anything. However, now they have had groups of people sometimes as many as 9-11 people in the backyard just hanging out using the pool. The last straw came when my LL, his wife, their 2 kids and kid's friends showed up during my son's birthday party. We were having the party in the backyard and they just showed up to use the pool. I very politely told the LL that we were having a bithday party and that it wasn't really a good time for them to come over to use the pool. LL responded by telling me that it's HIS pool and he and his family will use it whenever they feel like it. I was shocked. The kids even helped themselves to my son's birthday cake and the LL sat there smoking and swearing and embarring my poor son in front of his friends. I am not sure if there is anything I can do about this since they are not actually entering the house. He has always given 2 days notice as stated in the lease when he enters the house but I don't know if this also covers the backyard and luxeries I am paying for. I am thinking about having the pool the drained just so I don't have to deal with them popping up all the time, but I really don't want to. Any advice would be appreciated.