Give Me Some Space!

Why can't I find 5 minutes of 'me' time? Every other Sunday night I get about 20 minutes between the time the kids get to bed and the time my husband gets home. That's about the sum total of my 'me' time. So guess what night he gets off early. Just about every time. Tonight I tucked the kids in and had just put my butt in the chair when I hear his key in the door. I know I should be thrilled to see him, but can't I be thrilled 20 minutes from now instead?
The only other 'me' time I have is Wednesday nights when the kids are at church. Since September there have been three Wednesday nights when they've both been gone and my husband hasn't had the night off. Three times. Just over an hour each time.
Maybe this is sounding horribly selfish, but I just need a little bit of space.