Rules for broker access?

This situation ought to be guided by your lease. If your lease is silent on the question, I'm not sure what happens or what recourse your LL has if you lock yourself in the apartment and don't permit access at night or at other times. Then again, that surely would sour your relationship with the LL and that might not be good for you in the longrun. Better to work out an agreeable compromise with the parties involved.

Rather than get too caught up in this, focus on getting the move done and putting this all behind you. Keep your eye on the ball.

The reply you got is backwards. The landlord has already soured his relationship with you by permitting this behavior. Be clear that allowable times are as you stated, and with 24 hour notice. And that if you can accommodate a non-standard time, you'll try to do so but you won't offer a guarantee of that and would need at least the same notice. Do not be bullied, otherwise you are setting yourself up for a problem at the move out and with security deposit, in which case I would advise sensing this as far in advance as you can and not paying the last month's rent and clearing everything remaining after the move-out.