Fee charged without service

yep tell them to go jump, they can either accept your lease or you'll be moving out.

being charged a brokers fee for this is ridiculous.

You're paying for the connection to the owner. That's the service, not finding you an apartment. If the original tenant had left they would have made the fee renting to someone else, correct? I suggest you negotiate since you've made the broker's job easier.
At the same time, I wouldn't push it too far, they could always rent it to someone else.

You're paying for the owners connection to you. That's the service, not finding the owner a tenant. If you leave, the broker will have to do work, the owner will have to find someone and potentially take a discount because a new tenant may factor in a fee, and the owner will have to take the gamble of someone less reliable without a proven history, possible vacancy time, cleaning and possible painting. I suggest you negotiate since the only beneficiary of the broker fee is the broker unless there is some larger relationship in which case I suggest you negotiate anyway.