Are unsubscribed contacts automatically removed from lists?

Hi,

I'd like to confirm something:

I sent out an email and when looking at the reports, saw a few "unsubscribe me"s. Do I need to manually remove these people from the contact lists or by clicking the button, did they remove themselves?

Thanks!