We have given your feedback to our developers regarding this feature. It is worth noting that if we change the way the Contacts collection works, there will no longer be an easy way to remove a list from the contacts choices. The current way to change list selection provides a simple way to add and remove list selection simply by adding or removing the ContactList node corresponding to the list that you want to change.
It is unlikely this implementation will change, however we have noted your recommendation and our Product Management group will look at this seriously.
Dave Berard Senior Product Manager, Constant Contact