It would be nice if you could see the tag or list data from the Contact List page instead of having to export the data and manually search for those individuals. Then you could sort, select, and delete right from the list page.
You should be able to find these contacts using Advanced Search! Just choose List "is blank" and then remove the entire list of contacts that appear as the results.