Automated email series not counting as email sent when creating a segment

I'm trying to do some housekeeping on our contact list by creating a segment of contacts who were not sent any emails within 1) the past 180 days and 2) the past year. However, the resulting segment shows me a bunch of contacts who have received emails from an automated series within that timeframe. How can I generate a segment of contacts who truly weren't sent any emails at all in that timeframe? Thanks!