Created segment does not find contacts that did not open a specific email

A segment created to allow email to only contacts that have not opened or clicked an RSVP button does not find those contacts, though review of reporting shows 100+ contacts have not yet opened the specific email. I need to send a new invite email to those contacts that have not opened the previous email. I don't want to send a new email to contacts that have already RSVP'D...