The most likely cause of that would be that you do not have documented permission to email some of those contacts. If you have confirmed opt-in turned on, then contacts who have not opted in yet can show in the list, however they are not counted as 'active' contacts, and your emails will not go to those contacts. Furthermore if they signed up through an event, they may have not checked the 'send me emails' checkbox. This would leave them in your account, but they would not be active and could not be sent to. You can see both the "Awaiting Confirmation" and the "No Permissions Set" categories on the left side of your contacts page, slightly below the "Active" contacts list.
... View more