I appreciate the frustraton this is causing you. I reached out to my upper level of support in order to better understand the issue and what I learned is when it comes to the reply-to address our system has very little control on where the end recipient will redirect their auto-reply. This is controlled at the server (i.e. Exchange server) or client (Gmail, Yahoo, etc.) level. What this means is while we give the option to choose a Reply-To Address, if a particular server or email client is set up to only reply back to the "From Address," we have to defer to the restrictions of that client.
A workaround is to leave the From Name as your boss but change the From Address to another address (like an email@example.com) or to the address you have as the Reply-To.
I'm not sure if this is something Constant Contact will ever be able to mitigate but I will certainly let our product teams know this is a concern from our customer base.
Thank you for your reply.
Unfortunate that this option isn't fully availale but understand that its out of Constant Contact hands.
I believe it would be advisable to let users know for sure.