Thank you for reaching out to the Community! I can understand how helpful it would be to simplify your invitations. At this time there is not a way to permanently remove this map. You can however delete it from the Landing Page and individual invitations of your event. This can be done in your Landing Page by clicking the trash can above the Event Location block.
While on the other hand, for you email this information can be selected and deleted by your keyboard in the Event Information Block. It is however important to mention any new email, this information will have to be deleted again.