What will happen if I setup a new account in WHM with a quota of 100MB then I setup 11 email accounts with each having mailbox quotas of 10MB? Please explain how Hosting quotas relates to mailbox quotas.

At what point will I receive an error of exceeding limits, on creation of the eleventh email account or when the total of mailbox filesize plus filesize all web files reach 100MB? Or something completely different?

You will see the error when you will try to create 11th mailbox.
You can't overuse quota. Also mailboxes are located at your disk space. All space used by them is used space at your disk quota.