I spent some years working at an email-specific agency. They sent more than 5 billion emails during that time for some very big clients, and we did most of the data analysis. They also did a lot of usability testing.
I can tell you that on just about every metric - brand recall, clickthrough rates, ROI, unsubscribes - HTML outperformed plain text. There were exceptions, both in terms of types of campaign and types of sender - say, if your list includes lots of Hacker News readers.
Ok, I am aware of the higher click rates, but that is part of my point. Well crafted imagery is no doubt more catchy and attention getting than plain text. However, is visual stimulation (manipulation?) supposed to be a part of communication medium? Between the reasons for why HTML should be in emails, where do you rank recipients well-being compared to ROI?
Unfortunately, an un-branded email looks suspicious and unprofessional nowadays. You'll get more complaints and unsubs than ever versus a well-crafted (not over-the-top) engaging layout.
Images and fonts are a very valid and useful tools for communication. Even the most old-school, respectable newspapers use images, graphics, and a variety of fonts--even in their most serious news stories.
That's fair enough, but you are very much in a minority.