Only Muslims from eastern countries(India, Pak, Malaysia(duh), etc. consider it offensive. From purely legal POV, there is no precedence from authentic Islamic history on people being punished for eating in front of fasting Muslims. On the contrary, there are a number of precedences where fasting Muslims were offered food, and they would refuse saying "I am fasting" and the other people(Muslims and non Muslims) would simply continue eating.

From religious POV, a fasting Muslim actually gets reward when others are eating in front of them.