It's the written Chinese that is difficult. Since most of society is based around written text it's an important part of learning a language.
Having studied Japanese for three years full time I know the obstacles. I've yet to read a full novel. It's too painful. I just started reading one with the help of my Japanese partner who can fill me in on the reading and meaning when I try to read.
Technology certainly helps for Japanese.. when I hear a word I don't understand (e.g. from my wife), I enter it as it sounds and the dictionary gives me a nice (often short) list of translations, with hiragana, katakana, and kanji. And, due to my RtK studies (see earlier posts) I often get an 'aha' moment when I see the kanji.
If what you described works similarly for Chinese then yes, it's a real game changer.