I've been living in China and picking up Mandarin as I go. I can see how it would be a nightmare in a classroom, but through immersion I've been finding it pretty easy.

Ignoring tones, the spoken language itself is simple compared to English. As for the tones themselves that's a hurdle that takes a couple of weeks to develop an ear for by osmosis but it's natural after that (and I'm assuming that in a classroom you'd be deliberately tackling them and trying to force your brain through tone-shaped holes would be the main reason it'd suck).

When it comes to reading, naturally it's a given that understanding shit you need in your daily life would make retention pretty straightforward... but then again my reading ability is currently limited to things relating to the subway system and restauranting as a result. So while not difficult I'm going to wind up with huge gaps everywhere in my written Chinese. Planning on enrolling in proper classes once I'm more fluent to fix the errors I'm bound to pick up doing what I'm doing.

tl;dr: Agree that the difficulty of Chinese is overrated, but I can see how people get that impression.