Can someone comment on data processing and statistics on GAE? Any good experiences? Any bad ones? I'm not an expert on GAE by any means, but I ported my app away from GAE to Django + MySQL because the app was data and statistics oriented (fantasy football) and I found myself wasting a lot of time on data processing, data cleaning, import, and export. Switching to a SQL database in this case felt like getting out of handcuffs (not that I know how that feels).
The GAE data store is really nice for persisting objects that come from users filling in HTML forms, or equivalent. One user can only type in data so fast and that kind of volume is fine on GAE. Plus you get great read scalability.
However, if you're creating data in some sort of simulation or loading it in from elsewhere, use a relational database (you can connect to external servers from your GAE app). Loading a few million rows into the GAE data store would cost you a fortune and the lack of joins or proper SQL probably means you have to get them all back out into memory again before you can do anything useful.
QueryTree uses the GAE data store as a central hub for user account info and settings, then shards lots of MySQL instances for actual data work, which seems to work well.
An SQL query can do just about anything, it'll do it right next to the data, and a lot of time and effort has gone into making that fast. For an app with data up to a certain size and usage, I'd totally go with SQL for all of the reasons you mentioned.
What worries me with SQL is that queries can do just about anything, and it'll do it right next to the data. That means that you want your database machine(s) to be hulking monsters, and sharding / replication gets complicated.
In my personal experience, the App Engine datastore exposes fewer and simpler operations which scale horizontally more or less perfectly. It's harder to write for initially, but it scales up incredibly smoothly.
