This is a really great idea, especially when done right. The difficulty with machine learning and AI is understanding the pitfalls inherent in selecting data and training systems. You can fool yourself pretty easily into thinking you've got something that works when you really don't. That said it sounds like they're doing things well, I have no doubt this will have a positive impact in demystifying the "magic" of ML/AI and making all those Google products I use better!
"And then (this is hard for coders) trusting the systems to do the work."
Like you say, it can be easy to think that something works when it really doesn't. I hope that the above quote isn't meant to be interpreted as "believe the results are correct." Evaluation is paramount when working on these systems to avoid making such mistakes. I assume Google is including evaluation in their machine learning training, but it would have been nice to see that pointed out in the article for folks who may have an interest in machine learning but don't know what's important to focus on.
One big problem with ML is that it's highly based on your training set. There's been a few papers published in computational linguistics that discuss how poorly ML based sentiment analysis is if you try and apply the data to domains outside the training set. For instance, if you train the sentiment data on movie reviews (which is actually a data set commonly used for that purpose) and try and apply it to Twitter or the Web, the results are terrible. But, people keep on trying it.
