Evaluators form their opinions based on the guidelines, given by Google. These are the so-called Search Quality Raters guidelines.
So far they were available only to professionals, but a few days ago Google published the full text of the document. Google motivated their decision with the enormous number of mobile searches and the imposed algorithm changes.
Google’s Mimi Underwood explains: ‘(…) ratings from evaluators do not determine individual site rankings (…)’. She adds that the guidelines, used by the evaluators ‘(…) reflect what Google thinks search users want (…)’.
Underwood also hinted that Google will update the document permanently, as users’ demands and behavior are changing over time.