I had an opportunity to sit down with Jennifer Slegg yesterday, and chat with her about her findings in the leaked Google Quality Evaluator’s Guide that she was able to obtain, and there are a number of very interesting tidbits. Jennifer just posted a great and extensive writeup on what she’s discovered (see: “Latest Google Search Quality Rater’s Guide: Mobile Rewrite“), and she just gave a great presentation titled “Competing With Google in a Featured Snippet World” here at the State of Search Conference in Dallas, Texas about her discoveries.
It may be worthwhile to give just a super-quick background: for many years now, Google has employed “Quality Evaluators” — individuals they hire from various communities all over the world to go through and rate their search engine results. The “Evaluators” or “Raters” or “Reviewers” are typically short-term employees that are asked to conduct searches and to provide ratings based on a number of quality factors. Google has incorporated these grades into the search algorithm, and the baking-in of this enduser feedback has affected the rankings of webpages.
In 2005, Henk van Ess leaked the first quality evaluation guide that the search marketing community became aware-of — an internal document that Google used to instruct these employees on how to assess the search results. Since that first guide was leaked, there have been subsequent editions that have been leaked at various times and in various places with interesting updates on details that Google apparently focuses upon. So, the copy that Jennifer Slegg has found is yet the newest disclosure of some of these internal criteria that Google is using to assess and rate webpages.
So, fast forward to the present. The newly obtained quality rating guide has apparently evolved, and some of the areas of particular interest involve focus upon Featured Snippets and Mobile (Eric Enge is also speaking at the State of Search Conference, and it’s worthwhile to also read his article on How to Get Featured Snippets).
In the current version, Google continues to have raters judge a page’s E-A-T (“expertise, authoritativeness and trustworthiness”) or the lack of it. That hasn’t changed. Apparently, for queries where freshness and recency is more important, the raters are asked to give lower ratings to stale stuff. Google’s document recognizes that some webpages reflect the current date or reflect a more recent date, even when the content is really old. If uncertain of whether the content is recent, the raters are actually told to check the page via the Internet Wayback Machine!
Perhaps the biggest thing to change in the new version of the guidelines are around Mobile. We all know that Google has been amping-up their focus around user-experience for mobile users, and that they have told us that “mobile-friendliness is a ranking factor”. The new rater’s guidelines that Jennifer has exposed make this abundantly clear and provide further insights into how Google may be assessing the various elements that can contribute to “mobile-friendliness”. In Jennifer’s blog post, she listed the following things that can negatively impact a webpage’s rating due to how it works on mobile devices:
- Submission forms are difficult to enter data into;
- Layout is bad in small screen sizes;
- Website is difficult to use on mobile — navigation is hard to use, Flash content, and elements that don’t resize for mobile or can’t be viewed on mobile;
- Usability is impacted by internet connectivity in the mobile version – stuff involving switching networks, app openings and swapping, voice commands, and webpage load times;
I’m very glad to’ve had the chance to sit down with Jennifer, because she provided some greater context and emphasis in some areas than what I saw in her blog post or heard in her presentation.
What’s very telling about the newly leaked guide is that all quality evaluators are required to view the pages they’re rating through mobile devices — only! They are tasked with considering webpages as though viewed on an Android phone, even if they’re testing sites on a desktop. (I imagine this means they’re asked to change the user-agent to mimic an Android’s and to resize a Chrome browser window to much smaller to mimic the phone interface.)
As Jen notes in her post, the differences between how a mobile-version site appears from iPhone to Android is probably minimal, but this directive could introduce a slight bias towards sites that function better for Android.
She told me that the evaluators are also testing app installs — so, when a site invites the user to perform the install, does it go smoothly?
Fascinatingly, webpages that are not mobile friendly are getting the very lowest rating. These are being rated as “Fails to Meet”, putting them in a category along with spammy content!
As an aside, this makes me wonder whether these particular ratings are intended to only impact mobile search results — if so, then this might not affect the desktop search results. However, her findings around the Featured Snippets evaluation guidelines might suggest that these ratings values could be affecting both the mobile and desktop search simultaneously. It would seem very odd to a user to submit a search, receive back a Featured Snippet, and then find a different response when they submitted the same search on a different device.
Featured snippets guidelines
In Google Webmaster Help sessions, John Mueller has indicated that some webmasters have complained that their site isn’t providing Featured Snippets in search results when perhaps it should. However, those webmasters just don’t realize that they have indeed been getting Featured Snippets treatment — so, they’re unaware of how to check. Jennifer provided some how-to steps at the end of her presentation on how to go about doing this.
I was slightly amused when Jennifer told me that the quality guide focuses a lot of attention on evaluating Featured Snippets. “Shouldn’t this simply be a completely automated thing by now?”, I naively asked. “I mean, after all, Google performs heavy usability testing prior to launching such features, right?” Jen said: “How often have we seen the funky ones, right? That’s why they’re testing this. They would test unless they were curating these for some reason.” I’m sure she’s right, and it makes sense. They likely need to continue to evolve the snippet invocation criteria, selection criteria, and presentation interface.
Google attempts to categorize search queries as either “Know Queries” or “Know Simple Queries”. Know Queries are things that will not return a Featured Snippet because they are complex, too broad, nonspecific or have vague answers. Here are things that should be categorized as Know Queries:
- Info queries that are broad, complex and or in-depth and do not have a short answer;
- Queries on controversial topics where there are different answers;
- Where there’s no right answer;
Some examples of queries for which you will not see Featured Snippets: “barack obama”, “new york city” and “what nickel is used for”.
Qualifications for Know Simple Queries:
- They seek a very specific answer, like a a fact, diagram, etc;
- This answer has to be correct and complete and can be displaced in a small amount of space: the size of a mobile screen;
- If most people would agree on the answer;
- If the answer could be expressed in a 1 or 2 sentence answer;
How to tell if featured snippets have been shown with your site’s content:
- Login to your Google Search Console account;
- Go to Search Analytics
- Go under the Clicks & Impressions section; Select “Position”, also.
- Search for phrases that include: How, Why, When, or What; These are implied Simple Know queries.
- If you have a high position on the first page of search results for any of those How/Why/When/What queries, then those are keyword phrases for which your site’s content might have shown up in a Featured Snippet.
- Click on the link symbol adjacent to the query and Google will pop a window open with the search results for that query and you can see what shows up for that Implied Simple Know query.
Jennifer went on to touch on some ways that one might be able to optimize in order to not only have content appear under a Featured Snippet, but also to increase clickthroughs to the website from Google search — essentially, how to compete with Google’s Featured Snippet.
She pointed out that in instances where the source of the Featured Snippet doesn’t also have an image or video associated with it, they will deliver up an image from the very top of the Image Search results. So, there’s an opportunity to insert one’s image into a competitor’s Featured Snippet if one works in optimizing an image targeted to the same query. To optimize, optimize the ALT text of your image, use the OpenGraph image tag on the page, and use the keyword in the image filename.
Jennifer mentioned that there are interesting instances when the answers for Featured Snippets can change back-and-forth across a few days. She cited Glenn Gabe as having identified that a query for “What color are aliens?” had resulted in Featured Snippets responses with different colors over a few days. (See Glenn’s piece, “The Curious Case of The Disappearing and Reappearing Google Featured Snippet“)
One secret weapon for supplying a Featured Snippet can be through setting up a YouTube video targeted to the query. It must contain voice audio within the video, and that voice must answer the Know Simple question in a clear, succinct phrase that would translate easily into one or two sentences. Google pulls the Featured Snippet text in these cases from the video’s transcript, not from the video’s written description.
Jennifer warns that if you change your content when you have Featured Snippets, you may risk losing them. She refers to instances where people tried to change their site out of unnecessary fears of the Google Panda update, only to lose the advantage they have of Featured Snippets presentation — this can happen from removing pages with small amounts of text content.
I’m glad that Jennifer analyzed the newly leaked Google Rater’s Guide and provided this information out to the community! Also, a big thanks to her for coming to speak at the State of Search Conference!