Rich snippets guidelines

Webmaster level: All

Traditional, text-only, search result snippets aim to summarize the content of a page in our search results. Rich snippets (shown above) allow webmasters to help us provide even better summaries using structured data markup that they can add to their pages. Today we're introducing a set of guidelines to help you implement high quality structured data markup for rich snippets.

Once you've correctly added structured data markup to you site, rich snippets are generated algorithmically based on that markup. If the markup on a page offers an accurate description of the page's content, is up-to-date, and is visible and easily discoverable on your page and by users, our algorithms are more likely to decide to show a rich snippet in Google’s search results.

Alternatively, if the rich snippets markup on a page is spammy, misleading, or otherwise abusive, our algorithms are much more likely to ignore the markup and render a text-only snippet. Keep in mind that, while rich snippets are generated algorithmically, we do reserve the right to take manual action (e.g., disable rich snippets for a specific site) in cases where we see actions that hurt the experience for our users.

To illustrate these guidelines with some examples:

• If your page is about a band, make sure you mark up concerts being performed by that band, not by related bands or bands in the same town.
• If you sell products through your site, make sure reviews on each page are about that page's product and not the store itself.
• If your site provides song lyrics, make sure reviews are about the quality of the lyrics, not the quality of the song itself.

In addition to the general rich snippets quality guidelines we're publishing today, you'll find usage guidelines for specific types of rich snippets in our Help Center. As always, if you have any questions or feedback, please tell us in the Webmaster Help Forum.

Posted by Jeremy Lubin, Consumer Experience Specialist, & Pierre Far, Webmaster Trends Analyst

Keeping you informed of critical website issues

Webmaster level: All

Having a healthy and well-performing website is important, both to you as the webmaster and to your users. When we discover critical issues with a website, Webmaster Tools will now let you know by automatically sending an email with more information.

We’ll only notify you about issues that we think have significant impact on your site’s health or search performance and which have clear actions that you can take to address the issue. For example, we’ll email you if we detect malware on your site or see a significant increase in errors while crawling your site.

For most sites these kinds of issues will occur rarely. If your site does happen to have an issue, we cap the number of emails we send over a certain period of time to avoid flooding your inbox.  If you don’t want to receive any email from Webmaster Tools you can change your email delivery preferences.

We hope that you find this change a useful way to stay up-to-date on critical and important issues regarding your site’s health. If you have any questions, please let us know via our Webmaster Help Forum.

Posted by John Mueller, Webmaster Trends Analyst, Google Zürich

Structured Data Testing Tool

Webmaster level: All

Today we’re excited to share the launch of a shiny new version of the rich snippet testing tool, now called the structured data testing tool. The major improvements are:

• We’ve improved how we display rich snippets in the testing tool to better match how they appear in search results.

• The brand new visual design makes it clearer what structured data we can extract from the page, and how that may be shown in our search results.

• The tool is now available in languages other than English to help webmasters from around the world build structured-data-enabled websites.

Here’s what it looks like:
The new structured data testing tool works with all supported rich snippets and authorship markup, including applications, products, recipes, reviews, and others.

Try it yourself and, as always, if you have any questions or feedback, please tell us in the Webmaster Help Forum.

Written by Yong Zhu on behalf of the rich snippets testing tool team

Answering the top questions from government webmasters

Webmaster level: Beginner - Intermediate

Government sites, from city to state to federal agencies, are extremely important to Google Search. For one thing, governments have a lot of content — and government websites are often the canonical source of information that’s important to citizens. Around 20 percent of Google searches are for local information, and local governments are experts in their communities.

That’s why I’ve spoken at the National Association of Government Webmasters (NAGW) national conference for the past few years. It’s always interesting speaking to webmasters about search, but the people running government websites have particular concerns and questions. Since some questions come up frequently I thought I’d share this FAQ for government websites.

Question 1: How do I fix an incorrect phone number or address in search results or Google Maps?

Although managing their agency’s site is plenty of work, government webmasters are often called upon to fix problems found elsewhere on the web too. By far the most common question I’ve taken is about fixing addresses and phone numbers in search results. In this case, government site owners really can do it themselves, by claiming their Google+ Local listing. Incorrect or missing phone numbers, addresses, and other information can be fixed by claiming the listing.

Most locations in Google Maps have a Google+ Local listing — businesses, offices, parks, landmarks, etc. I like to use the San Francisco Main Library as an example: it has contact info, detailed information like the hours they’re open, user reviews and fun extras like photos. When we think users are searching for libraries in San Francisco, we may display a map and a listing so they can find the library as quickly as possible.

If you work for a government agency and want to claim a listing, we recommend using a shared Google Account with an email address at your .gov domain if possible. Usually, ownership of the page is confirmed via a phone call or post card.

Question 2: I’ve claimed the listing for our office, but I have 43 different city parks to claim in Google Maps, and none of them have phones or mailboxes. How do I claim them?

Use the bulk uploader! If you have 10 or more listings / addresses to claim at the same time, you can upload a specially-formatted spreadsheet. Go to www.google.com/places/, click the "Get started now" button, and then look for the "bulk upload" link.

If you run into any issues, use the Verification Troubleshooter.

Question 3: We're moving from a .gov domain to a new .com domain. How should we move the site?

We have a Help Center article with more details, but the basic process involves the following steps:

• Make sure you have both the old and new domain verified in the same Webmaster Tools account.
• Use a 301 redirect on all pages to tell search engines your site has moved permanently.
• Don't do a single redirect from all pages to your new home page — this gives a bad user experience.
• If there's no 1:1 match between pages on your old site and your new site (recommended), try to redirect to a new page with similar content.
• If you can't do redirects, consider cross-domain canonical links.
• Make sure to check if the new location is crawlable by Googlebot using the Fetch as Google feature in Webmaster Tools.
• Use the Change of Address tool in Webmaster Tools to notify Google of your site's move.
• Have a look at the Links to Your Site in Webmaster Tools and inform the important sites that link to your content about your new location.
• We recommend not implementing other major changes at the same time, like large-scale content, URL structure, or navigational updates.
• To help Google pick up new URLs faster, use the Fetch as Google tool to ask Google to crawl your new site, and submit a Sitemap listing the URLs on your new site.
• To prevent confusion, it's best to retain control of your old site’s domain and keep redirects in place for as long as possible — at least 180 days.

What if you’re moving just part of the site? This question came up too — for example, a city might move its "Tourism and Visitor Info" section to its own domain.

In that case, many of the same steps apply: verify both sites in Webmaster Tools, use 301 redirects, clean up old links, etc. In this case you don't need to use the Change of Address form in Webmaster Tools since only part of your site is moving. If for some reason you’ll have some of the same content on both sites, you may want to include a cross-domain canonical link pointing to the preferred domain.

Question 4: We've done a ton of work to create unique titles and descriptions for pages. How do we get Google to pick them up?

First off, that's great! Better titles and descriptions help users decide to click through to get the information they need on your page. The government webmasters I’ve spoken with care a lot about the content and organization of their sites, and work hard to provide informative text for users.

Google's generation of page titles and descriptions (or "snippets") is completely automated and takes into account both the content of a page as well as references to it that appear on the web. Changes are picked up as we recrawl your site. But you can do two things to let us know about URLs that have changed:

• Submit an updated XML Sitemap so we know about all of the pages on your site.
• In Webmaster Tools, use the Fetch as Google feature on a URL you’ve updated. Then you can choose to submit it to the index.
• You can choose to submit all of the linked pages as well — if you’ve updated an entire section of your site, you might want to submit the main page or an index page for that section to let us know about a broad collection of URLs.

Question 5: How do I get into the YouTube government partner program?

For this question, I have bad news, good news, and then even better news. On the one hand, the government partner program has been discontinued. But don’t worry, because most of the features of the program are now available to your regular YouTube account. For example, you can now upload videos longer than 10 minutes.

Did I say I had even better news? YouTube has added a lot of functionality useful for governments in the past year:

• You can now broadcast live streaming video to YouTube via Hangouts On Air (requires a Google+ account).
• You can link your YouTube account with your Webmaster Tools account, making it the "official channel" for your site.
• Automatic captions continue to get better and better, supporting more languages.

I hope this FAQ has been helpful, but I’m sure I haven’t covered everything government webmasters want to know. I highly recommend our Webmaster Academy, where you can learn all about making your site search-engine friendly. If you have a specific question, please feel free to add a question in the comments or visit our really helpful Webmaster Central Forum.

Posted by Jason Morrison, Search Quality Team

Site Errors Breakdown

Webmaster level: All

Today we’re announcing more detailed Site Error information in Webmaster Tools. This information is useful when looking for the source of your Site Errors. For example, if your site suffers from server connectivity problems, your server may simply be misconfigured; then again, it could also be completely unavailable!  Since each Site Error (DNS, Server Connectivity, and Robots.txt Fetch) is comprised of several unique issues, we’ve broken down each category into more specific errors to provide you with a better analysis of your site’s health.

Site Errors will display statistics for each of your site-wide crawl errors from the past 90 days.  In addition, it will show the failure rates for any category-specific errors that have been affecting your site.

If you’re not sure what a particular error means, you can read a short description of it by hovering over its entry in the legend.  You can find more detailed information by following the “More info” link in the tooltip.

We hope that these changes will make Site Errors even more informative and helpful in keeping your site in tip-top shape.  If you have any questions or suggestions, please let us know through the Webmaster Tools Help Forum.

Written by Cesar Cuenca and Tiffany Wang, Webmaster Tools Interns

Configuring URL Parameters in Webmaster Tools

Webmaster Level: Intermediate to Advanced

We recently filmed a video (with slides available) to provide more information about the URL Parameters feature in Webmaster Tools. The URL Parameters feature is designed for webmasters who want to help Google crawl their site more efficiently, and who manage a site with -- you guessed it -- URL parameters! To be eligible for this feature, the URL parameters must be configured in key/value pairs like item=swedish-fish or category=gummy-candy in the URL http://www.example.com/product.php?item=swedish-fish&category=gummy-candy.

Guidance for common cases when configuring URL Parameters. Music in the background masks the ongoing pounding of my neighbor’s construction!

URL Parameter settings are powerful. By telling us how your parameters behave and the recommended action for Googlebot, you can improve your site’s crawl efficiency. On the other hand, if configured incorrectly, you may accidentally recommend that Google ignore important pages, resulting in those pages no longer being available in search results. (There's an example provided in our improved Help Center article.) So please take care when adjusting URL Parameters settings, and be sure that the actions you recommend for Googlebot make sense across your entire site.

Written by Maile Ohye, Developer Programs Tech Lead

Introducing the Structured Data Dashboard

Webmaster level: All

Structured data is becoming an increasingly important part of the web ecosystem. Google makes use of structured data in a number of ways including rich snippets which allow websites to highlight specific types of content in search results. Websites participate by marking up their content using industry-standard formats and schemas.

To provide webmasters with greater visibility into the structured data that Google knows about for their website, we’re introducing today a new feature in Webmaster Tools - the Structured Data Dashboard. The Structured Data Dashboard has three views: site, item type and page-level.

Site-level view
At the top level, the Structured Data Dashboard, which is under Optimization, aggregates this data (by root item type and vocabulary schema).  Root item type means an item that is not an attribute of another on the same page.  For example, the site below has about 2 million Schema.Org annotations for Books (“http://schema.org/Book”)

Itemtype-level view
It also provides per-page details for each item type, as seen below:

Google parses and stores a fixed number of pages for each site and item type. They are stored in decreasing order by the time in which they were crawled. We also keep all their structured data markup. For certain item types we also provide specialized preview columns as seen in this example below (e.g. “Name” is specific to schema.org Product).

The default sort order is such that it would facilitate inspection of the most recently added Structured Data.

Page-level view
Last but not least, we have a details page showing all attributes of every item type on the given page (as well as a link to the Rich Snippet testing tool for the page in question).

Webmasters can use the Structured Data Dashboard to verify that Google is picking up new markup, as well as to detect problems with existing markup, for example monitor potential changes in instance counts during site redesigns.

Posted by Thomas Biggs & Andrei Pascovici, Webmaster Tools Team

New Crawl Error alerts from Webmaster Tools

Webmaster level: All

Today we’re rolling out Crawl Error alerts to help keep you informed of the state of your site.

Since Googlebot regularly visits your site, we know when your site exhibits connectivity issues or suddenly spikes in pages returning HTTP error response codes (e.g. 404 File Not Found, 403 Forbidden, 503 Service Unavailable, etc). If your site is timing out or is exhibiting systemic errors when accessed by Googlebot, other visitors to your site might be having the same problem!

When we see such errors, we may send alerts –- in the form of messages in the Webmaster Tools Message Center –- to let you know what we’ve detected. Hopefully, given this increased communication, you can fix potential issues that may otherwise impact your site’s visitors or your site’s presence in search.

As we discussed in our blog post announcing the new Webmaster Tools Crawl Errors feature, we divide crawl errors into two types: Site Errors and URL Errors.

Site Error alerts for major site-wide problems

Site Errors represent an inability to connect to your site, and represent systemic issues rather than problems with specific pages. Here are some issues that might cause Site Errors:

• Your DNS server is down or misconfigured.
• Your web server itself is firewalled off.
• Your web server is refusing connections from Googlebot.
• Your web server is overloaded, or down.
• Your site’s robots.txt is inaccessible.

These errors are global to a site, and in theory should never occur for a well-operating site (and don’t occur for the large majority of the sites we crawl). If Googlebot detects any appreciable number of these Site Errors, regardless of the size of your site, we’ll try to notify you in the form of a message in the Message Center:

Example of a Site Error alert

The alert provides the number of errors Googlebot encountered crawling your site, the overall crawl error connection rate for your site, a link to the appropriate section of Webmaster Tools to examine the data more closely, and suggestions as to how to fix the problem.

If your site shows a 100% error rate in one of these categories, it likely means that your site is either down or misconfigured in some way. If your site has an error rate less than 100% in any of these categories, it could just indicate a transient condition, but it could also mean that your site is overloaded or improperly configured. You may want to investigate these issues further, or ask about them on our forum.

We may alert you even if the overall error rate is very low — in our experience a well configured site shouldn’t have any errors in these categories.

URL Error anomaly alerts for potentially less critical issues

Whereas any appreciable number of Site Errors could indicate that your site is misconfigured, overloaded, or simply out of service, URL Errors (pages that return a non-200 HTTP code, or incorrectly return an HTTP 200 code in the case of soft 404 errors) may occur on any well-configured site. Because different sites have different numbers of pages and different numbers of external links, a count of errors that indicates a serious problem for a small site might be entirely normal for a large site.

That’s why for URL Errors we only send alerts when we detect a large spike in the number of errors for any of the five categories of errors (Server error, Soft 404, Access denied, Not found or Not followed). For example, if your site routinely has 100 pages with 404 errors, we won’t alert you if that number fluctuates minimally. However we might notify you when that count reaches a much higher number, say 500 or 1,000. Keep in mind that seeing 404 errors is not always bad, and can be a natural part of a healthy website (see our previous blog post: Do 404s hurt my site?).

A large spike in error count could be because something has changed on your site — perhaps a reconfiguration has changed the permissions for a section of your site, or a new version of a script is crashing regularly, or someone accidentally moved or deleted an entire directory, or a reorganization of your site causes external links to no longer work. It could also just be a transient spike, or could be because of external causes (someone has linked to non-existent pages), so there might not even be a problem; but when we see an unusually large number of errors for your site, we’ll let you know so you can investigate:

Example of a URL Error anomaly alert

The alert describes the category of web errors for which we’ve detected a spike, gives a link to the appropriate section of Webmaster Tools so that you can see what pages we think are problematic, and offers troubleshooting suggestions.

Enable Message forwarding to send alerts to your inbox

We know you’re busy, and that routinely checking Webmaster Tools just to check for new alerts might be something you forget to do. Consider turning on Message forwarding. We’ll send any Webmaster Tools messages to the email address of your choice.

Let us know what you think, and if you have any comments or suggestions on our new alerts please visit our forum.

Written by Brian Malcolm, Webmaster Tools team

Adding associates to manage your YouTube presence

Webmaster level: All

Many organizations have multiple presences on the web. For example, Webmaster Tools lives at www.google.com/webmasters, but it also has a Twitter account and a YouTube channel. It's important that visitors to these other properties have confidence that they are actually associated with the Webmaster Tools site. However to date it has been challenging for webmasters to manage which users can take actions on behalf of their site in different services.

Today we're happy to announce a new feature in Webmaster Tools that allows webmasters to add "associates" -- trusted users who can act on behalf of your site in other Google products. Unlike site owners and users, associates can't view site data or take any site actions in Webmaster Tools, but they are authorized to perform specific tasks in other products.

For this initial launch, members of YouTube's partner program that have created a YouTube channel for their site can now link the two together. By doing this, your YouTube channel will be displayed as the "official channel" for your website.

Management within Webmaster Tools

To add or change associates:

1. On the Webmaster Tools home page, click the site you want.
2. Under Configuration, click Associates.
   
3. Click Add a new associate.
4. In the text box, type the email address of the person you want to add.
5. Select the type of association you want.
   
6. Click Add.
   

Management within YouTube

It’s also possible for users to request association from a site’s webmaster.

1. Log in to your YouTube partner account.
2. Click on the user menu and choose Settings > Associated Website.
3. Fill in the page you would like to associate your channel with.
   
4. Click Add. If you’re a verified owner of the site, you’re done. But if someone else in your organization manages the website, the association will be marked Pending. The owner receives a notification with an option to approve or deny the request.
   
5. After approval is granted, navigate back to this page and click Refresh to complete the association.

Through associates, webmasters can easily and safely allow others to associate their website with YouTube channels. We plan to support integration with additional Google products in the future.

If you have more questions, please see the Help with Associates article or visit our webmaster help forum.

Posted by Konstantin Roslyakov, Software Engineer

Download to Google Spreadsheet from Webmaster Tools

Webmaster level: All

Webmaster Tools now has a new download option for exporting your data directly to a Google Spreadsheet. The download option is available for most of our data heavy features, such as Crawl errors, Search queries, and Links to your site. If you enjoy digging into the data from Webmaster Tools but don’t want to use Python scripts or the API, we’ve added new functionality just for you. Now when you click a download button from a Webmaster Tools feature like Search queries, you'll be presented with the "Select Download Format" option where you can choose to download the data as "CSV" or "Google Docs."

Choosing "CSV" initiates a download of the data in CSV format which has long been available in Webmaster Tools and can be imported into other spreadsheet tools like Excel. If you select the new “Google Docs” option then your data will be saved into a Google Spreadsheet and the newly created spreadsheet will be opened in a new browser tab.

We hope the ability to easily download your data to a Google Spreadsheet helps you to get crunching on your site's Webmaster Tools data even faster than you could before. Using only a web browser you can instantly dive right into slicing and dicing your data to create customized charts for detecting significant changes and tracking longer term trends impacting your site. If you've got questions or feedback please share it in the Webmaster Help Forum.

Written by Jonathan Simon, Webmaster Trends Analyst

Easier domain verification

Webmaster level: All

Today we’re announcing a new initiative that makes it easier for users to verify domains for Google services like Webmaster Tools and Google Apps.

First, some background on this initiative. To use certain Google services with your website or domain, you currently have to verify that you own the site or domain, since these services can share sensitive data (like search queries) or operate Internet-facing services (like hosted email) on your behalf.

One of our supported verification methods is domain verification. Currently this method requires a user to manually create a DNS TXT record to prove their ownership. For many users, this can be challenging and difficult to do.

So now, in collaboration with Go Daddy and eNom, we’re introducing a simple, automated solution for domain verification that guides you through the process in a few easy steps.

If your domain name records are managed by eNom or Go Daddy, in the Google site verification interface you will see a new, easier verification method as shown below.

   

Selecting this method launches a pop-up window that asks you to log in to the provider using your existing account with them.

  

The first time you log in, you’ll be asked to authorize the provider to access the Google site verification service on your behalf.

 

Next you’ll be asked to confirm that you wish to verify the domain.

   

And that’s it! After a few seconds, your domain should be automatically verified and a confirmation message displayed.

 

Now eNom and Go Daddy customers can more quickly and easily verify their domains to use with Google services like Webmaster Tools and Google Apps.

We’re also happy to share that Bluehost customers will be able to enjoy the same capability in the near future. And we look forward to working with more partners to bring easier domain verification to even more users. (Interested parties can contact us via this form.)

If you have any questions or feedback, as always please let us know via our webmaster help forum.

Posted by Anthony Chavez, Product Manager

Navigation, Dashboard and Home page

Webmaster level: All

After announcing Webmaster Tools spring cleaning earlier this quarter, it’s time to do the job. There are a few changes coming along: an updated navigation, revamped dashboard, and a compact view for the home page site-list.

Here's the new sample Webmaster Tools Dashboard for www.example.com

We’ve regrouped the features in Webmaster Tools to create an improved navigation structure (shown on the left-hand side of the above image). We distinguished the following groups: Configuration, Health, Traffic and Optimization. Each group represents a related set of functionality:

• Configuration: Things you configure and generally don’t change very often.
• Health: Where you look to make sure things are OK.
• Traffic: Where you go to understand how your site is doing in Google search, who’s linking to you; where you can explore the data about your site.
• Optimization: Where you can find ideas to enhance your site, which enables us to better understand and represent your site in Search and other services.

If you have a moment, please take time to familiarize yourself with the new Webmaster Tools navigation. Some features were slightly renamed, such as HTML Suggestions became HTML Improvements, however the functionality remains the same.

Hoping you’ll find the new navigation useful, we also think you’ll like the new Dashboard. At the top of the Dashboard you can see recent, important, prioritized messages regarding your site. Just below that, you’ll find another section which provides a brief summary of the current status of your site. There are three widgets displayed: Crawl Errors, Search Queries and Sitemaps, each representing a different navigation group: Health, Traffic and Optimization (respectively). We know your time is valuable. With the new Dashboard, we've surfaced more messages and charts to let you see how your site is doing at a glance. Take a quick look before diving into the details.

Finally, those of you who manage a large number of sites can choose to view your site-list in a 'Compact' layout, without the large site-preview thumbnails. Don't worry, if you want the more expanded layout you can always switch back.

Compact layout of the Home page

If you have questions or comments about these changes please post them in our Help Forum.

Written by Kamila Primke, Software Engineer, Webmaster Tools

Even more Top Search Queries data

Webmaster level: All We recently updated the Top Search Queries data to take into account the average top position, we enabled programmatic download and we made sure you could still get all the queries that drive traffic to your site. Well, now it’s time to give you more search queries data!

First, and most important, you can now see up to 90 days of historical data. If you click on the date picker in the top right of Search queries, you can go back three months instead of the previous 35 days.

And after you click:

In order to see 90 days, the option to view with changes will be disabled. If you want to see the changes with respect to the previous time period, the limit remains 30 days. Changes are disabled by default but you can switch them on and off with the button between the graph and the table. Top search queries data is normally available within 2 or 3 days.

Another big improvement in Webmaster Tools is that you can now see basic search query data as soon as you verify ownership of a site. No more waiting to see your information.

Finally, we're now collecting data for the top 2000 queries for which your site gets clicks. You may see less than 2000 if we didn’t record any clicks for a particular query in a given day, or if your query data is spread out among many countries or languages. For example, a search for [flowers] on Google Canada is counted separately from a search for [flowers] on google.com. Nevertheless, with this change 98% of sites will have complete coverage. Let us know what you think. We hope the new data will be useful.

Written by Javier Tordable, Tech Lead, Webmaster Tools

Webmaster Tools spring cleaning

Webmaster level: All

Webmaster Tools added lots of new functionality over the past year, such as improvements to Sitemaps and Crawl errors, as well as the new User Administration feature. In recent weeks, we also updated the look & feel of our user interface to match Google's new style. In order to keep bringing you improvements, we occasionally review each of our features to see if they’re still useful in comparison to the maintenance and support they require. As a result of our latest round of spring cleaning, we'll be removing the Subscriber stats feature, the Create robots.txt tool, and the Site performance feature in the next two weeks.

Subscriber stats reports the number of subscribers to a site’s RSS or Atom feeds. This functionality is currently provided in Feedburner, another Google product which offers its own subscriber stats as well as other cool features specifically geared for feeds of all types. If you are looking for a replacement to Subscriber stats in Webmaster Tools, check out Feedburner.

The Create robots.txt tool provides a way to generate robots.txt files for the purpose of blocking specific parts of a site from being crawled by Googlebot. This feature has very low usage, so we've decided to remove it from Webmaster Tools. While many websites don't even need a robots.txt file, if you feel that you do need one, it's easy to make one yourself in a text editor or use one of the many other tools available on the web for generating robots.txt files.

Site performance is a Webmaster Tools Labs feature that provides information about the average load time of your site's pages. This feature is also being removed due to low usage. Now you might have heard our announcement from a couple of years ago that the latency of a site's pages is a factor in our search ranking algorithms. This is still true, and you can analyze your site's performance using the Site Speed feature in Google Analytics or using Google's PageSpeed online. There are also many other site performance analysis tools available like WebPageTest and the YSlow browser plugin.

If you have questions or comments about these changes please post them in our Help Forum.

Written by Jonathan Simon, Webmaster Trends Analyst

Crawl Errors: The Next Generation

Webmaster level: All

Crawl errors is one of the most popular features in Webmaster Tools, and today we’re rolling out some very significant enhancements that will make it even more useful.

We now detect and report many new types of errors. To help make sense of the new data, we’ve split the errors into two parts: site errors and URL errors.

Site Errors

Site errors are errors that aren’t specific to a particular URL—they affect your entire site. These include DNS resolution failures, connectivity issues with your web server, and problems fetching your robots.txt file. We used to report these errors by URL, but that didn’t make a lot of sense because they aren’t specific to individual URLs—in fact, they prevent Googlebot from even requesting a URL! Instead, we now keep track of the failure rates for each type of site-wide error. We’ll also try to send you alerts when these errors become frequent enough that they warrant attention.

View site error rate and counts over time

Furthermore, if you don’t have (and haven’t recently had) any problems in these areas, as is the case for many sites, we won’t bother you with this section. Instead, we’ll just show you some friendly check marks to let you know everything is hunky-dory.

A site with no recent site-level errors

URL errors

URL errors are errors that are specific to a particular page. This means that when Googlebot tried to crawl the URL, it was able to resolve your DNS, connect to your server, fetch and read your robots.txt file, and then request this URL, but something went wrong after that. We break the URL errors down into various categories based on what caused the error. If your site serves up Google News or mobile (CHTML/XHTML) data, we’ll show separate categories for those errors.

URL errors by type with full current and historical counts

Less is more

We used to show you at most 100,000 errors of each type. Trying to consume all this information was like drinking from a firehose, and you had no way of knowing which of those errors were important (your homepage is down) or less important (someone’s personal site made a typo in a link to your site). There was no realistic way to view all 100,000 errors—no way to sort, search, or mark your progress. In the new version of this feature, we’ve focused on trying to give you only the most important errors up front. For each category, we’ll give you what we think are the 1000 most important and actionable errors.  You can sort and filter these top 1000 errors, let us know when you think you’ve fixed them, and view details about them.

Instantly filter and sort errors on any column

Some sites have more than 1000 errors of a given type, so you’ll still be able to see the total number of errors you have of each type, as well as a graph showing historical data going back 90 days. For those who worry that 1000 error details plus a total aggregate count will not be enough, we’re considering adding programmatic access (an API) to allow you to download every last error you have, so please give us feedback if you need more.

We've also removed the list of pages blocked by robots.txt, because while these can sometimes be useful for diagnosing a problem with your robots.txt file, they are frequently pages you intentionally blocked. We really wanted to focus on errors, so look for information about roboted URLs to show up soon in the "Crawler access" feature under "Site configuration".

Dive into the details

Clicking on an individual error URL from the main list brings up a detail pane with additional information, including when we last tried to crawl the URL, when we first noticed a problem, and a brief explanation of the error.

Details for each URL error

From the details pane you can click on the link for the URL that caused the error to see for yourself what happens when you try to visit it. You can also mark the error as “fixed” (more on that later!), view help content for the error type, list Sitemaps that contain the URL, see other pages that link to this URL, and even have Googlebot fetch the URL right now, either for more information or to double-check that your fix worked.

View pages which link to this URL

Take action!

One thing we’re really excited about in this new version of the Crawl errors feature is that you can really focus on fixing what’s most important first. We’ve ranked the errors so that those at the top of the priority list will be ones where there’s something you can do, whether that’s fixing broken links on your own site, fixing bugs in your server software, updating your Sitemaps to prune dead URLs, or adding a 301 redirect to get users to the “real” page. We determine this based on a multitude of factors, including whether or not you included the URL in a Sitemap, how many places it’s linked from (and if any of those are also on your site), and whether the URL has gotten any traffic recently from search.

Once you think you’ve fixed the issue (you can test your fix by fetching the URL as Googlebot), you can let us know by marking the error as “fixed” if you are a user with full access permissions. This will remove the error from your list.  In the future, the errors you’ve marked as fixed won’t be included in the top errors list, unless we’ve encountered the same error when trying to re-crawl a URL.

Select errors and mark them as fixed

We’ve put a lot of work into the new Crawl errors feature, so we hope that it will be very useful to you. Let us know what you think and if you have any suggestions, please visit our forum!

Written by Kurt Dresner, Webmaster Tools team

Safely share access to your site in Webmaster Tools

Webmaster Level: All

We just launched a new feature that allows you as a verified site owner to grant limited access to your site's data and settings in Webmaster Tools. You've had the ability to grant full verified access to others for a couple of years. Since then we've heard lots of requests from site owners for the ability to grant limited permission for others to view a site's data in Webmaster Tools without being able to modify all the settings. Now you can do exactly that with our new User administration feature.

On the Home page when you click the "Manage site" drop-down menu you'll see the menu option that was previously titled "Add or remove owners" is now "Add or remove users."


Selecting the "Add or remove users" menu item will take you to the new User administration page where you can add or delete up to 100 users and specify each user's access as "Full" or "Restricted." Users added via the User administration page are tied to a specific site. If you become unverified for that site any users that you've added will lose their access to that site in Webmaster Tools. Adding or removing verified site owners is still done on the owner verification page which is linked from the User administration page.


Granting a user "Full" permission means that they will be able to view all data and take most actions, such as changing site settings or demoting sitelinks. When a user’s permission is set to "Restricted" they will only have access to view most data, and can take some actions such as using Fetch as Googlebot and configuring message forwarding for their account. Restricted users will see a “Restricted Access” indicator at various locations within Webmaster Tools.



To see which features and actions are accessible for Restricted users, Full users and site owners, visit our Permissions Help Center article.

We hope the addition of Full and Restricted users makes management of your site in Webmaster Tools easier since you can now grant access within a more limited scope to help prevent undesirable or unauthorized changes. If you have questions or feedback about the new User administration feature please let us know in our Help Forum.

Written by Jonathan Simon, Webmaster Trends Analyst

What’s new with Sitemaps

Webmaster level: All

Sitemaps are a way to tell Google about pages on your site. Webmaster Tools’ Sitemaps feature gives you feedback on your submitted Sitemaps, such as how many Sitemap URLs have been indexed, or whether your Sitemaps have any errors. Recently, we’ve added even more information! Let’s check it out:


The Sitemaps page displays details based on content-type. Now statistics from Web, Videos, Images and News are featured prominently. This lets you see how many items of each type were submitted (if any), and for some content types, we also show how many items have been indexed. With these enhancements, the new Sitemaps page replaces the Video Sitemaps Labs feature, which will be retired.

Another improvement is the ability to test a Sitemap. Unlike an actual submission, testing does not submit your Sitemap to Google as it only checks it for errors. Testing requires a live fetch by Googlebot and usually takes a few seconds to complete. Note that the initial testing is not exhaustive and may not detect all issues; for example, errors that can only be identified once the URLs are downloaded are not be caught by the test.

In addition to on-the-spot testing, we’ve got a new way of displaying errors which better exposes what types of issues a Sitemap contains. Instead of repeating the same kind of error many times for one Sitemap, errors and warnings are now grouped, and a few examples are given. Likewise, for Sitemap index files, we’ve aggregated errors and warnings from the child Sitemaps that the Sitemap index encloses. No longer will you need to click through each child Sitemap one by one.

Finally, we’ve changed the way the “Delete” button works. Now, it removes the Sitemap from Webmaster Tools, both from your account and the accounts of the other owners of the site. Be aware that a Sitemap may still be read or processed by Google even if you delete it from Webmaster Tools. For example if you reference a Sitemap in your robots.txt file search engines may still attempt to process the Sitemap. To truly prevent a Sitemap from being processed, remove the file from your server or block it via robots.txt.

For more information on Sitemaps in Webmaster Tools and how Sitemaps work, visit our Help Center. If you have any questions, go to Webmaster Help Forum.

Written by Kamila Primke, Software Engineer, Webmaster Tools

Update to Top Search Queries data

Webmaster level: All

Starting today, we’re updating our Top Search Queries feature to make it better match expectations about search engine rankings. Previously we reported the average position of all URLs from your site for a given query. As of today, we’ll instead average only the top position that a URL from your site appeared in.

An example
Let’s say Nick searched for [bacon] and URLs from your site appeared in positions 3, 6, and 12. Jane also searched for [bacon] and URLs from your site appeared in positions 5 and 9. Previously, we would have averaged all these positions together and shown an Average Position of 7. Going forward, we’ll only average the highest position your site appeared in for each search (3 for Nick’s search and 5 for Jane’s search), for an Average Position of 4.

We anticipate that this new method of calculation will more accurately match your expectations about how a link's position in Google Search results should be reported.

How will this affect my Top Search Queries data?
This change will affect your Top Search Queries data going forward. Historical data will not change. Note that the change in calculation means that the Average Position metric will usually stay the same or decrease, as we will no longer be averaging in lower-ranking URLs.

Check out the updated Top Search Queries data in the Your site on the web section of Webmaster Tools. And remember, you can also download Top Search Queries data programmatically!

We look forward to providing you a more representative picture of your Google Search data. Let us know what you think in our Webmaster Forum.

Posted by Chris Anderson, Google Analytics team, and Susan Moskwa, Webmaster Trends Analyst

Download search queries data using Python

Webmaster level: Advanced

For all the developers who have expressed interest in getting programmatic access to the search queries data for their sites in Webmaster Tools, we've got some good news. You can now get access to your search queries data in CSV format using a open source Python script from the webmaster-tools-downloads project. Search queries data is not currently available via the Webmaster Tools API, which has been a common API user request that we're considering for the next API update. For those of you who need access to search queries data right now, let's look at an example of how the search queries downloader Python script can be used to download your search queries data and upload it to a Google Spreadsheet in Google Docs.

Example usage of the search queries downloader Python script
1) If Python is not already installed on your machine, download and install Python.
2) Download and install the Google Data APIs Python Client Library.
3) Create a folder and add the downloader.py script to the newly created folder.
4) Copy the example-create-spreadsheet.py script to the same folder as downloader.py and edit it to replace the example values for “website,” “email” and “password” with valid values for your Webmaster Tools verified site.
5) Open a Terminal window and run the example-create-spreadsheet.py script by entering "python example-create-spreadsheet.py" at the Terminal window command line:

python example-create-spreadsheet.py

6) Visit Google Docs to see a new spreadsheet containing your search queries data.


If you just want to download your search queries data in a .csv file without uploading the data to a Google spreadsheet use example-simple-download.py instead of example-create-spreadsheet.py in the example above.

You could easily configure these scripts to be run daily or monthly to archive and view your search queries data across larger date ranges than the current one month of data that is available in Webmaster Tools, for example, by setting up a cron job or using Windows Task Scheduler.

An important point to note is that this script example includes user name and password credentials within the script itself. If you plan to run this in a production environment you should follow security best practices like using encrypted user credentials retrieved from a secure data storage source. The script itself uses HTTPS to communicate with the API to protect these credentials.

Take a look at the search queries downloader script and start using search queries data in your own scripts or tools. Let us know if you have questions or feedback in the Webmaster Help Forum.

Written by Jonathan Simon, Webmaster Trends Analyst

Clicks and impressions for authors

Webmaster Level: All
(Cross-posted on the Inside Search Blog)

With the latest improvements to the way authorship annotations look in search and the addition of authorship to Google News, authors have been really excited about getting more visibility, and users benefit from seeing the name, photo, and way to connect with the person who created the content.

Authors have also been giving us a lot of feedback on what else they'd like to see, so today we're introducing “Author Stats” in Webmaster Tools that shows you how often your content is showing up on the Google search results page. If you associate your content with your Google Profile either via e-mail verification or a simple link, you can visit Webmaster Tools to see how many impressions and clicks your content got on the Google search results page. Check out what Matt Cutts would see for his content:

To see your information, go to google.com/webmasters and login with the same username you use for your Google+ Profile. On the left hand panel, you can see “Author Stats” under the “Labs” section. This is an experimental feature so we’re continuing to iterate and improve, but we wanted to get early feedback from you. You can e-mail us at authorship-pilot@google.com if you run into any issues or have feedback.

If you’re a content creator interested in learning more about authorship, check out our Help Center.

Posted by Javier Tordable, Software Engineer