Traffic

CPX

PTP

DOWNLOAD OUR MOVIE COLLECTION

Thursday, February 28, 2008

iGoogle Gadgets for Webmaster Tools



Update: The described feature is no longer available.

When you plan to do something, are you a minimalist, or are you prepared for every potential scenario? For example, would you hike out into the Alaskan wilderness during inclement weather with only a wool overcoat and a sandwich in your pocket - like the naturalist John Muir (and you thought Steve McQueen was tough)?

Or are you more the type of person where even on a day hike, you bring a few changes of clothes, 3 dehydrated meals, a couple of kitchen appliances, a power inverter, and a foot- powered generator, because, well, you never know when the urge will arise to make toast?

The Webmaster Tools team strives to serve all types of webmasters, from the minimalist to those who use every tool they can find. If you're reading this blog, you've probably had the opportunity to use the current version of Webmaster Tools, which offers as many features as possible just shy of the kitchen sink. Now there's something for those of you who would prefer to access only the features of Webmaster Tools that you need: we've just released Webmaster Tools Gadgets for iGoogle.

Here's the simple process to start using these Gadgets right away. (Note: this assumes you've already got a Webmaster Tools account and have verified at least one site.)

1. Visit Webmaster Tools and select any site that you've validated from the dashboard.
2. Click on the Tools section.
3. Click on Gadgets sub-section.
4. Click on the big "Add an iGoogle Webmaster Tools homepage" button.
5. Click the "Add to Google" button on the following confirm page to add the new tab to iGoogle.
6. Now you're in iGoogle, where you should see your new Google Webmaster Tools tab with a number of Gadgets. Enjoy!

You'll notice that each Gadget has a drop down menu at the top which lets you select from all the sites you have validated to see that Gadget's information for the particular site you select. A few of the Gadgets that we're currently offering are:

Crawl errors - Does Googlebot encounter issues when crawling your site?



Top search queries - What are people searching for to find your site?



External links - What websites are linking to yours?




We plan to add more Gadgets in the future and improve their quality, so if there's a feature that you'd really like to see which is not included in one of the Gadgets currently available, let us know. As you can see, it's a cinch to get started.

It looks like rain clouds are forming over here in Seattle, so I'm off for a hike.

Wednesday, February 27, 2008

Cross-submissions via robots.txt on Sitemaps.org


Last spring, the Sitemaps protocol was expanded to include the autodiscovery of Sitemaps using robots.txt to let us and other search engines supporting the protocol know about your Sitemaps. We subsequently also announced support for Sitemap cross-submissions using Google Webmaster Tools, making it possible to submit Sitemaps for multiple hosts on a single dedicated host. So it was only time before we took the next logical step of marrying the two and allowing Sitemap cross-submissions using robots.txt. And today we're doing just that.

We're making it easier for webmasters to place Sitemaps for multiple hosts on a single host and then letting us know by including the location of these Sitemaps in the appropriate robots.txt.

How would this work? Say for example you want to submit a Sitemap for each of the two hosts you own, www.example.com and host2.google.com. For simplicity's sake, you may want to host the Sitemaps on one of the hosts, www.example.com. For example, if you have a Content Management System (CMS), it might be easier for you to change your robots.txt files than to change content in a directory.

You can now exercise the cross-submission support via robots.txt (by letting us know the location of the Sitemaps):

a) The robots.txt for www.example.com would include:
Sitemap: http://www.example.com/sitemap-www-example.xml

b) And similarly, the robots.txt for host2.google.com would include:
Sitemap: http://www.example.com/sitemap-host2-google.xml

By indicating in each individual host's robots.txt file where that host's Sitemap lives you are in essence proving that you own the host for which you are specifying the Sitemap. And by choosing to host all of the Sitemaps on a single host, it becomes simpler to manage your Sitemaps.

We are making this announcement today on Sitemaps.org as a joint effort. To see what our colleagues have to say, you can also check out the blog posts published by Yahoo! and Microsoft.

Tuesday, February 26, 2008

Leap day hackathon for Google Gadgets, Maps, and more



If you've got JavaScript skills and you'd like to implement such things as Google Gadgets or Maps on your site, bring your laptops and come hang out with us in Mountain View.

This Friday, my team (Google Developer Programs) is hosting a hackathon to get you started with our JavaScript APIs. There will be plenty of our engineers around to answer questions. We'll start with short introductions of the APIs and then break into groups for coding and camaraderie. There'll be food, and prizes too.

The featured JavaScript APIs:When: Friday, February 29 - two sessions (you're welcome to attend both)
  • 2-5:30 PM
  • 6-10 PM
Where: The Googleplex
Building 40
1600 Amphitheatre Pkwy
Mountain View, CA 94043
Room: Seville Tech Talk, 2nd floor

See our map for parking locations and where to check in. (Soon, you too, will be making maps like this! :)

Just say yes and RSVP!

And no worries if you're busy this Friday; future hackathons will feature other APIs and more languages. Check out the Developer Events Calendar for future listings. Hope to see you soon.

Tuesday, February 12, 2008

7 must-read Webmaster Central blog posts

Our search quality and Webmaster Central teams love helping webmasters solve problems. But since we can't be in all places at all times answering all questions, we also try hard to show you how to help yourself. We put a lot of work into providing documentation and blog posts to answer your questions and guide you through the data and tools we provide, and we're constantly looking for ways to improve the visibility of that information.

While I always encourage people to search our Help Center and blog for answers, there are a few articles in particular to which I'm constantly referring people. Some are recent and some are buried in years' worth of archives, but each is worth a read:

  1. Googlebot can't access my website
    Web hosters seem to be getting more aggressive about blocking spam bots and aggressive crawlers from their servers, which is generally a good thing; however, sometimes they also block Googlebot without knowing it. If you or your hoster are "allowing" Googlebot through by whitelisting Googlebot IP addresses, you may still be blocking some of our IPs without knowing it (since our full IP list isn't public, for reasons explained in the post). In order to be sure you're allowing Googlebot access to your site, use the method in this blog post to verify whether a crawler is Googlebot.
  2. URL blocked by robots.txt
    Sometimes the web crawl section of Webmaster Tools reports a URL as "blocked by robots.txt", but your robots.txt file doesn't seem to block crawling of that URL. Check out this list of troubleshooting tips, especially the part about redirects. This thread from our Help Group also explains why you may see discrepancies between our web crawl error reports and our robots.txt analysis tool.
  3. Why was my URL removal request denied?
    (Okay, I'm cheating a little: this one is a Help Center article and not a blog post.) In order to remove a URL from Google search results you need to first put something in place that will prevent Googlebot from simply picking that URL up again the next time it crawls your site. This may be a 404 (or 410) status code, a noindex meta tag, or a robots.txt file, depending on what type of removal request you're submitting. Follow the directions in this article and you should be good to go.
  4. Flash best practices
    Flash continues to be a hot topic for webmasters interested in making visually complex content accessible to search engines. In this post Bergy, our resident Flash expert, outlines best practices for working with Flash.
  5. The supplemental index
    The "supplemental index" was a big topic of conversation in 2007, and it seems some webmasters are still worried about it. Instead of worrying, point your browser to this post on how we now search our entire index for every query.
  6. Duplicate content
    Duplicate content—another perennial concern of webmasters. This post talks in detail about duplicate content caused by URL parameters, and also references Adam's previous post on deftly dealing with duplicate content, which gives lots of good suggestions on how to avoid or mitigate problems caused by duplicate content.
  7. Sitemaps FAQs
    This post answers the most frequent questions we get about Sitemaps. And I'm not just saying it's great because I posted it. :-)

Sometimes, knowing how to find existing information is the biggest barrier to getting a question answered. So try searching our blog, Help Center and Help Group next time you have a question, and please let us know if you can't find a piece of information that you think should be there!

Tuesday, January 22, 2008

Feeling lucky at PubCon

Last month, several of us with Webmaster Central hit the "good times" jackpot at PubCon Vegas 2007. We realize not all of you could join us, so instead of returning home with fuzzy dice for everyone, we've got souvenir conference notes.

Listening to the Q&A, I was pleased to hear the major search engines agreeing on best practices for many webmaster issues. In fact, the presentations in the duplicate content session were mostly, well, duplicate. When I wasn't sitting in on one of the many valuable sessions, I was chatting with webmasters either at the Google booth, or at Google's "Meet the Engineers" event. It was exciting to hear from so many different webmasters, and to help them with Google-related issues. Here are a few things that were on the minds of webmasters, along with our responses:

Site Verification Files and Meta Tags
Several webmasters asked, "Is it necessary to keep the verification meta tag or HTML file in place to remain a verified owner in Webmaster Tools?" The answer is yes, you should keep your verification file or meta tag live to maintain your status as a verified owner. These verification codes are used to control who has access to the owner-specific tools for your site in Webmaster Tools. To ensure that only current owners of a site are verified, we periodically re-check to see if the verification code is in place, and if it is not, you will get unverified for that site. While we're on the topic:

Site Verification Best Practices
  • If you have multiple people working on your site with Webmaster Tools, it's a good idea to have each person verify the site with his or her own account, rather than using a shared login. That way, as people come and go, you can control the access appropriately by adding or removing verification files or meta tags for each account.
  • You may want to keep a list of these verification codes and which owner they are connected to, so you can easily control access later. If you lose track, you can always use the "Manage site verification" option in Webmaster Tools, which allows you to force all site owners to reverify their accounts.
Subdomains vs. Subdirectories
What's the difference between using subdomains and subdirectories? When it comes to Google, there aren't major differences between the two, so when you're making that decision, do what works for you and your visitors. Following PubCon, our very own Matt Cutts outlined many of the key issues in a post on his personal blog. In addition to those considerations, if you use Webmaster Tools (which we hope you do!), keep in mind that you'll automatically be verified for deeper subdirectories of any sites you've verified, but subdomains need to be verified separately.

Underscores vs. Dashes
Webmasters asked about the difference between how Google interprets underscores and dashes in URLs. In general, we break words on punctuation, so if you use punctuation as separators, you're providing Google a useful signal for parsing your URLs. Currently, dashes in URLs are consistently treated as separators while underscores are not. Keep in mind our technology is constantly improving, so this distinction between underscores and dashes may decrease over time. Even without punctuation, there's a good chance we'll be able to figure out that bigleopard.html is about a "big leopard" and not a "bigle opard." While using separators is a good practice, it's likely unnecessary to place a high priority on changing your existing URLs just to convert underscores to dashes.

Keywords in URLs
We were also asked if it is useful to have relevant keywords in URLs. It's always a good idea to be descriptive across your site, with titles, ALT attributes, and yes, even URLs, as they can be useful signals for users and search engines. This can be especially true with image files, which otherwise may not have any text for a search engine to consider. Imagine you've taken a picture of your cat asleep on the sofa. Your digital camera will likely name it something like IMG_2937.jpg. Not exactly the most descriptive name. So unless your cat really looks like an IMG_2937, consider changing the filename to something more relevant, like adorable-kitten.jpg. And, if you have a post about your favorite cat names, it's much easier to guess that a URL ending in my-favorite-cat-names would be the relevant page, rather than a URL ending in postid=8652. For more information regarding issues with how Google understands your content, check out our new content analysis feature in Webmaster Tools, as well as our post on the URL suggestions feature of the new Google Toolbar.

Moving to a new IP address
We got a question about changing a site's IP address, and provided a few steps you can take as a webmaster to make sure things go smoothly. Here's what you can do:
  1. Change the TTL (Time To Live) value of your DNS configuration to something short, like five minutes (300 seconds). This will tell web browsers to re-check the IP address for your site every five minutes.
  2. Copy your content to the new hosting environment, and make sure it is live on the new IP address.
  3. Change your DNS settings so your hostname points to the new IP address.
  4. Check your logs to see when Googlebot starts crawling your site on the new IP address. To make sure it's really Googlebot who's visiting, you can verify Googlebot by following these instructions. You can then log into Webmaster Tools and monitor any crawl errors. Once Googlebot is happily crawling on the new IP address, you should be all set as far as Google is concerned.
  5. To make sure everyone got the message of your move, you may want to keep an eye out for visits to your old IP address before shutting it down.
Proxies
A few webmasters were concerned that proxy services are being indexed with copies of their content. While it's often possible to find duplicate copies of your content in our results if you look hard enough, the original source is most likely going to be ranked higher than a proxy copy. However, if you find this not to be the case, please drop us some URLs in the Webmaster Help Group. There are many Googlers including myself who monitor this group and escalate issues appropriately.

It was great talking with webmasters at the conference -- we hope those of you unable to join us found this post useful. If you want to continue to talk shop with me, other Googlers, and your fellow webmasters, join the follow-up conversation in the Webmaster Help Group.

Update: Additional PubCon notes from Jonathan Simon are available in our discussion group.

Tuesday, January 15, 2008

Sitemaps FAQs



Last month, Trevor spoke on the Sitemaps: Oversold, Misused or On The Money? panel at Search Engine Strategies in Chicago. After receiving a lot of great questions at the conference in addition to all the feedback we receive in our Help Group, we've pulled together a FAQ:

Q: I submitted a Sitemap, but my URLs haven't been [crawled/indexed] yet. Isn't that what a Sitemap is for?
A: Submitting a Sitemap helps you make sure Google knows about the URLs on your site. It can be especially helpful if your content is not easily discoverable by our crawler (such as pages accessible only through a form). It is not, however, a guarantee that those URLs will be crawled or indexed. We use information from Sitemaps to augment our usual crawl and discovery processes. Learn more.

Q: If it doesn't get me automatically crawled and indexed, what does a Sitemap do?
A: Sitemaps give information to Google to help us better understand your site. This can include making sure we know about all your URLs, how often and when they're updated, and what their relative importance is. Also, if you submit your Sitemap via Webmaster Tools, we'll show you stats such as how many of your Sitemap's URLs are indexed. Learn more.

Q: Will a Sitemap help me rank better?
A: A Sitemap does not affect the actual ranking of your pages. However, if it helps get more of your site crawled (by notifying us of URLs we didn't previously didn't know about, and/or by helping us prioritize the URLs on your site), that can lead to increased presence and visibility of your site in our index. Learn more.

Q: If I set all of my pages to have priority 1.0, will that make them rank higher (or get crawled faster) than someone else's pages that have priority 0.8?
A: No. As stated in our Help Center, "priority only indicates the importance of a particular URL relative to other URLs on your site, and doesn't impact the ranking of your pages in search results." Indicating that all of your pages have the same priority is the same as not providing any priority information at all.

Q: Is there any point in submitting a Sitemap if all the metadata (<changefreq>, <priority>, etc.) is the same for each URL, or if I'm not sure it's accurate?
A: If the value of a particular tag is the same for 100% of the URLs in your Sitemap, you don't need to include that tag in your Sitemap. Including it won't hurt you, but it's essentially the same as not submitting any information, since it doesn't help distinguish between your URLs. If you're not sure whether your metadata is accurate (for example, you don't know when a particular URL was last modified), it's better to omit that tag for that particular URL than to just make up a value which may be inaccurate.

Q: I've heard about people who submitted a Sitemap and got penalized shortly afterward. Can a Sitemap hurt you?
A: Only if it falls on you from a great height. (Seriously, though: if it ever happened that someone was penalized after submitting a Sitemap, it would have been purely coincidental. Google does not penalize you for submitting a Sitemap.)

Q: Where can I put my Sitemap? Does it have to be at the root of my site?
A: We recently enabled Sitemap cross-submissions, which means that you can put your Sitemap just about anywhere as long as you have the following sites verified in your Webmaster Tools account:
  • the site on which the Sitemap is located
  • the site(s) whose URLs are referenced in the Sitemap
Note that cross-submissions may not work for search engines other than Google. Learn more about Sitemap cross-submissions.

Q: Can I just submit the site map that my webmaster made of my site? I don't get this whole XML thing.
A: There's a difference between a (usually HTML) site map built to help humans navigate around your site, and an XML Sitemap built for search engines. Both of them are useful, and it's great to have both. A site map on your domain can also help search engines find your content (since crawlers can follow the links on the page). However, if you submit an HTML site map in place of a Sitemap, Webmaster Tools will report an error because an HTML page isn't one of our recognized Sitemap formats. Also, if you create an XML Sitemap, you'll be able to give us more information than you can with an HTML site map (which is just a collection of links). Learn more about supported Sitemap formats.

Q: Which Sitemap format is the best?
A: We recommend the XML Sitemap protocol as defined by sitemaps.org. XML Sitemaps have the advantage of being upgradeable: you can start simple if you want (by just listing your URLs), but—unlike a text file Sitemap—you can easily upgrade an XML Sitemap later on to include more metadata. XML Sitemaps are also more comprehensive than an Atom or RSS feed submitted as a Sitemap, since feeds usually only list your most recent URLs (rather than all the URLs you want search engines to know about).

Q: If I have multiple URLs that point to the same content, can I use my Sitemap to indicate my preferred URL for that content?
A: Yes. While we can't guarantee that our algorithms will display that particular URL in search results, it's still helpful for you to indicate your preference by including that URL in your Sitemap. We take this into consideration, along with other signals, when deciding which URL to display in search results. Learn more about duplicate content.

Q: Does the placement of a URL within a Sitemap file matter? Will the URLs at the beginning of the file get better treatment than the URLs near the end?
A: No, and no.

Q: If my site has multiple sections (e.g. a blog, a forum, and a photo gallery), should I submit one Sitemap for the site, or multiple Sitemaps (one for each section)?
A: You may submit as few or as many Sitemaps as you like (up to these limits). Organize them in whatever way you find easiest to maintain. If you create multiple Sitemaps, you can use a Sitemap Index file to list them all. Learn more.

If your question isn't covered here, you can find even more questions and answers in our Sitemaps Help Group.

Wednesday, January 9, 2008

Remove your content from Google



Confused about the best uses of robots.txt, nofollow, URL removal tool? Wondering how to keep some of your pages off the web? Our webspam lead, Matt Cutts, talks about the best ways to stop Google from crawling your content, and how to remove content from the Google index once we've crawled it.



We love your feedback. Tell us what you think about this video in our Webmaster Help Group.

* Note from Matt: Yes, robots.txt has been around since at least 1996, not 2006. It's hard for me to talk for 12-13 minutes without any miscues. :)



Update: for more information, please see our Help Center articles on removing content.

Monday, January 7, 2008

Growing the Webmaster Help Groups Team



One of the ways in which we continuously work to improve our service for users everywhere is to localize and adjust our tools for different regions. In November, we had a chance to introduce our international Webmaster Help Group Team and its guides from Dublin. Today I'm very happy to announce that we've launched three more Google monitored groups in Hebrew, Turkish and Hungarian.

Keeping with tradition, the new guides want to introduce themselves, so please meet our new Webmaster Help Group guides :-)


Turkish Webmaster Help Group
Merhaba! I am Bahar and I am very happy about our brand new Turkish Webmaster Help Group. I will be helping out on any topics related to this new group. Having an engineering background, I've worked in different parts of the IT field as a C and C++ developer and now I work for Google in Dublin. It was quite a big change for me to move here, leaving sunny Istanbul behind. But it was very easy to get used to this new life since I have met really friendly people here who are very enthusiastic about the job they have been doing. You're going to meet a couple of them as well when you keep reading further. Please do not hesitate to have a look at the group and drop your posts either to introduce yourselves or to discuss Webmaster related issues with me and hopefully soon many other helpful, open-minded members!
- Bahar

Hebrew Webmaster Help Group
I’m Alon and I’ll be looking after the Hebrew Webmaster Help Group. Originally coming from the wonderful Israeli city of Holon, now I spend my days and nights in the green island called Ireland. So let me tell you a bit about myself by ranking keywords and their importance in my life: family, social activism, work, web, friends, Tel-Aviv, coffee and rollerblading, to mention a few. Prior to joining Google I graduated in economics and worked as a freelancer in the IT field. I love the online sphere, and for a couple of years now I've been volunteering for an organization facilitating international webmaster communication. I’m looking forward to seeing many of you in our Hebrew Webmaster Help Group.
- Alon

Hungarian Webmaster Help Group
Sziasztok, my name is Tibor, and I am very excited about the launch of the Hungarian Webmaster Help Group. I grew up in the heart of Hungary, in the prosperous and pleasant city of Székesfehérvár, right between Lake Balaton and Budapest. I have an engineering and management background, seasoned with marketing and design experience, and have worked in various assignments, including dozens of web related projects (the first in '94). Today I am a happy owner of a small blog, but have some ambitious plans to start some bigger projects too, to get first hand experience of the constantly growing arsenal of Google's Webmaster Tools! I work primarily on search, dealing with Turkish and Lithuanian languages besides Hungarian. I also oversee trends in a dozen other languages scattered across Europe, the Middle East and Asia. Just like my colleague János, I also enjoy old-school times with StarCraft, and my "weaknesses" also include Guitar Hero and the Nintendo Wii. As Webmaster Tools is now available in Hungarian, I look forward to hearing from you in the Hungarian Webmaster Help Group and to engage myself in vibrant discussions :-)
- Tibor

Hungarian Webmaster Help Group
Sziasztok! My name is János and I'm in charge of the Hungarian Webmaster Help Group, too. I studied linguistics and worked in the IT industry before I ended up with Google in Ireland. Currently my primary work focus is search in several languages. In the past I've been lucky to support a wide range of different markets including Russian, Turkish, Polish and many more. Well, I like learning languages, as you might guess. I'm running my own website where I develop content, sometimes I need to troubleshoot it as well - no crawling/indexing issues yet though - I'm a happy Webmaster Tools user ;-) When I don't work, I'm usually at home surfing the Internet or leading my zealous Protoss forces against other races in the StarCraft universe. Please drop by the Hungarian Help Group and introduce yourself, too! My guide colleague Tibor and I are eagerly waiting to help you with webmaster issues.
- János

Thursday, January 3, 2008

Looking back on 2007

Written by Liza Ma, Webmaster Tools Team

2007 has been a great year of change and growth for the Webmaster Central team. Without the webmaster community helping us drive our vision to create better tools we wouldn't have accomplished much. All of us with the Webmaster Central team would like to say a big thank you to everyone! Before we look ahead to 2008 and begin to incorporate your feedback from the blog posts and discussion groups, we would like to take a minute to reflect upon some of our bigger feats of the last twelve months.

Our blog has been very active since its launch in August 2006, and we hope you've found it helpful. For those who are new readers, you can find useful information about our Webmaster Tools updates here, as well as articles on how to make your sites more search engine-friendly and user-friendly. Outside of the US, we have readers from the UK, France, Canada, India and Germany, just to name a few—it just goes to show what a big and widespread community you are! In fact, we also have a dedicated group of readers for our Webmaster Central German and China blogs. For those keeping track, 47.28% of you are using Firefox to read our blog and 45.06% are using Internet Explorer, and our most popular post this past December answered your questions on meta tags and web search. Here's a snapshot of what our top search queries look like for December:

January
  • Made preparations for our big debut (see February)

February
March

April

May

June

July
August
September
    • Introduced Subscriber stats
    • Revamped the UI
    • Added support for 2 new languages: Romanian & Turkish

October
  • Went all out for Halloween in true Google fashion
  • Launched new features in Webmaster Tools—and a lot of them:
  • Improved data freshness to keep you better informed about your site's performance

November

December
  • Another Webmaster Tools update loaded with new features:
    • Improved Sitemap summary pages to help you identify issues in your Sitemaps
    • Content analysis
    • Video Sitemaps
    • Added support for 2 new languages: Czech and Hungarian

It was a busy year, but we had a lot of fun. I hope you'll stick with us as we embark on more great adventures in 2008. Remember, you can always reach out to us through our discussion group or at various search conferences throughout the year and around the globe!