Written by Prashanth Koppula, Product Manager, and Matt Cutts, Software Engineer

When Google originally introduced Supplemental Results in 2003, our main web index had billions of web pages. The supplemental index made it possible to index even more web pages and, just like our main web index, make this content available when generating relevant search results for user queries. This was especially useful for queries that did not return many results from the main web index, and for these the supplemental index allowed us to query even more web pages. The fewer constraints we're able to place on sites we crawl for the supplemental index means that web pages that are not in the main web index could be included in the supplemental. These are often pages with lower PageRank or those with more complex URLs. Thus the supplemental index (read more - and here's Matt's talk about it on video) serves a very important purpose: to index as much of the relevant content that we crawl as possible.

The changes we make must focus on improving the search experience for our users. Since 2006, we've completely overhauled the system that crawls and indexes supplemental results. The current system provides deeper and more continuous indexing. Additionally, we are indexing URLs with more parameters and are continuing to place fewer restrictions on the sites we crawl. As a result, Supplemental Results are fresher and more comprehensive than ever. We're also working towards showing more Supplemental Results by ensuring that every query is able to search the supplemental index, and expect to roll this out over the course of the summer.

The distinction between the main and the supplemental index is therefore continuing to narrow. Given all the progress that we've been able to make so far, and thinking ahead to future improvements, we've decided to stop labeling these URLs as "Supplemental Results." Of course, you will continue to benefit from Google's supplemental index being deeper and fresher.

Written by Maile Ohye
Today we're launching our Message Center, a new way for webmasters to receive personalized information from Google in our webmaster console. Should we need to contact you, you'll see a notification in your Webmaster Tools dashboard.


Initially the messages will refer to search quality issues, but over time we'll use the Message Center as a communication channel for more types of information. Here's an example: informing the site owner about hidden text, a violation in our webmaster guidelines.


For our webmasters outside the U.S., we’re also pleased to tell you that Message Center is capable of providing information in all supported Webmaster Tools languages (French, Italian, German, Spanish, Danish, Dutch, Swedish, Russian, Chinese-Simplified, Chinese-Traditional, Korean, Japanese, etc.), across all countries.

Right now the number of sites we’re contacting is small, but we hope to expand this program over time. We’re also really happy that the Message Center lets us communicate with webmasters in an authenticated way. As time goes on, we’ll keep looking for even more ways to improve communication with site owners, but right now, why not claim your site in our webmaster tools so that we can give you a heads-up of any issues that we see?

Given helpful suggestions from our discussion group, we've improved feedback for sitemaps in Webmaster Tools. Now, minor problems in a sitemap will be reported as "warnings," and will appear instead of, or in addition to, more serious "errors." (Previously all problems were listed as errors.) Warnings allow us to provide feedback on portions of your sitemap that may be confusing or inaccurate, while saving the real "error" alarm for problems that make your sitemap completely unreadable. We hope the additional information makes it even easier to share your sitemaps with Google.

The new set of warnings includes many problems that we had previously classified as errors, including the "incorrect namespace" and "invalid date" examples shown in the screenshot above. We also crawl a sample of the URLs listed in your sitemap and report warnings if the Googlebot runs into any trouble with them. These warnings might suggest a widespread problem with your site that warrants further investigation, such as a stale sitemap or a misconfigured robots.txt file.
Please let us know how you like this new feedback. Tell us what you think via the comments below, or in the discussion group. We also appreciate suggestions for additional warnings that you would find useful.

Written by Mark Berghausen, Search Quality Team

We occasionally get questions on the Webmaster Help Group about how webmasters should work with Adobe Flash. I thought it would be worthwhile to write a few words about the search considerations designers should think about when building a Flash-heavy site.

As many of you already know, Flash is inherently a visual medium, and Googlebot doesn't have eyes. Googlebot can typically read Flash files and extract the text and links in them, but the structure and context are missing. Moreover, textual contents are sometimes stored in Flash as graphics, and since Googlebot doesn't currently have the algorithmic eyes needed to read these graphics, these important keywords can be missed entirely. All of this means that even if your Flash content is in our index, it might be missing some text, content, or links. Worse, while Googlebot can understand some Flash files, not all Internet spiders can.

So what's an honest web designer to do? The only hard and fast rule is to show Googlebot the exact same thing as your users. If you don't, your site risks appearing suspicious to our search algorithms. This simple rule covers a lot of cases including cloaking, JavaScript redirects, hidden text, and doorway pages. And our engineers have gathered a few more practical suggestions:

1. Try to use Flash only where it is needed. Many rich media sites such as Google's YouTube use Flash for rich media but rely on HTML for content and navigation. You can too, by limiting Flash to on-page accents and rich media, not content and navigation. In addition to making your site Googlebot-friendly, this makes you site accessible to a larger audience, including, for example, blind people using screen readers, users of old or non-standard browsers, and those on limited low-bandwidth connections such as on a cell phone or PDA. As a bonus, your visitors can use bookmarks effectively, and can email links to your pages to their friends.
2. sIFR: Some websites use Flash to force the browser to display headers, pull quotes, or other textual elements in a font that the user may not have installed on their computer. A technique like sIFR still lets non-Flash readers read a page, since the content/navigation is actually in the HTML -- it's just displayed by an embedded Flash object.
3. Non-Flash Versions: A common way that we see Flash used is as a front page "splash screen" where the root URL of a website has a Flash intro that links to HTML content deeper into the site. In this case, make sure there is a regular HTML link on that front page to a non-Flash page where a user can navigate throughout your site without the need for Flash.
   

If you have other ideas that don't violate these guidelines that you'd like to ask about, feel free to ask them in the Webmaster Help Group under Crawling, Indexing, and Ranking. The many knowledgeable webmasters there, along with myself and a cadre of other Googlers, will do our best to clear up any confusion.

Update: See our additional blog posts about Flash Indexing at Google.

Note: The information in this post may be outdated. See our latest post about reporting spam.

Posted by Selina & Jos, Search Quality Team, Dublin

In the Dutch market, the concept of so-called 'startpages' is hugely popular. In this article we will give some background information on them, and give those of you who may be startpage webmasters a few tips on how to create unique and informative startpages.

What's a startpage?

Basically, it's a webpage with a lot of links about a specific topic. The startpages are hosted on a startpage domain and each separate startpage is maintained by an individual webmaster. The links on startpages are usually ordered by categories related to the topic of the page. Besides hyperlinks, startpages often contain text, animations and pictures. Startpages are quite unique to the Dutch market, and offer a simple interface for novice users to create their own web portals, with a unique approach to user-generated content.

The whole startpage concept began in September 1998 with the launch of Startpagina.nl, which was set up to be an online linkbook for the inexperienced Internet user. Since then, Startpagina.nl has become a huge success, mainly because an enormous number of volunteers created and maintained the different startpages covering lots of interesting and diverse topics. Since Startpagina.nl emerged, lots of other startpage domains have been created, and are still being created today. The fact that there are still new startpage domains appearing and that the number of individual startpages on these domains is still increasing shows the continued popularity of startpages in the Dutch market.

Creating useful startpages

As a search engine, we love to have useful and diverse pages showing up in the search results we present to our users. We thought it would be a good idea to highlight some of the best practices we've seen in creating value-added startpages.

1. Create your startpage for users, and not for search engines. This involves making sure that all your text on the page is visible to users, and writing full sentences as descriptions instead of just keywords.
2. Try to deliver unique, informative and on-topic content. The structure of startpages is pretty straightforward and does not leave much room for variation. However, you can make a difference. Try to find a topic you know a lot about that has not been fully covered yet. Create good categories that are related to your topic and give a relevant title to every category. Then, find links that are related to the categories on your page and label every link with an anchor text that is relevant. For example, instead of naming your links 'link1', 'link2' et cetera, you can choose names that make clear where the link is pointing to. And you can write a short description for every category.
3. Don't create startpages out of commercial intent or for the sole purpose of exchanging links. Of course there is nothing wrong with trying to monetize your startpage, but a page with only banners and affiliate links is not the best user experience and therefore not recommended. The same goes for startpages that are created as part of a link network. For example, pages that have all links pointing to a particular website and to other startpages that are also pointing to that same website. These kind of link schemes have no added value for the user and go against the Google webmaster guidelines.

With this post, we hope to have provided potential startpage webmasters with some helpful guidelines that will help to create the type of startpages the Dutch speaking people love!

On a final note, we would like to encourage you to fill in a paid links form if you come across a startpage that is involved in buying and selling links for the purpose of search engine manipulation. To report other forms of bad behavior, you can send a spam report. We'll review each report we get and use this feedback to enhance our algorithms and improve our search results. As always, we really appreciate your feedback and your help to provide the best search experience.

Startpagina's

Gepost door Selina en Jos, Search Quality Team, Dublin

Op de Nederlandstalige markt zijn de zogenaamde startpagina's bijzonder populair. In dit artikel willen we, naast het geven van wat achtergrondinformatie over startpagina's, toekomstige startpaginabeheerders een aantal tips geven voor het creëren van unieke en informatieve startpagina's.

Wat is een startpagina?

Een startpagina is een webpagina met een verzameling links gerelateerd aan een specifiek onderwerp. De startpagina's worden gehost op een startpagina domein en elke individuele startpagina wordt beheerd door een webmaster. De links op een startpagina zijn meestal opgedeeld in verschillende categorieën die relevant zijn voor het specifieke onderwerp van de startpagina. Naast een indeling in hyperlinks vind je op een startpagina vaak tekst, animaties en plaatjes. Het concept van startpagina's is redelijk specifiek voor de Nederlandstalige markt en komt nauwelijks voor in andere markten. Startpagina's hebben een simpele interface die het, ook voor de onervaren internetgebruikers, eenvoudig maakt om een eigen webpagina te creëren.

Het startpagina concept kwam tot stand in september 1998 met de lancering van Startpagina.nl, dat werd opgezet als een soort van linkboek voor de onervaren internet gebruiker. Startpagina.nl bleek al gauw een enorm succes. Dit succes was vooral te danken aan het enorme aantal vrijwilligers dat meehielp om startpagina's te creëren en beheren. Dat er nu, bijna negen jaar later, nog steeds nieuwe startpagina domeinen verschijnen en dat het aantal individuele startpagina's op deze domeinen nog steeds groeit toont aan dat de startpagina's onverminderd populair zijn.

Een waardevolle startpagina creëren

Als zoekmachine vinden we het fantastisch om waardevolle pagina's met unieke content en diversiteit in onze zoekresultaten te hebben. Het leek ons daarom een goed idee om een aantal tips te geven die kunnen helpen bij het creëren van startpagina's met toegevoegde waarde.

1. Maak een startpagina voor internetgebruikers en niet voor zoekmachines. Zorg dat alle tekst zichtbaar is en gebruik volledige zinnen in plaats van enkel een aantal keywords.
2. Probeer unieke, informatieve en aan je onderwerp gerelateerde inhoud aan je bezoekers te presenteren. Hoewel de opzet van een standaard startpagina niet heel veel ruimte biedt voor variatie, kun jij als beheerder het verschil maken! Begin met het zoeken naar een onderwerp waar je veel over weet en waar naar jouw idee nog niet genoeg informatie over te vinden is. Maak vervolgens relevante categorieën aan die gerelateerd zijn aan het onderwerp en geef elke categorie een relevante naam. Zoek vervolgens de links die je op je startpagina wil plaatsen en geef elke link een anchor tekst die omschrijft waar de link je bezoeker naar toe stuurt. Noem je links niet link1, link2, en link3, maar geef ze een naam die relevant is voor de inhoud van de pagina waar de link naar verwijst. Als extra aanvulling kan voor iedere categorie een korte beschrijving worden toegevoegd.
3. Maak geen startpagina's vanuit een puur commercieel oogpunt. Er is niets mis met te proberen om wat te verdienen met je startpagina, maar vergeet niet dat je bezoekers niet zitten te wachten op een pagina met alleen reclamebanners en affiliate links. Hetzelfde geldt voor startpagina's die enkel worden aangemaakt als onderdeel van een linknetwerk. Een voorbeeld hiervan zijn startpagina's waarbij alle links verwijzen naar eenzelfde website en naar andere startpagina's die ook allemaal naar dezelfde website verwijzen. Dit soort startpagina's hebben geen enkele waarde voor je bezoekers en gaan bovendien in tegen de Google Richtlijnen voor Webmasters.

We hopen dat we met deze eerste Nederlandstalige post potentiële startpaginabeheerders hebben kunnen voorzien van een aantal nuttige tips die er voor zorgen dat zij het soort startpagina's kunnen gaan creëren waar onze Nederlandstalige gebruikers van houden!

Tot slot willen we iedereen aanmoedigen om een paid link formulier in te vullen, wanneer je een startpagina tegenkomt die links koopt en verkoopt om daarmee zoekmachines te manipuleren. Andere zaken die ingaan tegen de Google Richtlijnen voor Webmasters kun je melden door een spamrapport in te sturen. Wij bekijken elk rapport dat wordt ingestuurd en deze informatie wordt gebruikt om onze algoritmes en zoekresultaten verder te verbeteren. Zoals altijd wordt jullie feedback en hulp om onze gebruikers te voorzien van de meest relevante zoekresultaten enorm gewaardeerd!

Posted by Luisella & Stefano, Search Quality Team, Dublin

Un paio di chiarimenti...

Ciao! Siamo appena rientrati da un breve soggiorno in Italia. Tempo fantastico! Abbiamo partecipato come spettatori al Search Engine Strategies conference a Milano nei giorni 29 e 30 maggio. La conferenza è stato davvero una fantastica opportunità per parlare con molti di voi! Ci ha fatto molto piacere esserci e vorrei ringraziare tutti quelli che si sono fermati semplicemente a salutare o a discutere di strategie dei motori di ricerca. Abbiamo avuto la possibilità di parlare con diversi dei partecipanti e con alcuni dei più importanti attori del mondo SEO e Web Search Marketing in Italia. Discussioni utili e fruttuose per molti aspetti. Si e' parlato di come il mercato Web si stia sviluppando in Italia, di strategie SEO e di evangelizzazione (la traduzione italiana suona veramente forte).

Un buon numero di voi è saltato fuori con domande interessanti, e mi piacerebbe ora esporre un caso per poi fornire un paio di chiarificamenti che siano chiari e concisi.

Allora partiamo. Questa è la situazione in cui un webmaster potrebbe ritrovarsi: ho ottimizzato questo sito utilizzando tecniche non in accordo con le linee guida di Google. Ce la siamo cavata per un po', e questo ci ha aiutato a raggiungere la seconda posizione nei risultati di ricerca per alcune parole chiave. Ad un certo punto però, abbiamo ricevuto una email dal team della qualità della ricerca di Google che diceva che il nostro sito non sarebbe stato momentaneamente più presente nell'indice (nelle email c'è sempre almeno un esempio delle tecniche utilizzate). Abbiamo allora sistemato il sito togliendo tutto ciò che non era conforme alle linee guida e dopo alcuni giorni il nostro sito era di nuovo presente nell'indice. Come è possibile che non è più posizionato in seconda posizione nonostante il fatto che abbiamo rimosso tutto ciò che non era conforme alle linee guida?!

Va bene, lasciatemi fare un paio di domande prima di rispondere.

• Non avete ottimizzato il sito utilizzando quelle tecniche al fine di posizionarlo il meglio possibile artificialmente?
• Non pensavate che quelle tecniche avrebbero funzionato, almeno in una prospettiva di breve periodo?

Quindi se c'è stato un utilizzo di tecniche spam, incoraggiamo il sito che ha ricevuto la notifica da Google a prendere la cosa seriamente. Molti ripuliscono il proprio sito dalle tecniche scorrette di ottimizzazione dopo aver ricevuto una nostra notifica, ma noi dobbiamo anche tenere in considerazione che oltre a quelle presenti sul sito (per esempio testo nascosto, redirecting doorway page, etc) spesso ci sono anche tecniche utilizzate al di fuori del sito stesso come link popularity artificiali per guadagnarsi un’ottima posizione nelle pagine dei risultati di ricerca di Google.

Quindi, per rendere la questione più chiara possibile, una volta che ognuna delle manipolazioni sopra citate, inserite ai fini del posizionamento, e’ stata rimossa, il sito torna ad occupare la posizione che merita sulla base dei suoi contenuti e della sua link popularity naturale. C'è in oltre da evidenziare che il posizionamento del vostro sito dipende anche dagli altri siti relazionati al vostro per argomento trattato e tali siti nel frattempo potrebbero essere stati ottimizzati correttamente, va da sé che questo avrebbe un impatto anche sulla vostra posizione.

Notate che non c’è alcun tipo di penalizzazione preventiva applicata a quei siti che, ora puliti, hanno però visto in precedenza un utilizzo di tecniche non consentite. E questo è un punto a cui teniamo particolarmente: non rimangono né malus né macchie nella storia di un sito.

E' per questo motivo che insistiamo fermamente nel consigliare di lavorare sodo sui propri contenuti in modo che siano una risorsa che abbia valore per gli utenti, essendo proprio il buon contenuto una delle risorse più importanti che alimentano una link populary naturale e tutti dovremmo ormai sapere quanto una tale popolarità possa essere solida.

Qualità della ricerca, qualità dei contenuti e l'esperienza dei tuoi lettori.

Tra le varie conversazioni sulla qualità della ricerca, una su tutte ricorreva più spesso. Mi riferisco alle landing page e come scrivere per i motori di ricerca, due temi che spesso viaggiano in coppia quando si parla di risultati organici di ricerca.

Pensiamo allora al tuo visitatore che ha cercato qualcosa con Google e ha trovato la tua pagina. Ora, che tipo di accoglienza gli stai riservando? Una buona esperienza di ricerca consiste nel trovare una pagina che contiene l'informazione necessaria per rispondere alla domanda posta all'inizio.

Tuttavia un errore frequente nello scrivere per motori di ricerca é dimenticare proprio il visitatore e focalizzare l'attenzione solo sulla sua domanda. In effetti potremmo sostenere, "é con quella chiave di ricerca che hanno trovato la mia pagina!"

Alla fine dei conti, esasperare un comportamento del genere potrebbe portare a creare pagine fatta "su misura" per rispondere a quella ricerca ma con ben poco contenuto. Pagine del genere spesso utilizzano tecniche quali, tra l'altro, pure ripetizioni di parole, contenuti duplicati e in generale minimo contenuto. Ricapitolando, possono anche essere a tema con la domanda posta - ma per il tuo visitatore, sono inutili. In altri termini, hai finito per creare una pagina scritta solo per i motori di ricerca e ti sei dimenticato del visitatore. Il risultato é che l'utente trova pagine all'apparenza a tema ma in realtà completamente insignificanti.

Queste pagine "insignificanti", fatte artificialmente per generare traffico dai motori, non rappresentano una buona esperienza di ricerca. Anche se non adottano tecniche scorrette, quali ad esempio testo o links nascosti, sono fatte solo ed esclusivamente per posizionarsi per specifiche parole chiave, o combinazioni di parole, ma in realtà non offrono autonomamente alcun valore come risultato di una ricerca.

Un primo approccio per capire se stai causando una cattiva esperienza di ricerca ai tuoi utenti é controllare che le pagine trovate siano davvero utili. Queste pagine avranno contenuto a tema, che risponde alla domanda originalmente posta dall'utente ed in generale sono significative e rilevanti. Potresti cominciare con il controllo delle pagine che ricevono più visite e passare poi a rivedere tutto il sito. E per concludere, un consiglio: in generale, anche quando si vuole ottimizzare la pagina affinché il motore la trovi facilmente, bisogna ricordarsi che i visitatori sono il tuo pubblico e che una pagina scritta per i motori di ricerca non soddisfa necessariamente le aspettative del visitatore in termini di qualità e contenuti. Allora se stai pensando a come scrivere per il motore di ricerca, pensa invece ai tuoi utenti e a qual é il valore che stai offrendo loro!

We're back from SES Milan!

...with a couple of clarifications

Ciao everybody! We just got back from Italy—great weather there, I must say! We attended SES in Milan on the 29th and 30th of May. The conference was a great opportunity to talk to many of you. We really had a good time and want to thank all the people who stopped by to simply say "hi" or to talk to us in more detail about search engine strategies. This gave us a chance to talk to many participants and many of the big Italian actresses and actors in the SEO and web marketing worlds. We discussed recent developments in the Italian internet market, SEO strategies and evangelizing.

A number of you have raised interesting questions, and we'd like to go through two of these in more detail.

This is a situation a webmaster might find himself/herself in: I optimized this site using some sneaky techniques that are not in accordance with Google´s Webmaster Guidelines. I got away with it for a while and it helped me to rank in second position for certain keywords. Then, suddenly, I got an email from Google saying my site has been banned from the index because of those techniques (in these emails there is always an example of one of the infractions found). I now have cleaned up the site and after some days the site was back in the index.
Why on earth doesn't my site rank in the second position anymore, even though I've already paid for the sneaky techniques we used?

OK, before answering let me ask you a couple of questions:

• Didn't you optimize your site with those techniques in order to artificially boost the ranking?
• Didn't you think those techniques had worked out (in a short term perspective at least)?

So, if there has been spamming going on, we encourage a site that has gotten an email from Google to take this notification seriously. Many people clean up their sites after receiving a notification from us. But we must also take into account that besides the shady SEO techniques used on a particular site (for instance hidden text, redirecting doorway pages, etc) there are often off-site SEO techniques used such as creating artificial link popularity in order to gain a high position in Google's SERPs.

So, to make it straightforward, once those manipulations to make a site rank unnaturally high are removed, the site gains the position it merits based on its content and its natural link popularity. Note that of course the ranking of your site also depends on other sites related to the same topic and these sites might have been optimized in accordance to our guidelines, which might affect the ranking of your site.

Note that a site does not keep a stain or any residual negative effect from a prior breach of our webmaster guidelines, after it has been cleaned up.

That is why we first and foremost recommend to work hard on the content made for the audience of your site, as the content is a decisive factor for building natural link popularity. We all know how powerful a strong natural link popularity can be.

Search quality, content quality and your visitor's experience.

During our conversations about search-related issues, another topic that came up frequently was landing pages and writing for search engines, which are often related when we consider organic search results.

So, think of your visitors who have searched for something with Google and have found your page. Now, what kind of welcome are you offering? A good search experience consists of finding a page that contains enough information to satisfy your original query.

A common mistake in writing optimized content for search engines is to forget about the user and focus only on that particular query. One might say, that's how the user landed on my page!

At the end of that day, exaggerating this attitude might lead to create pages only made to satisfy that query but with no actual content on them. Such pages often adopt techniques such as, among others, mere repetition of keywords, duplicate content and overall very little value. In general, they might be in line with the keywords of the query – but for your visitor, they’re useless. In other words, you have written pages solely for the search engine and you forgot about the user. As a result, your visitor will find a page apparently on topic but totally meaningless.

These “meaningless” pages, artificially made to generate search engine traffic, do not represent a good search experience. Even though they do not employ other not recommendable techniques, such as for examples hidden text and links, they are very much made solely for the purpose of ranking for particular keywords, or a set of keywords, but actually are not offering a satisfying search result in itself.

A first step to identify if you are causing a bad search experience for your visitor consists of checking that the pages that he or she finds are actually useful. They will have topical content, that satisfies the query for which your visitor has found it and are overall meaningful and relevant. You might want to start with the pages that are most frequently found and extend your check up to your entire site. To sum up, as general advice, even if you want to make a page that is easily found via search engines, remember that the users are your audience, and that a page optimized for the search engine does not necessarily meet the user's expectations in terms of quality and content. So if you find yourself writing content for a search engine, you should ask yourself what the value is for the user!

Posted by Ríona MacNamara, Webmaster Tools Team

In den letzten Tagen gab es nochmals Versuche, deutsche Webmaster durch falsche E-Mails von Google zu verunsichern. Diese E-Mails stammen nicht von Google. Seit einigen Wochen hat Google die Benachrichtigung von Webmastern durch E-Mails eingestellt. Google arbeitet derzeit an einem zuverlässigeren Webmaster-Kommunikationsprozess.

We've noticed that someone is again trying to spoof the emails that Google sends to webmasters to alert them with issues about their site. These emails are not coming from Google, and in fact several weeks ago we temporarily discontinued sending these emails to webmasters while we explore different, secure ways of communicating with webmasters. Watch this space for more news - but in the meantime, you can safely assume that any email message you receive is not, in fact, from us.