Push Button Paradise

Yahoo! Site Explorer is a great tool, and simple too. I don't have any inside information, but I'm pretty sure a more advanced version is coming at some point.

How is it used? Let's look at a few quick examples. Say your web site isn't getting indexed very well by Slurp, the friendly neighborhood Yahoo! crawler. Go to Site Explorer and enter in your site's main URL. You'll see what I call and "index eye view" of your site. Like this.

I see that at the moment, 385 of my pages are available, as are 273 inlinks. I can quickly scan through both lists, as well as filter by omitting subdomains.

Note that a common mistake is to throttle the crawler too much. Something around 1 document per second should be ideal. Longer delays will likely result in large chunks of your site getting skipped.

Another example beyond vanity searches: Let's say you hear about a site called annualcreditreport.com. Is it legit? Site Explorer helps you answer.

Look at who's linking to that site: 34,683 inlinks--possibly a good sign. Just on the first page, a half-dozen of the inlinks come directly from ftc.gov, the Federal Trade Commission. Clicking through to one of those links shows a definitive answer:

Your Access to Free Credit Reports

Here are the details about your rights under the FCRA and the Fair and Accurate Credit Transactions (FACT) Act, which established the free annual credit report program.

Other features, like partial URLs work about like you'd expect. Try a few experiments. There's also an API to hack against.

-m

posted at: 00:18 | under: 2006-01 | 0 comment(s)