A few months back, I blogged that Google was becoming more choosy on the pages it crawled and that having a few high-quality pages on a site might be a better strategy than a lot of low-quality pages.
Our suggested solution was to add unwanted URLs and directories to your robots.txt file. However, that’s an imperfect solution because search engines don’t always obey the directions in robots, nor do they act quickly.
Fortunately, there’s a new, improved solution….
In late April, Google announced that it was adding a URL Removal tool to its Webmaster console.
The URL Removal Tool
Late last night, I decided to give it? a try and entered some directories that robots.txt had failed to remove:
The Results
When I woke up in the morning and Google was telling me its job was finished:
I checked the SERPs and it was true – the pesky URLs really were gone. The URL removal tool had done more good in 8 hours than robots.txt had in 8 weeks.
I’d still suggest using robots.txt, particularly to deal with Yahoo and MSN, but Google’s URL removal works and works fast.
?