I’ve talked before about how having less pages indexed is often better and after doing SEO audits on several sites last week I was reminded of how important this was. Sites with only 200 or 300 genuine pages of content had over 1000 pages indexed in Google.
Lots of Joomla components produce 100s and sometimes 1000s of unnecessary extra URLs that are then indexed by Google. A simple example is Mostree which is a very robust component but because it was built in pre-Ajax days, it creates separate 4 extra pages for each listing: recommend, rate, review and report listing.
In this post we’re using JReviews, Fireboard and MosTree as examples. You can cut-and-paste the rules from this post into your robots.txt file. However, please read the "important notes" section first.
Important notes
1) If you add a rule that contains wildcards (*) then you are removing all the URLs from Google that contain that word. For example:
- If you use JReviews for a mouthwash review site, then Disallow: /*list*/ will remove all the unnecessary lists but also your opinion on Listerine.
- If you use Mosets Tree for file listings, then Disallow: /*contact*/ will remove some surplus contact pages but also your information about Jodie’s Fosters so-so Sci-Fi Movie called Contact.
2) These rules are for sites with Search Engine Friendly URLs turned on. Change the rules to fit your own uppercase / lowercase settings.
3) Be careful. Robots.txt files can have a big impact on your site.
Mosets Tree
Disallow: /*review*/
Disallow: /*rate*/
Disallow: /*recommend*/
Disallow: /*report*/
Disallow: /*Owner*/
Disallow: /*Contact*/
Disallow: /*List_Alpha*/
Disallow: /*New_Listing*/
Disallow: /*Featured_Listing*/
Disallow: /*Popular_Listing*/
Disallow: /*Most_Reviewed*/
Disallow: /*Most_Favoured*/
Disallow: /*Top_Rated*/
Disallow: /*Most_Rated*/
Disallow: /*Recently_Updated*/
Disallow: /*Add_Category*/
Disallow: /*Add_Listing*/
JReviews
Disallow: /*list*/
Disallow: /*myreviews*/
Disallow: /*reviewrank*/
Disallow: /*rss*/
Disallow: /*order*/
Disallow: /option=com_jreviews/
Fireboard
Disallow: /profiles/
Disallow: /*action=reminder*/
Disallow: /*action=stats*/
Disallow: /*;msg*/
Any Kind of Calendar or Photo Gallery
Disallow: /calendar/
Disallow: /photogallery/
For SEO Club members we’re building a longer list covering more components. Click here to view.