Forum Discussion

vaishnavi's avatar
vaishnavi
Expert
10 years ago

Archive boards should not get listed in search engine results

Hi ,

I have a separate category "Archive" for some boards which are not supported(and archived) now.

The archive boards seem to have strong SEO attached to them, so users are still arriving at these boards, even though they are no longer supported.

 

Could anyone guide me on how to hide some boards from SEO crawlers?

Any pointers?

 

Thanks,

Vaishnavi

 

  • VarunGrazitti's avatar
    VarunGrazitti
    Boss
    10 years ago

    vaishnavi - You could get your robots.txt updated and this should solve the purpose. This document should help you. This doc will guide you to basic robots.txt knowledge.

     

    The robots.txt lives on the root, and here is the URL for robots file for Lithium community: http://community.lithium.com/robots.txt

     

    yourcomunity.com/robots.txt

     

    I hope this helps.

     

     

  • Fellsteruk's avatar
    Fellsteruk
    Boss
    10 years ago

    Hi,

     

    Is your "Archive" category public or private?, if public then its gonna get crawled, ive had some sucsess with Google Webmaster tools, do you use this? You can feed google with URL's you want to be removed from their index, you could add the now archived boards, this should help reduce the organic search traffic.

     

    Thanks  

  • vaishnavi's avatar
    vaishnavi
    Expert
    10 years ago

    Thank you Fellsteruk, VarunGrazitti, OlivierS for your help.

     

    So there are 2 ways to hide boards from SEO crawlers - robots.txt and google's webmasters tools.

    Both the ways would require manual data entry by community admin/moderator.

     

    Is there any way where community admin can configure the robots.txt from settings? 

    This would be easy for community admin to update robots.txt when a new board is moved udner "Archive" category.

     

    how to enable configurable robots.txt in admin?

    Any pointers?

     

    Regards,

    Vaishnavi

     

    • VarunGrazitti's avatar
      VarunGrazitti
      Boss
      10 years ago

      vaishnavi - This cannot be done via admin as this is not a Lithium exclusive thing. You'd need to submit your robots.txt file to lithium support and they will update it on the server for you.

       

      For your second question, you wouldn't have to update the robots.txt every time, you just need to disallow Archive category in it and moving forward, everything inside it would be disallowed. Whenever you archive a board, it will be inheriting disallowed property from Archive category. So, it is just a one time change. 

       

      I hope this helps.