Our commuinty had an archive of old content. While the content is not displayed in the navigation of the commuinty,customers are still viewing it. Is there something that we should do with the robots.txt file to prevent this?
if the content has already been indexed by the SE I believe the robots.txt won't by of any use.
As a reminder, hiding the board from the navigation will only prevent the board from appearing from any menu or board list. It's content will still be searchable.
You should remove the content manually from the SE console.
Another option would be to make the content private so that the crawler won't be able to crawl the board (hence it should remove the URLs from the cache).
Welcome to the Technology board!Curious about our platform? Looking to connect on social technology? You've come to the right place!