Knowledge Base Article

Aurora SEO: Regulate content crawling by search engines using robots.txt

When you publish content in the community, search engines (web robots or web crawlers) crawl these newly published pages to discover and gather information from them. After crawling the content, th...

Updated 5 months ago

Version 5.0

Helper

6 months ago

Hello, I have read through the article, but im still not 100% confident on making a custom rule.

We have found that google is returning results for posts that we have archived and we dont want this to happen.

Is it possible to disallow certain boards through this feature
Is 'googlebot' all that is needed for google searches and then I would have the / URL related to the specific boards I want removed from search?

Thanks in advance for the help