Aurora SEO: Regulate content crawling by search engines using robots.txt
Updated 5 months ago
Version 5.0Hello, I have read through the article, but im still not 100% confident on making a custom rule.
We have found that google is returning results for posts that we have archived and we dont want this to happen.
Is it possible to disallow certain boards through this feature
Is 'googlebot' all that is needed for google searches and then I would have the / URL related to the specific boards I want removed from search?
Thanks in advance for the help