Google indexing in a closed community

[lcgolden]

Since we have a closed community, we have the challenge of figuring out how to allow Google searches to find TKB articles.  The idea is that someone who isn't logged in to our Community does a Google search, finds an article, but is prompted to log in/create an account before being taken to the article.  

The way we manage our permissions is to have a default role that allows all community members to search the TKB. The Default role allows only access to a collection of public information.

I've been poring through the permissions but don't see anything that would give us this outcome and am hoping someone else has some widsom you can share.

[PaoloT]

The idea is that someone who isn't logged in to our Community does a Google search, finds an article, but is prompted to log in/create an account before being taken to the article.

Hi @lcgolden

as far as I am aware - google (and other search engines) only index content that is accessible to the web crawler bot. The bot is equivalent to an anonymous user, so the content should be viewable for an anonymous user to be crawled and indexed by the bot. It's probably (I haven't checked but I would expect so) also part of the search engines's guidelines to only index content that is publicly accessible.

You could probably do a customization to display a call to action to invite the user to log in or register, but they shouldn't be prevented from accessing the content if that content has been indexed by a search engine.

Alternatively, you could think of publishing public, "teaser" content articles (that can be indexed) and then keep the full content in a private area only accesible upon registration.

Hope it helps,

[OlivierS]

@lcgolden I've no idea if that would work or not, but try allowing bots crawling your TKB folder by updating your robots.txt file

[lcgolden]

Thanks for the input! I'll see if this can be done by modifying robots.txt. We actually do have a few teaser articles already, but the goal at this point is to have all articles indexed.

[PaoloT]

Hi @lcgolden

robots.txt is usually tweaked to prevent Google (and other search engines) to prevent unintentional access/indexing of "private" areas. It is not used the other way around.

If the article is not visible for an anonymous user, it will not be indexed and this is exactly how it's supposed to work. If the content is indexed, it is not private anymore by definition.