Forum Discussion

aheywoo's avatar
aheywoo
Adept
11 years ago

Search crawl of Forums taking a long time

Hi, we're trying to do a search crawl and index our Forums content using the Google Search Appliance and also the SharePoint search crawler.

We noticed that the crawls were taking a long time (over 30 hours) and that an incremental crawl takes as long as a full crawl.  The incremental crawl should only re-index forum posts that were modified since the last time it crawled them so it's not working properly.

We've narrowed this down to the fact that the 'Last-Modifed' field is missing from each Forum post's response header, so the crawler doesn't know if the page has been modified or not.

Here's an example repsonse header from one of our forum posts: 

 

 Lithium2.png

 

And here's one from another website that does include the Last-Modified field.

 

Lithium1.png

 

This means that the search crawler has to download the whole page and re-index it every time, which causes resource issues and the 30 hour crawl.

Incidentally, this is also the field that a browser uses to determine if the page needs to be downloaded again, of if it can use the one held in it's own cache, so it really should be there.

Do you know if the 'Last-Modified' date can be switched on in the reponse header at all?

Is this a known bug in the forums?

Cheers.

4 Replies

Sort By
  • aheywoo's avatar
    aheywoo
    Adept
    11 years ago

    We opened a support case for this and got this back:

     

    “...it has been determined that the platform does not offer this feature and there is not a clear workaround.”

     

    So how do we get this onto the radar for a future release?  It should be an easy fix and will benefit everyone using the forums, not just search crawlers.

     

    As a help, if I was writing it in .NET (the only code I know) and the pages were being rendered dynamically, then I would use HttpResponse.AddHeader to add the required header.

    I don't know what the Lithium forums are written in so can't help you there.  Anyone...?

     

     

  • PaoloT's avatar
    PaoloT
    Lithium Alumni (Retired)
    11 years ago

    Hi aheywoo 

     

    have you considered posting this idea in the Customer Ideas Exchange ? In that way, it can get more visibility with the product management and get more votes within the community.

     

    I am not part of the engineering department so I cannot really comment on the implementation steps, however based on my personal experience it is likely to be more involved than a one liner change - taking into account that you need to calculate the right date to insert into the header for each page served etc...

     

    Thanks for your suggestion!

  • aheywoo's avatar
    aheywoo
    Adept
    11 years ago
    Thanks Paulo, I'll post it there.

    And yes, you're right. It certainly wouldn't be 1 line; but still, not a hard fix.
  • JasonL's avatar
    JasonL
    Lithium Alumni (Retired)
    11 years ago

    hi aheywoo 

    Honestly haven't tried but one possibility is to use the Page Init scripts

    http://community.lithium.com/t5/Developers-Knowledge-Base/Using-a-FreeMarker-Page-Initialization-Script/ta-p/125767

     

    Conceivably, it should be possible to add custom code in Page Init to:

    • check if you're on the Forum Topic Page
    • if so, make rest api call to get the message of current topic, last update time
    • use the http.response freemarker context object to add the Last-Modified header to the response

    http://community.lithium.com/t5/Developers-Knowledge-Base/http-response-FreeMarker-context-object/ta-p/9325

     

    Suggest reach out to Lithium Professional Services (via Support or your Account Executive) for deeper feasibility/analysis.