Forum Discussion

aheywoo's avatar
aheywoo
Adept
11 years ago

SharePoint crawl...

I'm trying to do a search crawl from SharePoint on all our Lithium communities.

We currently crawl verious external sites, but in each case we have access to the underlying database.  Is it possible to get read-only access to the database for our communities so that I could do the same in this case?

 

Alternatively, I could pull all the data using the REST API, but I have a some questions:

I see that there is a 25 message limit when getting messages.  Can this be increased?

Is there a limit (throttling of any sort) as to how much data we can pull at one time or when sending multiple requests one after the other?

 

And then for the incremental crawl thereafter, is it possible to get all new/modified messages after a supplied date?

 

Thanks for any help.

 

 

3 Replies

Sort By
  • YuriK's avatar
    YuriK
    Khoros Expert
    11 years ago

    Hey Aheywoo,

     

    As far as I know, we don't give any access to the DB. 

     

    As for the REST API, you can control the size of a page using the page_size parameter (our default maximum is 1000). You can see this in the pagination section of the RestApi Response Documentation. We don't currently do any throttling as far as I know, see this post for a discussion on throttling.

     

    There are a few ways to see messages ordered by date, you can take a look at the Recent Posts call as an example of this. Let me know if this call doesn't meet your needs and I will provide alternatives.

     

    Hope this helps,

     

    Yuri

     

     

  • PaoloT's avatar
    PaoloT
    Lithium Alumni (Retired)
    11 years ago

    Hi aheywoo 

     

    In addition to the excellent pointers YuriK already gave you, have a look at the posts stream call , which has a position parameter. With the position parameter you should be able to create a tool that pulls posts incrementally. Please note that streams support must be enabled by our support department, if needed.

     

    Thanks,