Forum Discussion

kevinutoft's avatar
kevinutoft
Contributor
4 years ago

API Content and User Permissions

Hello, I am working on a solution to build out an external search index with Khoros being one source of many. I having been trying unsuccessfully to use the API to extract out all content and users ...
  • snaffle's avatar
    snaffle
    4 years ago

    Hi Kevin,

    I don't actually remember that being an issue for us and can't find any reference to it in emails/documentation so you might be surprised.

    Having said that, it was fairly early on in our community and we may have made the decision to not worry about historical date, but I can't remember for sure... I have a feeling that Inbenta was able to do a spider crawl of the existing community for historical data.

    What I would have a look at though, is Khoros' Bulk Data API tool - https://developer.khoros.com/khoroscommunitydevdocs/reference/bulk-data-api - which provides an interface for exporting large amounts of data from the system.

    It does though limit each export to 7 days of data so I'd imagine you'd want to look at writing a script to automate that depending on how much historical data you have... not sure if it's the right solution for you but it may help.

    Cheers

    Nathan

snaffle's avatar
snaffle
Expert
4 years ago

Hi there,

I have in the past used Khoros as a search source for the Inbenta federated search engine.

The way we did it then and I believe is still the recommended way of doing it is to use the "Subscribe to Community events" feature of the API... doing this we were able to pull in every post and reply made on the community.

You can have a look at the Docs on that here - https://developer.khoros.com/khoroscommunitydevdocs/docs/subscribe-to-community-events

We were also able to develop a custom script for Inbenta in PHP which queried the API for custom user field data which was then matched against their posts.

Hope that helps.

Cheers

Nathan

  • kevinutoft's avatar
    kevinutoft
    Contributor
    4 years ago

    Nathan,

    Thanks for the reply. I had not considered the subscription as an option. It appears that using this would help in a lot of ways. By only making the API calls we need for a single post/message it would greatly reduce the time it takes and the timeout issues I was experiencing. 

    However, after looking over the documentation, I don't believe this would capture any current/existing messages in the databases to date, it would only trigger on new messages. In that case I still think I would need a way to do a mass export of the nature I was describing above. 

    Was there another way you handled posts/messages that already existed in the database?

    Thanks for your help,

    Kevin

    • snaffle's avatar
      snaffle
      Expert
      4 years ago

      Hi Kevin,

      I don't actually remember that being an issue for us and can't find any reference to it in emails/documentation so you might be surprised.

      Having said that, it was fairly early on in our community and we may have made the decision to not worry about historical date, but I can't remember for sure... I have a feeling that Inbenta was able to do a spider crawl of the existing community for historical data.

      What I would have a look at though, is Khoros' Bulk Data API tool - https://developer.khoros.com/khoroscommunitydevdocs/reference/bulk-data-api - which provides an interface for exporting large amounts of data from the system.

      It does though limit each export to 7 days of data so I'd imagine you'd want to look at writing a script to automate that depending on how much historical data you have... not sure if it's the right solution for you but it may help.

      Cheers

      Nathan