Forum Discussion

hphemanth's avatar
hphemanth
Contributor
10 years ago

Daily Batch Processing and Incremental Updates

We are using this URL to get the data from Lithium server using REST APIs.

String url = "http://XXXX.xx.com/restapi/vc/boards/id/"+categoryType+"/search/messages?

q=date%3A%5B1402951097%20TO%201405543103%5D

&collapse_discussion=false

&sort_by=-date

&page_size=1000

&page=1

&restapi.response_format=json

&restapi.format_detail=full_list_element

&restapi.response_style=view";

 

Whenever we change the date , we get different output.

Questions :

1. Based on our new approach, we decided to hit the REST API as a dialy batch job to reterive the result and process the output everyday. How do we do it. what would be the API URL.

2. We are limiting the certain amount of records per call everytime we hit the Lithium Server.

Is there a way to get incremental updates ( everytime, I pass the request, can I able to get the data from where I left last time)

 

Thanks,

Hemanth

 

 

  • PaoloT's avatar
    PaoloT
    Lithium Alumni (Retired)
    10 years ago

    Hi hphemanth 

     

    did you have a look at the streams APIs ? You may need to enable them for your community via Support. They allow you to read the stream of posts (or topics) from a given timestamp. This may be more suitable for your needs.

     

    Thanks,

    • hphemanth's avatar
      hphemanth
      Contributor
      10 years ago

      Hi Paolo ,

      You mean the timestamp for each day, as I'm going to run it as a daily batch jobs.

      can you give me an example URL.

      Will timestamp have a From and To parameters

       

      We currently use

      String url = "http://<myurl.com>/restapi/vc/boards/id/"+categoryType+"/search/messages?
      q=date%3A%5B1402951097%20TO%201405543103%5D&
      collapse_discussion=false
      &sort_by=-date
      page_size=1000
      &page=1

       

      Can you please suggest, how we can change the current URL for the given Timestamp.