Forum Discussion

synthesis's avatar
12 years ago

How do I get older/more records with the REST API?

I work for HP. We have a gazillion product forums. I wanted to harvest a bunch of data with a call like this:

 

http://h30499.www3.hp.com/restapi/vc/categories/id/bsc-412/posts/recent?page_size=1000&page=1&restapi.response_format=json&restapi.format_detail=full_list_element&restapi.response_style=view

 

The max I can get at a time is 1000 records. If I keep increasing the "page" parameter it starts returning me zero results when page equals 3.

 

What if I want to pull 10,000 records?

 

If 2000 is the max, how do I get to older posts because 2000 records only takes me back to June of this year for the products I'm interested in.

 

Is there another way to bulk request more records?

 

I understand why you'd want to have a cap, but how do I work around it?

 

Thank you!

9 Replies

  • If you put your logic in a loop like this, you can make y as large as you want. It will automatically end the loop when you run out of data.

     

    <#list 1..3 as y>
    <#assign username = restadmin("/users?page_size=1000&page=${y}") />

    ....

    </#list>

     

    This may not be the optimal way to do this. I'm very new at this, but it works.

  • synthesis's avatar
    synthesis
    Guide
    12 years ago

    >> If you put your logic in a loop...

     

    That is is exactly what I'm doing. It doesn't work for me.

     

    We have two Lithium boards I'm working with: One for commerical products and one for consumer. They BOTH return 0 results on any call with "page_size" greather than two (3 or greater). I KNOW there are more records than 2000 because we've been using Lithium for a long time and have products that go back many years.

     

    This still smells of an API cap.

     

    What else can I try/do?

  • dementad's avatar
    dementad
    Genius
    12 years ago

    I just ran the loop on a subset of my users in an endpoint and got 2794 records back. I'm not running into this limit.

  • synthesis's avatar
    synthesis
    Guide
    12 years ago

    Hmmm....

     

    Maybe I'm doing something wrong. Again, it works on the first two calls but returns 0 on everyone after that. Here's the calls I'm making:

     

    for (int i = 1; i <= LITHIUM_CONSUMER_API_CALLS; i++)
    {
        ProcessLithiumData("http://h30434.www3.hp.com",
            String.Format(@"restapi/vc/categories/id/InkJet/posts/recent?page_size={0}&page={1}&restapi.response_format=json&restapi.format_detail=full_list_element&restapi.response_style=view", LITHIUM_RECORDS_PER_CALL, i));
    }

     

    Any ideas? If I issue the calls from a browser I get the same result: First two work, next ones all fail.

     

    Could be their is no cap on listing users. Can you run against a category/board of yours and see if you can pull more than 2000?

     

    Notice: This is HP's consumer forum that contains all the Photosmart, Deskjet, Officejet, and Laser printers.

     

     

  • dementad's avatar
    dementad
    Genius
    12 years ago

    synthesis I will see if I can pull *all messages", but I'm doing my call with an end point and not exactly as you are. Specifically, I ran this query on our very small private forum (I knew I had more than 2000 but less than 3000 registered users). As I mentioned before this returned more than 2000 users.

     

    <#--
    Get list of registered users without the Staff role = registered customers
    -->
    <#list 1..3 as y>
      <#assign username = restadmin("/users?page_size=1000&page=${y}") />
      <#list username.users.user as x>
        <#assign has_role = false />
        <#list restadmin("/users/id/${x.id}/roles").roles.role as role>
          <#if role.name?? && (role.name == "Staff")>
            <#assign has_role = true />
          </#if>
        </#list>
        <#if (has_role == false)>
          ${x.login}</br>
        </#if>
      </#list>
    </#list>

     

  • YuriK's avatar
    YuriK
    Khoros Expert
    12 years ago

    Hey Synthesis,

    I think you will have to open a support ticket for this. There's a config set that limit's the full set of result to 2000 overall at the moment. You will need to let us know what you want the limit to be. (Performance degradation happens pretty quickly once you get further down the pages, so I would suggest setting the limit to something around 5000 or whatever you want)

     

    There's a workaround for this without needing the config change, however as you go further and further down pages, performance will decrease exponentially. In fact, by page 6 I am seeing significant delay (4 seconds or so).

     

    The workaround is to iterate over the boards in the category and get the threads from each board. For example I used the following to get to page 3: http://h30499.www3.hp.com/restapi/vc/categories/id/bsc-412/boards/id/bsc-420/threads?page_size=1000&page=3&restapi.response_format=json&restapi.format_detail=full_list_element&restapi.response_style=view

    Hope this helps,

    Yuri

  • synthesis's avatar
    synthesis
    Guide
    12 years ago

    Yuri,

     

    Thank for your help on this!

     

    What is the configuration parameter that needs to be set in order to increase the 2000 limit? I have access to the person who handles this for HP, I just need to tell him WHAT to set (and HOW to set it).

     

    ~Synthesis

  • YuriK's avatar
    YuriK
    Khoros Expert
    12 years ago

    Hey Synthesis,

     

    This is something you will need to file a support ticket for. Let them know that they can contact me if they are unsure about which configuration needs to be changed.

     

    Thank you,

     

    Yuri

  • YuriK's avatar
    YuriK
    Khoros Expert
    12 years ago
    Hey Synthesis,

    You should be all set up to get up to 10,000 messages.

    Yuri