Forum Discussion

dfeasey's avatar
dfeasey
Mentor
2 years ago

How to use a Cursor in LiQL

For those of you struggling to find out information about how to use Cursors to return large datasets from Khoros, I offer the following example.

This is a Python function, but concept will work in other functional languages.

Idea here is to recursively call the function from within the same function until one is on the last "page" and then it exits the loop and returns the aggregated results.

Let me know if this is helpful!

 

def get_kdata(q="SELECT id, login, email, rank.name FROM users \
WHERE roles.name = 'Moderator'",c=None, lmt=50): #query string, cursor, limit
    nq= q +" LIMIT "+ str(lmt)
    if(c):
        nq = nq +" CURSOR '" + c +"'"
    resp = khoros.query(nq)

    if(resp["status"]=="success"):
        itms = resp['data']['items']
        size = int(resp['data']['size'])

        while(size == lmt):
            print("page of " + str(lmt) + "...")  # ... and counting
            try: #recurse with next cursor
                c = resp['data']['next_cursor']  
                kd = get_kdata(q,c,lmt)
            except: #no remainder so no next_cursor in response
                break

            [itms.append(i) for i in kd['items']]
            c = kd['cursor']
            return ({'items': itms, 'cursor': c})

        return({'items': itms, 'cursor': c})

    return False

 

  • It occurs to me that additional error handling may be needed. Let's say you set the limit to 50 and there are exactly 500 items.

    On the last page you would meet the criteria of size == lmt, but no next_cursor would be included in the response. So should probably wrap that in a try statement.

    *UPDATED* the code above to reflect a fix

    • dfeasey's avatar
      dfeasey
      Mentor

      This approach should also work with BulkData. I'm currently pulling each day into a separate file and aggregating with Power BI.