Forum Discussion

prasath's avatar
prasath
Helper
9 months ago

Pagination in Search API v2

Hi,
I'm using Search API v2 to get users detail and below is my python script.

import requests
url = "https://[COMMUNITY DOMAIN]/api/2.0/search?q=SELECT%20distinct%20id%20FROM%20users%20where%20last_visit_time%20%3E%3D%202024-02-01T00%3A00%3A00%20order by%20id%20asc"
headers = {'Authorization': '<access_token>'}
response = requests.get(url, headers=headers)

I successfully fetched the results using the Search API v2 to obtain user details. Furthermore, I received the next_cursor token for pagination. However, when attempting pagination using this token, I encountered an issue where subsequent requests with the cursor parameter appended to the URL or passed as a parameter yielded the same results as the initial API call.

1. adding cursor parameter to the url itself.
url = "https://[COMMUNITY DOMAIN]/api/2.0/search?q=SELECT%20distinct%20id%20FROM%20users%20where%20last_visit_time%20%3E%3D%202024-02-01T00%3A00%3A00%20order by%20id%20asc&cursor=<next_cursor>"
2. pass next_cursor as parameter
request.get(url,headers, params={'cursor':<next_cursor>})

Could someone assist me in resolving this issue, or perhaps suggest an alternative approach to retrieving all the records?

  • betty4920taylor's avatar
    betty4920taylor
    Contributor
    7 months ago

    Hello prasath,

    It looks like you’re facing a common issue with cursor-based pagination. When using the next_cursor token, it’s important to ensure that each subsequent request is fetching new data rather than repeating the initial results.

    import requests

# Initialize variables
base_url = "https://[COMMUNITY DOMAIN]/api/2.0/search"
query = "SELECT distinct id FROM users where last_visit_time >= '2024-02-01T00:00:00' order by id asc"
headers = {'Authorization': '<access_token>'}
params = {'q': query}
next_cursor = None

# Function to fetch data with pagination
def fetch_data(cursor):
    if cursor:
        params['cursor'] = cursor
    response = requests.get(base_url, headers=headers, params=params)
    if response.status_code == 200:
        data = response.json()
        next_cursor = data.get('next_cursor')
        users = data.get('users')
        # Process your data here
        return next_cursor, users
    else:
        print(f"Error: {response.status_code}")
        return None, []

# Loop to handle pagination
while True:
    next_cursor, users = fetch_data(next_cursor)
    if not users or not next_cursor:
        break
    # Add your logic to handle the users data

    This script uses a function to handle the fetching of data and includes the cursor in the parameters only if it exists. The loop continues to call this function with the new next_cursor until there are no more records to fetch.

    Remember to replace [COMMUNITY DOMAIN] with the actual domain and <access_token> with your valid authorization token. Also, ensure that the next_cursor is being correctly extracted from the API response and that the API endpoint supports cursor-based pagination as expected.

    If you continue to face issues, it might be helpful to check the API documentation for any specific requirements or limitations regarding pagination.