Forum Discussion

prasath's avatar
prasath
Helper
9 months ago

API v2 Pagination Error

Hi, I'm new to the Khoros community and I'm attempting to retrieve users data from February 2023 until the present using pagination with cursor which has around 1.6 million records. The pagination...
vk-9's avatar
vk-9
Ace

Hi prasath,

Thank you for sharing your script and the issues you've encountered. It seems that the error messages you're receiving are related to the pagination limit in the URL. The limit you've set is currently at 10,000 records per page, which might be causing the server to return a 403 Forbidden error or a 504 Gateway Time-out, especially after several iterations.

To address this, I would recommend adjusting the limit in the URL to 1,000 records per page, which aligns with the pagination best practices for the Khoros community API. This change should help alleviate the issues you're facing with the pagination process.

Additionally, please keep in mind the rate limits imposed by the Khoros community API. Requests to the Author and Conversation APIs are limited to 60 calls in 60 seconds, while requests to the Analytics Reports API are rate-limited to 20 requests per minute. Ensuring that your script adheres to these rate limits will help prevent rate-limiting errors and ensure smooth API interactions.

For more information on rate limits and pagination with the Khoros community API, you can refer to the official documentation at the following URLs:

prasath's avatar
prasath
Helper
9 months ago

 

Hi vk-9  Thank you for your response. I'm currently facing an issue with extracting 1.6 million records efficiently. Fetching 1000 records per request with my script seems to be taking too much time. I've discovered that the API supports bulk data retrieval, and I've attempted to implement it using the Python example provided in the documentation.
Document1 
Document 2 
However, I'm encountering difficulties in retrieving the specific fields I need, such as id, sso_id, and last_visit_time from the response also getting huge volume of data for one day.

Regarding the second document, I'm unable to utilize it as I lack the necessary privileges to obtain the username and password required to instantiate the Khoros object.

Also how can i use 'https://[COMMUNITY DOMAIN]/api/2.0/search?' API to get bulk data,  is this different than 'https://api.lithium.com/lsi-data/v2/data/export/community/ea.prod' ?

vk-9 , could you please assist me with this? Below is the script I'm currently using for bulk data retrieval.

 

 

import requests
import json
access_token = 'access_token'  # Place the "Bulk API access token" from Community Analytics here
client_id = 'client_id' # Place the "Client ID" from Community Analytics here
response = requests.get(
    'https://api.lithium.com/lsi-data/v2/data/export/community/ea.prod',
    params={'fromDate': '20240201', # Set the start date in YYYYMMDD format
           'toDate': '20240202'},   # Set the end date in YYYYMMDD format
    auth=(access_token, ''),
    headers={'client-id': client_id, # TODO: place the "Client ID" from Community Analytics here
             'Accept': 'application/json' }  # OPTIONAL: this could also be set to "application/json" to receive JSON results, or even left unset to receive CSV
	)
data = response.json() # Use "response.json()" incase you need json response (set 'Accept' header as mentioned above).


with open('ahq_user_data.json', 'w', encoding='utf-8') as f:
    json.dump(data, f, ensure_ascii=False, indent=4)

 

 

 

  • vk-9's avatar
    vk-9
    Ace
    9 months ago

    Hi prasath ,

    I see you're facing challenges with efficiently extracting 1.6 million records and fetching 1000 records per request is taking too long. You've attempted to implement bulk data retrieval using the Python example provided in the documentation. 

    1. Bulk data retrieval provides all the data within the specified parameters, which means it does not inherently support fetching specific fields. You would need to retrieve the entire dataset and then filter out the specific fields you need from the response data.

    2. If you're unable to instantiate the Khoros object due to lacking privileges, consider reaching out to Khoros support. They can assist you in obtaining the necessary access credentials or provide guidance on how to proceed. Requesting a Khoros support ticket can help address issues related to access permissions and authentication.   

    If You have required permissions   Then try Below procedure: 

    Solution:

    1. Get Bulk Data API Connection Information:

      • Sign into the Community Analytics (formerly known as Lithium Social Intelligence or LSI) user interface.
      • Click on your username in the top-right corner to retrieve your connection information, including:
        • Community ID
        • Client ID
        • Access Token

    2. Supply Connection Information in Python SDK:

      • There are two ways to supply this information:
        • During Instantiation: Supply the connection information when creating the Khoros object. 

     

    bulk_data_settings = {
    'community_id': 'example.prod',
    'client_id': 'ay0CXXXXXXXXXX/XXXX+XXXXXXXXXXXXX/XXXXX4KhQ=',
    'token': '2f25XXXXXXXXXXXXXXXXXXXXXXXXXa10dec04068',
}
khoros = Khoros(defined_settings=settings, bulk_data_settings=bulk_data_settings, auto_connect=False)​

     

       3. Query the Bulk Data API:

    • Use the query() method with from_date, to_date, and export_type parameters to perform queries.
    • Example:

     

    results = khoros.bulk_data.query(from_date='20221025', to_date='20221101', export_type='json')​

     

       4. Manipulate Retrieved Data:

    • After querying, manipulate the data using methods like:
      • Filtering by user type
      • Filtering by action
      • Counting actions, logins, and views    

     

    Thank You! 

    • prasath's avatar
      prasath
      Helper
      9 months ago

      Hi vk-9 , I appreciate your assistance.

      Could you please provide guidance on how to achieve bulk data retrieval for the users collection? My understanding is that the two APIs I mentioned yield different results:

      Is there an API available to retrieve bulk data specifically from the users collection?
      basically  I want to achieve the bulk data retrieval for the below request

      https://answers.ea.com:443/api/2.0/search?q=SELECT id, sso_id, email, banned, deleted FROM users WHERE last_visit_time > 2023-02-01T00:00:00 order by id ASC limit 1000