API v2 Pagination Error

Hi prasath ,

I see you're facing challenges with efficiently extracting 1.6 million records and fetching 1000 records per request is taking too long. You've attempted to implement bulk data retrieval using the Python example provided in the documentation. 

1. Bulk data retrieval provides all the data within the specified parameters, which means it does not inherently support fetching specific fields. You would need to retrieve the entire dataset and then filter out the specific fields you need from the response data.

2. If you're unable to instantiate the Khoros object due to lacking privileges, consider reaching out to Khoros support. They can assist you in obtaining the necessary access credentials or provide guidance on how to proceed. Requesting a Khoros support ticket can help address issues related to access permissions and authentication.   

If You have required permissions   Then try Below procedure: 

Solution:

1. Get Bulk Data API Connection Information:

   • Sign into the Community Analytics (formerly known as Lithium Social Intelligence or LSI) user interface.
   • Click on your username in the top-right corner to retrieve your connection information, including:
     • Community ID
     • Client ID
     • Access Token

2. Supply Connection Information in Python SDK:

   • There are two ways to supply this information:
     • During Instantiation: Supply the connection information when creating the Khoros object. 

bulk_data_settings = {
    'community_id': 'example.prod',
    'client_id': 'ay0CXXXXXXXXXX/XXXX+XXXXXXXXXXXXX/XXXXX4KhQ=',
    'token': '2f25XXXXXXXXXXXXXXXXXXXXXXXXXa10dec04068',
}
khoros = Khoros(defined_settings=settings, bulk_data_settings=bulk_data_settings, auto_connect=False)​

   3. Query the Bulk Data API:

• Use the query() method with from_date, to_date, and export_type parameters to perform queries.
• Example:

results = khoros.bulk_data.query(from_date='20221025', to_date='20221101', export_type='json')​

   4. Manipulate Retrieved Data:

• After querying, manipulate the data using methods like:
  • Filtering by user type
  • Filtering by action
  • Counting actions, logins, and views    

Thank You!