Forum Discussion

prasath's avatar
prasath
Helper
2 years ago

API v2 Pagination Error

Hi, I'm new to the Khoros community and I'm attempting to retrieve users data from February 2023 until the present using pagination with cursor which has around 1.6 million records. The pagination...
prasath's avatar
prasath
Helper
2 years ago

 

Hi vamsiu  Thank you for your response. I'm currently facing an issue with extracting 1.6 million records efficiently. Fetching 1000 records per request with my script seems to be taking too much time. I've discovered that the API supports bulk data retrieval, and I've attempted to implement it using the Python example provided in the documentation.
Document1 
Document 2 
However, I'm encountering difficulties in retrieving the specific fields I need, such as id, sso_id, and last_visit_time from the response also getting huge volume of data for one day.

Regarding the second document, I'm unable to utilize it as I lack the necessary privileges to obtain the username and password required to instantiate the Khoros object.

Also how can i use 'https://[COMMUNITY DOMAIN]/api/2.0/search?' API to get bulk data,  is this different than 'https://api.lithium.com/lsi-data/v2/data/export/community/ea.prod' ?

vamsiu , could you please assist me with this? Below is the script I'm currently using for bulk data retrieval.

 

 

import requests
import json
access_token = 'access_token'  # Place the "Bulk API access token" from Community Analytics here
client_id = 'client_id' # Place the "Client ID" from Community Analytics here
response = requests.get(
    'https://api.lithium.com/lsi-data/v2/data/export/community/ea.prod',
    params={'fromDate': '20240201', # Set the start date in YYYYMMDD format
           'toDate': '20240202'},   # Set the end date in YYYYMMDD format
    auth=(access_token, ''),
    headers={'client-id': client_id, # TODO: place the "Client ID" from Community Analytics here
             'Accept': 'application/json' }  # OPTIONAL: this could also be set to "application/json" to receive JSON results, or even left unset to receive CSV
	)
data = response.json() # Use "response.json()" incase you need json response (set 'Accept' header as mentioned above).


with open('ahq_user_data.json', 'w', encoding='utf-8') as f:
    json.dump(data, f, ensure_ascii=False, indent=4)

 

 

 

vamsiu's avatar
vamsiu
Ace
2 years ago

Hi prasath ,

I see you're facing challenges with efficiently extracting 1.6 million records and fetching 1000 records per request is taking too long. You've attempted to implement bulk data retrieval using the Python example provided in the documentation. 

1. Bulk data retrieval provides all the data within the specified parameters, which means it does not inherently support fetching specific fields. You would need to retrieve the entire dataset and then filter out the specific fields you need from the response data.

2. If you're unable to instantiate the Khoros object due to lacking privileges, consider reaching out to Khoros support. They can assist you in obtaining the necessary access credentials or provide guidance on how to proceed. Requesting a Khoros support ticket can help address issues related to access permissions and authentication.   

If You have required permissions   Then try Below procedure: 

Solution:

  1. Get Bulk Data API Connection Information:

    • Sign into the Community Analytics (formerly known as Lithium Social Intelligence or LSI) user interface.
    • Click on your username in the top-right corner to retrieve your connection information, including:
      • Community ID
      • Client ID
      • Access Token

  2. Supply Connection Information in Python SDK:

    • There are two ways to supply this information:
      • During Instantiation: Supply the connection information when creating the Khoros object. 

 

bulk_data_settings = {
    'community_id': 'example.prod',
    'client_id': 'ay0CXXXXXXXXXX/XXXX+XXXXXXXXXXXXX/XXXXX4KhQ=',
    'token': '2f25XXXXXXXXXXXXXXXXXXXXXXXXXa10dec04068',
}
khoros = Khoros(defined_settings=settings, bulk_data_settings=bulk_data_settings, auto_connect=False)​

 

   3. Query the Bulk Data API:

  • Use the query() method with from_date, to_date, and export_type parameters to perform queries.
  • Example:

 

results = khoros.bulk_data.query(from_date='20221025', to_date='20221101', export_type='json')​

 

   4. Manipulate Retrieved Data:

  • After querying, manipulate the data using methods like:
    • Filtering by user type
    • Filtering by action
    • Counting actions, logins, and views    

 

Thank You! 

  • prasath's avatar
    prasath
    Helper
    2 years ago

    Hi vamsiu , I appreciate your assistance.

    Could you please provide guidance on how to achieve bulk data retrieval for the users collection? My understanding is that the two APIs I mentioned yield different results:

    Is there an API available to retrieve bulk data specifically from the users collection?
    basically  I want to achieve the bulk data retrieval for the below request

    https://answers.ea.com:443/api/2.0/search?q=SELECT id, sso_id, email, banned, deleted FROM users WHERE last_visit_time > 2023-02-01T00:00:00 order by id ASC limit 1000