Forum Discussion

elbranscomb's avatar
elbranscomb
Executive
3 years ago

Bulk Data API - new user questions

Hi, we have just started using the Bulk Data API to get data from production and our team have a couple of questions that hopefully are simple to answer. We can't find the answers in the docs or here so sorry if these are covered somewhere else. 

1. fromDate and toDate is what we’re using to fetch the data. Just not entirely sure if it maps to the timestamp of event.time.ms or if it’s another timestamp (i.e. when event was processed, when event was made available for export, etc).
2. Does this endpoint paginate results in any way or is it always one giant dump?

Thanks!

Bulk Data API
  • JavidH's avatar
    JavidH
    3 years ago
    1. You are right. The fromDate and toDate are mapped to the event.time.ms timestamp.
      event.time.ms - represent the time of user activity occurred in the community (in UTC).
    2. Bulk API won’t support pagination, it is always whole data served as a stream for the requested timestamp.

    Hope this helps.

7 Replies

Sort By
  • JavidH's avatar
    JavidH
    Khoros Staff
    3 years ago
    1. You are right. The fromDate and toDate are mapped to the event.time.ms timestamp.
      event.time.ms - represent the time of user activity occurred in the community (in UTC).
    2. Bulk API won’t support pagination, it is always whole data served as a stream for the requested timestamp.

    Hope this helps.

  • MarkSchwanke's avatar
    MarkSchwanke
    Boss
    3 years ago

    elbranscomb It's been some years since I used sql. Wondering if you know of any document or trick for setting up connection to do some simple queries locally. I looked for a little walk me through it document here but didn't find one unless I'm missing something.

  • MarkSchwanke's avatar
    MarkSchwanke
    Boss
    3 years ago

    I’m looking to find out how to set up database on my local machine and do some queries.  I just couldn’t find any steps on selecting application and then where to get the settings to connect the database to run the queries.  Once I get it set up I’m sure I’ll remember how to do the queries after a little practice.  

  • SudhakaraS's avatar
    SudhakaraS
    Khoros Alumni (Retired)
    3 years ago

    Hi MarkSchwanke Bulk Data API supports data exports in CSV and JSON formats. Once you download data in CSV format,  it is easy to import CSV data to any SQL data system. There are many external tools available across different operating systems for the same.