Forum Discussion

tmarshall's avatar
tmarshall
Advisor
9 years ago

Bulk Data API - Too Many Fields Error

Hello Lithium Developers and NaokiT,

 

More questions about the Bulk Data API and how to best use it so it will not break our mapping structure.

 

If we use the curl script and specify all the available fields (52 of them) I receive the following error.

{"code":"InvalidArgument" message:"Too many fields in parameter"}

 

1) If I do not specify the fields will the order always stay the same? Probably not as new fields are added (i.e. sso.user.id was added recently)? You can see in the attached that since Nov 2015 there have been more fields added the the structure has changed. 

 

2) I figured out that I can specify 20 fields in the CURL script, but anything over 20 will result in the error. Is there a CURL script to specify the fields we do NOT need? I would not need 6 of 52 fields for example.

 

How can we maintain the structure and not have our file break if new fields are added in the future? Is there a way to specify or keep the structure intact so we do not have to maintain our mapping structure if/when Lithium adds more fields?

 

3) If we have to run 3 separate scripts so we can get the specificity we need? What would be the unique key we can use to merge all the data back together? However, the best case would be ability to specify all fields and not have to merge the fields later? Otherwise, can you clarify how often new fields are added to the build data api?

 

Thanks,

Tim

 

 

 

 

 

  • NaokiT's avatar
    NaokiT
    9 years ago

    tmarshall

     

    1)  While we do not often add columns or change the order, it will happen on occasion.  We disclose these changes in the Lithium release notes, but I also try to post the on the Bulk Data API TKB a few weeks before it happens.  We strongly recommend that you not count on field order to map your data, but instead do the mapping by header name.  The header names have never changed.  So ingest the full file - preferably as a json file by adding "Accept: application/json".  Then parse it on your side for the fields that you need.  Or if you're using some sort of BI tool there should be some function like HLookup in Excel.

     

    2) That's correct you are capped at 20, but inclusive only and not exclusive.  We can expand the number of fields if that helps, but the exclusive feature will probably not come till later this year.

     

    3) If you have to do the mappings across 3 exports than document.id is the common key you can join across all data.  But document.id will only be in files from March 4, 2016

  • NaokiT's avatar
    NaokiT
    Lithium Alumni (Retired)
    9 years ago

    tmarshall

     

    1)  While we do not often add columns or change the order, it will happen on occasion.  We disclose these changes in the Lithium release notes, but I also try to post the on the Bulk Data API TKB a few weeks before it happens.  We strongly recommend that you not count on field order to map your data, but instead do the mapping by header name.  The header names have never changed.  So ingest the full file - preferably as a json file by adding "Accept: application/json".  Then parse it on your side for the fields that you need.  Or if you're using some sort of BI tool there should be some function like HLookup in Excel.

     

    2) That's correct you are capped at 20, but inclusive only and not exclusive.  We can expand the number of fields if that helps, but the exclusive feature will probably not come till later this year.

     

    3) If you have to do the mappings across 3 exports than document.id is the common key you can join across all data.  But document.id will only be in files from March 4, 2016

    • tmarshall's avatar
      tmarshall
      Advisor
      9 years ago

      Thanks NaokiT. I'll work with our BI guy to understand if we can map to name as this sounds like the best way to have the script to continue to work if/when items are added on the Lithium side.

       

      Can you please clarify where to put the JSON text in the following script and what that looks like? I've put it in a few places and have not been successful.

       

      curl https://api.lithium.com/lsi-data/v1/data/export/community/mycompany.prod?fromDate=201603150000&toDate=201603160000  -H "client-id: cooluniquekeyforustogetdata=" -u "secondcooluniquekey:" >> C:\Users\tmarshall\bulk_test\prod_20160315-test1.txt