Forum Discussion

tmarshall's avatar
tmarshall
Advisor
9 years ago

Bulk Data API - Too Many Fields Error

Hello Lithium Developers and NaokiT,   More questions about the Bulk Data API and how to best use it so it will not break our mapping structure.   If we use the curl script and specify all the av...
  • NaokiT's avatar
    9 years ago

    tmarshall

     

    1)  While we do not often add columns or change the order, it will happen on occasion.  We disclose these changes in the Lithium release notes, but I also try to post the on the Bulk Data API TKB a few weeks before it happens.  We strongly recommend that you not count on field order to map your data, but instead do the mapping by header name.  The header names have never changed.  So ingest the full file - preferably as a json file by adding "Accept: application/json".  Then parse it on your side for the fields that you need.  Or if you're using some sort of BI tool there should be some function like HLookup in Excel.

     

    2) That's correct you are capped at 20, but inclusive only and not exclusive.  We can expand the number of fields if that helps, but the exclusive feature will probably not come till later this year.

     

    3) If you have to do the mappings across 3 exports than document.id is the common key you can join across all data.  But document.id will only be in files from March 4, 2016