Forum Discussion

luk's avatar
luk
Boss
6 years ago

API & FreeMarker Issue Tracker

As there is no public issue tracker (afaik) or reporting workflow (other than support tickets and ideas), I'd like to collect some things that were found trough hour-long trial end error (some including workarounds) so others trying to customize the platform hopefully don't have to jump trough the same hoops as I had to or at least might be able to find some information or workarounds regarding things that don't behave as advertised or expected.

This topic is NOT a place to rant about the platform, the devs or the product (it's software after all, and software has bugs, we know that very well...).

I will make a new reply to this post for each issue and link them in the topic message for easier access.

Hope it helps somebody!

PS: If you know about something similar and it's not yet listed below, please feel free to add your own reply to the original topic message using the template below as a rough guideline of what information to provide. If you'd like to comment on a bug, try to reply to that specific "issue" message (so technically a threaded view could be aggregated, it's not right here, I know...) and reference (via permalink!) the issue you're commenting on, otherwise it's going to be very confusing =)...
______


Issues:

  • #1: API v2 function count(topic) does not work.
  • #2: API v2 response structure should be normalized.
  • #3: FreeMarker context object apiv2.toJson() fails with "augmented" API response objects.

 

<!-- Issue Template -->
<p>
	<strong>Issue</strong>: #<br />
	<strong>Type</strong>: BUG | INCONSISTENCY | FEATURE<br />
	<strong>Area</strong>: REST API v1 | REST API v2 | FreeMarker<br />
	<strong>Collection</strong>: API collection if applicable<br />
	<strong>Platform Version</strong>: &lt;= xx.x (if fixed &gt;= xx.x)<br />
	<strong>Summary</strong>: Summary sentence goes here.<br />
	<strong>Ping</strong>: @mention a user if applicable<br />
</p>

<p><strong>ISSUE SUMMARY</strong>: Short summary of issue, what is expected and what is actually happening.</p>

<p>Detailed description/explanation/workaround</p>

 

  • Issue: #1
    Type: BUG
    Area: REST API v2
    Collection: messages
    Platform Version: <= 19.7
    Summary: The function count(topic) does not work as documented.
    PingSuzieH 

    The docs state that "count(topic): The count of topics on a community, category, or board", which would theoretically allow a query like:

     

    // query1
    SELECT count(topic) FROM messages WHERE board.id = '<board.id>'

     

    if the above worked, it would shorten/make more readable the "regular" query of getting a topic count on a node, which at the same time is the workaround:

     

    // query2
    SELECT count(*) FROM messages WHERE depth = 0 AND board.id = '<board.id>'

     

    ISSUE: The expected output from query1 would be the SAME count as returned by query2, but the count returned is the same as if depth = 0 was omitted, e.g. the total count of messages in the node.

  • Issue: #3
    Type: BUG
    Area: REST API v2 | FreeMarker
    Collection: all
    Platform Version: <= 19.7
    Summary: FreeMarker context object apiv2.toJson() fails with "augmented" API response objects.
    Ping: DougS

    ISSUE SUMMARY: If an API v2 response object is modified (after fetching it) with FreeMarker, the apiv2.toJson() method fails with an error.

    Due to a variety of shortcomings with API v2 we still have to rely on API v1 calls either because the data is completely unavailable with v2 or due to performance reasons (e.g. v1 is much faster than aggregating the same data with v2).

    As we try to work primarily with API v2, we often fetch the necessary data with API v1 and then modify an existing API v2 response object with the data from v1 (this is the "augmenting" part mentioned in the summary, can also be thought of as "extending" the object or overwriting existing properties/values).

    Doing this helps keeping the code more consistent/reusable (because field names in v1 and v2 differ in many cases which would complicate things when reading properties/values while rendering the markup).

    The problem I came along when doing this kind of response object "augmentation" relates to the FreeMarker context object apiv2 which has a toJson() method that allows printing the response object as JSON which is very useful for debugging, e.g. seeing what is in the actual response within the component which can be different from what we get in the API Browser due to permission differences when logged in as an Administrator.

    Trying to use apiv2.toJson() on a modified response object will throw the following error:

    No compatible overloaded variation was found; declared parameter types and argument value types mismatch.
    The FTL type of the argument values were: extended_hash (wrapper: f.c.AddConcatExpression$ConcatenatedHashEx).
    The Java type of the argument values were: freemarker.ext.beans.HashAdapter.
    The matching overload was searched among these members:
    - lithium.template.FreeMarkerRestMethods$RestV2ResponseParserMethod.toJson(List),
    - lithium.template.FreeMarkerRestMethods$RestV2ResponseParserMethod.toJson(lithium.api.lion.RestCallTemplateModel)

    Tip: You seem to use BeansWrapper with incompatibleImprovements set below 2.3.21. If you think this error is unfounded, enabling 2.3.21 fixes may helps. See version history for more.

    A practical example/use case of this technique is the following:

    Goal: We need the total count of kudos for a topic, that includes kudos given to replies!

    API v2 unfortunately is inconsistent here with regards to v1 as there is NO direct way (at least afaik) to get the sum of all kudos for a topic aggregated from topic message and replies to it. Although indirectly this is still doable with v2 by:

    1. Fetching the kudos from all messages related to a topic with:

     

    SELECT kudos.sum(weight) FROM messages WHERE topic.id = '4701' AND kudos.sum(weight) > 0

     

    2. looping trough all the kudos objects and summing up their kudos values with:

     

    <#assign kudos = 0 />
    <#list response.data.items as obj>
        <#assign kudos = kudos + obj.kudos.sum.weight?number />
    </#list>

     

    BUT the v2 call ALONE (without the looping) is roughly 15x slower than fetching the same value with API v1:

     

    <#assign kudos = rest('/threads/id/4701/kudos/count').value?number />

     

    Considering we most likely do that for each topic in a collection, a performance discrepancy like this quickly adds up and seems unnecessary. An ideal-world solution would be if we could simply specify topic.kudos.sum(weight) (which we can today, it just does NOT return the total count but only the kudos count of the topic message...) and get the aggregated value in one API call with all the other needed data. => FEATURE REQUEST

    I hope that explains the performance part and why we are modifying v2 response objects, which works fine on a data level (e.g. we have no issues accessing the "augmented" properties/values when looping over the response object for rendering markup.

    A simple example to reproduce the issue at hand:

     

    <#-- fetch total kudos count for topic with v1 -->
    <#assign query = "/threads/id/4701/kudos/count" />
    <#assign count = rest(query).value?number />
    
    <#-- fetch topic message object with v2 -->
    <#assign query = "SELECT id, kudos.sum(weight) FROM messages WHERE id = '4701'" />
    <#assign response = liql(query) />
    
    <#-- augment the response object with total kudos count from v1 call -->
    <#assign response = response + {
    	'data': {
    		'items': [
    			response.data.items[0] + { 'kudos': {'sum': {'weight': count}} }
    			]
    	}
    } />
    
    <#-- accessing the overwritten value works -->
    ${response.data.items[0].kudos.sum.weight?number}
    
    <#-- but this throws an error -->
    <#attempt>
    	${apiv2.toJson(response)}
    <#recover>
    	${.error}
    </#attempt>

     

    • SuzieH's avatar
      SuzieH
      Khoros Alumni (Retired)

      Hi luk Thank you very much for compiling this. I will make sure this data gets into the right hands. To all, please be sure to file Support Cases for these issues.

       

      • PAULEM's avatar
        PAULEM
        Advisor

        Hi SuzieH 

        Can we pin luk's issue tracker post to the top of the Developer Discussion board feed so that we don't lose it as other posts arrive?  This is a great initiative!

        Just a thought ...

         

  • Issue: #2
    Type: INCONSISTENCY
    Area: REST API v2
    Collection: all
    Platform Version: <= 19.x
    Summary: API v2 error responses should contain an empty items array/sequence.
    Ping: DougS

    ISSUE SUMMARY: It would be much easier and result in less code bloat and code repetition if API v2 error responses would contain an items property like the successful responses, the value could simply be an empty sequence/array.

    The meta-structure of a successful API v2 response looks like this:

     

    {
      "status" : "success",
      "message" : "",
      "http_code" : 200,
      "data" : {
        "type" : "messages",
        "list_item_type" : "message",
        "size" : 1,
        "items" : [
          { ... },
          { ... }
        ]
      },
      "metadata" : { }
    }

     

    while an error response looks like this:

     

    {
      "status" : "error",
      "message" : "Invalid query syntax",
      "data" : {
        "type" : "error_data",
        "code" : 604,
        "developer_message" : "...",
        "more_info" : ""
      },
      "metadata" : { }
    }

     

    it is easy to see that there are several inconsistencies in the response structure that MUST then be handled with FreeMarker or JS when dealing with the response which leads to unnecessary checks/error handling (not because of the actual error, but because of the structure of the response object!) and subsequently code-bloat which could be simply avoided by having a consistent (even if meaningless) response structure, e.g. the properties of a successful response should always be present, also if there is an error, like:

     

    {
      "status" : "error",
      "message" : "Invalid query syntax",
      // because this is the http code the browser actually gets,
      // but it's missing on error responses!
      "http_code" : 200,
      "data" : {
        "type" : "error_data",
        "code" : 604,
        "developer_message" : "...",
        "more_info" : "",
        "size": 0, // just give us a 0 result size
        "items": [] // just give us an empty sequence, no results
      },
      "metadata" : { } // out of interest, is this ever used?
    }

     

    This seems to be a rather easy to implement suggestion because it just ADDS properties to the error response, so hopefully not many backward-compatibility issues should arise.

  • SuzieH   Khoros Freemarker version is a bit out-dated. It does not support Array?map function. Maybe Khoros should upgrade to the latest stable version? It is currently using FreeMarker 2.3.26-incubating

     

    2.3.31
    2.3.30
    2.3.29
    2.3.28
    2.3.27 (incubating at Apache)
    2.3.26 (incubating at Apache)
    2.3.25 (incubating at Apache)
    2.3.24 (incubating at Apache)
    2.3.23

    • SuzieH's avatar
      SuzieH
      Khoros Alumni (Retired)

      Good question peterlu
      RayC is a FreeMarker upgrade anywhere on our Community roadmap?

  • Issue: #?
    Type: BUG
    Area: REST API v2
    Collection: settings
    Platform Version: >= 22.12?
    Summary: Per Khoros Support, Community 22.12 broke the API V2 field-setting capability.
    Ping: not sure who's in charge

    ISSUE SUMMARY: Kudos to keithkelly for reporting that publicly, it seems that if you do a query like SELECT * FROM settings WHERE id = 'c_custom_field_name' AND user.id = '409' won't work properly, not confirmed for all types of requests, but at least POST requests to update a setting value seem to be affected. If someone has more info, please let me know. For some reason I also can't find the settings collection anymore in the API v2 docs, I was pretty sure it was there at some point, was this removed?

    • Details about this in case it helps anyone who stumbles across it:

      Bug ID: 00407790   Submitted 2/10/23

      Setting Custom Board Property (is no longer working)

      Description:

      I have some PostMan scripts that I used a couple months ago to set the "c_dc_scope" board property successfully.  However, this is no longer working, even when I set the user to have Administrator permissions.

      This acts successful, but the value isn't updated

       

      Workaround provided by Khoros Support (and worked) was to use:

       

      curl --location --request POST 'https://[COMMUNITY URL]/restapi/vc/boards/id/[BOARD ID]/settings/name/custom.dc_scope/set?value=foo' \
      --header 'Accept: application/json' \
      --header 'Content-Type: application/json' \
      --header 'li-api-session-key: SESSION_KEY' \
      --data-raw '{
      "data": {
      "type": "board",
      "c_dc_scope": "foo"
      }
      }' 

       

       

  • This post is great and really helpful! Thank you mate!

    • luk's avatar
      luk
      Boss

      Allow me to disagree, we (normal users) do not have the permissions to edit/update TKB articles here, so we couldn't change it anymore.

  • <#assign qry = "SELECT id FROM images WHERE visibility = 'public' AND owner.id = '100' limit 1" />
    
    <#assign response = (restBuilder().asIdentityUser('/users/id/100').liql(qry).data.items[0])!{} />
    response user 100 image = ${response.id}

     

    I am getting a permission error. If I remove "asIdentityUser('/users/id/100')", then it is working fine. I am testing this endpoint code in incognito mode.

    It is complaining about some permission issue when I test it in incognito mode.

    Any ideas? The logic is fetch user 100's public image, run the api as user identity 100.

  • Hello peterlu,

    That's expected behavior if the currently authenticated user doesn't have the switch user permission, and using an Incognito Window would mean they're anonymous.