Forum Discussion

snaffle's avatar
snaffle
Expert
10 years ago

Advice on performance for external API query

Hi all,

I'm looking for a bit of advice on improving the performance of a very simple API call that I've used to develop a Joomla CMS Module in PHP for use on a client's main website.

The purpose of the module is to display the 3 most recent posts for a specified "Custom Tag" which the Community admins apply to various posts.

We have two categories in our community, and the module displays posts from both categories so there are 2 API calls as follows:

http://saneforums.org/restapi/vc/categories/id/lived-experience-forum/posts/for/metadata/key/modbar.anxiety/value/true?message_viewer.topic_sort_order=last_post_date&restapi.format_detail=full_list_element&restapi.response_style=view&page_size=3

and

http://saneforums.org/restapi/vc/categories/id/carers-forum/posts/for/metadata/key/modbar.anxiety/value/true?message_viewer.topic_sort_order=last_post_date&restapi.format_detail=full_list_element&restapi.response_style=view&page_size=3

Each API call is loaded in to a separate PHP variable which I then use PHP's Simple XML Parser on to extract the data I need to display in the module.

You can look at the module in action here (down on the right hand side, heading "From SANE Forums"):

https://www.sane.org/mental-health-and-illness/facts-and-guides/anxiety-disorder

The problem is that the pages which have the module on them are loading very slowly at around 8 to 10 seconds while all of other pages load instantly. The site is hosted on a pretty serious setup at Rackspace so it's not a general hosting issue, and as I said it only happens on pages with the module assigned.

So I can only assume that the page load delay is coming from the query to the Lithium API and I'm wondering if anyone has any advice on that front.

One option I've thought of is to have a separate script activated by Cron which calls the API and generates a static XML file which the module would access instead of doing the real-time query - does that sound like a possible solution?

Would appreciate any advice that anyone might have.

Thanks

Nathan

  • luk's avatar
    luk
    10 years ago

    snaffle a few elaborations on the points made above:

     

    a) Cache the output of your Lithium API call, to do that you have to create an endpoint where you could also merge the data from the two calls into one dataset and then cache it for later reuse, which you do with something like that 

     

    <#assign data = appcache.set("externalfeed", yourdatainavariable) />

    Depending on how frequent the data should refresh, you have to contact support so they can adjust (lower) the cache expiry time for the appcache. In your endpoint you can then check if there is something stored for that cache-key and if so return that, else make the API calls and regenerate the data, e.g.

     

    <#if appcache.get("externalfeed", "")?has_content>
        <#-- output your data, be aware that you cannot just print freemarker node objects, so you have to convert them somehow to json/xml, which is not easy btw... -->
    <#else>
        <#-- re-fetch the data and put it into the cache -->
        ...
        <#assign data = appcache.set("externalfeed", yourdatainavariable) />
    </#if>

    But this is basically putting the work from your PHP script to Lithium which requires some FreeMarker/Lithium API knowledge.

     

    If we say that is not available then options to maybe slightly improve the performance is

     

    a) to simply use AJAX/JavaScript to fetch the content from the PHP script after the page has initially loaded and inject it to the right place, so the main content will be there while the AJAX call is getting the "slow" content from Lithium. This would be the easiest solution but doesn't actually change the performance, it just changes the perception =).

    b) Go away from parsing the response XML in PHP and just use the restapi.response_format=json URL parameter for your API calls that will return JSON instead of XML, which you can parse much more efficiently in PHP with json_decode(data, true); but I guess this will not significantly improve the performance of the whole thing...so just a micro-optimization and more elegant solution.

     

    if you have FreeMarker knowledge you can try to use API v2 message resource together with the restd context object inside an endpoint, the restd context object can actually return the data as a JSON string inside FreeMarker code (but you will not be able to modify the fetched data when doing that!) and API v2 would maybe allow to combine your two calls into one with an appropriate query, which means you can just cache and output the response without further worries about how to output it...

     

     

     

3 Replies

  • OlivierS's avatar
    OlivierS
    Lithium Alumni (Retired)
    10 years ago

    snaffle some ideas on top of my head:

     

    - check if you could cache the data (there are some information about caching somewhere on this community)

    - make the use of LiQL from our new API v2 (I think your calls are made with our v1)

    - use AJAX (if not already) to get / display the data, so it won't impact too much your page load

     

     

  • luk's avatar
    luk
    Boss
    10 years ago

    snaffle a few elaborations on the points made above:

     

    a) Cache the output of your Lithium API call, to do that you have to create an endpoint where you could also merge the data from the two calls into one dataset and then cache it for later reuse, which you do with something like that 

     

    <#assign data = appcache.set("externalfeed", yourdatainavariable) />

    Depending on how frequent the data should refresh, you have to contact support so they can adjust (lower) the cache expiry time for the appcache. In your endpoint you can then check if there is something stored for that cache-key and if so return that, else make the API calls and regenerate the data, e.g.

     

    <#if appcache.get("externalfeed", "")?has_content>
        <#-- output your data, be aware that you cannot just print freemarker node objects, so you have to convert them somehow to json/xml, which is not easy btw... -->
    <#else>
        <#-- re-fetch the data and put it into the cache -->
        ...
        <#assign data = appcache.set("externalfeed", yourdatainavariable) />
    </#if>

    But this is basically putting the work from your PHP script to Lithium which requires some FreeMarker/Lithium API knowledge.

     

    If we say that is not available then options to maybe slightly improve the performance is

     

    a) to simply use AJAX/JavaScript to fetch the content from the PHP script after the page has initially loaded and inject it to the right place, so the main content will be there while the AJAX call is getting the "slow" content from Lithium. This would be the easiest solution but doesn't actually change the performance, it just changes the perception =).

    b) Go away from parsing the response XML in PHP and just use the restapi.response_format=json URL parameter for your API calls that will return JSON instead of XML, which you can parse much more efficiently in PHP with json_decode(data, true); but I guess this will not significantly improve the performance of the whole thing...so just a micro-optimization and more elegant solution.

     

    if you have FreeMarker knowledge you can try to use API v2 message resource together with the restd context object inside an endpoint, the restd context object can actually return the data as a JSON string inside FreeMarker code (but you will not be able to modify the fetched data when doing that!) and API v2 would maybe allow to combine your two calls into one with an appropriate query, which means you can just cache and output the response without further worries about how to output it...

     

     

     

  • snaffle's avatar
    snaffle
    Expert
    10 years ago

    Thanks to you both for your suggestions.

     

    I do have some Freemarker/Lithium API knowledge but am not yet up to speed on version 2 of the API.

     

    It's not using Ajax at the moment to retrieve the data and as this is an issue largely of perception, I think implementing Ajax will be my first step and down the track I will reviste the Endpoint and version 2 API suggestion.

     

    Thanks again for your help.

     

    Cheers

     

    Nathan