Forum Discussion

shrugupt's avatar
shrugupt
Adept
6 years ago

Help with scraping OOTB component

Trying to scrape recommendations component using :

<#assign markup>
<@delegate />
</#assign>
${markup}

<#-- LOGIC -->

<#assign ids = [] /> 

<#assign idmatches = markup?matches("-p\\/(\\d+)") />

<#list idmatches as id>
<#-- only add an id once -->
<#-- the ?groups build-in contains a sequence with [0] = the whole match
and [0+groupn] items with their match, in this case the <topicid> -->
<#if !ids?seq_contains(id?groups[1])>
<#assign ids = ids + [id?groups[1]]/>
</#if>
</#list>

<#-- DISPLAY -->
<#if markup?contains("Recommendations")>
<span>RC</span>
<#list ids as id>
<#assign convo = rest("2.0", "/search?q=" + "select subject,board,replies.count(*) from messages where id='${id}'"?url).data.items[0]/>
${convo.subject}
${convo.board.id}
${convo.replies.count}
</#list>
</#if>

 

Which works but I want to omit the need for "${markup}" without which I'm unable to get the HTML on DOM in order to get the data (the topic ids basically)

  • PerBonomi's avatar
    PerBonomi
    Boss
    6 years ago

    My non-expert guess is that until you explicitly render it it doesn't, for all intents and purposes, exist. One of those Freemarker quirks.

    Wrap it in a hidden div?

    • shrugupt's avatar
      shrugupt
      Adept
      6 years ago

      Tried that, but it ends up creating a hidden div around the entire overriden component, i.e my content below ${markup} also gets hidden.

      So I've to grab the class in which just the Recommendations component resides and hide that (using CSS/JS) 😕

  • AndrewF's avatar
    AndrewF
    Khoros Oracle
    6 years ago

    I believe this has something to do with the way we implement <@delegate> internally combined with the way FreeMarker's block <#assign> works. Are you able to get around it by forcing a read, like <#assign dummyvariable = markup[0..]> ?

    I had some other ideas for improvements:

    • I'd recommend moving your <#if markup?contains("Recommendations")> as high as possible so that you don't do more work than necessary.
    • Instead of using ?seq_contains, use our utils.sequence.unique(ids) utility method at the end.
    • If you only intend to build ids so that you can iterate over it later, consider interleaving your "LOGIC" and "DISPLAY" loops so that you can space out some of your API calls.
    • You don't need to escape a backslash in a regular expression string. ?matches("-p/(\\d+)") is fine.

    Note that the DOM is not related here -- at the point this template is being processed, there is no HTML DOM. FreeMarker is maintaining its own data structure that represents the parsed template, and it is building pure text. Later on the other end of the wire, a browser will parse the text into a DOM.

    • shrugupt's avatar
      shrugupt
      Adept
      6 years ago

      Hi Andrew, 

      Thanks for the inputs.

      I'm actually hiding the Recommendations OOB using their class through CSS.

      My other issue is that the DISPLAY seems to be getting executed twice. Once *with* the IDs and once without it (chronologically). The "${markup?length}" if printed is around 11k and 1k in the cases. Which I find to be strange.

      Which is why I'm having to use "<#if markup?contains("Recommendations")>"

      😕