Problems with Freemarker - utils.digest.hmac Hmac SHA256 and base 64 encoding?

cblown call me daft but are you not better simply storing the hash output you generated using PHP and setting the variable to that, rather than trying to generate this in FreeMarker?

If the challenge and secret key are unlikely to change then it shouldn't cause any problems. This would also be a bit more secure as the challenge string and secret key are not stored then on the community only the hashed output.

There is cleary something different in the way the hash is genered between PHP and FreeMarker, for example I notice that you wrap the entire code in base64_encode for PHP:

echo base64_encode(hash_hmac('sha256', "the quick brown fox jumps over the lazy dog", base64_decode("12345678901234567890") , true));

but don't do the same in FreeMarker, you're only decoding the key:

<#assign signature = utils.digest.hmac("HmacSHA256", utils.base64.decode("12345678901234567890”), "the quick brown fox jumps over the lazy dog") />

There could be something in that which we're missing.

The access token expires so we have to hash another challenge each time to get a new token. The Freemarker routine returns base 64 as per the docs so no need to encode it like we do for the php. The php and freemarker agree on the hash, it’s only when we try to pass a byte array to the freemarker (the output from the base 64 decode should be straight binary) The fact I can’t decode and re-encode in Base 64 using Freemarker is somehow related.. 

Nice problem, don't know the solution though =)...I would ping DougS for things like this!

Hi cblown

I think I might know what's going on here, but please correct me if I've got something wrong:

Base64 encoding was designed to encode binary data into an ASCII string (a radix-64 representation). The string "12345678901234567890" is not a base-64-encoded string, so decoding it using base64 will not necessarily generate the same result on different implementations of the Base64 spec (since the spec says nothing about this). Try any number of online Base64 decoding tools and try decoding the string "12345678901234567890" -- you will either get different results in different tools, or you may get an error (because this is not a valid use of Base64 decoding).

If you encode the string "12345678901234567890" using base64, you will get the same results in the Lithium's Freemarker context implementation that you get in any other valid Base64 encoding algorithm (for UTF-8 charset you would get "MTIzNDU2Nzg5MDEyMzQ1Njc4OTA="). If you then decode the base-64 encoded string (in this case "MTIzNDU2Nzg5MDEyMzQ1Njc4OTA=") you will get the same result on  in Lithium's Freemarker context implementation that will get in any other valid Base64 decoding algorithm (for UTF-8 charset you would get "12345678901234567890").

Why is the 3rd party auth system trying to decode the string "12345678901234567890" (which isn't a base-64 encoded string), and can that be changed? (because it's not going to behave correctly on different implementations of Base64).

-Doug

DougS - thanks for the reply. The key I'm using above is just an example and was a bad example in hindsight :) - I'll PM'd you the real key.. I'm thinking now the key might be the problem and as you say each implementation is potentially different if the encode string is not 100% right. 

I did some further testing using your example MTIzNDU2Nzg5MDEyMzQ1Njc4OTA= since we've confirmed its encoded. It works as expected. I suspect the key we received from the 3rd party site, as you mentioned, not properly encoded, although with php we can get a token using this key. 

PHP

echo base64_encode(hash_hmac('sha256', "the quick brown fox jumps over the lazy dog", "12345678901234567890" , true));
echo "<br>";
echo base64_encode(hash_hmac('sha256', "the quick brown fox jumps over the lazy dog", base64_decode("MTIzNDU2Nzg5MDEyMzQ1Njc4OTA=") , true));

Returns

XIDINszyYHIP5vszuWRnyP/4/p/y0COFZHV8tzcxPh4=
XIDINszyYHIP5vszuWRnyP/4/p/y0COFZHV8tzcxPh4=

Freemarker

<#assign signature = utils.digest.hmac("HmacSHA256", "12345678901234567890", "the quick brown fox jumps over the lazy dog") />
<#assign signature2 = utils.digest.hmac("HmacSHA256", utils.base64.decode("MTIzNDU2Nzg5MDEyMzQ1Njc4OTA="), "the quick brown fox jumps over the lazy dog") />
${signature}
<br>
${signature2}

Returns

XIDINszyYHIP5vszuWRnyP/4/p/y0COFZHV8tzcxPh4= 
XIDINszyYHIP5vszuWRnyP/4/p/y0COFZHV8tzcxPh4=

Actually, you know what I don't think the key is all that sensitive on its own, its a testing key anyway and you need an app Id and other stuff.. So here it is again with the real key :)

PHP

echo base64_encode(hash_hmac('sha256', "the quick brown fox jumps over the lazy dog", base64_decode("2ZKF2RDckw3M++TwLVP8t+AQIU66Rj3mCykTbMwBnOWUpAnM") , true));

Result

2IweFAZF3mu13aizUcTjRV450W1LEwISUJnRnOapOUk=

Freemarker

<#assign signature = utils.digest.hmac("HmacSHA256", utils.base64.decode("2ZKF2RDckw3M++TwLVP8t+AQIU66Rj3mCykTbMwBnOWUpAnM"), "the quick brown fox jumps over the lazy dog") />
${signature}

Result

w868e7nxYVl+qNTNMg6Zb4rLOh02x0jC5i8LYfkeVuE=

Today we've tried various charsets since the HMAC routines accept charset parameters for the string if the input isn't UTF-8. No luck so far. I'm really not sure what specifically is the problem, but as you can see from my last post we're are getting different results with that key. Unfortunately the unencoded version of that key is a bunch of garble and not nice ascii like our example. 

The raw unencoded hex is 

0000000 d9 92 85 d9 10 dc 93 0d cc fb e4 f0 2d 53 fc b7
0000010 e0 10 21 4e ba 46 3d e6 0b 29 13 6c cc 01 9c e5
0000020 94 a4 09 cc                                    
0000024

Any ideas how I can get this back into Freemarker as a UTF-8 string? And don't say base 64 encode it :)

This answer is misleading. 

The problem is that base64.encode and digest.hmac take only strings as input.

Most Base64 suites take a binary array.   Like Spring:

https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/util/Base64Utils.html#encode-byte:A-

Why Base64 encode a binary array that will not encode to a string?  It increases the entropy of the result.

cblown Please support the following Product Idea to add versions of the methods base64.encode and digest.hmac that take byte arrays as input:

https://community.khoros.com/t5/Li-Product-Ideas/Add-versions-of-digest-hmac-and-base64-encode-that-take-byte/idi-p/529244