Unable to use non-ASCII characters in sendRoomMessage()

[heffebaycay]

Hello,

I tried integrating hipchat-api to send messages to a specific Hipchat room on an Hipchat cloud account and I ran across a funny issue:

• Sending simple messages like Hello world! works fine.
• Sending a message containing a character not from the ASCII charset (like Hello é world!) fails with a 400 error from Hipchat server, letting me know it failed to parse the request body as UTF-8.

Here's the response from the Hipchat server:

FoxHttpResponse(responseBody=FoxHttpResponseBody(body={
  "error": {
    "code": 400,
    "message": "The request body cannot be decoded as UTF-8: 'utf8' codec can't decode byte 0xe9 in position 90: invalid continuation byte",
    "type": "Bad Request"
  }
}), responseCode=400, responseHeaders=FoxHttpHeader(headerEntries=[HeaderEntry(name=X-Ratelimit-Reset, value=1514665166), HeaderEntry(name=X-Ratelimit-Limit, value=100), HeaderEntry(name=Access-Control-Expose-Headers, value=Date, ETag, Link, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-Backoff), HeaderEntry(name=Server, value=nginx), HeaderEntry(name=X-Ratelimit-Remaining, value=98), HeaderEntry(name=Access-Control-Allow-Origin, value=*), HeaderEntry(name=Connection, value=keep-alive), HeaderEntry(name=Content-Length, value=204), HeaderEntry(name=Date, value=Sat, 30 Dec 2017 20:15:56 GMT), HeaderEntry(name=Content-Type, value=application/json)]), foxHttpClient=ch.viascom.groundwork.foxhttp.FoxHttpClient@45da906e, foxHttpRequest=ch.viascom.groundwork.foxhttp.FoxHttpRequest@3b6031c2)

If I look at the raw request using Wireshark, I notice that the é character is encoded as 0xe9 (latin-1) instead of being encoded as 0xc3 0xa9 (UTF-8). Since the content-type of the request is set to application/json; charset=UTF-8, it's understandable for the Hipchat server to get confused.

I can reproduce the issue with the following snippet (I opened a web server on port 8080):

FoxHttpClient client = new FoxHttpClientBuilder(new GsonParser()).build();

client.setFoxHttpLogger(new SystemOutFoxHttpLogger(true, "FoxHttp-Logger"));

String[] bodies = {"Hello world!", "Hello é world!"};

String url = "http://localhost:8080/test";

for (String body : bodies) {
    FoxHttpRequestBuilder requestBuilder = new FoxHttpRequestBuilder(url, RequestType.POST, client);
    MessageRequestBody messageRequestBody = new MessageRequestBody(body);
    requestBuilder.setRequestBody(new RequestObjectBody(messageRequestBody, ContentType.APPLICATION_JSON));
    requestBuilder.build().execute();
}

Here are the raw data extracted via Wireshark:

Hello world!

0000   7b 22 6d 65 73 73 61 67 65 22 3a 22 48 65 6c 6c  {"message":"Hell
0010   6f 20 77 6f 72 6c 64 21 22 7d                    o world!"}

Hello é world!

0000   7b 22 6d 65 73 73 61 67 65 22 3a 22 48 65 6c 6c  {"message":"Hell
0010   6f 20 e9 20 77 6f 72 6c 64 21 22 7d              o . world!"}

(see how é got encoded as 0xe9).

Is this a limitation of FoxHttp or am I doing something wrong?

[itsmefox]

Hi @heffebaycay

This problem is as you mentioned a problem on the side of FoxHttp. I'm working on a solution for this problem on FoxHttp first and then update HipChat-API so please check viascom/FoxHttp#5 for further updates on the progress first.

Cheers, Patrick