Byte Order Mark appearing at the beginning of API calls

Question

I've been using the API in python for the past few months to look at data. Today, my code broke because the result was no longer valid JSON. It appears that the Zero Width No Break Space character/Byte Order Mark (U+FEFF) is appearing as the first character in the response.

A python script to replicate:

import requests
url="https://api.stackexchange.com/2.2/sites/?filter=%21%2AL1%2AAY-85YllAr2%29&pagesize=1&page=1"
response = requests.get(url)
response.text

This returns the following:

u'\ufeff{"items":[{"site_state":"normal","api_site_parameter":"stackoverflow","name":"Stack Overflow"}],"has_more":true,"quota_max":300,"quota_remaining":282}'

Can that leading character be removed? It looks like this was reported on stackapps already, though the question didn't contain enough detail.

The server also says that it is returning application/json data (I've formatted this to look nicer):

>>> response.headers
{
  'content-length': '142', 
  'content-encoding': 'gzip', 
  'cache-control': 'private', 
  'access-control-allow-credentials': 'false', 
  'date': 'Thu, 03 Jul 2014 02:10:26 GMT', 
  'access-control-allow-origin': '*', 
  'access-control-allow-methods': 'GET, POST', 
  'content-type': 'application/json; charset=utf-8'
}

And response.json() errors, as expected with invalid json

>>> response.json()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\requests\models.py", line 763, in json
    return json.loads(self.text, **kwargs)
  File "C:\Python27\lib\json\__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "C:\Python27\lib\json\decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python27\lib\json\decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Answer 1

This should be fixed now. There was an early fix stripping out invalid codepoints that inadvertently starting adding that preamble to each encoded string.

Answer 2

UTF-8 doesn't need a BOM (it is independent of byte order) and a BOM should not be used with UTF-8. Clearly, not all systems handle this bogus character transparently (my Python and Objective-C code broke too). If this isn't an official breaking change in the API, then the API should be reverted to its original behaviour.

If this change is intended for future API versions, then it should be a major API version number increment so it doesn't break all existing clients.

Answer 3

I had your same problem with Python, I simply solved by doing:

string_response.decode('utf-8-sig')

The utf-8-sign handle automatically the BOM, if not present will keep working.

Here you can see the change I introduced on celeryproject.org to keep working stackoverflow queering:

https://github.com/celery/celeryproject/commit/2525781ad5e6e956d79572c60ab0f2d2de0f7ff6

More details here

Convert UTF-8 with BOM to UTF-8 with no BOM in Python

By the way, Greg Hewgill is right, this was changed by day to night breaking all clients, should have been handled with a API version change.

Answer 4

The api should not send an BOM, UTF-8 encoding is already signaled in HTTP header. U+FEFF is the BOM for all unicode encodings. In UTF-8 the BOM results in a sequence of the tree bytes (0xEF 0xBB 0xBF) - that usually allows the reader to identify the file as an UTF-8 file - if it's read from a hard disk or any other stream without meta data. Normally, the BOM is used to signal the endianness of the encoding, but since endianness is irrelevant to UTF-8, the BOM is unnecessary.

According to the Unicode standard, the BOM for UTF-8 files is not recommended:

2.6 Encoding Schemes

... Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature. See the “Byte Order Mark” subsection in Section 16.8, Specials, for more information.