24

I've been using the API in python for the past few months to look at data. Today, my code broke because the result was no longer valid JSON. It appears that the Zero Width No Break Space character/Byte Order Mark (U+FEFF) is appearing as the first character in the response.

A python script to replicate:

import requests
url="https://api.stackexchange.com/2.2/sites/?filter=%21%2AL1%2AAY-85YllAr2%29&pagesize=1&page=1"
response = requests.get(url)
response.text

This returns the following:

u'\ufeff{"items":[{"site_state":"normal","api_site_parameter":"stackoverflow","name":"Stack Overflow"}],"has_more":true,"quota_max":300,"quota_remaining":282}'

Can that leading character be removed? It looks like this was reported on stackapps already, though the question didn't contain enough detail.

The server also says that it is returning application/json data (I've formatted this to look nicer):

>>> response.headers
{
  'content-length': '142', 
  'content-encoding': 'gzip', 
  'cache-control': 'private', 
  'access-control-allow-credentials': 'false', 
  'date': 'Thu, 03 Jul 2014 02:10:26 GMT', 
  'access-control-allow-origin': '*', 
  'access-control-allow-methods': 'GET, POST', 
  'content-type': 'application/json; charset=utf-8'
}

And response.json() errors, as expected with invalid json

>>> response.json()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\requests\models.py", line 763, in json
    return json.loads(self.text, **kwargs)
  File "C:\Python27\lib\json\__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "C:\Python27\lib\json\decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python27\lib\json\decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
  • @AwesomePoodles, the server is already returning application/json and response.json() fails. – Andy Jul 3 '14 at 3:01
  • IIRC, Isn't FEFF a UTF-16 BOM?? (UTF-8 is 0xEF,0xBB,0xBF?) try setting the response.encoding to UTF-16.? (This looks like a malformed or mismatched BOM bug. Doesn't Python seamlessly handle proper BOM's??) – Awesome Poodles Jul 3 '14 at 3:17
  • @AwesomePoodles, it is a BOM. That's listed under the old names on my link in the question. I'll edit to include that name too. I just pulled the title of the page for the character name – Andy Jul 3 '14 at 3:22
  • 1
    Yes, but isn't that the BOM for UTF-16? NOT UTF-8? See latest edit of comment(s). – Awesome Poodles Jul 3 '14 at 3:23
  • @AwesomePoodles, Maybe? According to the link, yes. But it also mentions that it will look like this for python: Python source code u"\uFEFF", which is exactly what I'm seeing in my response. Forcing the encoding to UTF-16 didn't help. – Andy Jul 3 '14 at 3:31
  • Related question and answer: u'\ufeff' in Python string. .decode() might work... – Awesome Poodles Jul 3 '14 at 3:53
  • I can query the API just fine from javascript. – John Dvorak Jul 3 '14 at 8:41
  • 1
    @AwesomePoodles Encoding confusion detected… The API returns text in the Unicode character set which is encoded in UTF-8. The first character is U+FEFF ZERO WIDTH NO-BREAK SPACE. It is sent as the three-byte sequence {0xef, 0xbb, 0xbf} which is the encoding of this Unicode character in UTF-8. This is not a byte-order mark since a byte order is a feature of encodings that have a byte order, such as UTF-16. If the API returned a byte sequence beginning with {0xfe, 0xff, 0x00, 0x7b, 0x00, 0x22} and declared the encoding as UTF-16, that would also be valid. But the character U+FEFF is invalid. – Gilles Jul 3 '14 at 9:18
  • @JanDvorak, The BOM is still there in JS; I checked. But most browsers (tested on FF) handle it, malformed or not, if you use the normal mime types (text/plain or application/json). Python apparently isn't as tolerant? – Awesome Poodles Jul 3 '14 at 9:55
  • 1
    @Awesome: that's because browser devs have been trained to be over-tolerant (HTML should always be parsed ignoring errors to the best of their ability); the JSON standard doesn't allow for a leading ZWNBS. The use of ZWNBS as marker at the start is a Microsoft 'invention' to aid encoding detection instead, but that aid has no place here where both the JSON standard (defaulting to UTF-8) and the Content-Type header (containing a charset parameter) already communicate the encoding. – Martijn Pieters Jul 4 '14 at 7:53
10

This should be fixed now. There was an early fix stripping out invalid codepoints that inadvertently starting adding that preamble to each encoded string.

  • 1
    Thank you! I can confirm my stuff works once again. :) – Greg Hewgill Jul 3 '14 at 18:11
  • Well I leave the changes in my code just in case the next fix will introduce it again. :-) – Mauro Rocco Jul 3 '14 at 20:31
7

UTF-8 doesn't need a BOM (it is independent of byte order) and a BOM should not be used with UTF-8. Clearly, not all systems handle this bogus character transparently (my Python and Objective-C code broke too). If this isn't an official breaking change in the API, then the API should be reverted to its original behaviour.

If this change is intended for future API versions, then it should be a major API version number increment so it doesn't break all existing clients.

  • Unfortunately, Microsoft has hit upon the idea to add the character to UTF-8-encoded files to aid encoding detection. Excel will open CSV as encoded in the configured codepage unless the character sequence at the start indicates UTF-8-with-marker, as do many other of their tools (Notepad in particular). – Martijn Pieters Jul 4 '14 at 7:46
3

I had your same problem with Python, I simply solved by doing:

string_response.decode('utf-8-sig')

The utf-8-sign handle automatically the BOM, if not present will keep working.

Here you can see the change I introduced on celeryproject.org to keep working stackoverflow queering:

https://github.com/celery/celeryproject/commit/2525781ad5e6e956d79572c60ab0f2d2de0f7ff6

More details here

Convert UTF-8 with BOM to UTF-8 with no BOM in Python

By the way, Greg Hewgill is right, this was changed by day to night breaking all clients, should have been handled with a API version change.

  • 1
    ...handled with a version change if it was intentional – Aza Jul 3 '14 at 16:54
3

The api should not send an BOM, UTF-8 encoding is already signaled in HTTP header. U+FEFF is the BOM for all unicode encodings. In UTF-8 the BOM results in a sequence of the tree bytes (0xEF 0xBB 0xBF) - that usually allows the reader to identify the file as an UTF-8 file - if it's read from a hard disk or any other stream without meta data. Normally, the BOM is used to signal the endianness of the encoding, but since endianness is irrelevant to UTF-8, the BOM is unnecessary.

According to the Unicode standard, the BOM for UTF-8 files is not recommended:

2.6 Encoding Schemes

... Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature. See the “Byte Order Mark” subsection in Section 16.8, Specials, for more information.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .