3

Summary

When data is requested with curl, gzip'ed data is returned, I would expect JSON to be returned. You probably want your API to work with curl without too much effort.

Repro

curl "http://api.stackexchange.com/2.2/questions/3853722/answers?order=desc&sort=activity&site=stackoverflow" | hd
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   644  100   644    0     0    973      0 --:--:-- --:--:-- --:--:--   974
00000000  1f 8b 08 00 00 00 00 00  04 00 ad 94 5d 6f d3 30  |............]o.0|
00000010  14 86 ff ca 64 b4 5e 85  c6 9f b1 5d 54 21 ae b8  |....d.^....]T!..|
00000020  40 48 dc 20 24 10 8a dc  c4 49 4d 9b 8f d9 ce c2  |@H. $....IM.....|
00000030  36 ed 86 ff c6 ff c2 4d  83 d6 6d 4d e9 24 ee 12  |6......M..mM.$..|
00000040  9f e3 d7 e7 9c c7 af ef  80 f1 ba 72 60 f1 ed 0e  |...........r`...|
00000050  34 7d ad 2d 58 dc 01 ab  db ce 2b 6f 9a 1a 2c 30  |4}.-X.....+o..,0|
00000060  62 11 e8 9c b6 a9 c9 c1  82 52 89 08 81 e3 8a bf  |b........R......|
00000070  69 35 58 84 f4 d2 38 af  ad ce 41 04 54 96 e9 d6  |i5X...8...A.T...|
00000080  a7 56 f9 10 4a 42 66 6b  9b c2 6c 75 6a 2a 55 ee  |.V..JBfk..luj*U.|
00000090  b2 d7 de b7 6e 11 c7 66  ee bc ca 36 73 53 95 9d  |....n..f...6sS..|
000000a0  9d 67 4d 15 7f 66 a6 63  f3 1f 6d f9 d6 2d 11 16  |.gM..f.c..m..-..|
000000b0  b3 72 89 82 60 6e 5c bb  55 37 69 ad aa dd f6 2f  |.r..`n\.U7i..../|
0

Technical details

This seems to be caused because curl does not send an Accept-Encoding heading, and assumes, therefore, that that the response will not be encoding. You send back gzip encoded data together with a Content-Encoding heading. (Use --verbose flag to see the headers)

Now, it would appear that it is curl that is in the wrong; according to section 14.3 of https://www.ietf.org/rfc/rfc2616.txt, if Accept-Encoding is missing you can do whatever you want. However, I imagine curl (and old versions of curl) are sufficiently common that you aren't allowed to ignore its bugs.

Interestingly (or perhaps pedantically), you also have the same (and in this case standard-violating) behaviour for an empty Accept-Encoding heading curl -H 'Accept-Encoding;', and an accept encoding that doesn't allow gzip curl -H 'Accept-Encoding: identity'. Again this behaviour is not technically wrong, but you are ignoring recommendations.

Work around

This behaviour can be worked around by using the --compressed option.

Possible actions

In an ideal world you might return uncompressed data in these cases, but one appreciates that you might be deliberately forcing clients to support compression.

Another option would be to return 406 errors, as this might be easier to understand. I don't know if this has implications for buggy clients that aren't include Accept-Encodings.

Alternatively, you could do nothing and refer to this post as a workaround.

5

This is not a bug; it is .

The API will only return compressed data, by design. Once upon a time, requests without the proper Accept-Encoding header returned an error, but they had to stop doing that for practical reasons.

The API "Compression" doc states:

...we guarantee that ==> all <== responses are compressed, either with GZIP or DEFLATE. ...

(Emphasis mine)

and then:

History

There is a way to remain in compliance with the HTTP spec, which is to reject all requests that do not list "gzip" or "deflate" in their Accept-Encoding header. Unfortunately, this does not work in practice as far too many proxies (affecting ~1% of users in our experience) will strip out this header.

We experimented with this approach during the beta period of API version 1.0, and found that it effectively banned a small but non-trivial number of potential users. That many users access the Stack Exchange network from corporate machines exacerbates the problem.


This question is a duplicate and/or related to these questions on Stack Apps:

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .