SE API multiple-escapes the returned data (at least display names, userlinks, and post content)

Question

The Stack Exchange API will, unless you explicitly disable it, apply HTML escapes to certain strings. However, it will apply this transformation multiple times prior responding with the data.

This is most easily visible when querying users. For any username containing the letter Ö, such escaping is applied:

$ curl -s 'https://api.stackexchange.com/2.3/users/1525238?site=stackoverflow' | gunzip -c  | jq '.items|map([.link,.display_name])'
[
  [
    "https://stackoverflow.com/users/1525238/ayberk-%c3%96zg%c3%bcr",
    "Ayberk Özgür"
  ]
]

When I then ask for more users including the previous user ID, the display name is escaped twice. In the following example I added user ID 1 to the user IDs in the URL:

$ curl -s 'https://api.stackexchange.com/2.3/users/1;1525238?site=stackoverflow' | gunzip -c  | jq '.items|map([.link,.display_name])'
[
  [
    "https://stackoverflow.com/users/1/jeff-atwood",
    "Jeff Atwood"
  ],
  [
    "https://stackoverflow.com/users/1525238/ayberk-214zg252r",
    "Ayberk Özgür"
  ]
]

The Ö and ü HTML escapes have been escaped again, now to &#214 and &#252. Moveover, note that the changed display name now bleeds into the user link, where the URL slug has now changed from ayberk-%c3%96zg%c3%bcr to ayberk-214zg252r.

This can affect other API fields too, e.g. others have seen this affect post bodies, and the escaping can end up being applied more than once.

I tried adding a 3rd new user to the mix, but the escaping then disappears again:

$ curl -s 'https://api.stackexchange.com/2.3/users/1;2;1525238?site=stackoverflow' | gunzip -c  | jq '.items|map([.link,.display_name])'
[
  [
    "https://stackoverflow.com/users/1/jeff-atwood",
    "Jeff Atwood"
  ],
  [
    "https://stackoverflow.com/users/2/geoff-dalgas",
    "Geoff Dalgas"
  ],
  [
    "https://stackoverflow.com/users/1525238/ayberk-%c3%96zg%c3%bcr",
    "Ayberk Özgür"
  ]
]

but this could be due to the request having been handled by a different server or a cache invalidation in the intervening space.

I can, however, reproduce the issue reliably by querying for any user ID that requires escaping, then adding 1 additional user to the query.

For those areas where HTML escaping is not an issue or can trivially be applied manually, you can disable the 'safe' option in the filter; the minimal 'unsafe' variant of the default filter is y49uLr, and this makes the problem go away:

$ curl -s 'https://api.stackexchange.com/2.3/users/1525238?filter=y49uLr&site=stackoverflow' | gunzip -c  | jq '.items|map([.link,.display_name])'
[
  [
    "https://stackoverflow.com/users/1525238/ayberk-%c3%96zg%c3%bcr",
    "Ayberk Özgür"
  ]
]
$ curl -s 'https://api.stackexchange.com/2.3/users/1;1525238?filter=y49uLr&site=stackoverflow' | gunzip -c  | jq '.items|map([.link,.display_name])'
[
  [
    "https://stackoverflow.com/users/1/jeff-atwood",
    "Jeff Atwood"
  ],
  [
    "https://stackoverflow.com/users/1525238/ayberk-%c3%96zg%c3%bcr",
    "Ayberk Özgür"
  ]
]

Answer 1

Okay - we finally figured this out.

Everything looked totally fine on the surface, which is why it took so long to identify what was happening.

Basically, we cache the request data correctly, but then when we run the methods to properly encode the data, they are also encoding the same object as that which is cached.

To fix it, I changed the code so that we're only ever running the method to encode the data on a cloned version of the object we're caching. Gotta love it when a bug that takes a week to figure out is fixed in just a couple lines of code.