23

The Stack Exchange API will, unless you explicitly disable it, apply HTML escapes to certain strings. However, it will apply this transformation multiple times prior responding with the data.

This is most easily visible when querying users. For any username containing the letter Ö, such escaping is applied:

$ curl -s 'https://api.stackexchange.com/2.3/users/1525238?site=stackoverflow' | gunzip -c  | jq '.items|map([.link,.display_name])'
[
  [
    "https://stackoverflow.com/users/1525238/ayberk-%c3%96zg%c3%bcr",
    "Ayberk Özgür"
  ]
]

When I then ask for more users including the previous user ID, the display name is escaped twice. In the following example I added user ID 1 to the user IDs in the URL:

$ curl -s 'https://api.stackexchange.com/2.3/users/1;1525238?site=stackoverflow' | gunzip -c  | jq '.items|map([.link,.display_name])'
[
  [
    "https://stackoverflow.com/users/1/jeff-atwood",
    "Jeff Atwood"
  ],
  [
    "https://stackoverflow.com/users/1525238/ayberk-214zg252r",
    "Ayberk Özgür"
  ]
]

The Ö and ü HTML escapes have been escaped again, now to &#214 and &#252. Moveover, note that the changed display name now bleeds into the user link, where the URL slug has now changed from ayberk-%c3%96zg%c3%bcr to ayberk-214zg252r.

This can affect other API fields too, e.g. others have seen this affect post bodies, and the escaping can end up being applied more than once.

I tried adding a 3rd new user to the mix, but the escaping then disappears again:

$ curl -s 'https://api.stackexchange.com/2.3/users/1;2;1525238?site=stackoverflow' | gunzip -c  | jq '.items|map([.link,.display_name])'
[
  [
    "https://stackoverflow.com/users/1/jeff-atwood",
    "Jeff Atwood"
  ],
  [
    "https://stackoverflow.com/users/2/geoff-dalgas",
    "Geoff Dalgas"
  ],
  [
    "https://stackoverflow.com/users/1525238/ayberk-%c3%96zg%c3%bcr",
    "Ayberk Özgür"
  ]
]

but this could be due to the request having been handled by a different server or a cache invalidation in the intervening space.

I can, however, reproduce the issue reliably by querying for any user ID that requires escaping, then adding 1 additional user to the query.

For those areas where HTML escaping is not an issue or can trivially be applied manually, you can disable the 'safe' option in the filter; the minimal 'unsafe' variant of the default filter is y49uLr, and this makes the problem go away:

$ curl -s 'https://api.stackexchange.com/2.3/users/1525238?filter=y49uLr&site=stackoverflow' | gunzip -c  | jq '.items|map([.link,.display_name])'
[
  [
    "https://stackoverflow.com/users/1525238/ayberk-%c3%96zg%c3%bcr",
    "Ayberk Özgür"
  ]
]
$ curl -s 'https://api.stackexchange.com/2.3/users/1;1525238?filter=y49uLr&site=stackoverflow' | gunzip -c  | jq '.items|map([.link,.display_name])'
[
  [
    "https://stackoverflow.com/users/1/jeff-atwood",
    "Jeff Atwood"
  ],
  [
    "https://stackoverflow.com/users/1525238/ayberk-%c3%96zg%c3%bcr",
    "Ayberk Özgür"
  ]
]
2
  • 8
    It's not just display names, it's post content too example. It's probably lots of things and can be ridiculous: code: 'ESOCKET', (source) Credit: double-beep
    – Makyen
    Commented Mar 21, 2023 at 23:30
  • 5
    My uninformed theory on what seems to be happening is: Data is loaded from cache -> data is escaped -> then escaped data is put back into the cache. This would add one level of escaping for every time a entity is loaded during it's cache lifetime.
    – mousetail
    Commented Mar 22, 2023 at 9:59

1 Answer 1

8

Okay - we finally figured this out.

Everything looked totally fine on the surface, which is why it took so long to identify what was happening.

Basically, we cache the request data correctly, but then when we run the methods to properly encode the data, they are also encoding the same object as that which is cached.

To fix it, I changed the code so that we're only ever running the method to encode the data on a cloned version of the object we're caching. Gotta love it when a bug that takes a week to figure out is fixed in just a couple lines of code.

1
  • Oh the old shared mutable data structure problem! Yes, when you cache, make sure you always clone the cached value on reuse. Don’t ever trust that the the next piece of code operating on it will remember to do this! :-) (and sorry for the looooong delay here on my part, my SE inbox is snowed under). Commented Jun 6, 2023 at 20:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .