76

TL;DR: We’re making a substantial repayment on our JavaScript technical debt. Apologies in advance for any bugs as we work through it!

An administrative note: Rather than posting bug reports as answers to this post, in this case we'd prefer you to post a separate question for each bug you find.


Stack Overflow's JavaScript setup hasn't substantially changed since, like, 2008. Here are a few fun facts about the way we build our JS.

  • The process to install third party libraries is:

    1. Download a copy of the library's code
    2. Paste that code into a folder
    3. git commit

    Seriously! We have a third-party folder in the repo. It contains (very old versions of) a variety of libraries. Some of those libraries have multiple versions in subfolders. Some of those libraries have had ad-hoc edits made to them. It's difficult to upgrade dependencies and it's difficult to manage transitive dependencies.

  • If you want to split your code up into multiple files, you need to manually include each of those files in the correct order using script tags.

    • This goes for files in the aforementioned third-party folder too.
    • It’s painful enough to discourage developers from splitting code up. We have quite a lot of 5000-line monster files.
    • In many cases, splitting code into files to make it more maintainable can be actively detrimental to performance.
    • Code gets written in shared files and then included everywhere. Most of our pages are downloading way more JS than they actually need.
  • Files communicate with each other through global variables.

    • It's difficult to track down all the usages of a given function.
    • Intellisense tooling doesn't know which globals are available in which contexts. The namespace is cluttered up with irrelevant definitions and it's easy to accidentally use a function on a page where it isn't actually available.
    • Consistency across the codebase in this regard is not high. Some files use TS namespaces, others write to a global StackExchange object, others install variables directly into the global scope…
  • We have an idiosyncratic home-made file-loading/module system (called StackExchange.using) with a variety of similar-but-different usage patterns.

We've bolted things on to this system over the years — most new code is written in TypeScript, for example — but the overall story of how we build and deliver JS hasn't really changed.

Anyway, that's enough background. You may be aware that we've recently integrated Webpack into our JS build pipeline. I'm writing today to let you know that the second phase of this migration is underway: we are rewriting our code to use ES modules. This'll help us manage dependencies (both internal dependencies and third-party ones, using NPM) and make it easier to write reliable JavaScript as time goes on.

However, this is one of those big change-every-file migrations, and of course each file has its own idiosyncrasies. I've set things up so we can mostly work one file at a time, and we will obviously test each piece of the migration before we ship it, but Stack Overflow is a complicated 13-year-old system and it's difficult to catch every possible edge case in testing. In other words,

There Will Be Bugs

The work on this has already begun. I've built out all of the infrastructure (I think), and we're working file by file. Every developer in the department is converting a handful of files each, and in most cases each of those conversions can and will be shipped independently.

We have quite a lot of files to convert (hundreds but not thousands) and we're working on this migration in parallel with other ongoing projects, so I'm not expecting it to be a particularly quick job. I'm about to go on a six week sabbatical and I'd be astonished (and delighted) if the work was 100% complete by the time I'm back! But hopefully it'll be mostly done by that point.

The ask for you Metazens is simple: please let us know if you notice something's broken. In other words, keep doing what you're doing! We really appreciate it when you inform us about bugs.

Thank you for your patience and help!

18
  • 1
    Any reason for the administrative note? Are there some problems with the usual feedback procedure for announcements? Or do you just want to experiment?
    – Glorfindel Mod
    Dec 7, 2021 at 16:16
  • 14
    Good question, thanks! Every developer in the department is working on a few files each, plus additionally I'll be going on a six week sabbatical in a couple of weeks, so this post probably isn't the best channel to reach the developer who's best equipped to fix a given bug. Dec 7, 2021 at 16:28
  • 2
    "In many cases, splitting code into files to make it more maintainable can be actively detrimental to performance" Has the team considered just keeping code split out for work locally and then combining it when minifying and uploading to the production environment?
    – TylerH
    Dec 7, 2021 at 18:46
  • 3
    @TylerH That's exactly what we're currently working on. Dec 7, 2021 at 18:48
  • 1
    Will actual ES modules be served to the users, or will they be compiled by Webpack, or however it works? I mean, all supported browsers support modules natively, or does this list only apply to the Stacks design system? Dec 7, 2021 at 23:43
  • 2
    Given that I have 13 userscripts that hit this page alone, this ought to get interesting. Dec 8, 2021 at 4:28
  • 1
    Is there any chance this will affect (positively or negatively) Stack Exchange's accessibility in China (behind the Great Firewall)? Dec 8, 2021 at 9:49
  • 3
    @SebastianSimon They're compiled by Webpack. Webpack takes a (possibly large) number of source files (modules) and bundles them into a (usually small) number of output assets. We could serve ES modules directly (at a 1-1 correspondence to input files) - we no longer support any legacy browsers, as you noted - but that wouldn't help with the performance issue I mentioned in my post. Dec 8, 2021 at 18:34
  • 2
    @RebeccaJ.Stones I'm not anticipating any problems there. Our actual delivery mechanism isn't changing (the CDN and websites are remaining where they are), only the way we build the assets. Dec 8, 2021 at 18:37
  • 4
    @gnat We can and frequently do let risky things bed in on Meta for a few hours. Meta runs on a different (smaller) web tier than the rest of the network. At time of writing we don't have any way of deploying JavaScript changes to the primary web tier without shipping them to the entire network. (I learned this the hard way a few weeks ago.) Webpack has some sophisticated tools to manage versions though, so that's something we'd like to set up in the future. Dec 8, 2021 at 20:40
  • 2
  • 1
    @BenjaminHodgson Is the work (migration of JavaScript) done and on a different note, is your sabbatical done? Jan 31 at 19:28
  • 5
    @RandomPerson this work is still ongoing, we're migrating each script file individually over time
    – Kyle Pollard StaffMod
    Jan 31 at 19:51
  • 3
    @RandomPerson I got back from my sabbatical today! Happy to be back. (But not as happy as when I was on break 🙃) Jan 31 at 20:18
  • 1
    Highly likely strongly related to the recent rework: StackExchange.realtime.reloadPosts() is broken due to a change in data Mar 4 at 23:40

4 Answers 4

53

Quite a few user scripts rely on certain (undocumented?) properties of the StackExchange object. For example, the popular AutoReviewComments - Pro-forma comments for SE uses StackExchange.ready and StackExchange.options.

It's not your job to ensure backwards compatibility of these user-contributed scripts, but do you have any useful information you can share upfront? Or do we need to experience any bugs ourselves and wait for them to be posted on Meta?

8
  • 1
    Auto review comments is pretty broken at this point and my attempts to submit patches haven't been accepted. I am now using this replacement that I wrote: github.com/stephenostermiller/stack-exchange-comment-templates Dec 7, 2021 at 18:37
  • A JS change that would break all my user scripts would be the removal of jQuery. Dec 7, 2021 at 18:39
  • 4
    No plans as of now to remove jQuery. Though maybe we can safely upgrade to a newer version that what we are using (1.12.4).
    – Yaakov Ellis StaffMod
    Dec 8, 2021 at 19:24
  • 1
    @StephenOstermiller auto review comments broke recently?
    – Yaakov Ellis StaffMod
    Dec 8, 2021 at 19:25
  • 8
    Thank you for pointing this out. There's certainly a risk that these changes would break user scripts and leave no alternative option for maintainers. Breaking changes are frustrating, especially for maintaining user scripts. We don't want to irreparably break user scripts! We're working on a plan to keep the most important JS objects in place - we'll follow up soon with a separate Meta post to work with the community on understanding how this was being used with the goal of finding a stable solution here.
    – Kyle Pollard StaffMod
    Dec 8, 2021 at 20:11
  • 1
    @KylePollard we actually created and maintain a set of TypeScript type definitions for the global objects as a package - it would be very helpful to know precisely what changed so as those methods/nested objects/properties could be explicitly marked as deprecated so that the transition is reflected without trial and error? I understand it might not be possible, but if it is, it would save a lot of hassle. Dec 8, 2021 at 20:20
  • @YaakovEllis auto review comments doesn't work for close votes any more because of changes SE made some years ago to HTML and CSS. This patch would fix it: github.com/stephenostermiller/SE-AutoReviewComments/commit/… but it looks the author isn't active anymore. Dec 8, 2021 at 20:48
  • 3
    We've provided a follow up on providing a JavaScript API for userscripts as well as documenting essential Stack Exchange Userscripts and the dependencies they utilize. We'd love to hear your feedback there!
    – Kyle Pollard StaffMod
    Dec 14, 2021 at 16:17
34

This process sounds like a lot of work. I wish you well in doing it.

When you release each of these changes, please tell us what changed, at least what area of functionality changed (e.g. just add/edit an answer on a general announcement question, which interested users can follow).

These changes are almost certainly going to break userscripts; some will probably break multiple times. Keeping us informed as to what changed, even if that's just a fairly broad statement of something like "Made adjustments to StackExchange.realtime", would significantly help to reduce the amount of effort which userscript maintainers expend and the overall impact on users. It also allows us to be more proactive about finding areas where our userscripts may have broken, without needing to either test everything or wait for users to encounter problems and report them.

Informing us what changed will also allow users to focus on testing your code without userscripts, which should get you more useful bug reports. That will, hopefully, allow you to squash the bugs faster.

While detailed change information would be helpful, I don't feel it's reasonable to ask for that. However, just the broad strokes of the changes in each notable release would be quite helpful in reducing the impact on both userscript maintainers and users.

3
  • 8
    hm. In a sense - to some extent, something dangerously close to a public changelog?
    – Journeyman Geek Mod
    Dec 7, 2021 at 18:17
  • 3
    It would be quite literally a public changelog, being that it would publicly log the changes. That's fine though right? The reasoning is pretty solid. The devs don't need to be super specific, but general directions of what changed would be helpful. Dec 8, 2021 at 15:09
  • 2
    I understand how frustrating it is to have an unstable API for userscripts and how undocumented changes would frustrate maintenance. We're collecting more information on providing a JavaScript API for userscripts and would love to work on providing a stable and documented solution going forward.
    – Kyle Pollard StaffMod
    Dec 14, 2021 at 16:21
13

EDIT: This issue is reported as fixed in this comment. Thank you!


There have already been three bug reports from desperate souls for whom essentially all JavaScript stopped working, apparently because of the use of optional chaining (whatever that is). Would you consider avoiding this particular feature?

I understand that officially, these browsers are unsupported or too old (meaning 1 year old, really!). But many users simply cannot install newer versions for one reason or another (e.g., on a work computer), and for them, this is an extremely disruptive change (show stopper): it makes the site essentially read-only. It’s quite different from the usual minor degradation of some elements of the UI that one might expect with an old browser.

2
  • Sadly not solved for me and my browser (Firefox 52.9.0) still not able to Accept all cookies, vote, comment, show more HNQ and so on
    – undefined
    Dec 15, 2021 at 7:23
  • So happy this is working again!!!! Thx! FF 64.11.0. I will say that BWC can be interpreted liberally. If it was not possible to reinstate the same functionality, say for Inbox and Achievements (expanded drop down), a simple fallback to open to the user profile page, inbox tab is better UX than nothing/breaking.
    – Ian W
    Dec 15, 2021 at 9:25
7

I've been wanting to ask: Do you use any automated testing framework for your frontend code?

The description sounds a lot like test-driven development.

2
  • 11
    We don't have any yet. This is one of the big pieces of tech debt that we need to make in order to allow us to put that in place (which is one of the reasons for this work).
    – Yaakov Ellis StaffMod
    Dec 8, 2021 at 19:28
  • 4
    You know, having an automated testing framework in place first would have been a really good idea before overhauling everything; that would have been my preferred priority. But that's so 90's.
    – Ian W
    Dec 10, 2021 at 6:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .