We're Migrating our JavaScript

Question

TL;DR: We’re making a substantial repayment on our JavaScript technical debt. Apologies in advance for any bugs as we work through it!

An administrative note: Rather than posting bug reports as answers to this post, in this case we'd prefer you to post a separate question for each bug you find.

---

Stack Overflow's JavaScript setup hasn't substantially changed since, like, 2008. Here are a few fun facts about the way we build our JS.

• The process to install third party libraries is:

  1. Download a copy of the library's code
  2. Paste that code into a folder
  3. git commit

  Seriously! We have a third-party folder in the repo. It contains (very old versions of) a variety of libraries. Some of those libraries have multiple versions in subfolders. Some of those libraries have had ad-hoc edits made to them. It's difficult to upgrade dependencies and it's difficult to manage transitive dependencies.

• If you want to split your code up into multiple files, you need to manually include each of those files in the correct order using script tags.

  • This goes for files in the aforementioned third-party folder too.
  • It’s painful enough to discourage developers from splitting code up. We have quite a lot of 5000-line monster files.
  • In many cases, splitting code into files to make it more maintainable can be actively detrimental to performance.
  • Code gets written in shared files and then included everywhere. Most of our pages are downloading way more JS than they actually need.

• Files communicate with each other through global variables.

  • It's difficult to track down all the usages of a given function.
  • Intellisense tooling doesn't know which globals are available in which contexts. The namespace is cluttered up with irrelevant definitions and it's easy to accidentally use a function on a page where it isn't actually available.
  • Consistency across the codebase in this regard is not high. Some files use TS namespaces, others write to a global StackExchange object, others install variables directly into the global scope…

• We have an idiosyncratic home-made file-loading/module system (called StackExchange.using) with a variety of similar-but-different usage patterns.

We've bolted things on to this system over the years — most new code is written in TypeScript, for example — but the overall story of how we build and deliver JS hasn't really changed.

Anyway, that's enough background. You may be aware that we've recently integrated Webpack into our JS build pipeline. I'm writing today to let you know that the second phase of this migration is underway: we are rewriting our code to use ES modules. This'll help us manage dependencies (both internal dependencies and third-party ones, using NPM) and make it easier to write reliable JavaScript as time goes on.

However, this is one of those big change-every-file migrations, and of course each file has its own idiosyncrasies. I've set things up so we can mostly work one file at a time, and we will obviously test each piece of the migration before we ship it, but Stack Overflow is a complicated 13-year-old system and it's difficult to catch every possible edge case in testing. In other words,

The work on this has already begun. I've built out all of the infrastructure (I think), and we're working file by file. Every developer in the department is converting a handful of files each, and in most cases each of those conversions can and will be shipped independently.

We have quite a lot of files to convert (hundreds but not thousands) and we're working on this migration in parallel with other ongoing projects, so I'm not expecting it to be a particularly quick job. I'm about to go on a six week sabbatical and I'd be astonished (and delighted) if the work was 100% complete by the time I'm back! But hopefully it'll be mostly done by that point.

The ask for you Metazens is simple: please let us know if you notice something's broken. In other words, keep doing what you're doing! We really appreciate it when you inform us about bugs.

Thank you for your patience and help!

Answer 1

Quite a few user scripts rely on certain (undocumented?) properties of the StackExchange object. For example, the popular AutoReviewComments - Pro-forma comments for SE uses StackExchange.ready and StackExchange.options.

It's not your job to ensure backwards compatibility of these user-contributed scripts, but do you have any useful information you can share upfront? Or do we need to experience any bugs ourselves and wait for them to be posted on Meta?

Answer 2

This process sounds like a lot of work. I wish you well in doing it.

When you release each of these changes, please tell us what changed, at least what area of functionality changed (e.g. just add/edit an answer on a general announcement question, which interested users can follow).

These changes are almost certainly going to break userscripts; some will probably break multiple times. Keeping us informed as to what changed, even if that's just a fairly broad statement of something like "Made adjustments to StackExchange.realtime", would significantly help to reduce the amount of effort which userscript maintainers expend and the overall impact on users. It also allows us to be more proactive about finding areas where our userscripts may have broken, without needing to either test everything or wait for users to encounter problems and report them.

Informing us what changed will also allow users to focus on testing your code without userscripts, which should get you more useful bug reports. That will, hopefully, allow you to squash the bugs faster.

While detailed change information would be helpful, I don't feel it's reasonable to ask for that. However, just the broad strokes of the changes in each notable release would be quite helpful in reducing the impact on both userscript maintainers and users.

Answer 3

EDIT: This issue is reported as fixed in this comment. Thank you!

---

There have already been three bug reports from desperate souls for whom essentially all JavaScript stopped working, apparently because of the use of optional chaining (whatever that is). Would you consider avoiding this particular feature?

I understand that officially, these browsers are unsupported or too old (meaning 1 year old, really!). But many users simply cannot install newer versions for one reason or another (e.g., on a work computer), and for them, this is an extremely disruptive change (show stopper): it makes the site essentially read-only. It’s quite different from the usual minor degradation of some elements of the UI that one might expect with an old browser.

Answer 4

I've been wanting to ask: Do you use any automated testing framework for your frontend code?

The description sounds a lot like test-driven development.