Unfortunately, there are differences between the code that runs on the server to make the final version of a post and the code that runs in the browser during the preview. These were once pretty close to the same, but they have drifted apart over time.
There have been some recent changes to the server-side code in order to insert
<span class="math-container">...</span> around MathJax delimiters (I'm not sure why, but I spect it is so that other parts of the system can recognize the math for things like hot questions and the difference displays for past edits). It seems that the code that is doing this is produce incorrect results, as the page you linked to includes
<img src="https://cdn.sstatic.net/Sites/math/img/logo.svg?v=2ad3f9715b74" alt="test <span class="math-container">$2$</span>"></p>
which is invalid HTML. The quotation marks for the math-container element end the quotation marks for the
alt attribute, which means the
$2$ is parsed as the end of the
img tag, and so the
$2$ now falls outside the
alt text, and is displayed by MathJax. This doesn't happen in the preview because the math-container tags are not inserted during previews.