I just found that the Stack Overflow font is bigger than the font on math.stackexchange. Look at last lines on two images.
Why do they use different fonts?
Stack Overflow, as the first site on the network, always had sans-serif, I think. It's pretty common for websites. Sans-serif is essentially the default for Stack Exchange sites.
The math site did not start out with serifs. This changed when the site got a design because Jin, the designer at the time, "believe[d] the serif typeface on the header and questions work well for the academic look." It also works well because MathJax is serif.
Other sites use serifs for other reasons. For example, Mi Yodeya, the site about Judaism, uses serifs because it makes Hebrew more legible.
Jin also did the design for English Language and Usage, which uses serifs. The site design was chosen to "feature beautiful typography and invoke a vintage/warm feeling." It also works better for legibility, especially with IPA.
The network design was majorly simplified in 2018 and many people requested that the font style not be switched on various sites. I'm not sure if any sites were switched to sans-serif as a result of this, but this change did introduce a lot of sans-serif elements into the design on all serif sites (such as tags being sans-serif).
Some sites decided that serif fonts were more appropriate for their subject, so they have our serif font stack rather than the default sans serif stack.
There's more information on this answer about our fonts but I haven't checked whether the exceptions are still correct (I don't think they are).