I've actually noticed this problem a lot on my computer too, and I've noticed that this only happens on my laptop, not on an iPad or on an Android smartphone.
While it doesn't disrupt my interaction with the site, it often causes the site to lag as I scroll up or down, but that's just as bad as it gets. I noticed that this happens because of the ad that is loading. Alternatively, you may have an adblocker installed, so that could be causing a slowdown in the page-loading.
So, to answer your question, this has nothing to do with your Internet connection, or StackExchange; it simply depends on your computer HDD/SDD speed, extra software interference like adblockers, background tasks runnning that slow overall PC performance and reduce RAM availability for your browser operation, and/or browser speed itself.
What I recommend doing is clicking the "X" button after the page has completely loaded; I've found that it causes no problem to the site if you do that.