5

I extract all external links in the text blocks in Stack Overflow posts and comments. 14.2% of the links on Stack Overflow are broken links. Posts with broken links receive fewer votes after the links become broken. After accumulation, broken links are common across the posts with different vote scores. Therefore, viewers would also encounter broken links in the posts with high vote scores.

We are glad to hear your comments, suggestions and feedbacks!

We present some of the key findings in our research as below:

Motivation

  • A large number of links that point to the third-party websites are shared on Stack Overflow. This shows that the knowledge base of Stack Overflow also comprised of these external resources that are out of the control and management of Stack Overflow.
  • Viewers frequently complaints the broken links problems in comments. For example, comments to the accepted answer (authentication error for mifare card “6982:Security status not satisfied”) that solves the authentication error for a Mifare card. Six comments complain about the broken links.

Approach

We extract all external links in the text blocks in Stack Overflow posts and comments since the launch of Stack Overflow. We then perform the link availability test using Scrapy. To identify the broken links, we obtain the HTTP response status code (i.e., response code) that is returned to the request for the resource referenced by the link. To avoid the IP being banned from the website, we obtain a list of proxies from Free Proxy List and make different requests using different proxies. For the same website, we set a 15 seconds delay for different requests. To mitigate the intermittent behavior, we perform one link availability test using an elastic compute service located in Virginia, U.S. in Dec 2019. For the links that are not responded with 2xx response code, in Jan 2020, we perform another link availability test using an elastic compute service located in Singapore. We identify the broken links that are not responded with 2xx response code in both trials.

Findings

Prevalence of broken links

14.2% (i.e., 1,687,995) of the links on Stack Overflow are broken links. 13.5% (i.e., 2,156,095) of the posts and comments with links have broken links. 10.8% (i.e., 2,493,328) of the occurrences of the links are broken links.

Response code of the broken links on Stack Overflow

top 10 response code for broken links on Stack Overflow

What are the intended roles of the shared broken links?

We randomly sampled a statistically representative sample of 384 questions, 384 answers, and 384 comments using a 95% confidence level with 5% confidence interval. We performed an open coding process to check the discussion context where the broken links are shared.

Intended roles for broken links

What are the impacts of broken links on Stack Overflow?

Can the posts with broken links help users solve the programming related problems before the links are broken?

The answer is yes! Posts with broken links can help users resolve the programming related problems before the links become broken.

  • 10.1% (i.e., 297,305) of the links in accepted answers are broken links. 10.7% (i.e., 338,705) of the accepted answers have broken links. 44.4% (i.e., 297,305) of the broken links in answers are posted in accepted answers. This shows that broken links are common in accepted answer. The answers with broken links can solve questioners' problems when the links were posted, i.e., before the links were broken.

  • This figure shows that viewers would vote on the posts with broken links as the posts without broken links before the links were broken, as there is no significant difference in the number of votes per post between the posts current with broken links and the posts currently without broken links in the first 30 days after their creation.

Votes before the links become broken

Can the posts with broken links help users solve the programming related problems after the links are broken?

The posts current with broken links could not solve viewers' problems as posts without broken links. This figure shows that viewers would vote less on the posts with broken links than the posts without broken links after the links were broken, as there is a significant difference in the number of votes per post between the posts with or without invalid links in the last 30 days before the collection of the dataset.

Votes after the links become broken

Are viewers less likely to encounter broken links in the posts with higher vote scores?

The answer is NO!

These figures show that broken links are common across questions and answers with different vote scores. Viewers can randomly encounter the posts with broken links with the posts with any vote scores. However, the usefulness of the posts with broken links are less than the posts without broken links as they receive less vote scores. Viewers would waste time in reading the posts with broken links and finally vote on the posts without broken links.

Broken links in posts with different votes

How do users deal with the posts with broken links?

Only 5.8% (i.e. 103,792) of the broken links were removed. 5.6% (i.e., 90,606) of the posts with broken links removed the broken links. 6.7% (i.e., 164,404) of revisions that changed the links in posts removed broken links. This shows that broken links attract limited attention on Stack Overflow. 1.57% (i.e., 25,443) of the posts with broken links are notified of the broken links with via one or more comments (i.e., notified posts). 14.3% (i.e., 3,648) of the notified posts removed the broken links. The proportion of the notified posts that removed broken links among all notified posts is 2.47 times larger than the proportion of the posts that removed broken links among the posts with broken links. This shows that when notified of the broken links, users are more likely to repair the broken links. We suggest Stack Overflow could design a mechanism to notify broken links in posts.

Do users follow community norms when posting links?

21.1% (i.e., 81) answers and 15.1% (i.e., 58) questions with broken links follow the community norms, i.e., quote the content of any links in the posts into quotation boxes, code blocks, or code snippets. This shows that users commonly not follow the Stack Overflow community norms. 10.2% (i.e., 39) answers and 12.5% (i.e., 48) questions with broken links quote the code of broken links. 5.5% (i.e., 21) answers and 0.3% (i.e., 1) questions with broken links quote the text of broken links. This shows that users pay more attention to the permanent visit of their code. 5.5% (i.e., 21) answers and 2.3% (i.e., 9) questions also quote other links that are not broken links. 70.1% (i.e., 60) answers and 84.5% (i.e., 49) questions that follow the community norms quote the broken links. Viewers cannot obtain the content of the broken links via a click but viewers can obtain the content of the broken links via the quotations from these links. This also shows that the importance of broken links, i.e., when users need to quote an important link, users would quote the broken links. We suggest researchers design a tool to identify the links that need quotation. Researchers also could design a tool to automatically retrieve the most relevant part from the shared links.

What are the characteristics of the websites that are referenced by broken links on Stack Overflow?

50% (i.e., 844,002) of broken links reference to the top 0.3% (i.e., 414) websites in terms of the number of the broken links referencing to them. 308,737 (i.e., 46.9%) of the websites that are referenced by Stack Overflow are referenced by broken links.

The websites that are referenced by fewer links on Stack Overflow are more likely to have broken links. For example, github.com is the second most commonly shared external websites on Stack Overflow (1,870,707 links on Stack Overflow reference to github.com). 6.6% of the links that reference to github.com are broken links. The broken links that reference to github.com account for 7.3% of the broken links on Stack Overflow. In contrast, among 468,577 links that reference the websites with 1–3 links that are shared on Stack Overflow, 43% of them are broken links. This proportion is 6.5 times higher than the proportion of broken links among the links that reference to github.com . The broken links that reference the websites with 1–3 links that are shared on Stack Overflow account for 16% of the total number of broken links. This indicates that the websites with 1–3 links that are shared on Stack Overflow contribute 2.2 times broken links on Stack Overflow more than github.com.

Broken links thet point to the websites with different numbers of links that are shared on Stack Overflow

What are the characteristics of the broken links that are associated with different Stack Overflow tags?

The top 10 tags in terms of the number of associated broken links corresponds to 55.4% of the broken links on Stack Overflow.

Web development related technologies are more likely to have broken links.

Top 10 tags in terms of the number of broken links

The broken links in each tag are uniformly introduced by different websites. Focusing on repairing the broken links that reference to specific websites cannot help with the broken links problem associated with a certain tag.

websites distribution for broken links associated with each tag

Implications:

  • To help viewers be aware of the broken links before clicking them, Stack Overflow could detect the broken links and mark the posts with broken links.
  • To maintain the crowdsourced knowledge on Stack Overflow, Stack Overflow moderators could adjust the gamification system to encourage users to update broken links, e.g., reward badges or reputation scores.
  • Stack Overflow could learn from Google to archive snapshots of links when they are posted and replace the broken links on Stack Overflow with the copies in the archive.
  • Users are encouraged to post the code in a more permanent site, code blocks, and Stack Snippets, and not to remove the examples in the links in questions.

Please kindly share your suggestions! Thank you!


Update for "correlation between the year the link was posted and the percentage of link-rot"

Trendlines of broken links on Stack Overflow

The links that were posted earlier are more likely to be broken.

Trendlines of broken links on Stack Overflow

The proportions of broken links among the links that were posted per month and the month the links are posted are significantly correlated with Pearson’s correlation coefficient = -0.97 (p-value < 0.05). 22.9% of the links that were posted in Aug 2008 (the month when the Stack Overflow website was established) are broken. Links that were posted earlier are more likely to be broken.

Stack Overflow posts made in May 2017 have a total of 103,411 broken links (the spike in Figure 2). The number of broken links posted in May 2017 is 5.43 times greater than the average number of broken links that were posted per month. We manually check the links that were posted in May 2017 and observe that 78.8% (i.e., 882,362) of the links were posted by the URL Rewriter Bot. As is indicated in meta.stackexchange.com, the URL Rewriter Bot is used by Stack Overflow to update the schema of the links, i.e., replacing HTTP with HTTPS for security and privacy concern without checking their validity. However, 87,086 (i.e., 84.2% of the broken links posted in May 2017) broken links were posted by the URL Rewriter Bot.

20
  • 1
    Six comments complain about the broken links. That is six missed opportunities to fix those links ...
    – rene
    Sep 9 '20 at 7:06
  • 4
    Interesting findings, but what are practical implications and solutions? Why was this endeavour carried out? Your post could use some introduction.
    – Luuklag
    Sep 9 '20 at 7:41
  • I also wonder to what extent posts have lost their value due to the loss of the external resource. And is there a correlation between the year the link was posted and the percentage of link-rot?
    – Luuklag
    Sep 9 '20 at 7:51
  • .0004 votes per post per day is not a meaningful difference. Breaking is evenly distributed. Hence both broken & unbroken links are irrelevant. PS The "Implications" are not implications.
    – philipxy
    Sep 9 '20 at 7:54
  • As it should be @philipxy. Links should be meant as supplementary information, the essence should be contained within the post body. So this is expected behaviour.
    – Luuklag
    Sep 9 '20 at 7:58
  • @Luuklag Thank you for your comments! I update the question and add more details related to the proportion of broken links among the links that were posted per month. Please kindly advise!
    – lim1718
    Sep 9 '20 at 8:08
  • @Luuklag It "should" be in the sense that it's desired & requested. But that doesn't mean we can "expect" it. PS The irrelevance of links suggests irrelevance of the rest of the post.
    – philipxy
    Sep 9 '20 at 8:14
  • @philipxy To check whether the differences between the posts with broken links and the posts without broken links are statistically significant in the number of votes per post in the last 30 days before the collection of dataset, we perform a Mann Whitney test. As a result, the difference between posts with broken links and the posts without broken links in the number of votes per post in the last 30 days before the collection of the dataset is significant(pvalue < 0.05).
    – lim1718
    Sep 9 '20 at 8:18
  • "Statistically significant" is not "meaningful". (And my comment presumed the .0004.)
    – philipxy
    Sep 9 '20 at 8:18
  • Have you differentiated between examples and actual links to sites. e.g. mysite.com/path and wikipidea.org/invalidpath? In which case the number of actual "broken links" is much lower than suggested. I see so many postings with examples of web addresses that are not links to be followed. Sep 9 '20 at 8:21
  • @BrianTompsett-汤莱恩 Sorry for that we do not exclude these links. This is because these example links might be useful for viewers to understand the questions. If the content of the example links is the same to the viewers, viewers do not need to post another question.
    – lim1718
    Sep 9 '20 at 8:24
  • 1
    I really admire your time and dedication, great effort. But I don't see the point. OK, so there are broken links. What about it? We already know about it. SE knows about it. They even made experiment with "broken links review" in the past, where the system detected broken links and created review queue for them where users could suggest correct links instead. It failed, and doubt they'll revive it now. Sep 9 '20 at 8:24
  • 1
    Given the, er, nontraditional numbering for the figures and tables, I'm guessing this post is an excerpt/reduction of a larger report. Is that available more widely? If so, that context may help in addressing concerns about why you went to what is clearly a significant effort. Sep 9 '20 at 9:02
  • 1
    @lim1718 thanks for the reply, but again: what's your goal? Also, do you represent a company, or working with other people on some big project that involves more sites? Sep 9 '20 at 9:12
  • 1
    @Shadow Wizard Wearing Mask Thanks for your comments! We are researchers from a university. This post is a part of my research paper related to the knowledge maintenance on SO. I am happy to add more details required by the SO community. I would appreciate it if the community could provide feedbacks and shed light on our future research! I will public my findings once the paper is published. Thank you!
    – lim1718
    Sep 9 '20 at 10:20

You must log in to answer this question.

Browse other questions tagged .