I extract all external links in the text blocks in Stack Overflow posts and comments. 14.2% of the links on Stack Overflow are broken links. Posts with broken links receive fewer votes after the links become broken. After accumulation, broken links are common across the posts with different vote scores. Therefore, viewers would also encounter broken links in the posts with high vote scores.
We are glad to hear your comments, suggestions and feedbacks!
We present some of the key findings in our research as below:
Motivation
- A large number of links that point to the third-party websites are shared on Stack Overflow. This shows that the knowledge base of Stack Overflow also comprised of these external resources that are out of the control and management of Stack Overflow.
- Viewers frequently complaints the broken links problems in comments. For example, comments to the accepted answer (authentication error for mifare card “6982:Security status not satisfied”) that solves the authentication error for a Mifare card. Six comments complain about the broken links.
Approach
We extract all external links in the text blocks in Stack Overflow posts and comments since the launch of Stack Overflow. We then perform the link availability test using Scrapy. To identify the broken links, we obtain the HTTP response status code (i.e., response code) that is returned to the request for the resource referenced by the link. To avoid the IP being banned from the website, we obtain a list of proxies from Free Proxy List and make different requests using different proxies. For the same website, we set a 15 seconds delay for different requests. To mitigate the intermittent behavior, we perform one link availability test using an elastic compute service located in Virginia, U.S. in Dec 2019. For the links that are not responded with 2xx response code, in Jan 2020, we perform another link availability test using an elastic compute service located in Singapore. We identify the broken links that are not responded with 2xx response code in both trials.
Findings
Prevalence of broken links
14.2% (i.e., 1,687,995) of the links on Stack Overflow are broken links. 13.5% (i.e., 2,156,095) of the posts and comments with links have broken links. 10.8% (i.e., 2,493,328) of the occurrences of the links are broken links.
Response code of the broken links on Stack Overflow
What are the intended roles of the shared broken links?
We randomly sampled a statistically representative sample of 384 questions, 384 answers, and 384 comments using a 95% confidence level with 5% confidence interval. We performed an open coding process to check the discussion context where the broken links are shared.
Intended roles for broken links
What are the impacts of broken links on Stack Overflow?
Can the posts with broken links help users solve the programming related problems before the links are broken?
The answer is yes! Posts with broken links can help users resolve the programming related problems before the links become broken.
10.1% (i.e., 297,305) of the links in accepted answers are broken links. 10.7% (i.e., 338,705) of the accepted answers have broken links. 44.4% (i.e., 297,305) of the broken links in answers are posted in accepted answers. This shows that broken links are common in accepted answer. The answers with broken links can solve questioners' problems when the links were posted, i.e., before the links were broken.
This figure shows that viewers would vote on the posts with broken links as the posts without broken links before the links were broken, as there is no significant difference in the number of votes per post between the posts current with broken links and the posts currently without broken links in the first 30 days after their creation.
Can the posts with broken links help users solve the programming related problems after the links are broken?
The posts current with broken links could not solve viewers' problems as posts without broken links. This figure shows that viewers would vote less on the posts with broken links than the posts without broken links after the links were broken, as there is a significant difference in the number of votes per post between the posts with or without invalid links in the last 30 days before the collection of the dataset.
Are viewers less likely to encounter broken links in the posts with higher vote scores?
The answer is NO!
These figures show that broken links are common across questions and answers with different vote scores. Viewers can randomly encounter the posts with broken links with the posts with any vote scores. However, the usefulness of the posts with broken links are less than the posts without broken links as they receive less vote scores. Viewers would waste time in reading the posts with broken links and finally vote on the posts without broken links.
How do users deal with the posts with broken links?
Only 5.8% (i.e. 103,792) of the broken links were removed. 5.6% (i.e., 90,606) of the posts with broken links removed the broken links. 6.7% (i.e., 164,404) of revisions that changed the links in posts removed broken links. This shows that broken links attract limited attention on Stack Overflow. 1.57% (i.e., 25,443) of the posts with broken links are notified of the broken links with via one or more comments (i.e., notified posts). 14.3% (i.e., 3,648) of the notified posts removed the broken links. The proportion of the notified posts that removed broken links among all notified posts is 2.47 times larger than the proportion of the posts that removed broken links among the posts with broken links. This shows that when notified of the broken links, users are more likely to repair the broken links. We suggest Stack Overflow could design a mechanism to notify broken links in posts.
Do users follow community norms when posting links?
21.1% (i.e., 81) answers and 15.1% (i.e., 58) questions with broken links follow the community norms, i.e., quote the content of any links in the posts into quotation boxes, code blocks, or code snippets. This shows that users commonly not follow the Stack Overflow community norms. 10.2% (i.e., 39) answers and 12.5% (i.e., 48) questions with broken links quote the code of broken links. 5.5% (i.e., 21) answers and 0.3% (i.e., 1) questions with broken links quote the text of broken links. This shows that users pay more attention to the permanent visit of their code. 5.5% (i.e., 21) answers and 2.3% (i.e., 9) questions also quote other links that are not broken links. 70.1% (i.e., 60) answers and 84.5% (i.e., 49) questions that follow the community norms quote the broken links. Viewers cannot obtain the content of the broken links via a click but viewers can obtain the content of the broken links via the quotations from these links. This also shows that the importance of broken links, i.e., when users need to quote an important link, users would quote the broken links. We suggest researchers design a tool to identify the links that need quotation. Researchers also could design a tool to automatically retrieve the most relevant part from the shared links.
What are the characteristics of the websites that are referenced by broken links on Stack Overflow?
50% (i.e., 844,002) of broken links reference to the top 0.3% (i.e., 414) websites in terms of the number of the broken links referencing to them. 308,737 (i.e., 46.9%) of the websites that are referenced by Stack Overflow are referenced by broken links.
The websites that are referenced by fewer links on Stack Overflow are more likely to have broken links. For example, github.com is the second most commonly shared external websites on Stack Overflow (1,870,707 links on Stack Overflow reference to github.com). 6.6% of the links that reference to github.com are broken links. The broken links that reference to github.com account for 7.3% of the broken links on Stack Overflow. In contrast, among 468,577 links that reference the websites with 1–3 links that are shared on Stack Overflow, 43% of them are broken links. This proportion is 6.5 times higher than the proportion of broken links among the links that reference to github.com . The broken links that reference the websites with 1–3 links that are shared on Stack Overflow account for 16% of the total number of broken links. This indicates that the websites with 1–3 links that are shared on Stack Overflow contribute 2.2 times broken links on Stack Overflow more than github.com.
What are the characteristics of the broken links that are associated with different Stack Overflow tags?
The top 10 tags in terms of the number of associated broken links corresponds to 55.4% of the broken links on Stack Overflow.
Web development related technologies are more likely to have broken links.
The broken links in each tag are uniformly introduced by different websites. Focusing on repairing the broken links that reference to specific websites cannot help with the broken links problem associated with a certain tag.
Implications:
- To help viewers be aware of the broken links before clicking them, Stack Overflow could detect the broken links and mark the posts with broken links.
- To maintain the crowdsourced knowledge on Stack Overflow, Stack Overflow moderators could adjust the gamification system to encourage users to update broken links, e.g., reward badges or reputation scores.
- Stack Overflow could learn from Google to archive snapshots of links when they are posted and replace the broken links on Stack Overflow with the copies in the archive.
- Users are encouraged to post the code in a more permanent site, code blocks, and Stack Snippets, and not to remove the examples in the links in questions.
Please kindly share your suggestions! Thank you!
Update for "correlation between the year the link was posted and the percentage of link-rot"
Trendlines of broken links on Stack Overflow
The links that were posted earlier are more likely to be broken.
The proportions of broken links among the links that were posted per month and the month the links are posted are significantly correlated with Pearson’s correlation coefficient = -0.97 (p-value < 0.05). 22.9% of the links that were posted in Aug 2008 (the month when the Stack Overflow website was established) are broken. Links that were posted earlier are more likely to be broken.
Stack Overflow posts made in May 2017 have a total of 103,411 broken links (the spike in Figure 2). The number of broken links posted in May 2017 is 5.43 times greater than the average number of broken links that were posted per month. We manually check the links that were posted in May 2017 and observe that 78.8% (i.e., 882,362) of the links were posted by the URL Rewriter Bot. As is indicated in meta.stackexchange.com, the URL Rewriter Bot is used by Stack Overflow to update the schema of the links, i.e., replacing HTTP with HTTPS for security and privacy concern without checking their validity. However, 87,086 (i.e., 84.2% of the broken links posted in May 2017) broken links were posted by the URL Rewriter Bot.
