32

I think so far this has only been mentioned in comments, like about preventing link rot. So:

What smart ways to prevent images from disappearing can we come up with? Like maybe: how could we prevent image rot by requesting (some) images at some regular interval, or how could we get all (existing and future) images into the SE provided pro account at imgur.com?

Some thoughts:

  • Editing posts to move images to imgur.com bumps the question onto the front page.

  • When the blog's images were lost, some of you came up with some very smart solutions to get them back, even from the visitor's browser cache!

  • Some users explicitly want to host their images elsewhere, to be able to update them and manage them without having to change the URL in the post, or because it makes them feel "more in control" with the files. Also, some workplaces block imgur.com (like using Websense), making some want to use other image hosters.

  • Can we somehow tell if an image at imgur.com has been uploaded to the SE pro account, or to an unrelated (possibly free) account? Since May 2011, images uploaded to the SE pro account use the i.stack.imgur.com domain, which apparently is a commercial Imgur installation for the Stack Exchange network. Before that, the image URL did not reveal if images were uploaded to some free account or the SE account. But then, in June 2011, SE changed many existing URLs to use the new URLs, even for images that are not hosted on the SE account. That effectively means some non-SE images are no longer linked, and hence will not get any views any more, and will be gone after some months.

  • Does the SE pro account need a backup? Its TOS claims, of course: Imgur makes no guarantee of availability of service and reserves the right to change, withdraw, suspend, or discontinue [...]. (Remember Flickr accidentally deleted 4,000 images of a paid subscription, and officially did not provide a backup, nor any means to restore existing URLs? And one can use filmot.com as a proxy to Imgur, but that does not keep its own copies.) Images uploaded to i.stack.imgur.com do probably get backed up, as Waffles commented: Our main goal here is to have local backups of images you folk upload and dedicated servers.

  • tinypic.info (not .com) seems to have gone bad: The website at img5.tinypic.info appears to host malware – software that can hurt your computer or otherwise operate without your consent. And existing references to images (such as img5.tinypic.info/files/361e5z58gdhp3kt1ryr6.jpg) are now serving ads, no images. But luckily these are HTML, so not visible when embedded as an image.

  • Do we need to detect image rot as well? Detecting it might give us an idea if this is a problem? Detecting might be hard though: nowadays imgur.com throws a 404 Not Found (with an error image in the same response), but it used to send a 301 Moved Permanently result for that. Things are really difficult for image hosters that respond with 200 OK, like tinypic.com gets one a 302 Moved Temporarily followed by a 200 OK for the 404 image.

Not necessarily a problem; related: Could we have the ability to mark a change as minor in questions or answers?

15
  • 5
    +1 for for your last point. I'd be interested to see if anyone can come up with a method to detect image rot (from the data dump?) so we can measure the problem. Jan 21, 2011 at 13:16
  • 1
    Good point. It's truly a tragedy that the SO birthday cake photos are gone forever....
    – Pops
    Jan 21, 2011 at 17:50
  • @Arjan: Argh! I guess it's back to the data dump after all....
    – Pops
    Jan 22, 2011 at 19:24
  • @Popular Demand, the cake pictures are still visible in an answer, they are only gone in the question: (10k+ required) stackoverflow.com/questions/2420689/… Jan 23, 2011 at 10:56
  • @Gamecat, ah. I thought it had been hard-deleted. Sometimes I feel like people shouldn't really be allowed on Meta until they have 10k....
    – Pops
    Jan 24, 2011 at 15:48
  • @Bill - posted a rough analysis of link rot, based on the data dump. Jan 25, 2011 at 23:31
  • @Bill: since ImageShack is #2 in images. Can you (or any of the team), please, register the domain stream.imageshack.us/content.php?page=register_domain so ImageShack stop blocking the images? Please? Mar 14, 2011 at 22:19
  • @Eduardo Molteni: I can't do that myself, but you should post it as a new question tagged feature-request. Mar 14, 2011 at 22:24
  • @Eduardo, that is not preventing image rot, is it? I'd favor somehow ensuring the usage of the Stack Exchange Imgur account.
    – Arjan
    Mar 14, 2011 at 22:47
  • @Arjan: there are 6013 images already there. We cant ignore them. Mar 14, 2011 at 22:55
  • "Alan Schaaf, the man behind Imgur, generously provided us a network-wide “pro” account that keeps any images hosted through our websites around indefinitely." blog.stackoverflow.com/2010/08/new-image-upload-support Mar 29, 2011 at 17:01
  • True, @Jeff. But is there any need (and any way) to ensure all images are uploaded to that Pro account? And even then, though Alan has been working hard to restore SE images that were lost earlier, would it need a backup anyhow?
    – Arjan
    Mar 31, 2011 at 13:24
  • The imgur images here are no longer available. What happened?
    – M. Tibbits
    Jul 22, 2011 at 2:52
  • 1
    @M.Tibbits, see Images manually uploaded to Imgur are not displaying?, the solution being uploading the images again, to ensure they're on the SE Pro account that only SE controls.
    – Arjan
    Jul 22, 2011 at 7:07
  • i.stack.imgur.com/tUZYH.jpg 《 technically not an orphan anymore, but you get what I mean
    – bobobobo
    Nov 21, 2019 at 13:29

8 Answers 8

38
+200

Here's some preliminary results of rough digging for stale href and image links in the most recent data dump. Looking at posts only, ignoring comments for now.

Links

  • ~1 700 000 links in total, of which ~865 000 distinct.
  • Requested 'HTTP HEAD' on a general sample of ~22 000, spread over the full period covered by the dump, plus for the 1 000 most frequently appearing links.
  • 404s made up 2.5% of the general sample, which might imply several tens of thousands in the whole dataset. The 'rate' varies from around 1% in August 2010 to 4% in November 2008.
  • Only one 404 in the most frequently posted 1 000 links: http://www.json.org/json2.js (835th most frequent).
  • Have not pinged any Stack Overflow, Google, or Wikipedia links as yet.
  • Used the CreationDate attribute for timestamp.

Images

  • Similar analysis for images: ~40 000 images embedded, of which ~39 000 distinct.
  • Ran HTTP HEAD on a sample of 600 image links spread over the full period covered by the dump.
  • 404s made up 3.3% of the sample.

Here's a plot showing the relative portion of the links per month found to 404. The trend is vague, but frequency of stale-links-per-post in 2008 is, roughly, double that found in December 2010. relative proportion of stale links per month

The top 10 linked-to domains are:

                            n.links
                            -------
stackoverflow.com            146512
msdn.microsoft.com           144241
en.wikipedia.org              91993 
code.google.com               35804
java.sun.com                  29896

developer.apple.com           20935    
blogs.msdn.com                17001
php.net                       16962
rads.stackoverflow.com        16673       
github.com                    15897

Across the general link sample the response codes were:

ret   links
code  tried    %
----  ----- ----
200   16989 78.1
204       2  0.0
300       1  0.0
301    2498 11.5
302     803  3.7
303      16  0.1
304       1  0.0
307       5  0.0
400     130  0.6
401       4  0.0
403      47  0.2
404     521  2.4
405     219  1.0
408       1  0.0
410       9  0.0
500      30  0.1
501       3  0.0
502       1  0.0
503      60  0.3
FAIL    425  1.9

The top 10 image host domains were:

                            n.links
                            -------
imgur.com                     12778
imageshack.us                  6013
tinypic.com                    2301
freeimagehosting.net            954
flickr.com                      815

images-amazon.com               692
photobucket.com                 690
wikimedia.org                   621
ggpht.com                       483
google.com                      398

Across the general image sample the response codes were:

ret   links
code  tried    %
----  ----- ----
301      23  3.6
302      15  2.4
400       3  0.5
403       2  0.3
404      21  3.3
FAIL     12  1.9
OK      557 88.0

The fails are mostly timeouts (used 15 seconds) but also include around 90 'localhost' or 127.0.0.1 type links. Some are just 'garbage', but no analysis done as yet.

3
  • 13
    Excellent job! I really appreciate you taking the time to put this together. Jan 26, 2011 at 0:22
  • It seems images uploaded to the SE pro account now have i.stack.imgur.com as the domain?
    – Arjan
    May 7, 2011 at 13:27
  • What? No 418? What's the world coming to? Jul 7, 2011 at 7:50
8

Some stats on images:

SO       4,917,235 posts with 53,395 having images (1.1%)
SU         229,422 posts with  9,237 having images (4%)
SF         231,511 posts with  2,071 having images (0.9%)
UI           4,155 posts with    441 having images (10.6%)
photo       10,048 posts with    611 having images (6%)

apple.se has about the same proportion of images as photo.se, surprisingly. Query used was:

select count(*) from Posts 
where DeletionDate is null 
and dbo.RegexIsMatch(LOWER(Body), 'http://[^\s]+\.(gif|png|jpg)') = 1
6

In August 2010 I fetched a copy of all images in posts on SO/SF/SU/META, based on the URLs found in the data dump (probably the one of July).

Of the 18549 images for SO, 17437 could be downloaded, which means 6% of the images were broken.

I still have all those images on disk, in case more are broken by now and somebody wants to start an effort to re-upload them somewhere or such.

5
  • 1
    Just because an image could be downloaded doesn't mean that it wasn't broken. It's very well possible that you happily downloaded an error-image. Jul 21, 2011 at 15:22
  • @Code: Yeah, but I think you usually would more likely get a 404/... or a HTML page on error.
    – sth
    Jul 21, 2011 at 15:24
  • In my experience it's very common for image hosters to either directly return a 200 on error or 3xx followed by 200. Jul 21, 2011 at 15:26
  • In a previous version of my question (posted after you downloaded), Imgur was still sending 301 Moved Permanently for expired images too; nowadays it's sending 404 Not Found for unknown URLs so I guess it sends the same for expired URLs. Some others do indeed even send 200 OK, @Code. That aside: quite nice you've done some statistics and backed up! Maybe the same mechanism can be used for a May or June 2011 dump, to back up non-SE-account images that got their URL replaced?
    – Arjan
    Jul 21, 2011 at 19:56
  • It would be useful if you made a WARC with wget and uploaded said WARC file to archive.org so that it's eventually ingested in the wayback machine.
    – Nemo
    May 17, 2015 at 10:53
5

Now that a SE-specific i.stack.imgur.com URL is used, and the excellent JavaScript preview reveals that the team has put some smart realtime URL parsing in place: would it be feasible to show a warning while editing? Like:

                Warning: It seems you're referring to an image that is not hosted on our servers. To ensure images are consistently available, please use the image upload button from the toolbar.

Not enforcing the uploading still allows for dynamic images, where really required.

Opening the existing help text with the added warning makes things a bit huge, I think:

                Huge help and warning combined

The preview even recognizes site URLs to Replace trilogy raw links with the current question title?, and replaces too!

2
  • 1
    I'd reword that to: "To ensure images are consistently available, ..." The alternative ("backed up") is potentially confusing: the user isn't trying to back anything up. Why should they care if it's backed up? This explains why. Jul 23, 2011 at 12:19
  • The red-dotted line is just my browser's spell check. Still, it would be nice to highlight the troublesome URL, so I figured I'd leave it in the mockup... ;-)
    – Arjan
    Jul 23, 2011 at 12:52
4

Now that a SE-specific i.stack.imgur.com URL is used: would it be feasible to detect all references to non-SE images and automatically upload them to the Pro account while saving the post?

I see many problems:

  • saving posts might take much longer: confusing
  • uploading might fail; then what?
  • it would be SE (not the user) that might be uploading copyrighted material
  • should be smart about fake URLs (like: when I use http://example.org/image.png I get...)
  • this might also need to apply to bare links?
  • the image might actually be some dynamic content that is part of the problem description, or the solution (like in http://dummyimage.com/60x20/fff/f00.png the URL should not be changed)
  • no more fun dynamic content, like Pekka's clock (but: also no more tracking images; good!)

When implemented, it might even be retrospectively applied to existing posts?

3

Now that a SE-specific i.stack.imgur.com URL is used: what about rejecting posts that use images that do not refer to i.stack.imgur.com, just like LMGTFY links are automatically banned?

Oops! Your edit couldn't be submitted because:

  • To ensure images are kept forever, please use the image upload button in the toolbar. To explicitly refer to another URL, format the URL as code.

This would be annoying to those who do want to help, but don't bother about image rot. And the check might need to be disabled in certain cases, like:

  • on devices that do not support the editing toolbar
  • when editing posts for which one is not the owner; we might not want to burden the editor with changing the existing images in the post

(Some of the problems for automatically uploading would apply too.)

2

I would be very happy to be informed (var a top bar) of any link (including images) in one of my posts that is broke, so I can go and fix it with an edit.

2
  • Yes, too bad one cannot search the raw Markdown source anymore (and hence cannot search our own posts for known non-Imgur hosters).
    – Arjan
    Mar 16, 2011 at 13:02
  • Meanwhile, the search for URLs is in beta again.
    – Arjan
    Apr 13, 2011 at 16:02
1

There are obviously many very clever ways to go about this, however I think a decent first line of defense after an image has been detected as blocked/missing/whatever would be to try the imgur mirror over at filmot. Every image is mirrored over there with the same name, i.e. www.imgur.com/abc123 is the same page as www.filmot.com/abc123. This is probably worth noting.

3
  • 1
    Too late: new images that are uploaded to stack.imgur.com cannot be fetched this way. Also, filmot.com claims to be a proxy: if the original is gone, then so is the link at filmot.com.
    – Arjan
    May 8, 2011 at 11:06
  • Ah, gotcha. Well in that case it can still be used as a simple solution when imgur is down for some reason. May 8, 2011 at 19:20
  • No, too bad. If a website is down then a proxy to that very same website won't work either.
    – Arjan
    May 8, 2011 at 21:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .