Roadmap to HTTPS: serving and uploading HTTPS-images only

Question

This is a cross post from mSO, but affecting the whole network. 1. is already enabled network wide, 2. and 3. will be enabled shortly on mSE and another SE community (main site + meta) of our choosing for the first round of site-wide HTTPS testing. Read the full master plan!

As requested by @patrick, lets go kill dead images. We don't have on-demand review queues, and editing a list of posts works OK for small batches¹, let's take crowdcrafting.org / pybossa for a spin. So, without further ado:

I need a bosse²

• 2017-03-07 (meta.)security.SE processed, (only) 28 tasks added to ^^
• 2017-03-08 serverfault, superuser and their metas processed, 1069 tasks
• 2017-03-09 25 main SE sites processed, 7911 (somebody call the police) tasks
• 2017-03-09 deleted mSE posts have been removed from the crowdsourcing queue
• 2017-03-10 rest of the public SE sites done, 6225 tasks for the queue
• 2017-03-11 fixed 2 URLs on askubuntu.com which just received https support, but affected ~2k posts
• child metas for .SE sites will be processed after the meta move

---

In the next 6-8 weeks, we'll be rolling out some changes to address the mixed content issue on the first page load, namely:

1. Imgur URLs are converted from HTTP to HTTPS. This will be a HTML baking change, so no Markdown will be affected. This has already shipped, and all the old posts were rebaked.

2. Prevent submission of posts that contain HTTP images. Instead of the HTTP image the markdown editor will show an additional error... ... with an option to bring up the image uploading tool with the URL pre-populated, so you can easily upload it to Imgur (we can't do that on your behalf, because cc-by-sa, and whatnot):

3. After this is enabled, old posts that have HTTP images, accessible via HTTPS, will be edited accordingly. This will be an actual markdown change, attributed to the "URL Rewriter Bot"; posts won't be bumped. (example)

4. Rebake remaining posts with HTTP images, so that images linking to HTTP addresses will become links - most of those are dead anyway. This will be a HTML baking change, so no Markdown will be affected, but it'll remove any mixed content even when viewing old revisions.

---

_{[1] ... and inspired by Let's rescue wayward resource requests! (trial run) [2] ... depending on how this works out, we might use it for other stuff too}

Answer 1

Nice work on finally bringing HTTPS support in!

I want to note something on point 4 (emphasis mine):

Rebake remaining posts with HTTP images, so that images linking to HTTP addresses will become links - most of those are dead anyway.

If most of those links are dead anyway, why not go through the hassle once and check if they are actually dead or not. If they are, just remove them! Why would we need to rely on the community to fix one by one if we can do this automated at once?

---

It seems automating this is difficult, so m0sa set up a crowd sourcing project to let us do the work manually. Please contribute if you can! About 2500 posts to go...

Answer 2

Please make it so that protocol relative URLs (i.e. //foo.bar/fum.png) will still be possible (you write that you will require https:) and are left alone.

Answer 3

Is anyone else having problems with imageshack images? They keep coming back with failures.

Answer 4

I've written the following snippet (also posted at Help us fix broken images!) to filter and return the Crowdcrafting tasks by site.

It currently returns up to 100 tasks (the maximum the API allows). It looks like a lot of sites have fewer affected posts than that, there's no instant way to get further posts for the ones that do though. It is possible to paginate results through the API (see the last note here), so maybe I'll look at adding that later.

I've included links to the SE post (both view and edit links) and the Crowdcrafting task page so that you can hit "Done" on the task, which should eventually get you more tasks (it takes 2 "Done"s to remove the post from the queue I believe).

Just pick a site, hit "Get Tasks" and work through the links...

function getget(url, callback) {
  var xhr = new XMLHttpRequest();
  xhr.addEventListener('load', function() {
    var response = JSON.parse(xhr.responseText);
    callback(response);
  });
  xhr.open('GET', url);
  xhr.send();
}

function loadSites() {
  var url = 'https://api.stackexchange.com/2.2/sites?pagesize=300&filter=!*L6Sij27hkbD7Gso';
  getget(url, listSites);
}

function listSites(sites) {
  var goBtn = document.getElementById('getTasks');
  var sitesList = document.getElementById('sites');

  for (var i = 0; i < sites.items.length; i++) {
    var siteUrl = sites.items[i].site_url.replace(/^https?\:\/\//i, "");
    var opt = document.createElement('option');
    opt.value = siteUrl;
    opt.textContent = sites.items[i].name;
    sitesList.appendChild(opt);
  }

  goBtn.innerText = 'Get Tasks';
  goBtn.disabled = false;
}

function listTasks(tasks, el) {
  for (var i = 0; i < tasks.length; i++) {
    var task = tasks[i],
      info = task.info,
      taskID = task.id,
      // borked column headers again...
      postID = info['PostId'] || info['12'] || info['149'] || info['73'] || '',
      siteName = info.BaseHostAddress || info['meta.serverfault.com'] || info['askubuntu.com'] || info['sound.stackexchange.com'] || '';

    lastID = taskID;

    var span = document.createElement('span');
    span.innerText = 'Post ' + postID + ':';

    var seViewLink = document.createElement('a');
    seViewLink.href = '//' + siteName + '/questions/' + postID;
    seViewLink.innerText = 'View';

    var seEditLink = document.createElement('a');
    seEditLink.href = '//' + siteName + '/posts/' + postID + '/edit';
    seEditLink.innerText = 'Edit';

    var ccLink = document.createElement('a');
    ccLink.className = 'ccLink';
    ccLink.href = '//crowdcrafting.org/project/sehttpimagescleanup/task/' + taskID;
    ccLink.innerText = 'Crowdcrafting Task ' + taskID;

    var li = document.createElement('li');
    li.appendChild(span);
    li.appendChild(seViewLink);
    li.appendChild(seEditLink);
    li.appendChild(ccLink);
    results.appendChild(li);
  }
}

function init() {
  var goBtn = document.getElementById('getTasks');
  goBtn.addEventListener('click', function() {

    var results = document.getElementById('results');
    results.innerHTML = '';

    var site = document.getElementById('sites').value;
    var searchUrl = '//crowdcrafting.org/api/task?project_id=4667&limit=100&info=BaseHostAddress::' + site;
    getget(searchUrl, listTasks, results);

    // task columns are borked... let's run this a few times
    var searchUrl = '//crowdcrafting.org/api/task?project_id=4667&limit=100&info=meta.serverfault.com::' + site;
    getget(searchUrl, listTasks, results);
    var searchUrl = '//crowdcrafting.org/api/task?project_id=4667&limit=100&info=askubuntu.com::' + site;
    getget(searchUrl, listTasks, results);
    var searchUrl = '//crowdcrafting.org/api/task?project_id=4667&limit=100&info=sound.stackexchange.com::' + site;
    getget(searchUrl, listTasks, results);
  });
  loadSites();
}

// go!
init();

ul { list-style: none; margin: 1em 0; padding: 0; }
li { margin: 0; padding: .5em 0; }
span { display: inline-block; width: 6em; }
a { color: #fff; background-color: #03A7DD; border-radius: 4px; padding: .25em .5em; margin: 0 .5em 0 0; text-decoration: none; }
a.ccLink { background-color: #2B9884; }

<label>Site: <select id="sites"></select></label>
<button id="getTasks" disabled>Loading Sites...</button>
<ul id="results"></ul>

Note, links in Stack Snippets don't really work... just open them in a new tab (ctrl+click, middle-click, right-click+"Open in New Tab" or whatever)

Answer 5

This is nice =).

It would be nicer if someone can write a SEDE query that can find old posts that need fixing - i.e. posts with non-stack.imgur images included.

And also, while we're here, it would be nice if you guys could eliminate dependences to profile images from cdn.facebook and other trackersy domains.

Answer 6

I just decided to try the crowdcrafting.org link in the question above, and the first task I received contained a link to https://meta.security.stackexchange.com/questions/227.

Obviously, that's not going to work (unless I edit the URL or manually add a security exception for the invalid certificate), since it's still using the old meta.security hostname. I guess that something that should be fixed?

^{Also, it turns out I can't handle that task anyway, since I'm 22 rep points short of having edit privileges on security.SE meta. But that's not really something the crowdcrafting site could possibly know.}

Answer 7

I found a bug: First task I got was http://security.stackexchange.com/questions/17393 which was complaining about an img tag which was in backticks so it wasn't actually an image. This img tag doesn't need fixing, but it ended up in the queue.

Answer 8

I just wanted to say thanks for having the editor make it super easy to upload. It saves a lot of time...

I wish I wasn't rate limited on my edits because I was moving thru a lot...