These things redirect https to http (which causes my access token to be revoked and fail the request).

  1. Googlebot user agent
  2. Googlebot IP range (I think)

It would be really nice if this didn't happen, but can anyone think of a workaround. Perhaps I can send my access token in a different way that the API allows over HTTP. Or maybe Stackoverflow can change this behaviour?

  • 1
    Why are you impersonating Googlebot? "Doctor, it hurts when I do this." "Then don't do that!" – Stijn Aug 29 '14 at 17:36
  • 14
    I'm not impersonating Googlebot. This is Googlebot :) – Ali Afshar Aug 29 '14 at 18:05
  • 1
    (Just as an aside, if you're going to publish the results somewhere, even if just internally at Google: both Stack Exchange and Stack Overflow like a space in their name.) – Arjan Aug 30 '14 at 9:31
  • So bad for variable names :D – Ali Afshar Sep 2 '14 at 19:07

This is a confluence of different things, so this is gonna be a kind of complicated answer.

While we're working on getting https everywhere, SO is available over https but it's not really supported yet. Because Googlebot will, at least in our experience, prefer crawling over https we explicitly redirect request from googlebot ips to http. We'll probably stop doing this once our infrastructure is ready for https everywhere.

The API wants some requests to come over https but is ok with http for all unauthenticated traffic. With regards to the API, https everywhere is already good to go. We don't require it except for authenticated requests, because that would be a breaking change in the 2.x-family.

Mix the two, and API requests that require authentication that also come from known googlebot will be redirected to http and then fail. Woops.

We're looking at fixing this, there's no reason not to allow googlebot to index (or otherwise query) https://api.stackexchange.com. I'll update this answer once the fix is in.

Our routing was updated last night to fix this, no traffic to api.stackexchange.com should be redirected from https to http now; regardless of origin.

  • "we explicitly redirect request from googlebot ips to http" -- to ensure the search results don't list HTTPS URLs, I assume? – Arjan Aug 29 '14 at 18:28
  • @Arjan yep, we don't want users to land on pages w/ dodgy certs, slow ssl, or mixed content warnings. All of those things are possible until we're 100% ready for https everywhere. – Kevin Montrose Aug 29 '14 at 18:29
  • 2
    @AliAfshar Soon™. I don't see why this wouldn't be fixed by Monday, hopefully sooner. – Kevin Montrose Aug 29 '14 at 18:33
  • Smart thinking, Kevin! (Fun fact as for warnings of dodgy certs: some of our test users got frightened by this week's new Chrome "you're not browsing privately" warning when using a test server for which they already knew the certificate was wrong, but for which the brand new warning made them think something else was happening...) – Arjan Aug 29 '14 at 18:33
  • @KevinMontrose wow, Monday or sooner would be great. Thanks! – Ali Afshar Aug 29 '14 at 19:08
  • 3
    @AliAfshar this should be fixed now, I would appreciate confirmation on your end; it's somewhat difficult for us to fake google traffic :) – Kevin Montrose Aug 30 '14 at 19:06
  • Thanks very much Kevin. I'll let you know today. – Ali Afshar Sep 2 '14 at 15:44
  • Works. Thanks for fixing this so quickly @KevinMontrose and team. – Ali Afshar Sep 2 '14 at 19:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .