9

While doing some data analysis I put together an application to pull the list of moderators for a site from /users?tab=moderators (example).

When I attempt to grab this page, either through my application or, for example, wget, I receive a 302 response directing back to /users?tab=reputation&filter=all.

Request:

GET /users?tab=moderators HTTP/1.1
Accept-Encoding: gzip
User-Agent: Java/1.7.0_55
Host: stackoverflow.com
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

Response:

HTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=utf-8
Location: /users?tab=reputation&filter=all
X-Frame-Options: SAMEORIGIN
X-Request-Guid: <redacted>
Set-Cookie: <redacted>
Date: Sun, 12 Apr 2015 20:41:18 GMT
Content-Length: 153

<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="/users?tab=reputation&amp;filter=all">here</a>.</h2>
</body></html>

Why is this response being received, and what do I need to include in the request to avoid it? I do not run into this issue when directly requesting any other pages on the site, and I do not run into this issue visiting /users?tab=moderators in a browser (even when not logged in).

Improve this question
3
  • 2
    I'm not sure about the redirect, but using the API for this would probably make more sense anyway.
    – Tim Stone
    Apr 12, 2015 at 20:54
  • @TimStone Yeah; the API would make more sense. My application is already well-equipped to parse HTML but not the API responses, and this particular action is an afterthought. I'm trying to reduce extra work, although it'd probably be faster at this point to just switch to the API instead of trying to solve the request issue.
    – Jason C
    Apr 12, 2015 at 20:57
  • 1
    ^ And I did just that. Still curious about the 302 though.
    – Jason C
    Apr 12, 2015 at 21:29

1 Answer 1

Reset to default
4

The User Agent header needs love.

This curl statement works for me and returns the html content of the moderators page:

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" https://stackoverflow.com/users?tab=moderators

While using your User Agent:

curl -A "Java/1.7.0_55" https://stackoverflow.com/users?tab=moderators

<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="/users?tab=reputation&amp;filter=all">here</a>.</h2>
</body></html>

returns a 302 with a Location header.

Only an SE developer can explain why they handle routes under /users differently when an short user agent is coming in. I do recall that if you scrape on SE it is considered a good practice to include in your user agent who you are. For that reason I include the url to my MSE user profile in the User Agent. Adding your profile link does indeed resolve your issue as well.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .