5

I just had a question about wget usage (for Rip javadocs from a doc site to a local zip file), and wondered where to post it. Here the tags on the sites I checked:

If we don't limit us to the tagged questions, but search the word, we will find more on each site:

Most of them contain quite similar questions.

I finally found my answer (it is not possible) at ServerFault, but I'm still not sure on which site my question would be best.

Is there some clear guideline describing the limits?

Jeff said that it depends on my profession/role. Thus here is what my question would look like (if I didn't already found that there is no solution with current wget, and I would need some alternative program):

I'm a programmer which wants to download (once) a website containing Javadoc documentation (which I'm later using to program against the API).

These consists of a bunch of HTML files, and most of them contain a link like index.html?<filename> (where <filename> is the current file name). There is only one index.html, e.g. these URLs all refer to the same file (and give an identical result when downloading). Using wget -r, all these are downloaded and saved individually.

With --reject 'index.html\?*' wget will delete them after downloading, but it still will download them all, in effect downloading the same file some hundred or thousand times for bigger APIs, wasting both my bandwith as well as the one of the server.

How would I avoid downloading them?

This is my current command line:

 wget --no-parent --recursive --level=inf --page-requisites --wait=1 
   http://epaul.github.com/jsch-documentation/simple.javadoc/

So, I'm (for this concrete problem) a programmer and want to use the downloaded website for programming purposes, but the same problem could occur for a webmaster which wants to copy another website, a sysadmin which wants to provide a local mirror of some other website to his users, or some simple (Unix or Windows or Mac or ...) user which simply wants to download this website for offline reading.

I (now) don't want to create a program which will repeatedly download this website (or other similar ones), but if I would want to, the question and answers would not really differ, I think.

I'm not always a programmer, I also are sometimes a server administrator (for the server of an association), a webmaster (for this same association, and my own blog), and a power user (on different Linux systems at home and at the university - and I tested the command line on one of these).

Did I get really unlucky to hit the corner case where all these fields touch, or is there a problem with the delineation?
Or do I simply misunderstand this and it is totally clear where this question belongs?

Improve this question
7
  • Same place you put .htaccess questions, right?
    – random
    Jul 7, 2011 at 2:02
  • Huh ... .htaccess questions I would put on webmasters.stackexchange.com, where I didn't found a single wget question.
    – Paŭlo Ebermann
    Jul 7, 2011 at 2:06
  • that's odd... your permissions must be wrong. <- that was a joke, but there are in fact questions about wget on webmasters.
    – M. Tibbits
    Jul 7, 2011 at 2:08
  • Okay, mis-remembered. I looked in fact there, but there was no wget tag. I'll add it to the question.
    – Paŭlo Ebermann
    Jul 7, 2011 at 2:13
  • I would prefer SF or SU, but there is probably a crossover point for SO.
    – user7116
    Jul 7, 2011 at 2:43
  • based on your edit, this is definitely SU. The programming part is irrelevant, you are just "downloading documentation from a website as a technical user."
    – Jeff Atwood
    Jul 7, 2011 at 22:02
  • @Jeff: Thus, when I would want to write a program that does this repeatedly, it get's programming?
    – Paŭlo Ebermann
    Jul 8, 2011 at 0:17

3 Answers 3

Reset to default
11

Well, you are looking at this the wrong way, IMHO.

The question should not start

I have a question about wget and ...

but rather

I am a programmer and ...
I am a sysadmin and ...
I am a power user and ...
I am a webmaster and ...
I am a unix nut and ...

In other words, you aren't just using wget, you are accomplishing some specific task with wget. And the nature of that task always, always, always depends on your profession and background. That is what should determine where the question goes.

(below are all real examples copied and pasted from guess-which-site!)

For a sysadmin, it would be

I have Windows 2008 Server on which i store DB backups on daily basis. I want to be able to download new files only using wget, curl or windows built-in FTP doesn't matter.

For a programmer it would be

I built a rails deploy dashboard that kicks off a ruby script on a remote machine to update a deployed application. The command to run the script looks like this: ssh test-host-02 "wget -q -O - http://server/deploy.rb | sudo ruby"

For a power user it would be

What is the windows equivalent of wget?

For a webmaster it would be

I have a website where I post csv files as a free service. Recently I have noticed that wget and libwww have been scraping pretty hard and I was wondering how to circumvent that even if only a little.

For a unix nut it would be

Why doesn't cp have a progress bar like wget ? Please note that I don't ask how. I already know options like pv and rsync -P.

Yes, some overlap is of course unavoidable; this is how the world works. Things are shades of gray, not black and white.

I humbly submit that you are asking the wrong question. To determine what site you should ask a question on -- unless it's totally obvious like asking a bicycles question on http://bicycles.stackexchange.com -- the correct question to ask is not "where do wget questions belong", but rather

… which community do I belong to?

Improve this answer
1
  • 1
    Which community do I belong to? All of them. (Not professionally, but as a hobby, at least.) Thus, it is not that the question depends on the background, but more that my precise role depends on the question. (I'm adding more details about my wget question to the question.)
    – Paŭlo Ebermann
    Jul 7, 2011 at 13:03
3

SuperUser fits best in my eyes, since it's a question about the usage of an application that's not directly related to programming.

Improve this answer
1

I would recommend to put this kind of questions to SuperUser, Unix&Linux, ServerFault, Ask Ubuntu but NOT stackoverflow if you're not running "wget" via php or another application releated to a question

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .