3

I want to download a small dataset for one year. I didn't find a link that allows me to download 1 year of data or the last five years' data.

I tried "https://data.stackexchange.com/stackoverflow/query/" but results are not accurate; moreover I went through this list but I didn't get any option to download for specific years.

How can I get data for only last year from Stack Exchange?

Improve this question
12
  • 4
    You'd have to download it all, put it in a database and extract the last year from that.
    – Robert Longson
    Jul 24 at 0:40
  • Unfortunately my database do not have enough space to upload that big data. Is there any other way to do that?
    – I192058 Misbah Minhas
    Jul 24 at 0:43
  • 12
    Buy a bigger disk or use a cloud DB.
    – Robert Longson
    Jul 24 at 0:44
  • 3
    I vaguely recall google has this bigquery data set thing that might work. I'm not a dev so I'll leave it to someone who knows these things to poke at it and elaborate Not sure how updated it is either but cloud.google.com/blog/topics/public-datasets/… would be a starting point
    – Journeyman Geek
    Jul 24 at 3:02
  • 4
    Are you wanting data from Stack Exchange (i.e. all sites) or just a selected number of sites (e.g. just Stack Overflow)? What data are you looking to be contained in your dataset? How are you wanting your dataset stored (e.g. in a database, as text, as JSON, etc.)? How are you wanting that structured (e.g. if a database, are you wanting similar table layout to SE/SEDE)?
    – Makyen
    Jul 24 at 3:58
  • 1
    I want data from all sites.
    – I192058 Misbah Minhas
    Jul 24 at 4:27
  • 1
    Re "for one year": How do you define that? Is it questions posted in that time span? What about answers to older questions? And comments? What about changes to older questions and older answers?
    – This_is_NOT_a_forum
    Jul 24 at 9:29
  • @Robert Longson: A full import may not be required. It could be prefiltered on the XML level, e.g., using an (event-driven) XML parser and an XPath expression. At least it works OK as grep-like direct text search.
    – This_is_NOT_a_forum
    Jul 24 at 9:33
  • @This_is_NOT_a_forum sure but it depends whether the problem is that there's not enough disk space for the download or there's not enough disk space for the download and the database. If it's the latter your idea would work, if it's the former then buy a bigger disk is the answer.
    – Robert Longson
    Jul 24 at 9:39
  • 3
    Do you specifically need 1 year of data? Or do you just need a smaller dataset? There are some derivative works of the data dump that are great for this, depending on your full set of requirements.
    – AMtwo
    Jul 24 at 17:08
  • 3
    Somewhat miffed about the close votes. This is definitely a question about the software which serves these sites (specifically, the data dumps) and I expect future visitors might be able to post useful answers with various workarounds for the monolithic design of the dump data.
    – tripleee
    Jul 25 at 8:07
  • Originally asked on Stack Overflow main on 2023-07-14 (10 days before it was posted here), but only migrated to MSO 1 1/2 month later, on 2023-09-01.
    – This_is_NOT_a_forum
    Sep 1 at 14:50

1 Answer 1

Reset to default
7

How download small dataset (1 year 2023-2022) from stack exchange?

Not possible, only the entire data dump is downloadable (https://archive.org/download/stackexchange) (one file per SE site. SO is the largest by far).

The SE API isn't made for massively downloading content (https://api.stackexchange.com/docs/throttle).

Improve this answer
4
  • 3
    .... Actually you could selectively download files off the torrent... But depends on OPs constraints
    – Journeyman Geek
    Jul 24 at 6:16
  • @JourneymanGeekOnStrike can't select the year
    – Franck Dernoncourt
    Jul 24 at 6:19
  • 5
    True but you can select a file, extract the data you need, delete the file and go on. The only issue really is SO is huge
    – Journeyman Geek
    Jul 24 at 6:23
  • @JourneymanGeekOnStrike ok I've clarified that in the answer.
    – Franck Dernoncourt
    Jul 24 at 16:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .