3

I want to download a small dataset for one year. I didn't find a link that allows me to download 1 year of data or the last five years' data.

I tried "https://data.stackexchange.com/stackoverflow/query/" but results are not accurate; moreover I went through this list but I didn't get any option to download for specific years.

How can I get data for only last year from Stack Exchange?

12
  • 4
    You'd have to download it all, put it in a database and extract the last year from that. Commented Jul 24, 2023 at 0:40
  • Unfortunately my database do not have enough space to upload that big data. Is there any other way to do that? Commented Jul 24, 2023 at 0:43
  • 12
    Buy a bigger disk or use a cloud DB. Commented Jul 24, 2023 at 0:44
  • 3
    I vaguely recall google has this bigquery data set thing that might work. I'm not a dev so I'll leave it to someone who knows these things to poke at it and elaborate Not sure how updated it is either but cloud.google.com/blog/topics/public-datasets/… would be a starting point Commented Jul 24, 2023 at 3:02
  • 4
    Are you wanting data from Stack Exchange (i.e. all sites) or just a selected number of sites (e.g. just Stack Overflow)? What data are you looking to be contained in your dataset? How are you wanting your dataset stored (e.g. in a database, as text, as JSON, etc.)? How are you wanting that structured (e.g. if a database, are you wanting similar table layout to SE/SEDE)?
    – Makyen
    Commented Jul 24, 2023 at 3:58
  • 1
    I want data from all sites. Commented Jul 24, 2023 at 4:27
  • 1
    Re "for one year": How do you define that? Is it questions posted in that time span? What about answers to older questions? And comments? What about changes to older questions and older answers? Commented Jul 24, 2023 at 9:29
  • @Robert Longson: A full import may not be required. It could be prefiltered on the XML level, e.g., using an (event-driven) XML parser and an XPath expression. At least it works OK as grep-like direct text search. Commented Jul 24, 2023 at 9:33
  • @This_is_NOT_a_forum sure but it depends whether the problem is that there's not enough disk space for the download or there's not enough disk space for the download and the database. If it's the latter your idea would work, if it's the former then buy a bigger disk is the answer. Commented Jul 24, 2023 at 9:39
  • 3
    Do you specifically need 1 year of data? Or do you just need a smaller dataset? There are some derivative works of the data dump that are great for this, depending on your full set of requirements.
    – AMtwo
    Commented Jul 24, 2023 at 17:08
  • 3
    Somewhat miffed about the close votes. This is definitely a question about the software which serves these sites (specifically, the data dumps) and I expect future visitors might be able to post useful answers with various workarounds for the monolithic design of the dump data.
    – tripleee
    Commented Jul 25, 2023 at 8:07
  • Originally asked on Stack Overflow main on 2023-07-14 (10 days before it was posted here), but only migrated to MSO 1 1/2 month later, on 2023-09-01. Commented Sep 1, 2023 at 14:50

1 Answer 1

7

How download small dataset (1 year 2023-2022) from stack exchange?

Not possible, only the entire data dump is downloadable (https://archive.org/download/stackexchange) (one file per SE site. SO is the largest by far).

The SE API isn't made for massively downloading content (https://api.stackexchange.com/docs/throttle).

4
  • 3
    .... Actually you could selectively download files off the torrent... But depends on OPs constraints Commented Jul 24, 2023 at 6:16
  • @JourneymanGeekOnStrike can't select the year Commented Jul 24, 2023 at 6:19
  • 5
    True but you can select a file, extract the data you need, delete the file and go on. The only issue really is SO is huge Commented Jul 24, 2023 at 6:23
  • @JourneymanGeekOnStrike ok I've clarified that in the answer. Commented Jul 24, 2023 at 16:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .