7

I noticed that a query on the stackoverflow data is cached after first execution. This is quite good, but how to force a re-execution? Since the underlaying data changed the query results are no more consistent with the present situation.

Improve this question

3 Answers 3

Reset to default
9

The data on data.stackexchange.com is "cached"; the data sets are updated once every week. Queries are not done against the live SE databases. See the data.se.com FAQ:

How frequently is Stack Exchange Data Explorer updated?
The data is updated early every Sunday morning around 3:00 UTC.

Query results are cached until the next dataset update, as the result of any one query and parameter combination is not going to change until then anyway.

Improve this answer
7
  • Is it monthly or weekly?
    – Sajeetharan
    Commented Aug 20, 2017 at 5:09
  • These days it's weekly. When this answer was posted it was monthly.
    – Martijn Pieters
    Commented Aug 20, 2017 at 7:20
  • there is an AutoExpireCacheMinutes but I'm not sure what its setting is for production. I have doubts though if it allows for up to a week to be stored. My experience is that it isn't.
    – rene Mod
    Commented Aug 20, 2017 at 15:32
  • @rene: I run a set of queries weekly. Sometimes the weekly data dump process fails and I know it has failed because my queries are run instantly (== cache results are returned). The caches are cleared with the new data dump transfer being successful.
    – Martijn Pieters
    Commented Aug 20, 2017 at 15:34
  • @MartijnPieters interesting, I would have to pay attention to this then. Unfortunately I don;t have access to the settings table
    – rene Mod
    Commented Aug 20, 2017 at 15:37
  • I have a query that returns 9-day-old data... maybe this changed
    – xjcl
    Commented Jun 25, 2020 at 13:40
  • @xjcl: nope, it hasn't changed but sometimes the process fails.
    – Martijn Pieters
    Commented Jun 30, 2020 at 21:26
5

While it's true that the general problem is that the data simply isn't live, the results of executing a query with a particular set of parameters absolutely are cached, and there isn't any way that normal users can purge them.

This can have some impact on the apparent correctness of the results if you use something in your query that's based on the current time, since after the first execution the selected reference point is locked in until that cache is cleared.

Since the data is only updated every month or so, though, it typically doesn't make sense to use the current time in your query at a granularity where it would be noticed anyway (i.e. the current day or hour). As such, the visible impact of the cache is quite insignificant*.

Currently, the cache will persist from the time that query is first run to the time that the cache is cleared manually. This manual purge is part of the process which updates the site statistics on the front page of the site, so you can be guaranteed that the cache will be cleared every time the underlying data is updated.

* However, if someone feels that there are cases where this isn't true, feel free to provide examples and I'll take a look

Improve this answer
2

Data.SE is loaded with database dumps every month--it's not a cache you're encountering, and the data on data.SE is not updated live.

The last time it was updated was about a week back, so you probably need to wait ~three weeks for the next update.

Improve this answer
1
  • Data has been updated monthly of late. The faq also reflects this.
    – Martijn Pieters
    Commented Jul 9, 2012 at 7:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .