3

I'm currently working on changing http to https in questions and answers of mine. To not flood the homepage (SF&F doesn't have the volume SO has), I do about 5 a day. When I run the query the next day, the ones I've already fixed are still among the results. It would be nice to be able to get a new list of only the posts I still have to edit, but the SEDE is only refreshed on Sundays.

Is it possible to increase the frequency and refresh SEDE daily?

I'm sure this would come in handy for a lot of other queries as well. And while I can certainly work around it, simply by remembering which ones I've done already, that wouldn't work if we were to adapt the query to find all cases, regardless of user ID. And I can imagine a number of other queries that would benefit from a more frequent refresh.

4
  • 8
    As a workaround you could run the query once, download the file and each day delete the lines you have done. Commented Aug 13, 2021 at 14:54
  • 2
    Considering the answer, this seems like it might be a really good support question - particularly since rene is already offering suggestions on how to avoid your issue. :) That said, I'm not sure if a support version of this would be a duplicate or not.
    – Catija
    Commented Aug 13, 2021 at 15:05
  • 1
    Another alternative, for your specific case: vanilla site search. It doesn't look like search plays nice with excluding a bunch of domains, like the SEDE query supports, but it can still pull all of your posts with unsecure HTTP links with user:19561 url:"http://*"
    – zcoop98
    Commented Aug 13, 2021 at 15:18
  • @zcoop98 unfortunately, there are a lot of sites that (still) do not support https. One of those is isfdb.org, which is used a lot on Science Fiction & Fantasy, yielding a lot of posts that can't be fixed.
    – SQB
    Commented Aug 13, 2021 at 15:21

1 Answer 1

16

It is certainly possible.

I don't think it is wise to do.

Refreshing SEDE takes 6 to 8 hours (speed improvements incoming since April 2024) on Sunday. During that time you'll notice SEDE is struggling to perform queries as quickly as it does during the week.

When a refresh is on its way, there is always one database "extra", the famous _temp database as that is the one currently being restored. This is a pain for those queries that run over all databases. They either break or report stats over incomplete data.

The progress report on the refresh isn't great. You can't really see how far we are. Once a week this is manageable with a nifty query but having to deal with this every day is a bit too much of a pain.

For your use case: can't you rely on your browser to show you which links you already visited? Alternatively, use a sensible order by and add a "start from" paramater that you keep track of yourself.

4
  • 5
    Ah, I didn't know refreshing it takes such a long time. That certainly explains why it can't be done more frequently.
    – SQB
    Commented Aug 13, 2021 at 15:19
  • 1
    Alternatively, use a sensible order by and add a "start from" paramater that you keep track of yourself. That's exactly what my Broken Image Repairer does. One can either use created: or lastactive: for manual searches: meta.stackexchange.com/help/searching
    – Glorfindel Mod
    Commented Aug 13, 2021 at 20:08
  • thanks to work by Aaron (and potentially others), it should now only take about 2 hours to refresh SEDE (source). still, it is unlikely to have daily refresh, but Aaron has future plans for live SEDE (source).
    – starball
    Commented Apr 8 at 7:46
  • 3
    @starball To be clear, it’s not a future plan, it’s a fantasy in my head that I hope to propose at some point, and persuade people to let me do it. That has not happened, and what ultimately gets implemented - if anything - might be different than what’s in my head.
    – Aaron Bertrand Staff
    Commented Apr 8 at 11:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .