How do I automate the data explorer download? Has anybody done that? I want to build an app which would use this data. I want this data to be updated automatically. Is there an JDBC connection or an API call available which I can run daily.
2
-
2From what I recall, they're only updated weekly, so a daily download would be overkill.– Fish Below the IceCommented Mar 10, 2015 at 19:41
-
If you need data more current than the weekly data dump, you should presumably use the Stack Exchange API.– senshinCommented Mar 10, 2015 at 21:21
Add a comment
|
1 Answer
If you are referring to The Stack Exchange Data Explorer there used to be an option for an odata connection but that got switched off some time ago due to stabillity/configuration issues. It is not definitely cancelled but it isn't working at the moment as well.
If you are interested in the result of one or more queries you created on SEDE you can use the Download CSV link for each query. Scripting that with wget or curl is not hard.
The databases under SEDE are refreshed weekly, in the weekend. The maxium number of rows returned in SEDE is 50,000 (with some tricks you might get more)
If you are looking to get all data in your own database based on the The Internet Archive Stack Exchange Dump you can use the scripts from this question. Notice this data is updated quarterly-ish.
To have near-live access to questions, answers, comments, users and sites among other data you might explore the Stack API with its support site Stack Apps. Do notice that you are allowed only a certain amount of calls per day and you are throttled if you make to many calls per minute. For the current details check the rate limits.
-
I want to have the result of this query:
DECLARE @UserId int = ##UserId## SELECT c.Score, c.Id as [Comment Link], p.Id as [Post Link], c.Text, c.CreationDate, c.UserId as [User Link] FROM Comments c, Posts p WHERE p.Id = c.PostId AND c.UserId = @UserId ORDER BY c.Score DESCon my website on github pages (no server, only js).The option with csv does not work because I am serverless.Just creating an api link seems impossible because of 2 tables in use(comments and posts). Commented Nov 17, 2020 at 19:45 -
@user3123159 this will work
https://api.stackexchange.com/docs/comments-on-users#order=desc&sort=votes&ids=811&filter=!)Q2A3(brA9BlriV-A2Kh)6X3&site=stackoverflow&run=truebut if you want the question title, you have to do an extra call per comment to get the title of the posts from the posts endpointhttps://api.stackexchange.com/docs/posts-by-ids#order=desc&sort=activity&ids=243778&filter=!9_bDDt835&site=stackoverflow&run=truethat can all be done client side with a bit of javascript run in the browser– rene ModCommented Nov 17, 2020 at 21:06