Note: this is a rewrite of a heavily down-voted question.
What has lead me to ask this question
(Note to moderators: I am for taking this section out because I suspect it's not of general interest but it could make the new version of the question not recognizable from the original i.e. edited to be another question, or give me downvotes for asking a question being completely useless from the first glance)
In an earlier project, I had a task to elaborate a set of NLP (natural language processing) metrics to grade the quality of text excerpts summarizing the purpose of publicly funded software components.
One of the simplest metrics under consideration was for example text length. There was no exact requirements for acceptable minimum or maximum length.
Some vendors provided just a couple of sentences, some several pages. So my metric was to compute the median and define acceptable deviation from that, then manually analyze outliers.
In this context, I learnt that it is very hard to write short but precise and correct summaries for concepts based on many complexity abstraction layers.
I thought then how to confirm this with more texts and be able to know what is a "good" summary and how does it relate with the text length.
SO/SE sites appear a perfect data corpus for this purpose, because
- there are lots of questions being text pieces containing (maybe not always) summaries of complex and special topics
- the voting system provides a human-defined distribution for the text quality where a personal bias is possibly not so high for a larger number of votes.
Formally, as I assume there might be some quality function F=(T,V), question quality being a function of text postings T and votes V.
The background motivation, or hypothesis, is here that maybe well-written summaries tend to grow in their size which I searched to confirm and posted this question.
Here are two examples to demonstrate how context summaries, either questions, might tend to grow in their size if a researcher goes from general to special context as the knowledge base is growing.
Example 1: Biology.
Initially I've completely made it up but then started to search in this direction and found out there is indeed ongoing scientific research in this special area. Please excuse that the example is constructed because I am not a biologist. I think the example was so bad that it has contributed a lot to the question being down-voted. So feel free to skip to the second example.
- Are there parasites living in eggs?
- What are metabolism characteristics of egg parasites in warm climate versus cool climate?
- Where do Pacific turtles lay eggs? Did they do so a decade, longer time ago?
- How does the climate change (another scientific discussion) in the early 21th century influence metabolism of parasites?
If you try to ask a coherent question involving precise explanation of all above aspects towards some new inquiry, it will take readers more time to read and comprehend. With the growing knowledge complexity, not only text writing skills of the poster might influence the comprehension, but also the required background knowledge to follow the argumentation.
Example 2: Information technology
In IT, for somebody who just started to learn AWS and has some background in infrastructure deployment, it can be hard even to formulate a question.
- Version 1: I want to deploy container on AWS. <= some downvote, some ask for more details because too general
- Version 2: I want to deploy a container on AWS and I have created an image in ECR <= critics because not clear which service they want to use
- Version 3: I want to deploy a container on AWS, I have an image, but I do not understand the big picture, and do not know how I can approach either ECS or Fargate or EKS. I have read the docs but do not understand, also I want to automate all that, is that possible? <= community asks for more input
- Version 4: I want to deploy a container on AWS automatically, and would like to see an automation example for EC2 to see how it works <= community is sceptical about why you would want to deploy ONE container and you have not specified which language
- Version 5: I want to deploy a container on AWS automatically for learning and demonstration purposes, and would like to see an automation example for EC2 to see how it works, maybe with Python? => Answer: okay you can use Boto3, here is a Gist link I've googled up.
Again, my hypothesis is that on long term we will need more time to communicate about specialized topics, and thus get slower in any sort of progress. An indicator could be text length growing over time at SE sites for same high upvotes. Maybe though it is completely random, so my assumption is wrong.
The missing link was then to find out the distribution of text lengths at SE. First, I've thought of question titles in a first naive approach, but then I've got feedback that there is a hard limit of 150 characters.
The actual question
So is there a way to find out automatically for research purposes length of question posts on SE sites which have got an accepted answer (without dealing with dump data)?
Initial version of the post
This question might feel like funny, but exploring an answer from time to time might reveal something important, therefore worth answering. Namely, the knowledge is growing rapidly, and we create ever more and more terms, and work evermore inter-disciplinary. Thus a question to be asked specifically might involve more and more terms, getting on one hand overloaded, on the other hand showing the evolution of knowledge and questions. And getting less broad of course! An example of how I see it - imagine you study biology, turtles, their parasites and closely related phenomena.