-10

Note: this is a rewrite of a heavily down-voted question.

What has lead me to ask this question

(Note to moderators: I am for taking this section out because I suspect it's not of general interest but it could make the new version of the question not recognizable from the original i.e. edited to be another question, or give me downvotes for asking a question being completely useless from the first glance)

In an earlier project, I had a task to elaborate a set of NLP (natural language processing) metrics to grade the quality of text excerpts summarizing the purpose of publicly funded software components.

One of the simplest metrics under consideration was for example text length. There was no exact requirements for acceptable minimum or maximum length.

Some vendors provided just a couple of sentences, some several pages. So my metric was to compute the median and define acceptable deviation from that, then manually analyze outliers.

In this context, I learnt that it is very hard to write short but precise and correct summaries for concepts based on many complexity abstraction layers.

I thought then how to confirm this with more texts and be able to know what is a "good" summary and how does it relate with the text length.

SO/SE sites appear a perfect data corpus for this purpose, because

  • there are lots of questions being text pieces containing (maybe not always) summaries of complex and special topics
  • the voting system provides a human-defined distribution for the text quality where a personal bias is possibly not so high for a larger number of votes.

Formally, as I assume there might be some quality function F=(T,V), question quality being a function of text postings T and votes V.

The background motivation, or hypothesis, is here that maybe well-written summaries tend to grow in their size which I searched to confirm and posted this question.

Here are two examples to demonstrate how context summaries, either questions, might tend to grow in their size if a researcher goes from general to special context as the knowledge base is growing.

Example 1: Biology.

Initially I've completely made it up but then started to search in this direction and found out there is indeed ongoing scientific research in this special area. Please excuse that the example is constructed because I am not a biologist. I think the example was so bad that it has contributed a lot to the question being down-voted. So feel free to skip to the second example.

  • Are there parasites living in eggs?
  • What are metabolism characteristics of egg parasites in warm climate versus cool climate?
  • Where do Pacific turtles lay eggs? Did they do so a decade, longer time ago?
  • How does the climate change (another scientific discussion) in the early 21th century influence metabolism of parasites?

If you try to ask a coherent question involving precise explanation of all above aspects towards some new inquiry, it will take readers more time to read and comprehend. With the growing knowledge complexity, not only text writing skills of the poster might influence the comprehension, but also the required background knowledge to follow the argumentation.

Example 2: Information technology

In IT, for somebody who just started to learn AWS and has some background in infrastructure deployment, it can be hard even to formulate a question.

  • Version 1: I want to deploy container on AWS. <= some downvote, some ask for more details because too general
  • Version 2: I want to deploy a container on AWS and I have created an image in ECR <= critics because not clear which service they want to use
  • Version 3: I want to deploy a container on AWS, I have an image, but I do not understand the big picture, and do not know how I can approach either ECS or Fargate or EKS. I have read the docs but do not understand, also I want to automate all that, is that possible? <= community asks for more input
  • Version 4: I want to deploy a container on AWS automatically, and would like to see an automation example for EC2 to see how it works <= community is sceptical about why you would want to deploy ONE container and you have not specified which language
  • Version 5: I want to deploy a container on AWS automatically for learning and demonstration purposes, and would like to see an automation example for EC2 to see how it works, maybe with Python? => Answer: okay you can use Boto3, here is a Gist link I've googled up.

Again, my hypothesis is that on long term we will need more time to communicate about specialized topics, and thus get slower in any sort of progress. An indicator could be text length growing over time at SE sites for same high upvotes. Maybe though it is completely random, so my assumption is wrong.

The missing link was then to find out the distribution of text lengths at SE. First, I've thought of question titles in a first naive approach, but then I've got feedback that there is a hard limit of 150 characters.

The actual question

So is there a way to find out automatically for research purposes length of question posts on SE sites which have got an accepted answer (without dealing with dump data)?

=========

Initial version of the post

This question might feel like funny, but exploring an answer from time to time might reveal something important, therefore worth answering. Namely, the knowledge is growing rapidly, and we create ever more and more terms, and work evermore inter-disciplinary. Thus a question to be asked specifically might involve more and more terms, getting on one hand overloaded, on the other hand showing the evolution of knowledge and questions. And getting less broad of course! An example of how I see it - imagine you study biology, turtles, their parasites and closely related phenomena.

7
  • 5
    You're asking how long the title is? There's a max length of 150 characters.
    – Catija StaffMod
    Commented Aug 22, 2019 at 16:11
  • that is unfortunate :-)
    – J. Doe
    Commented Aug 22, 2019 at 16:11
  • 1
    @Catija that seems to be resized to 250, that is at least what SEDE tells me.
    – rene Mod
    Commented Aug 22, 2019 at 18:07
  • @rene the warning says 150, so if that's wrong...
    – Catija StaffMod
    Commented Aug 22, 2019 at 18:09
  • @Catija oh, the UI restricts it. There must be some titles with > 150 chars because SEDE barked at me when I had a column with nvarchar(150) ...
    – rene Mod
    Commented Aug 22, 2019 at 18:11
  • 2
    @Catija found it meta.stackexchange.com/a/48237/158100 and still 170 titles left that were already over 150 before the change went live.
    – rene Mod
    Commented Aug 22, 2019 at 18:42
  • I've asked this question without having the knowledge that besides the live quering tools there are also publicly released data dumps: archive.org/details/stackexchange
    – J. Doe
    Commented Jan 7, 2020 at 18:26

1 Answer 1

5

Okay, weird or not, here is the multi-database query that lists the questions ordered by body length, descending.

Today this question is the winner: Should we get rid of all the tag synonyms which differ by only one or more hyphens?

declare @sql nvarchar(max)

create table #result (site sysname
                    , hostname sysname
                    , title nvarchar(250)
                    , id integer
                    , bodylen integer)

create index ix_result_bl on #result(bodylen)

select @sql = N'insert into #result' + STRING_AGG(concat(N'
select ''', name ,N''' as site
     , ''',hostname ,N'''
     , title
     , id
     , len(body)
from ', quotename(name), N'.dbo.posts
where posttypeid = 1
and acceptedanswerid is not null
'),N' 
union all')
from (select convert(nvarchar(max),name) name
    -- , convert(nvarchar(max),sitename)  sitename
    -- , meta
    -- , domain
     ,  concat( 
        -- based on an idea from Glorfindel 
        -- https://meta.stackexchange.com/a/321181
        (case sitename
        WHEN 'Audio' THEN 'video'
        WHEN 'Beer' THEN 'alcohol'
        WHEN 'CogSci' THEN 'psychology'
        WHEN 'Garage' THEN 'mechanics'
        WHEN 'Health' THEN 'medicalsciences'
        WHEN 'Moderators' THEN 'communitybuilding'
        WHEN 'Photography' THEN 'photo'
        WHEN 'Programmers' THEN 'softwareengineering'
        WHEN 'Vegetarian' THEN 'vegetarianism'
        WHEN 'Writers' THEN 'writing'
        WHEN 'Br' THEN 'pt'
        WHEN 'Mathoverflow' THEN concat((meta+'.'), sitename)
        ELSE case when sitename = domain then null else sitename end
        end +'.')
        , (case 
           when sitename <> 'mathoverflow' then (meta+'.') 
           else null 
           end) 
        , (case 
           when sitename <> 'mathoverflow' then concat((domain + '.'), 'com') 
           else 'net' 
           end)
        ) hostname
from (
select name
, case parsename(name,1) 
  when 'Meta' then parsename(name,2)
  else parsename(name,1) 
  end [sitename]
, case parsename(name,1) 
  when 'Meta' then 'meta'
  else null
  end [meta]
  , coalesce(parsename(name,3), parsename(name,2)) [domain]
from sys.databases
where database_id > 5
-- (name not like '%.Meta' or name = 'StackExchange.Meta')
) dbs
) dbsall

exec (@sql)



select top 100
       site
     , concat(
       'https://'
       , hostname
       , '/q/'
       , id
       , '|'
       , title
       ) as [Question]
     , bodylen
from #result
order by bodylen desc

Here is the top 20 or so:

20 links to questions

SEDE refreshes on Sunday at 03:00 UTC. Do you know there is a tutorial and a Chatroom?

6
  • awesome, thank you! Can't understand why this question has got so many negative votes :-)
    – J. Doe
    Commented Aug 22, 2019 at 19:11
  • 1
    @J.Doe personally, I downvoted because I see three separate questions. Your title could be asking about the question content length. However, it could also be asking about the question with an accepted answer that's been around the longest. Your question body seems to refer to question title lengths. Commented Aug 22, 2019 at 23:12
  • @rene remarkably all the "winning" questions are just bloated up through large code and other quotations/listings. A pity that the field title is limited in length. Of we should look for longest sentences ending with the question mark...
    – J. Doe
    Commented Aug 23, 2019 at 4:14
  • 1
    @J.Doe I was pretty amazed I could get length out for all posts network-wide. Having to parse all post content to find code-blocks or sentences is something you have to do on the data-dump found on the Internet Archive and provide your own software, storage and processing capacity. That will never fly on SEDE.
    – rene Mod
    Commented Aug 23, 2019 at 6:45
  • @rene what you've done is great!
    – J. Doe
    Commented Aug 23, 2019 at 7:01
  • @TheWanderer I've changed the title and also reworked the post body to highlight the background of the question.
    – J. Doe
    Commented Jan 7, 2020 at 18:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .