22

For a while now, moderators have had the ability to move long comment threads to new chat rooms. "Out of the box", this is only possible when an automatic flag is raised after a certain number of comments are posted in a short time, though there exists a workaround that moderators can use to force it (via the browser console).

How has the use of this moderation tool affected people's behavior?

  • When that flag is raised, how often do moderators (a) move the comments to chat, (b) prune or purge the comments (without moving to chat), or (c) dismiss the flag?1

  • Once a comment thread has been moved to chat, how often do users actually go to that room and continue the discussion? (And how many others follow the link to see the comments, even if they don't participate?) Or do they continue adding comments on the post, ignoring the chat room? Or does moving the thread to chat usually shut it down?

  • How has comment flagging changed (if at all)?

  • Overall, are people commenting more or less, and by how much?

What are the trends across the network, and what tools can I use to answer these questions on an individual site? (I don't think moves to chat are recorded in SEDE.)

I ask mainly out of curiosity, as a moderator on a high-commenting site.

1 And is there a way for me to find out how often I do each of those, as a moderator, for comparison?

  • 1
    I thought anyone could do this using the "move this to chat" thing when it pops up... I'm not a mod anywhere but I can't say that I've ever been involved in a comment discussion that was moved to chat by a mod. In the couple of times I've been in a discussion where someone moved it to chat, I will admit to feeling disinterested in moving to chat for the most part, and generally don't continue the conversation at all. – Catija Jul 21 '15 at 2:34
  • @Catija anybody who's involved in a back-and-forth that goes on long enough can move (or copy, I think?) that comment thread, meaning the comments of just those two participants. The moderator tool allows moving all the comments. – Monica Cellio Jul 21 '15 at 2:39
  • 9
    @monica I've been doing a bit of research on this, I'll write something up soonish. – Taryn Jul 21 '15 at 2:52
  • 1
    Thanks @bluefeet! – Monica Cellio Jul 21 '15 at 2:54
  • 1
    I, for one, almost never visit a moved-to-chat link because if I did that it would publicly announce I entered the chat room, which is not something I want when I'm reading. </tangential-grumble> – Kevin Reid Jul 21 '15 at 3:03
  • 3
    @KevinReid good point. I generally copy the URL and edit it to go to the transcript, but I assume I'm in a small minority on that. – Monica Cellio Jul 21 '15 at 3:04
  • 3
    @KevinReid I tend to open chat links in an incognito window for that reason. – user259867 Jul 21 '15 at 3:09
  • 1
    Indeed you can do these things and I have. But they shouldn't be necessary just to read. (Actually, I consider this behavior of chat to be a privacy bug — someone could disguise a chat link in order to find out who clicks on it — but I haven't gotten around to complaining in particular.) – Kevin Reid Jul 21 '15 at 3:11
  • I just realized that this process seems kind of similar to free speech zones. – CRABOLO Jul 21 '15 at 3:19
  • @Dronehinge: Trigger warning much? :P – Nathan Tuggy Jul 21 '15 at 3:38
  • I am still a newly-mod, so I cannot speak from much experience. I got this flag twice in the last month, our team purged once and on the other discussion I thought the thread was worth keeping, so I moved it to chat. After I gave the OP the room owner and leaving a couple of messages myself, there was a very brief discussion and the chat is inactive since then. Not sure it helps anyone. – Martin - マーチン Jul 21 '15 at 13:14
17

You've got a lot of questions in this but I'll try to answer most of them. Basically, it appears that the new feature impacts smaller sites more than larger sites, like Stack Overflow. But overall, I'm not seeing a big difference in user behavior when comparing comments before versus after the implementation.

Over the past few weeks, I've been looking into the excessive comment discussion and the "too many comment" flags to see what, if anything, could be done and/or improved with them. While I'm still processing and analyzing data, when it comes to the move comments to chat function it seems to be a mixed bag of success.

As expected when looking at the network as a whole, Stack Overflow gets the majority of the "too many comment" flags. The site averages about 500 of these flags a month, the next closest site is Mathematics which gets about 50 a month. While SO gets the most flags, it falls way down on the list for the number of conversations moved into chat. I believe that disparity is due to the fact that SO mods have more higher priority flags in the queue, as a result these tend to sit and the discussion fizzles out before being moved to chat. But on sites where the flag queue is smaller, it seems that moderators can act on these discussions as they are happening and move it to chat faster.

Now for some numbers, which will hopefully answer some of your questions. I ran a query across the network to get the total number of “too many comment” flags, how many of those were moved to chat, the percentage moved to chat, the total number of those that moved to chat but the conversation continued under the post, and the percentage of those conversations. Overall, I’d say that the feature to move comments to chat is used quite a bit, but conversations are still taking place in the comments even after a moderator moves the discussion to chat.

Data from 2014-08-13 (implementation date) to present:

+------------------------------+---------+-----------------+-----------------+-----------------------------------+--------------------------------------+
|          Site Name           | # Flags | # Moved To Chat | % Moved To Chat | # Posts Moved But Comments Cont'd | % Posts Moved Yet Cont'd in Comments |
+------------------------------+---------+-----------------+-----------------+-----------------------------------+--------------------------------------+
| Politics                     |      19 |              17 | 89.47           |                                10 | 58.82                                |
| Game Developers              |       8 |               6 | 75              |                                 3 | 50                                   |
| Parenting                    |      19 |              13 | 68.42           |                                10 | 76.92                                |
| Astronomy                    |       3 |               2 | 66.67           |                                 1 | 50                                   |
| Computational Science        |       3 |               2 | 66.67           |                                 1 | 50                                   |
| SharePoint                   |      11 |               7 | 63.64           |                                 2 | 28.57                                |
| Biblical Hermeneutic         |       5 |               3 | 60              |                                 1 | 33.33                                |
| Hinduism Stack Exchange      |      17 |              10 | 58.82           |                                 7 | 70                                   |
| Philosophy                   |      52 |              29 | 55.77           |                                12 | 41.38                                |
| IT Security                  |      33 |              18 | 54.55           |                                12 | 66.67                                |
| Computer Science             |      23 |              12 | 52.17           |                                 7 | 58.33                                |
| Jewish Life and Learning     |     110 |              50 | 45.45           |                                16 | 32                                   |
| Worldbuilding Stack Exchange |     105 |              46 | 43.81           |                                26 | 56.52                                |
| Cryptography                 |      16 |               7 | 43.75           |                                 4 | 57.14                                |
| History                      |      19 |               8 | 42.11           |                                 4 | 50                                   |
| Photography                  |       5 |               2 | 40              |                                 1 | 50                                   |
| Salesforce                   |       5 |               2 | 40              |                                 2 | 100                                  |
| The Workplace                |     133 |              53 | 39.85           |                                36 | 67.92                                |
| Christianity                 |      56 |              22 | 39.29           |                                15 | 68.18                                |
| Physics                      |     177 |              64 | 36.16           |                                28 | 43.75                                |
| Bicycles                     |       6 |               2 | 33.33           |                                 0 | 0                                    |
| Personal Finance and Money   |      21 |               7 | 33.33           |                                 6 | 85.71                                |
| Ubuntu                       |     192 |              63 | 32.81           |                                22 | 34.92                                |
| Programmers                  |      96 |              31 | 32.29           |                                17 | 54.84                                |
| Graphic Design               |       7 |               2 | 28.57           |                                 2 | 100                                  |
| Apple                        |      29 |               8 | 27.59           |                                 7 | 87.5                                 |
| Drupal Answers               |      11 |               3 | 27.27           |                                 0 | 0                                    |
| Super User                   |     105 |              27 | 25.71           |                                10 | 37.04                                |
| Board and Card Games         |      24 |               6 | 25              |                                 0 | 0                                    |
| Code Review                  |      20 |               5 | 25              |                                 2 | 40                                   |
| Role-playing Games           |      61 |              14 | 22.95           |                                 8 | 57.14                                |
| German Language and Usage    |       9 |               2 | 22.22           |                                 1 | 50                                   |
| WordPress                    |      19 |               4 | 21.05           |                                 4 | 100                                  |
| User Experience              |      33 |               6 | 18.18           |                                 4 | 66.67                                |
| English Language and Usage   |     341 |              58 | 17.01           |                                32 | 55.17                                |
| Unix and Linux               |      83 |              13 | 15.66           |                                 6 | 46.15                                |
| Mathematics                  |     619 |              85 | 13.73           |                                31 | 36.47                                |
| Skeptics                     |      74 |              10 | 13.51           |                                 8 | 80                                   |
| Aviation Stack Exchange      |      15 |               2 | 13.33           |                                 2 | 100                                  |
| Academia                     |      77 |              10 | 12.99           |                                 8 | 80                                   |
| Mathematica                  |      33 |               4 | 12.12           |                                 4 | 100                                  |
| Travel                       |      35 |               4 | 11.43           |                                 2 | 50                                   |
| Electronics and Robotics     |      79 |               8 | 10.13           |                                 3 | 37.5                                 |
| Puzzling Stack Exchange      |     133 |              13 | 9.77            |                                10 | 76.92                                |
| Stack Overflow               |    6111 |             563 | 9.21            |                               169 | 30.02                                |
| Statistical Analysis         |      44 |               3 | 6.82            |                                 1 | 33.33                                |
| Science Fiction              |     158 |              10 | 6.33            |                                 8 | 80                                   |
| Code Golf                    |     129 |               7 | 5.43            |                                 4 | 57.14                                |
| English Language Learners    |      77 |               4 | 5.19            |                                 3 | 75                                   |
| MathOverflow                 |      72 |               2 | 2.78            |                                 1 | 50                                   |
| TeX - LaTeX                  |     101 |               2 | 1.98            |                                 1 | 50                                   |
+------------------------------+---------+-----------------+-----------------+-----------------------------------+--------------------------------------+

While we are moving a decent number of these conversations to chat, we are getting a high percentage of comments on the posts after the move. I'm still working on some numbers to see how often the chatrooms are used once the conversation is moved. In the meantime, I performed a very unscientific test of popping into rooms on a variety of sites to see if users continued chatting. I'd say that in about 30% of the rooms I checked, the conversations continued. (Very unscientific but it was something...right?) Most of the time, it appeared that the conversations had already stopped by the time it was moved to chat or the move ended the discussion.

How has comment flagging changed (if at all)?

It appears to be roughly the same. I queried the data from 2014-01-01 to present to get the monthly average of comment flags and across the network there isn’t much of a difference since this feature was implemented in August 2014 (note these are only the sites who average more than 50 comment flags after the implementation):

Before 2014-08-01 After 2014-08-01 Site Name 
----------------- ---------------- --------- 
13370             16952            Stack Overflow
394               673              Mathematics
179               269              Super User
209               266              Role-playing Games
276               230              Ubuntu    
87                201              Physics   
43                187              Code Review
197               181              Programmers
135               172              English Language and Usage
65                160              Science Fiction
234               152              The Workplace
92                137              Christianity
173               134              Jewish Life and Learning
58                127              Academia  
132               124              Computer Science
87                96               Unix and Linux
111               87               Gaming    
118               82               TeX - LaTeX
12                82               Chemistry 
78                76               Server Fault
8                 71               Puzzling Stack Exchange
40                65               Travel    
33                56               Electronics and Robotics
46                56               Skeptics  
16                52               Movies    
163               50               Software Recommendations Stack Exchange
21                50               German Language and Usage

Overall, are people commenting more or less, and by how much?

Users comment a lot both before and after it this was implemented but using the same timeframe as above 2014-01-01 to present and splitting the data based on the implementation date of 2014-08-01, here is the monthly average of the number of comments each site gets:

Before 2014-08-01 After 2014-08-01 Site Name 
----------------- ---------------- --------- 
724043            963819           Stack Overflow
18275             19437            Super User
9102              10340            Server Fault
66                101              Stack Apps
749               1378             Cooking   
1415              2158             Home Improvement
2899              2604             Game Developers
3777              4928             Gaming    
4846              5674             GIS       
70631             78682            Mathematics
1495              1439             Photography
5714              7582             Statistical Analysis
907               835              Web Apps  
1578              1538             Webmasters
3597              5248             Apple     
1134              708              Theoretical Computer Science
12749             14216            English Language and Usage
1445              1745             Personal Finance and Money
3068              3835             Role-playing Games
12986             12953            TeX - LaTeX
14944             19806            Ubuntu    
9207              10040            Unix and Linux
5273              4564             WordPress 
1016              1093             Bicycles  
5157              7062             Programmers
1801              3076             Android Enthusiasts
588               720              Board and Card Games
8105              12292            Physics   
238               185              Homebrew  
3214              4228             IT Security
375               460              Writers   
7965              10014            Electronics and Robotics
2154              2385             Graphic Design
3808              4810             Database Administrators
4260              6720             Science Fiction
523               528              Area 51 Discussions
5182              5546             Code Review
5789              3767             Code Golf 
534               487              Quantitative Finance
233               240              Project Management
1339              1460             Skeptics  
4028              3588             Drupal Answers
549               526              Fitness and Nutrition
391               696              Motor Vehicle Maintenance and Repair
598               926              Parenting 
3031              3252             SharePoint
1205              1365             Musical Practice and Performance
258               330              Software Quality Assurance and Testing
3016              3691             Jewish Life and Learning
448               516              Astronomy 
1027              1237             German Language and Usage
989               861              Japanese Language and Usage
327               514              Gardening and Landscaping
1296              1751             Philosophy
166               136              Personal Productivity
2154              3286             Travel    
1503              1453             Cryptography
951               844              Signal Processing
445               417              French Language and Usage
2429              2266             Christianity
1123              622              Bitcoin   
555               593              Linguistics
608               656              Biblical Hermeneutics
976               1231             History   
65                90               LEGO®     
280               337              Spanish Language and Usage
478               518              Computational Science
1336              1609             Movies    
481               520              Chinese Language and Usage
1148              2034             Biology   
65                108              Poker     
5377              6090             Mathematica
453               404              Cognitive Sciences
293               404              The Great Outdoors
128               142              Martial Arts
264               229              Sports    
3362              4338             Academia  
2140              2394             Computer Science
4046              3830             The Workplace
251               276              Windows Phone
1146              2203             Chemistry 
346               366              Chess     
909               1262             Raspberry Pi
278               334              Russian Language and Usage
811               771              Islam     
3453              3961             Salesforce
121               129              Patents   
2662              2143             User Experience
116               161              Genealogy and Family History
203               271              Robotics  
855               444              ExpressionEngine
536               612              Politics  
270               312              Audio-Video Production
856               1079             Anime and Manga
2608              4480             Magento   
3757              4823             English Language Learners
103               87               Sustainable Living
491               353              Tridion Stack Exchange
327               290              Reverse Engineering Stack Exchange
820               1071             Arduino Stack Exchange
957               930              Network Engineering Stack Exchange
142               199              Open Data Stack Exchange
185               139              Freelancing Stack Exchange
1336              2541             Blender Stack Exchange
6272              5965             MathOverflow
542               867              Space Exploration Stack Exchange
305               243              Sound Design Stack Exchange
200               180              Tor Stack Exchange
540               377              Pets Stack Exchange
193               136              Amateur Radio Stack Exchange
156               296              Italian Language Stack Exchange
6353              8521             Stack Overflow em Português
1394              1860             Aviation Stack Exchange
150               55               Ebooks Stack Exchange
153               31               Beer Stack Exchange
1886              1187             Software Recommendations Stack Exchange
622               287              Expatriates Stack Exchange
1434              470              Mathematics Educators Stack Exchange
583               342              Earth Science Stack Exchange
697               606              Joomla Stack Exchange
321               248              Data Science Stack Exchange
993               4162             Puzzling Stack Exchange
1249              701              Craft CMS Stack Exchange
1010              739              Buddhism Stack Exchange
1517              783              Hinduism Stack Exchange
324               93               Moderators Stack Exchange
93                389              Startups Stack Exchange
4646              6256             Stack Overflow на русском

I'm trying to get the chat usage info but the big question that sticks out to me is why so many conversations continue under a post after it's been moved to chat?

I can only guess, but in my experience as a SO mod these are a few things I've noticed:

  1. Users were already given the chance to move to chat when the notice pops up to them reminding them to "move extended discussions to chat" and they aren’t taking it at that point, so why would they move when a moderator got involved?
  2. Many of these conversations, at least on SO, involve new users and even experienced don't understand that moderators can bypass the rep requirement of 20 when they move it to chat. On more than one occasion, after I'd move a conversation I’d get a response back saying " can’t go to chat because they don’t have the rep".
  3. The chat URL can be blocked for some users at work/etc so migrating the conversation doesn't work for them.
  4. Some users don't want to go to chat, so they'll continue in the comments not matter what.

I'm still chewing on what I've seen, I think there might be a case to improve the usage of the feature but I'm not sure how yet. Some changes to the feature have been proposed already, including adding a move to chat on the mod menu, and I even suggested the possibility of making this a 30k privilege. Part of me wonders whether the feature would be used more if we were to move these conversations to chat earlier (without a flag) or if other users had the ability to move them... but that's a discussion for another day.

  • Thank you very much! I, too, am wondering about ways to improve this, and one area to look closer at is the comments that get made after a conversation has been moved to chat. Is it the same people continuing the discussion despite the chat room, is it other users coming late and not going to chat, is it actual requests for clarification (etc) that should be comments, or what? Some of each, of course, but first we need the broad view of what's actually happening with comments and linked chats, so I really appreciate the wowkr youdid to compile this. – Monica Cellio Jul 22 '15 at 18:10
  • (As a moderator, I would like it to be easier to move to chat earlier -- if people would actually move there. That chat transcripts are persistent, while comments get flushed, seems like it should encourage people to use chat -- for posterity! But that might not be widely known.) – Monica Cellio Jul 22 '15 at 18:11
  • 2
    One more factor, relevant on Mathematics and other MathJax-enabled sites: chat does not support MathJax. (There is a bookmarklet one can use, but those who do not use chat regularly are unlikely to have it.) – user259867 Jul 22 '15 at 18:22
  • Do comments support MathJax? (I should probably know; we have MathJax on Worldbuilding. But I haven't noticed it in comments.) – Monica Cellio Jul 22 '15 at 18:32
  • 2
    @MonicaCellio Yes, typing $y= x^2$ in a comment will work as one'd expect on a MathJax-enabled site. – user259867 Jul 22 '15 at 18:37
  • 1
    bluefeet, re #2, I wonder what the effects would be of dropping the 20-rep requirement for these rooms (only). (Also whether it's feasible to implement...) I know that "move to chat" adds everybody who commented to the room's access list, though most of them didn't need it since you need 50 rep to comment (unless you're the OP or your answer was converted to a comment). Instead of doing that, is it possible to change the threshold on the room or "add *" or something? And if we did, what do we think would happen? Would it attract chat trolls? Just wondering out loud here... – Monica Cellio Jul 23 '15 at 15:10
  • @MonicaCellio In regards to your first comment it is really mixed - some new people, some recurring people who don't want to chat, a real variety. As far as your last comment, dropping the rep requirement for anyone I think would be difficult and/or problematic. You don't want these rooms to be open to anyone for the same reasons that chat is currently limited. – Taryn Jul 23 '15 at 19:42
  • I've not knowingly been involved in a conversation moved to chat, but I have been warned it could be a good idea a couple of times. I mostly decided against it because of time zone issues and the "conversations" are over days, not minutes or hours. – Mark Hurd Jul 24 '15 at 4:04
13

The data you're still being owed because I promised bluefeet to get is how those chat rooms are actually used.

In all of the following, we only count rooms that where created via the "move comments to chat" moderator function. The word "messages" only refers to messages actually posted into chat, not the imported comments. The word "chatters" refers to the number of different users involved in posting those messages.

You'll notice slight discrepancies to some of bluefeet's numbers; I haven't dug deeply into this but my wild guess is that the comment migration wasn't logged in those cases for whatever reason.

So, here is Stack Overflow Chat:

Rooms with...  Number of rooms Avg # of comments Avg # of messages Avg # of chatters
-------------- --------------- ----------------- ----------------- -----------------
(all)          592             24.9              10.6              0.8
any chat       267             24                23.6              1.7
>= 10 messages 92              23.7              62.5              2.3
>= 2 chatters  152             23.6              39.9              2.3
>= 3 chatters  28              23.7              32.5              3.4

And this is Stack Exchange Chat (the chat for all sites except for SO and MSE):

Rooms with...  Number of rooms Avg # of comments Avg # of messages Avg # of chatters
-------------- --------------- ----------------- -----------------------------------
(all)          861             23.1              9.1               0.7
any chat       318             23.3              24.5              2
>= 10 messages 90              24.4              78.7              3
>= 2 chatters  187             23.8              40.5              2.6
>= 3 chatters  72              24.3              65                3.7

The following table breaks this down per site, only listing sites that have at least 4 such rooms. The columns correspond to the rows in the previous table (read as "number of rooms with…"), with the addition of the "% any chat" column, which is also what it's sorted by.

              # rooms any chat    % any chat >=10 messages >=2 chatters >=3 chatters
------------- ------- ----------- ---------- ------------- ------------ ------------
mathematica   4       4           100%       1             3            2
skeptics      11      9           82%        3             8            2
wordpress     4       3           75%        3             3            2
academia      11      8           73%        1             5            2
hinduism      10      7           70%        3             5            2
christianity  23      16          70%        8             15           7
hermeneutics  3       2           67%        1             2            1
rpg           14      9           64%        2             4            2
money         7       4           57%        0             2            0
crypto        7       4           57%        1             2            1
parenting     13      7           54%        1             7            1
puzzling      13      7           54%        4             5            4
codereview    6       3           50%        1             3            2
pt.SO         18      9           50%        2             4            0
scifi         11      5           45%        2             3            1
stackoverflow 592     267         45%        92            152          28
codegolf      7       3           43%        0             0            0
sharepoint    7       3           43%        2             2            0
ru.SO         10      4           40%        1             1            1
worldbuilding 46      18          39%        3             12           6
security      18      7           39%        1             2            1
unix          13      5           38%        1             3            0
askubuntu     63      24          38%        7             11           4
apple         8       3           38%        2             2            0
physics       67      25          37%        11            15           5
programmers   36      13          36%        2             6            1
judaism       51      18          35%        4             12           5
workplace     56      19          34%        1             7            4
boardgames    6       2           33%        1             1            1
stats         3       1           33%        1             1            1
drupal        3       1           33%        0             0            0
astronomy     3       1           33%        1             1            0
gamedev       6       2           33%        0             0            0
philosophy    29      9           31%        6             7            2
cs            11      3           27%        1             2            1
ell           4       1           25%        0             1            0
history       8       2           25%        1             1            1
electronics   8       2           25%        1             1            1
english       62      14          23%        0             5            1
superuser     28      6           21%        1             2            0
travel        5       1           20%        0             1            0
politics      17      3           18%        1             2            1
math          86      15          17%        2             6            1
ux            7       0           0%         0             0            0
  • 2
    Thanks very much for the data! So it looks like, on the SE network, of the rooms the rooms that get created by moving comments, more than a third (37%) see continued chat activity -- people go there and talk, more than half of the time with >2 participants. This is better utilization than I was expecting -- yay. – Monica Cellio Jul 23 '15 at 15:06
  • You're welcome. Note that it's >=2 participants (probably the two users who were arguing in the comments?), it drops of pretty sharply once you add a third user to the conversation. – balpha Jul 23 '15 at 15:16
  • Oh, right -- you did say >=, not >. Still, if they take the argument to chat, that's a win. :-) And nearly a quarter continue with >=3, which isn't trivial. – Monica Cellio Jul 23 '15 at 15:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .