You've got a lot of questions in this but I'll try to answer most of them. Basically, it appears that the new feature impacts smaller sites more than larger sites, like Stack Overflow. But overall, I'm not seeing a big difference in user behavior when comparing comments before versus after the implementation.
Over the past few weeks, I've been looking into the excessive comment discussion and the "too many comment" flags to see what, if anything, could be done and/or improved with them. While I'm still processing and analyzing data, when it comes to the move comments to chat function it seems to be a mixed bag of success.
As expected when looking at the network as a whole, Stack Overflow gets the majority of the "too many comment" flags. The site averages about 500 of these flags a month, the next closest site is Mathematics which gets about 50 a month. While SO gets the most flags, it falls way down on the list for the number of conversations moved into chat. I believe that disparity is due to the fact that SO mods have more higher priority flags in the queue, as a result these tend to sit and the discussion fizzles out before being moved to chat. But on sites where the flag queue is smaller, it seems that moderators can act on these discussions as they are happening and move it to chat faster.
Now for some numbers, which will hopefully answer some of your questions. I ran a query across the network to get the total number of “too many comment” flags, how many of those were moved to chat, the percentage moved to chat, the total number of those that moved to chat but the conversation continued under the post, and the percentage of those conversations. Overall, I’d say that the feature to move comments to chat is used quite a bit, but conversations are still taking place in the comments even after a moderator moves the discussion to chat.
Data from 2014-08-13 (implementation date) to present:
+------------------------------+---------+-----------------+-----------------+-----------------------------------+--------------------------------------+
| Site Name | # Flags | # Moved To Chat | % Moved To Chat | # Posts Moved But Comments Cont'd | % Posts Moved Yet Cont'd in Comments |
+------------------------------+---------+-----------------+-----------------+-----------------------------------+--------------------------------------+
| Politics | 19 | 17 | 89.47 | 10 | 58.82 |
| Game Developers | 8 | 6 | 75 | 3 | 50 |
| Parenting | 19 | 13 | 68.42 | 10 | 76.92 |
| Astronomy | 3 | 2 | 66.67 | 1 | 50 |
| Computational Science | 3 | 2 | 66.67 | 1 | 50 |
| SharePoint | 11 | 7 | 63.64 | 2 | 28.57 |
| Biblical Hermeneutic | 5 | 3 | 60 | 1 | 33.33 |
| Hinduism Stack Exchange | 17 | 10 | 58.82 | 7 | 70 |
| Philosophy | 52 | 29 | 55.77 | 12 | 41.38 |
| IT Security | 33 | 18 | 54.55 | 12 | 66.67 |
| Computer Science | 23 | 12 | 52.17 | 7 | 58.33 |
| Jewish Life and Learning | 110 | 50 | 45.45 | 16 | 32 |
| Worldbuilding Stack Exchange | 105 | 46 | 43.81 | 26 | 56.52 |
| Cryptography | 16 | 7 | 43.75 | 4 | 57.14 |
| History | 19 | 8 | 42.11 | 4 | 50 |
| Photography | 5 | 2 | 40 | 1 | 50 |
| Salesforce | 5 | 2 | 40 | 2 | 100 |
| The Workplace | 133 | 53 | 39.85 | 36 | 67.92 |
| Christianity | 56 | 22 | 39.29 | 15 | 68.18 |
| Physics | 177 | 64 | 36.16 | 28 | 43.75 |
| Bicycles | 6 | 2 | 33.33 | 0 | 0 |
| Personal Finance and Money | 21 | 7 | 33.33 | 6 | 85.71 |
| Ubuntu | 192 | 63 | 32.81 | 22 | 34.92 |
| Programmers | 96 | 31 | 32.29 | 17 | 54.84 |
| Graphic Design | 7 | 2 | 28.57 | 2 | 100 |
| Apple | 29 | 8 | 27.59 | 7 | 87.5 |
| Drupal Answers | 11 | 3 | 27.27 | 0 | 0 |
| Super User | 105 | 27 | 25.71 | 10 | 37.04 |
| Board and Card Games | 24 | 6 | 25 | 0 | 0 |
| Code Review | 20 | 5 | 25 | 2 | 40 |
| Role-playing Games | 61 | 14 | 22.95 | 8 | 57.14 |
| German Language and Usage | 9 | 2 | 22.22 | 1 | 50 |
| WordPress | 19 | 4 | 21.05 | 4 | 100 |
| User Experience | 33 | 6 | 18.18 | 4 | 66.67 |
| English Language and Usage | 341 | 58 | 17.01 | 32 | 55.17 |
| Unix and Linux | 83 | 13 | 15.66 | 6 | 46.15 |
| Mathematics | 619 | 85 | 13.73 | 31 | 36.47 |
| Skeptics | 74 | 10 | 13.51 | 8 | 80 |
| Aviation Stack Exchange | 15 | 2 | 13.33 | 2 | 100 |
| Academia | 77 | 10 | 12.99 | 8 | 80 |
| Mathematica | 33 | 4 | 12.12 | 4 | 100 |
| Travel | 35 | 4 | 11.43 | 2 | 50 |
| Electronics and Robotics | 79 | 8 | 10.13 | 3 | 37.5 |
| Puzzling Stack Exchange | 133 | 13 | 9.77 | 10 | 76.92 |
| Stack Overflow | 6111 | 563 | 9.21 | 169 | 30.02 |
| Statistical Analysis | 44 | 3 | 6.82 | 1 | 33.33 |
| Science Fiction | 158 | 10 | 6.33 | 8 | 80 |
| Code Golf | 129 | 7 | 5.43 | 4 | 57.14 |
| English Language Learners | 77 | 4 | 5.19 | 3 | 75 |
| MathOverflow | 72 | 2 | 2.78 | 1 | 50 |
| TeX - LaTeX | 101 | 2 | 1.98 | 1 | 50 |
+------------------------------+---------+-----------------+-----------------+-----------------------------------+--------------------------------------+
While we are moving a decent number of these conversations to chat, we are getting a high percentage of comments on the posts after the move. I'm still working on some numbers to see how often the chatrooms are used once the conversation is moved. In the meantime, I performed a very unscientific test of popping into rooms on a variety of sites to see if users continued chatting. I'd say that in about 30% of the rooms I checked, the conversations continued. (Very unscientific but it was something...right?) Most of the time, it appeared that the conversations had already stopped by the time it was moved to chat or the move ended the discussion.
How has comment flagging changed (if at all)?
It appears to be roughly the same. I queried the data from 2014-01-01 to present to get the monthly average of comment flags and across the network there isn’t much of a difference since this feature was implemented in August 2014 (note these are only the sites who average more than 50 comment flags after the implementation):
Before 2014-08-01 After 2014-08-01 Site Name
----------------- ---------------- ---------
13370 16952 Stack Overflow
394 673 Mathematics
179 269 Super User
209 266 Role-playing Games
276 230 Ubuntu
87 201 Physics
43 187 Code Review
197 181 Programmers
135 172 English Language and Usage
65 160 Science Fiction
234 152 The Workplace
92 137 Christianity
173 134 Jewish Life and Learning
58 127 Academia
132 124 Computer Science
87 96 Unix and Linux
111 87 Gaming
118 82 TeX - LaTeX
12 82 Chemistry
78 76 Server Fault
8 71 Puzzling Stack Exchange
40 65 Travel
33 56 Electronics and Robotics
46 56 Skeptics
16 52 Movies
163 50 Software Recommendations Stack Exchange
21 50 German Language and Usage
Overall, are people commenting more or less, and by how much?
Users comment a lot both before and after it this was implemented but using the same timeframe as above 2014-01-01 to present and splitting the data based on the implementation date of 2014-08-01, here is the monthly average of the number of comments each site gets:
Before 2014-08-01 After 2014-08-01 Site Name
----------------- ---------------- ---------
724043 963819 Stack Overflow
18275 19437 Super User
9102 10340 Server Fault
66 101 Stack Apps
749 1378 Cooking
1415 2158 Home Improvement
2899 2604 Game Developers
3777 4928 Gaming
4846 5674 GIS
70631 78682 Mathematics
1495 1439 Photography
5714 7582 Statistical Analysis
907 835 Web Apps
1578 1538 Webmasters
3597 5248 Apple
1134 708 Theoretical Computer Science
12749 14216 English Language and Usage
1445 1745 Personal Finance and Money
3068 3835 Role-playing Games
12986 12953 TeX - LaTeX
14944 19806 Ubuntu
9207 10040 Unix and Linux
5273 4564 WordPress
1016 1093 Bicycles
5157 7062 Programmers
1801 3076 Android Enthusiasts
588 720 Board and Card Games
8105 12292 Physics
238 185 Homebrew
3214 4228 IT Security
375 460 Writers
7965 10014 Electronics and Robotics
2154 2385 Graphic Design
3808 4810 Database Administrators
4260 6720 Science Fiction
523 528 Area 51 Discussions
5182 5546 Code Review
5789 3767 Code Golf
534 487 Quantitative Finance
233 240 Project Management
1339 1460 Skeptics
4028 3588 Drupal Answers
549 526 Fitness and Nutrition
391 696 Motor Vehicle Maintenance and Repair
598 926 Parenting
3031 3252 SharePoint
1205 1365 Musical Practice and Performance
258 330 Software Quality Assurance and Testing
3016 3691 Jewish Life and Learning
448 516 Astronomy
1027 1237 German Language and Usage
989 861 Japanese Language and Usage
327 514 Gardening and Landscaping
1296 1751 Philosophy
166 136 Personal Productivity
2154 3286 Travel
1503 1453 Cryptography
951 844 Signal Processing
445 417 French Language and Usage
2429 2266 Christianity
1123 622 Bitcoin
555 593 Linguistics
608 656 Biblical Hermeneutics
976 1231 History
65 90 LEGO®
280 337 Spanish Language and Usage
478 518 Computational Science
1336 1609 Movies
481 520 Chinese Language and Usage
1148 2034 Biology
65 108 Poker
5377 6090 Mathematica
453 404 Cognitive Sciences
293 404 The Great Outdoors
128 142 Martial Arts
264 229 Sports
3362 4338 Academia
2140 2394 Computer Science
4046 3830 The Workplace
251 276 Windows Phone
1146 2203 Chemistry
346 366 Chess
909 1262 Raspberry Pi
278 334 Russian Language and Usage
811 771 Islam
3453 3961 Salesforce
121 129 Patents
2662 2143 User Experience
116 161 Genealogy and Family History
203 271 Robotics
855 444 ExpressionEngine
536 612 Politics
270 312 Audio-Video Production
856 1079 Anime and Manga
2608 4480 Magento
3757 4823 English Language Learners
103 87 Sustainable Living
491 353 Tridion Stack Exchange
327 290 Reverse Engineering Stack Exchange
820 1071 Arduino Stack Exchange
957 930 Network Engineering Stack Exchange
142 199 Open Data Stack Exchange
185 139 Freelancing Stack Exchange
1336 2541 Blender Stack Exchange
6272 5965 MathOverflow
542 867 Space Exploration Stack Exchange
305 243 Sound Design Stack Exchange
200 180 Tor Stack Exchange
540 377 Pets Stack Exchange
193 136 Amateur Radio Stack Exchange
156 296 Italian Language Stack Exchange
6353 8521 Stack Overflow em Português
1394 1860 Aviation Stack Exchange
150 55 Ebooks Stack Exchange
153 31 Beer Stack Exchange
1886 1187 Software Recommendations Stack Exchange
622 287 Expatriates Stack Exchange
1434 470 Mathematics Educators Stack Exchange
583 342 Earth Science Stack Exchange
697 606 Joomla Stack Exchange
321 248 Data Science Stack Exchange
993 4162 Puzzling Stack Exchange
1249 701 Craft CMS Stack Exchange
1010 739 Buddhism Stack Exchange
1517 783 Hinduism Stack Exchange
324 93 Moderators Stack Exchange
93 389 Startups Stack Exchange
4646 6256 Stack Overflow на русском
I'm trying to get the chat usage info but the big question that sticks out to me is why so many conversations continue under a post after it's been moved to chat?
I can only guess, but in my experience as a SO mod these are a few things I've noticed:
- Users were already given the chance to move to chat when the notice pops up to them reminding them to "move extended discussions to chat" and they aren’t taking it at that point, so why would they move when a moderator got involved?
- Many of these conversations, at least on SO, involve new users and even experienced don't understand that moderators can bypass the rep requirement of 20 when they move it to chat. On more than one occasion, after I'd move a conversation I’d get a response back saying " can’t go to chat because they don’t have the rep".
- The chat URL can be blocked for some users at work/etc so migrating the conversation doesn't work for them.
- Some users don't want to go to chat, so they'll continue in the comments not matter what.
I'm still chewing on what I've seen, I think there might be a case to improve the usage of the feature but I'm not sure how yet. Some changes to the feature have been proposed already, including adding a move to chat on the mod menu, and I even suggested the possibility of making this a 30k privilege. Part of me wonders whether the feature would be used more if we were to move these conversations to chat earlier (without a flag) or if other users had the ability to move them... but that's a discussion for another day.
