37

The second private beta launch of EmbeddedSystems has been killed after about a week and a half.

Having had one disappointing experience with the maiden beta, and now a second (though with less personal buy-in, because of my experience with the first), the engineer in me feels compelled to start running a DFMA on this process. Here's where this exercise leads me.

In professional practice, I would never sign on the dotted line until the rules are made clear to me. I would put the work in up front to make sure I understood the needs, the timetable, the review steps, and the tools available to me to do the work.

In the Area51 process, I have very little of that. I don't know the rules, the metrics for success, and the timetable for review is simply unrealistic for the endeavor given those conditions.

I get that SE needs to somehow monetize our expertise to make this endeavor worth their bandwidth. I don't mind that. What I do mind is that in the Area51 process, SE feels like the executioner, not a partner. There are community coordinators like @JonEricson and @RobertCartaino who are generally encouraging, but certainly aren't playing the role of coaches who help move us through the process -- they just tell us when its over, and encourage us to start again.

Having been through two private betas (with the first the more successful of the two, in terms of engaging questions, IMO), and getting the feeling that critical mass in this area will be hard to come by, I'll say up front that I'd be very reticent to participate in a third without a clear review plan, some metrics, and an engaged community partner who can help move us to graduation to public beta.

This is a repost of a meta ES.SE post, but I'm reposting it here as a general discussion on private beta launches and graduations to public beta.

EDIT: Discussion has led me to request a feature to reinvite committers from previous launch attempts to new attempts. I added a reference to here at a feature request at the Area51 discussion area.

  • 1
    Embedded Systems, Robotics, Arduino, Raspberry Pi, EE... I'm reminded of Monty Python here (YouTube). – Deer Hunter Aug 13 '15 at 17:38
  • @DeerHunter, mind if I call you @Bruce? – Scott Seidman Aug 13 '15 at 17:42
  • 1
    I gave up on the area51 process a long time ago. The simple fact is that Stack Exchange has no need to improve it - they are satisfied with the few sites that come out gangbusters, eliminating the slower starting sites doesn't cost them a thing, and there's a very strong fear that sites which languish significantly hurt their brand image. At this point area51 is in maintenance mode - with nothing planned to replace it. This simply isn't going to change - they do not care. Certainly a few of the valued associates do, but the company itself has no love of its community or area51. – Adam Davis 'ze-zir-zem' Aug 14 '15 at 13:00
  • Hi Scott! I came across this through that discussion question in Area51. Might you be willing to link to it here in this question, so people can check it out? Thanks! – Sue Aug 14 '15 at 18:26
  • @Sue -- sure. It's done – Scott Seidman Aug 14 '15 at 18:29
26

I'm with you on just about… all of this.

A few months ago, I wrote a promising proposal called "An Objective Site Life Cycle" which accomplished about 90% of what you described above. It was ultimately pushed out in favor of splitting up the graduation process (baby steps), but the main tenet of this proposal was to present a completely objective and transparent process which listed exactly what communities would need to get to that next level (including things like starting an election, removing the 'beta' label, getting a custom design, etc, etc). It was a way to clearly convey:

You are ⇒here⇐ and here is what's
                                                    ⇓
                                               coming
                                                    ⇓
                                                   up
                                                    ⇓
                                                 next

A set of completely objective criteria would all but remove the Community Team from that adversarial role of judge and executioner. There would likely have to remain a thin editorial layer to account for process-gaming and other crazy sh—, but challenging communities to work towards very specific goals puts us back in that role of counselor and community advocate.

That doesn't guarantee that all sites will work — you can never really take 100% of that sting out of losing an experiment that just didn't work out — but when you know up front, "If you can do {this} by {x} date, you move on", communities are remarkably resilient and understanding when the "agreement" is known up front.

  • 1
    Thanks for your thoughts, @robertcartaino. Interestingly, its not even the transparency that I'm concerned about, though perhaps objectivity does come into play. I teach device design, and you can imagine my reaction when a team comes to me with "We'll know it when we see it" when we ask "How do you know its working?" Of course, we send them back to their needs and wants and make them work on their specs list. A spec, of course, is met or not, a binary choice. Inability to get to that point often indicates something about the grasp on the needs. – Scott Seidman Aug 12 '15 at 17:50
  • 1
    I know you've mentioned several times that "the correlation between a slow start and sites that never gain any traction" is high, but being my first private beta, I have to ask: What's the average duration of private betas? Has there ever been a significant improvement if private beta duration is extended considerably? – Roflo Aug 12 '15 at 22:36
  • @Roflo: We generally let private betas run for two weeks. I believe there have been a few sites that have gone on a little longer. But the relevant data is the dozens of sites that have started off slow which we have allowed to continue. Those sites have not fared particularly well, I'm afraid. – Jon Ericson Aug 13 '15 at 1:17
  • 1
    Thanks @JonEricson. Assuming you're in position to answer, does it represent a significant cost for SE to keep a private beta running for a couple of extra weeks? Is it a matter of human resources? Maybe the Community Staff is a valuable and scarce resource? Or maybe it's not as simple as that? – Roflo Aug 13 '15 at 14:06
  • 1
    @Roflo: I added an answer that describes some of the problems with keeping a site running in public beta. I'm reluctant to extend a private beta because I just don't think it will help. It seem wrong to me to encourage people to work on a site we are very likely to close. A far better use of your time would be to either gather momentum for a new proposal or figure out how to use existing site (SO, EE, Robotics, Raspberry Pi, Arduino, etc.) more effectively. – Jon Ericson Aug 13 '15 at 15:35
  • 1
    Where are these things published, if they are published at all? – Pavel Mar 5 '16 at 10:11
16

I closed both Embedded Systems private betas. In terms of the negative feelings closing a site causes, both were difficult decisions. In objective terms, however, I'm entirely comfortable. As I wrote the first time around:

Have you ever run across an awesome blog post and went to read the rest of the site only to discover the author gave it up after a week or two? That's how I feel about this site. To my (admitted unexpert) eye, the questions and answers so far are just what I'd hoped to see on this site. There's a lot to like. But the initial activity suggests a site that didn't gather the critical mass that Area 51 is supposed to build.

(I had to dig this up out of the data dump. One feature of private betas that we take seriously is making sure content is freely available even if the site gets closed.)

Building a site's user base is best done during the Area 51 process. Over the years, we've set up a handful private betas with the Field of Dreams approach. Instead of forcing proposals to complete each Area 51 phase, we've simply allowed the site to start with the promise that an audience will materialize. (Most of these cases were the result of an organization approaching us directly asking either for a site to be built or for one of the restrictions to pass to the next phase be lifted.) I'm not aware of any "Field of Dreams" sites (including our own) reaching graduation; most languish in beta if they ever go public at all.

The first week of a private beta is particularly critical since it sets the course of a site for months, if not years. Paradoxically, site activity usually decreases after going public. The exceptions are sites that start strong right out of the gate and never really slow down: Emacs, CiviCRM, Stack Overflow itself, etc. Much of the initial surge seems to be the result of the novelty effect; people put in more effort into using a thing while it's still new. Sites that don't benefit from a fast start tend to remain slow.

I am somewhat reluctant to publish objective numbers for private betas to hit because of the Hawthorne Effect. In addition, as Robert points out, we haven't done the work to figure out what level of activity has worked well and what level hasn't. However, setting objective numeric criteria (as described in Thinking, Fast and Slow) will reduce the subjective judgements we make, which are really likely to be wrong.

There are hidden risks of keeping a slow site around and those risks are heightened for a topic that's already being served by an existing site. Stack Exchange sites are community curated, which means ordinary users earn the right to edit, vote to close, review posts, and so on. Building up an invested community takes time so beta sites have lower reputation requirements for these actions. Instead of building a new site for embedded systems questions from scratch, most askers would be better off asking on an existing site that fields those questions. And of course, experts will find more questions to answer on more active sites.

Finally, I apologize for appearing to be more an executioner than an advocate. One way I can improve in the future, I suspect, would be to not assume that everyone was already familiar with the entire process. It's likely I overestimated the number of people who had experienced the first Embedded Systems closure.

  • Thanks for posting an answer Jon. Clarity in the process will be very helpful indeed for future proposals. – Roflo Aug 13 '15 at 16:01
  • I don't doubt that you take data availability seriously, but there are known exceptions where no dump was made available. – user259867 Aug 13 '15 at 16:03
  • Thanks for the input, Jon. There's a growing impetus for a relaunch. I'd love to hear about some of the actions that propagate from tossing some of the assumptions about how up on the process the committers are. Perhaps a better description of the process in the private beta invitation would help, and maybe even a followup a few days later. Do you have any feeling for how many committed users generally join a private beta? How did ES.SE stack up to "normal" groups in those terms? – Scott Seidman Aug 13 '15 at 18:47
  • Boy, that's a horrible sentence above, but I think the gist is there ;) – Scott Seidman Aug 13 '15 at 20:39
  • 5
    “topic that's already being served” is still debated. AFAIK there's a big chunk of [embedded.se] that neither Stack Overflow nor Electrical Engineering covers. If there wasn't, I wouldn't defend the site. I'd be happy to stick to Electrical Engineering, but my impression is that they don't want my software questions, and I know Stack Overflow doesn't want them (SO only wants the programming questions, not the integration questions). – Gilles Aug 13 '15 at 20:41
  • 1
    Turns out a bunch of interesting numbers are up on the Area51 site -- and I was just too dumb to see them there. The number of committers who joined the beta is interesting -- but the MORE interesting number to me is that about 10% of the members of the first beta joined the second!!! If we launch a third time, it would be fantastic if there were a reliable way to reach the members of the first and second launch to invite them. If that could be done, that would be a wonderful way for the SE Community to help us reach critical mass. – Scott Seidman Aug 13 '15 at 21:25
  • Jon, @RobertCartaino, I've added a suggestion in a new answer that might be considered a new feature request for Area51. I don't think it changes any of the "rules" or helps game the system, but can really add some robustness to the private betas of relaunches. – Scott Seidman Aug 14 '15 at 12:31
  • 2
    "I am somewhat reluctant to publish objective numbers for private betas" -- this is quite understandable. But I think the key is open communication, not objective numbers. I was part of the (as of today) closed Sexuality private beta. I really don't know why it failed (in soft, squishy, subjective terms). And if the subjective criteria vary so much from site to site, can someone at SE at least take the time to write a meaningful post-mortem for each site that fails, explaining why that specific decision was made? Help us learn your subjective reasoning. – Flimzy Aug 14 '15 at 19:35
  • @Flimzy: There should have been an announcement on meta sometime last week. (Unfortunately, you'll need to dig into the data dump, when it comes available, to read it.) On a personal note, many of the questions I read (and both that I answered) would have been good fits on the Health site. A lot of other questions weren't really good fits for our network at all. – Jon Ericson Aug 14 '15 at 19:44
  • 1
    @JonEricson: There was. I read and responded to it, and made the same complaint/request there. – Flimzy Aug 14 '15 at 19:55
13

When announcing site closure, Jon Ericson cited 74 questions in the first 9 days. On the other hand, Arabic Language was given green light with 63 questions in the first 9 days. Even now, the nearly month-old Arabic Language has only 89 questions, trailing the condemned Embedded Systems which has 109.

We can deduce that the decision is not a direct function of the question volume. Another factor likely to be in play is whether the site brings new participants and new audience to the Stack Exchange network, as opposed to moving them from one part of the network to another. This is a major handicap of the Embedded Systems proposal; I imagine it would need to demonstrate a high level of activity not only in general, but from users who are not already active on the network.

  • 16
    That was indeed an "exceptional" decision that we are (re)evaluating whether our reasoning panned out; it doesn't look like it did. Arabic never made up for that slow start, which only strengthens our premise for what the numbers tell us — that the correlation between a slow start and sites that never gain any traction is SO strong, ignoring those hard-earned lessons is becoming increasingly irresponsible on our part. I know it's tough to judge "success" by the numbers, but that is largely why Embedded Systems did not make it. – Robert Cartaino Aug 12 '15 at 18:19
  • 1
    @RobertCartaino Perhaps for betas with slow starts, a "growth" term might be a reasonable metric to determine possible graduation to public beta. I know in this case you guys generously laid it out as an increase in organic questions, but a healthier approach might have been a higher question count, a growing user base, and a certain percentage of >500 users on a more relaxed, but still realistic time frame. I know I wasn't going to be inviting the pros because of the maiden launch experience until I knew that this one had a shot, to avoid the "told you so's"... – Scott Seidman Aug 12 '15 at 18:39
  • 1
    An approach like that might afford small communities a more realistic shot of a viable stack. – Scott Seidman Aug 12 '15 at 18:40
  • 3
    @ScottSeidman I am addressing the time frame issue; there were some good points brought up during the private beta. But we have to be careful about coming up with a "healthier approach" when the question count has already proven to be a remarkably accurate indicator of success. Yes, we may have to add additional criteria to avoid gaming, but the goal isn't to lower the bar simply to create more sites. The goal is to accurately predict (based on our historical data) if a site has a reasonable chance of working at all. Otherwise we just piss a lot of people off when we have to close it later. – Robert Cartaino Aug 12 '15 at 19:00
  • @RobertCartaino -- thanks for looking into the time issue. The additional criteria I suggest based on user reputations wouldn't lower the bar, though. I don't think either of the ES launch attempts would have met this criteria (if it were well-specced), as well as the number of questions criteria. It would be a number harder to game as well, as the participation would have to be realer, with good questions, good answers, and good discussion to drive up reputations. It would be harder for a handful of high-rep users to drive up question count (and possibly disappear later!). – Scott Seidman Aug 14 '15 at 13:11
10

I've participated in several private betas, and my impression is very different. In the cases where the sites were closed, most of the community generally agreed that it didn't work out. Those were the easy cases, but this one looks more difficult as the community doesn't seem to agree with the decision.

From what I read just now, question volume seems to have been the deciding factor this time. Usually the problem is either question volume or quality, and while the latter is pretty much impossible to quantify, I think a general guideline could be created for the required volume.

Initial post volume in a private beta is a really useful indicator for several issues. It tells us something about the number of actually interested users that the A51 process managed to attract, and something about whether the scope and topic of the site are a reasonable fit for an SE Q&A site. Starting a site without enough active users is bound to fail, it's really hard to attract new users if a site looks very inactive.

Just to add some statistics, the two sites I'm a moderator on had 64 and 120 questions in the first two days. The initial volume of ES.SE with 20 questions on the first day and around 10 on the second looks low to me.

What could be useful is a statistic about all the private betas that a new site could compare themselves to. This would give the users of the new private beta a better idea on what kind of volume is expected. I wouldn't tie down SE to a specific value, it wouldn't be much use as there are several other more subjective factors aside from the question volume. But simply showing what other sites looked like when they started could be helpful.

  • I don't know that the beta users don't agree with the decision, so much as I (who won't deign to speak for other users) feel unsatisfied with the process. Observationally, there isn't much overlap between the top user list of the two launches, so the relaunch leaked users, and some feel question quality moved in the wrong direction. – Scott Seidman Aug 12 '15 at 14:50
  • would you recall how quickly the failed private betas failed? I think just including a better description of how QUICKLY these betas get shut down in the private beta invitation would get these off to a better start. – Scott Seidman Aug 12 '15 at 15:03
  • 3
    @ScottSeidman I think it's usually at some point in the second week. Sometimes SE gives a warning and another week to improve, but not always. – Mad Scientist Aug 12 '15 at 15:14
  • 1
    Thanks for the new stats. Part of the problem w/ the concept of ES.SE is the tremendously low pool of potential users. Maybe it just can't fit comfortably in the SE model. That said, because of this, its a tough area to get expert guidance in, so it would be worth some focus. With the small pool of SE users with expertise who are willing to consider a new group despite overlap to draw from, the "kill it and restart" model seems to alienate experts (judging from the low overlap in top users in the two attempts. I don't think multiple restarts will help get this group going. – Scott Seidman Aug 12 '15 at 15:51
  • 1
    I participated in the first Embedded site (proposed it, even) and I had a lot more enthusiasm the first time. With me, the very fact it was rebooted is what made me be less active. The novelty has already worn off. I wish "they" wouldn't expect Embedded 2 to be a novelty... – BenjiWiebe Aug 14 '15 at 4:24
  • 2
    @BenjiWiebe -- the carryover of 10% of the committers from the first launch to participants of the second private beta floored me. I think if Community can help us tap in to the committers list from the first and second private beta's to announce the launch of a third, that this could be a go. I understand the novelty issue -- I think I suffered it myself -- but I think the disappointment in the private beta process is really what held me back, and certainly prevented me from reaching out to nonparticipants. – Scott Seidman Aug 14 '15 at 12:14
7

After reading all the wonderful discussion, I can see one very powerful way for Community to help advocate relaunches. I have a data point of N=1 for ES.SE, but the 10% level of first launch committers who participated in the relaunch seems low. I can think of two possible reasons for it.

  1. Dissatisfaction with the first launch
  2. They simply didn't know about the second launch

It seems like an absolute no-brainer to me that sending private beta announcements to the committers from all previous launches (without, of course, changing any commitment phase rules) might substantially increase private beta participation.

So, is this something the Community can make happen? Not just for ES.SE, but for relaunches in general?

  • 1
    One issue is that we don't actually have a concept of a relaunch. We had at least two Artificial Intelligence proposals fail in private bata and two Machine Learning sites closed in private beta. The proposal that finally stuck was called Data Science. Not everyone committing to AI is interested in Data Science. – Jon Ericson Aug 14 '15 at 16:07
  • @JonEricson -- yeah, I can imagine that building this into the system would be quite the deal. Perhaps a way to flag other launches as "related" would be a good item to put on the wish list in the event of a rehaul. In the meantime, I was hoping there might be some sort of a more manual way to do this. – Scott Seidman Aug 14 '15 at 18:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .