What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 128 Stack Exchange communities.

Executive summary

Introduction of sparse hierarchy (partial inheritance relation) into the set of tags on SO, in order to allow more efficient question organization/filtering/tagging (especially via "Interesting"/"Ignored" mechanism).

The proposal is designed to achieve its goals while adhering to two principles:

  • As much backwards compatible as possible - e.g., all work-flows/GUIs not related to tags should remain 100% identical; and any work-flows related to tags should remain very similar (with exception of new tag creation for lower-rep users).

  • As far as data population, this is intended to be a gradual incremental process - e.g. instead of one-time massive hierarchical classification, it is intended as an incremental addition/improvement of relationships.

Why bother? (aka use cases)

  • As a use case #1, take a fictitious user "Larry Ceiling" . He is a Perl expert, and would like to ensure that every Perl question has good, complete and accurate answer. To do so, he will need to either use "Interesting Tags" mechanism, or to employ a search by tags.

    Currently, "Larry Ceiling" needs to use between 15 frequent Perl-related tags to ensure "good enough" coverage and at least 40 all perl-related tags to ensure full Perl coverage.

    Searching for "perl*" isn't good enough (because many Perl-related tags do not contain the word "Perl" - for example or or ).

    In addition, due to new tag additions, this list needs to be maintained by virtue of monitoring for new Perl-related tags.

    What Larry wishes is a way to simply say that he's interested in tag, and achieve the correct result.

  • A mirror use case is user named "Jon Biatlon", who would like to never see another C#, or .net question for whatever reason. I estimate the amount of C#/.net tags far exceeds Perl ones, and "Jon" will need to add every single one of them to "Ignored Tags" list or exclude from searches.

  • More use cases are in the examples in Benefits section.

Solution

Add an ability for a many-to-many transitive relation "contained in" (aka "sub-tag") between some tags.

Just to be clear - the relationship will be sparse. NOT every tag will need to have a parent.

  • This relation would only be applied to completely contained tags. For example, tag would be contained in tag; but [tag:OO programming] and tags would be completely independent. would be independent of .

  • The relation could be many-to-many, for example tag would be contained in both , and, if such exists, tag. However, I don't envision too many cases where multiple ancestors would exist - see next bullet point for the reason.

  • Some composite tags should be split up. For example, all of , and need to be changed to + specialized tag (Perl, boost or apache correspondingly). This is the reason I believe multiple ancestors would be rare.

  • Tag relationships can be specified by either a creation of a new tag, or via a special "relate tags" management screen.

  • When tagging a question; the GUI allows "searching" for tags by sub-string via auto-complete drop-down (as we do now), or by browsing a tag hierarchy, e.g. typing "C++>" will bring up a drop-down of all children of [C++] tag.

  • NOTE To increase backwards compatibility, we may want to only enable the "include descendant tags in searches" if the special form of the tag is used in Include/Exclude/Search (e.g. [perl] vs. [perl*]. Personally I don't think this is needed but may address the concerns by people that automated inclusion/exclusion of more questions may be disconcerning).

Proposed permissions

To shorten the post, I removed a detailed proposal for permissions required for creation of tags AND tag hierarchies - see edit history if you're genuinely interested. </Fermat>

Implementation - technical

Add 2 new tables (I don't know exact table structure of SE, so details may be off):

Table 1: *tag_contained* (parent-child relationship)
| parent_tag | tag | approved (could be >1 column) |

This table contains the relationship definitions. It is populated either when a new tag is created to be a child of an existing tag, or when a user with sufficiently high rep (say 250, or 3000 for "TOP" parent) proposes a new relationship, with "approved" value of false.

In addition it would contain newly proposed "top level" tags, with a special "TOP" parent tag value - but once approved, such records can probably be removed from the table.

When a user who can approve (10k? moderator?) approves it, "approved" is updated to true value.

This table is also used by the "tag population" GUI, by listing N-level drop-down menu instead of current 1-level auto-complete drop-down (better GUI approaches are welcome!)

Table 2: *tag_contained_expand*
| ancestor_tag | tag |

This table contains ALL transitively computed relationships. E.g. if the first table contains A->B, B->C, C->D, this table will contain A->B, A-C, A->D, B->C, B->D, C->D. This table is populated automatically based on all approved rows in the first table, either by a regular job or by a trigger. Since I do not anticipate an unduly high amount of tag relationships, this should not be overly intensive process.

The second table is actually used by search engine, include and ignore mechanisms - namely, a query of a type "where tag in (X,Y,Z)" will be expanded to join with tag_contained_expand.

It will also be used when tagging a question, by eliminating un-needed ancestor tags. E.g. a question with tag would not need tag due to "contained in" relationship between them, and thus the latter will be either automatically removed or suggested to be removed (I suggest automated + notification).

  • NOTE One additional technical detail - for Google's permission, the ancestor tags of question need to be included (hidden or visible?) on the question.

Benefits

  • Main benefit is, of course, much simplified searching and filtering as illustrated in use cases.

  • Some of the 35000 tags (especially composite ones) could be removed, as their only reason for existence is to denote hierarchy. Examples. , , .

  • Overly-specialized rarely used tags - - which are the main candidates for being someone's child - no longer pose a problem for either searches/filtering, OR for finding them when tagging a question.

  • Misspelled and differently-spelled tags pose less of a problem as well. Much easier to find assorted .net and dotnet related tags if they are all under a single [.net] hierarchy. I realize that tag-synonyms may solve this particular problem, but it's still a problem meanwhile.

  • A problem of finding tags that are related to some topic but have nothing at all common as far as wording is removed. is a lot easier to find in a list of perl's children than on its own, especially for people who don't necessarily remember that word.

Potential issues and resolutions

  • Issue: Tags that apply to multiple domains creating logistical nightmare. For example, C++/memory-allocations, Java/memory-allocation, etc...

    Solution: should be independent of either of those 2. The examples above should be 2 independent tags each.

  • Issue: It is very hard to organize 35000 tags into a tree, never mind do it correctly.

    Solution: The 3 components of "hard" assessment are sheer volume, the fact that most tags are really independent of each other, and the difficulty in placing certain tags under just one parent.

    First of all, a vast majority of the tags will not be in a single tree. There will be many top level tags and the forest would for most part be rather flat and sparse.

    Second, this is meant to be a gradual incremental process. 10 tags related first day. 5 tags second day. All Perl tags within 2 weeks (I'll volunteer :). etc...

    Third, while rare, multiple parents are allowed, thus eliminating "but it belongs to both!" headaches. Although, as noted, most cases when it appears that the tag belongs to multiple parents, either it should be independent in the first place, or it is a composite tag which should be split into two.

  • Issue: This proposal is too bloody long

    Solution: I should get more sleep.

Uniqueness

NOTE: this question is NOT a duplicate of this one, since it contains detailed proposal instead of a general idea. Some of the ideas are based on results of this discussion.

share|improve this question
4  
Ok, read. I like it. –  Tyler Carter Apr 4 '10 at 15:12
7  
Sounds like a right hot mess of adding complexity in what should be simple and straightforward. –  random Apr 4 '10 at 16:46
1  
@random - seamless complexity. Unless you're creating new tags, the system behavior is 100% backwards compatible other than including more posts in a search. Even underlying technology change is fairly simple/low impact –  DVK Apr 4 '10 at 17:35
    
possible duplicate of Tag hierarchy to browse through tags –  Brad Mace Aug 13 '12 at 18:32
add comment

6 Answers 6

I totally support hierarchie*s* in the underlying implementation.

I see very little need for hierarchies in the basic user interface itself (except for moderators and people explicitly using the tag hierarchy).

In the SQL space, right now this is increasingly obvious. If I want to see all SQL questions, this should include t-sql and pl/sql dialects etc. If I want to see all sql-server questions, this should include sql-server-2000, sql-server-2005, sql-server-2008. This is especially important because of the 5 tag limit. You cannot expect people to tag correctly at the moment since there is still no way to stop people repeatedly using the incorrect tags.

What we did for our technical support knowledge base was allow things to be tagged, but that tags could be multi-parented. So if people were looking at a problem related to an HP 970 printer driver and margins, it would show up under printing, drivers, hp, hp-970, margins, etc. And they didn't have to worry about the fact that there were multiple mount points for a tag or anything like that. They didn't have to tag all the way up and the search would automatically find everything under whatever rollups because we cached the full traversal.

For improved performance, we cached the tree traversal up so that the data entry and management was normalized but the denormalized cached data could be used to ensure performance.

share|improve this answer
    
A freeform, multiple-parent hierarchy does address a number of the issues. If it's practical to implement and maintain (not a trivial task in either respect) then this could work in tandem with or maybe even as a replacement of tag synonyms. –  Aarobot May 27 '10 at 16:11
add comment

You definitely need a multi-parent hierarchy.

The what-should-be-a-child-of-what discussion is not really one for here. The bottom line is that it is possible to come up with a hierarchy. Given that this is a user-managed environment, the hierarchy should eventually stabilize at the most intuitive structure. If it's not clear where a tag belongs, just keep it on the top level; we're after a hierarchy, not the perfect one.

The hierarchy already exists, we just want to formalize it. Consider that some people will just tag something with sql-server-2008, while others will tag it sql-server and sql as well. A lot of discussion can go into which one is better, but the proposed hierarchy would cause the two to be combined.

There are consistently new tags popping up, and people need to favourite and ignore appropriately, having a proper hierarchy would reduce the need to do this.

Search engines consideration - It may be disadvantageous to tag something only with a tag at the lowest level, possibly losing a keyword indicating the language used which would decrease the rank of the results. Worth a thought.

Some thoughts on dealing with the problem of people getting results for things they don't want to get results for:

  • You can have strong-inheritance (sql-server-2008 child of sql-server) and weak-inheritance (linq-to-sql child of linq). strong-inheritance automatically includes the child, weak-inheritance optionally includes the child. This default allows your new users to hit the ground running without knowing about inheritance, and they'll figure out that when they search for sql-server, sql-server-2008 also pops up. The inheritance structure explaining why a specific tag was included should probably also be displayed somewhere, with a link to an inheritance FAQ.

  • Degree of inheritance. An extension of the above, allowing more than 2 choices when specifying how strong the inheritance is. This would probably allow for a better hierarchy, but this is a lot more complicated for everyone.

  • You can specify whether you want to include children (strong/weak inheritance or both) when specifying a tag (as a favourite, ignored or in search), possibly using some affix.

  • Ignoring linq and favouriting sql (both with weak inheritance) probably should cause linq-to-sql to be ignored, similar to the way it's working now with multi-tagging.

  • Posting questions under certain tags (or tags high in the inheritance tree) should be demotivated (possibly with a warning when trying to use it) or disallowed. sql comes to mind, how many questions are there that fits under sql that wouldn't be better suited to be tagged under one of it's children? (I'm not saying there aren't any, just that there are few)

  • Tagging something with a tag and it's parent should probably be disallowed (at least in the case of strong-inheritance, demotivated in the case of weak-inheritance).

share|improve this answer
add comment

I think many-to-many hierarchies are too complicated to manage, and very complex tree-like hierarchies also may be problematic, but there are some cases where hierarchy is the only logical way.

As example, I had issues with LocalStorage and cache-manifest, both of them being part of HTML5 specification. I've tagged my post also html5, since this tag is more popular and tagging it so increases the chance of being answered, and also javascript, since problem is with AJAX application. There are even more tags that could be appropriate, such as offline, offline-cache.

Of course unnecessary tags are one case (offline-cache is, I think, often mistaken with cache-manifest). But another case is this HTML5, which contains of mechanisms such as Local Storage and cache-manifest, so it is logical that these both should be subtags of the tag html5.

The same, I think, should apply to the various jquery extensions, which have separate tags, but practically whoever asks such question will also use basic jquery tag.

So in my opinion it should be at least simple one-level hierarchy. Two levels would be even better, but the greater the hierarchy, the more important to manage this tags correctly. Very high reputation should be required to edit tag hierarchy.

share|improve this answer
add comment

Far too much complexity for the problems you propose to solve.

I went cross-eyed trying to read the whole post, but I was simply not convinced by the "problems" you posit with the current system: those may call for a better means of telling the system what we are and are not interested in. They don't call for a complete rewrite of the post sorting mechanism.

Remember that we're going to have to explain all this to the users who

  • want to search
  • want to set their interested and ignored subjects
  • obtain the ability to create tags

and that if they don't get it they're going to be here on meta asking about it. Over and over again.

There may be issues surrounding tag approval. Stack Overflow currently has more than 500 10k+ users. Can any one approve a new tag? What if other high rep users disagree? Are we going to end up with a voting system?

share|improve this answer
    
@dmckee - Users who want to search will not have anything changed (I just realized though that my proposal needs to be improved for Google's sake to include hidden ancestor tags). The only tangible change is that searching for, say "perl" , will return exactly the bigger set of results that a random user would intuitively expect. –  DVK Apr 4 '10 at 21:22
    
@dmckee - For people who want to set their interested/ignored subjects, again, the only difference is that more Qs will be returned/rejected - and heck, we can always make the "include ancestors" an optional behavior if there's a tangible confusion fact for that (e.g. the only search for ancestors is for "perl*" tag and not "perl"). –  DVK Apr 4 '10 at 21:24
    
@dmckee - however, they now get a small warning saying "you have perl and 16 other un-needed tags - you can just be interested in perl alone!) –  DVK Apr 4 '10 at 21:25
    
I agree that people who want to create tags would undergo steeper learning curve - but seriously, being asked "this is a new tag you're creating, please choose its parent" seems like a very small learning curve, no? –  DVK Apr 4 '10 at 21:26
    
Please note that there's no change in TAG creation - new tags do not need to be approved. Only the relationships of new tags need to be approved - again, the goal is to be as backwards compatible as possible. –  DVK Apr 4 '10 at 21:27
    
And as I noted in an update, a good automated compromise system for relationship update may be: this requires unanimous approval of 3 10k+ users who are in top N answering ranks for one of tags in question. Any disagreements are resolved the way tag merges are today - MSO or moderator decisions. –  DVK Apr 4 '10 at 21:29
    
But I'm perfectly comfortable with "if one high expertize user disagrees, err on the side of "no relationship", till that dissenting vote can be persuaded to join the yays (or never) –  DVK Apr 4 '10 at 21:30
    
I can try to do some mock-ups since my explanations don't seem to convey just how small the changes will for the users –  DVK Apr 4 '10 at 21:31
    
Good concerns overall, +1... hope I addressed some of them. I updated the proposal as well from ideas your concernes raised (perl vs. perl*; and google-centric listing of ancestor tags explicitly) –  DVK Apr 4 '10 at 21:34
    
@DVK The “perl vs. perl*” bugbear makes it effectively useless. Very annoying. –  tchrist May 18 at 23:13
add comment

Hierarchies are problematical. Heterarchies are less so.

What could work is a set of clustering tags -- eg:

[perl/] could mean [perl] and [perlmonks] and [mess-up-like-perl] and other tags.

[status/] could mean [status-complete], [status-norepro], [status-declined] etc

  • A base tag can be in zero or more clustering tags.

  • Members with over [5000?] rep points can edit/create the clusters.

  • No question can be tagged with a clustering tag, only with base tags
    as now.

  • No one needs to use clustering tags as all the base tags still work the same.

  • Those who want unusual clusters (eg [1990s-technology]) can argue amongst themselves until they have a clear definition of what base tags that may include.

  • Later, more nuanced searches could be built by allowing ANDing (etc) of
    base tags and clustering tags, eg:
    [perl/] and not [golf]

share|improve this answer
    
Note that in searches, favorites and ignores, your [perl/] example is [*perl*] now and [status/] is [status*]. –  Mark Hurd Apr 4 at 6:56
add comment

Sorry, but this sounds like a nightmare to use.

You basically hand-wave most of the issue away by saying that memory-management should be a top-level tag. Okay, great, let's use that as an example.

  • Should memory-management be a subset of memory?
  • Should memory-leaks be a subset of memory-management (keeping in mind that memory leaks are still possible in garbage-collected environments)? Should it be a subset of memory?
  • What about memory-usage? outofmemoryexception?
  • What about memory-allocation?
  • What about malloc, which is specific to C/C++? Does it go under memory-allocation, which is a general tag, or one of the C tags?

You say you don't predict many multiple-ancestor issues; I submit that the vast majority of tags fall under this umbrella. Even seemingly-innocuous tags like c# could technically go under both .net and mono. apache can go under linux and windows.

Not only is the maintenance a nightmare, but managing the search results are completely non-intuitive. If I search for wcf, I don't need a bunch of questions tagged with wcf-data-services popping up. Some 10k user might rightly decide that it's a subset of wcf, but I never use it so it has very little relevance to me (and wcf-data-services questions are, thankfully, rarely tagged with wcf).

It's going to be an endless source of confusion and arguments.

It never ends.

You know what's a lot simpler? Tagging a perl question with perl. We get 5 tags per question, it shouldn't be that hard. If users can't be bothered to tag their questions correctly, then I don't think Larry should really be worrying about them.

Hyper-specialized tags like visual-studio-settings (2 questions!) should just be merged into their "parent" tags.


Fine, since "memory" is an abstract concept and you seem content to just fire off answers to those as though everybody in the world would agree, here's another, more targeted example. Take the following tags:

c# vb.net .net linq linq-to-sql entity-framework nhibernate linq-to-xml linq-to-objects linqdatasource linqpad

Let's seriously start thinking about how to organize these.

Linq is definitely part of .NET. It's technically part of C# and VB.NET as well, and is different for both of them.

Linq to SQL could be thought of as a subset of Linq. But its scope goes far beyond Linq, it's an entire ORM framework. If linq-to-sql goes under linq, so should entity-framework. Does that make sense?

linq-to-nhibernate is definitely about nhibernate. But it is also about linq. If I search for questions on nhibernate, I probably do want to see questions on linq-to-nhibernate. But if I search for general questions on linq, I almost certainly don't want to see the linq-to-nhibernate or linq-to-sql questions.

linqdatasource is technically a subset of linq, linq-to-sql, and asp.net (being a web server control). But what is the actual likelihood that I want to see questions on linqdatasource if I search for either linq or asp.net? Almost nil in both cases.

And what about linq-to-xml and linq-to-objects? Surely you aren't going to tell me that linq-to-xml should go under xml or that linq-to-objects should go under objects. Are you? Well, it's hard to say; if I happen to be looking for questions about XML, Linq to XML is the de facto way of working with it in .NET now. So I don't want to search for all questions about .NET, just questions about .net and xml, which is a fairly logical search parameter. Except, if I'm not looking for .NET XML questions specifically, it makes no sense whatsoever to show me Linq to XML questions.

And linqpad. Well, it's definitely linq related. It's all about Linq. But rest assured that I don't want to see questions about linqpad when I search for linq. It is a genuine "containment" relationship, but nevertheless a completely useless one, I'm usually not interested in giving tech support on a program based around Linq to SQL/EF.

I repeat: There is no end to the pain and anguish that a tag hierarchy would cause.

I will take this conclusion one step further: The lack of a hierarchy is the raison d'être of a tag system. Adding hierarchy semantics to a flat tagging system is creating the very problem that the tagging system was supposed to solve/circumvent; all in order to solve what is basically a non-problem (is it really that hard to find questions about Perl or .NET, and does it matter if your favorite/ignored tags occasionally pick up one or two false positives?)

share|improve this answer
2  
@DVK: I disagree. As I explained and you actually quote in your comment, memory leaks can happen in garbage-collected environments where you don't "manage" memory at all. It makes no sense to search for memory-management and have .net questions coming up. –  Aarobot Apr 4 '10 at 15:43
    
What about malloc, which is specific to C/C++? Does it go under memory-allocation, which is a general tag, or one of the C tags? - BOTH! Thanks for finding a great example of multiply contained tag for me! –  DVK Apr 4 '10 at 15:43
4  
@DVK: You keep answering these rhetorical/hypothetical questions as if they are simple truths, but they are in fact just your opinion. Everything makes sense when there's only one person making the decisions. In real life, different people have different opinions and there is considerably more ambiguity than you want to admit. –  Aarobot Apr 4 '10 at 15:44
    
I disagree with your assessment that many tags have multiple parents. In your example, I'd say that apache is independent of both Linux and Windows, although it might theoretically qualifyt as being under web-development. Again, the priciple is to err on the side of NOT having a relationship unless one is 100% crystal clear. –  DVK Apr 4 '10 at 15:46
    
@Aaronaught - you're wrong according to Wikipedia. "Garbage collection is the automated allocation, and deallocation of computer memory resources for a program. This is generally implemented at the programming language level and is in opposition to manual memory management". So garbage collection - aka automated memory management - is in fact a part of memory management, alongside with "manual memory management". –  DVK Apr 4 '10 at 15:49
    
@Aaronaught - yes, I answered with my opinions (I removed most of my comments since they merely seem like nose after re-reading). If there's a difference of opinions, that's what rep limits and MSO discussions are for - but to repeat again, if there's a legitimate uncertainty, the logic is to err on the side of NOT relating the tags. My whole proposal is designed to deal more with multiple-branched obviously related tags (all flavors of visual-c++) as opposed to "ALL tags" –  DVK Apr 4 '10 at 16:01
    
@Aaronaught - on the other hand, you are disagreeing with that LIMITED APPLICABILITY proposal by trying to dredge up examples of tags that either are, or appear to you, to be hard to hierarchise, and using that as a reason against having ANY tags hierarchycal –  DVK Apr 4 '10 at 16:01
3  
@DVK: Quoting Wikipedia doesn't make the scenario any less ridiculous. A question about a memory leak in .NET is not a question about memory management. It's a question about a memory leak. I have no control over the memory management and don't really care about the implementation details of the garbage collector. –  Aarobot Apr 4 '10 at 16:02
2  
@DVK: Re: "That's what meta is for" - do we really want to spawn an entire class of new discussions/arguments based around a tag hierarchy, just so we can help Larry find the 3 questions that he missed with his umbrella perl search? –  Aarobot Apr 4 '10 at 16:04
    
@Aaronaught - just because you are not using the correct term doesn't make anything ridiculous. If you're not interested in manual memory management tags (or wcf-data-services tags) that is what ignore list is for - you use a combination of includes and ignores. –  DVK Apr 4 '10 at 16:04
    
Finally, @DVK, re: "limited applicability", see my updated example. Those are all very popular tags, so hardly "limited." –  Aarobot Apr 4 '10 at 16:05
3  
@DVK: I am not using the correct term? What are you on about? Memory leaks are simply not a fully-contained subset of memory management, any Java/.NET/python developer can tell you that, and you're demonstrating a perfect example of how contentious this will become. Why should I have to concoct absurd combinations of include/ignores just to get the same results I'm already able to get relatively easily with the existing tag system? –  Aarobot Apr 4 '10 at 16:08
    
@Aaronaught - What does a tag popularity have to do with unambiguity of its hierarchy? Matter of fact I'd expect the vast majority of relationtips to involve LESS used/popular tags. E.g. nhibernate1.0 (I made that up) is OBVIOUSLY under nhibernate, whereas the rest of your example (which I'm not qualified to agree or disagree with) is clearly not obvious based on your (to-me-seemingly-expert-sounding) objections. Again, the whole idea is to only put in the stuff that is 100% clear and un-ambiguous as, for example, defined by a set of top answerers for a specific tag. –  DVK Apr 4 '10 at 16:13
    
In other words, If I say linq should include linq-to-nhibernate, I don't count as I didn't exibit the necessary expertise (which I of course don't even have) on the topic. And the moderator who approves said relationship should pay attention to that. Whereas if I or better yet Sinan say that "mod-perl" shoud belong to perl tag, we are qualified, especially Sinan :) –  DVK Apr 4 '10 at 16:15
2  
@DVK: As I mentioned, frivolous tags like nhibernate1.0 should just be merged into their logical "parents". It's easy to do, and is one of the reasons (I assume) why lower-rep users are allowed to retag. As for the level of expertise required to make hierarchy decisions, (a) experts don't/shouldn't always agree, and (b) it seems antithetical to the Stack Overflow philosophy which is that content matters, not users. –  Aarobot Apr 4 '10 at 16:29
show 17 more comments

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .