Introduction of sparse hierarchy (partial inheritance relation) into the set of tags on Stack Exchange sites, in order to allow more efficient question organization/filtering/tagging (especially via "Interesting"/"Ignored" mechanism).
The proposal is designed to achieve its goals while adhering to two principles:
As much backwards compatible as possible - e.g., all work-flows/GUIs not related to tags should remain 100% identical; and any work-flows related to tags should remain very similar (with exception of new tag creation for lower-rep users).
As far as data population, this is intended to be a gradual incremental process - e.g. instead of one-time massive hierarchical classification, it is intended as an incremental addition/improvement of relationships.
Why bother? (aka use cases)
As a use case #1, take a fictitious user "Larry Ceiling" . He is a Perl expert, and would like to ensure that every Perl question has good, complete and accurate answer. To do so, he will need to either use "Interesting Tags" mechanism, or to employ a search by tags.
Currently, "Larry Ceiling" needs to use between 15 frequent Perl-related tags to ensure "good enough" coverage and at least 40 all perl-related tags to ensure full Perl coverage.
"perl*"isn't good enough (because many Perl-related tags do not contain the word "Perl" - for example autovivification or moose or dancer).
In addition, due to new tag additions, this list needs to be maintained by virtue of monitoring for new Perl-related tags.
What Larry wishes is a way to simply say that he's interested in perl tag, and achieve the correct result.
A mirror use case is user named "Jon Biathlon", who would like to never see another C#, or .net question for whatever reason. I estimate the amount of C#/.net tags far exceeds Perl ones, and "Jon" will need to add every single one of them to "Ignored Tags" list or exclude from searches.
More use cases are in the examples in Benefits section.
Add an ability for a many-to-many transitive relation "contained in" (aka "sub-tag") between some tags.
Just to be clear - the relationship will be sparse. NOT every tag will need to have a parent.
This relation would only be applied to completely contained tags. For example, stl tag would be contained in c++ tag; but oop and c++ tags would be completely independent. garbage-collection would be independent of java.
The relation could be many-to-many, for example visual-c++ tag would be contained in both c++, and, if such exists, visual-ms-languages tag. However, I don't envision too many cases where multiple ancestors would exist - see next bullet point for the reason.
Some composite tags should be split up. For example, all of perlre, boost-regex and apache-regex need to be changed to regex + specialized tag (Perl, boost or apache correspondingly). This is the reason I believe multiple ancestors would be rare.
Tag relationships can be specified by either a creation of a new tag, or via a special "relate tags" management screen.
When tagging a question; the GUI allows "searching" for tags by sub-string via auto-complete drop-down (as we do now), or by browsing a tag hierarchy, e.g. typing "C++>" will bring up a drop-down of all children of
NOTE To increase backwards compatibility, we may want to only enable the "include descendant tags in searches" if the special form of the tag is used in Include/Exclude/Search (e.g.
[perl*]. Personally I don't think this is needed but may address the concerns by people that automated inclusion/exclusion of more questions may be disconcerting).
To shorten the post, I removed a detailed proposal for permissions required for creation of tags AND tag hierarchies - see edit history if you're genuinely interested. </Fermat>
Implementation - technical
Add 2 new tables (I don't know exact table structure of SE, so details may be off):
Table 1: *tag_contained* (parent-child relationship) | parent_tag | tag | approved (could be >1 column) |
This table contains the relationship definitions. It is populated either when a new tag is created to be a child of an existing tag, or when a user with sufficiently high rep (say 250, or 3000 for "TOP" parent) proposes a new relationship, with "approved" value of false.
In addition it would contain newly proposed "top level" tags, with a special "TOP" parent tag value - but once approved, such records can probably be removed from the table.
When a user who can approve (10k? moderator?) approves it, "approved" is updated to true value.
This table is also used by the "tag population" GUI, by listing N-level drop-down menu instead of current 1-level auto-complete drop-down (better GUI approaches are welcome!)
Table 2: *tag_contained_expand* | ancestor_tag | tag |
This table contains ALL transitively computed relationships. E.g. if the first table contains A->B, B->C, C->D, this table will contain A->B, A-C, A->D, B->C, B->D, C->D. This table is populated automatically based on all approved rows in the first table, either by a regular job or by a trigger. Since I do not anticipate an unduly high amount of tag relationships, this should not be overly intensive process.
The second table is actually used by search engine, include and ignore mechanisms - namely, a query of a type "where tag in (X,Y,Z)" will be expanded to join with tag_contained_expand.
It will also be used when tagging a question, by eliminating un-needed ancestor tags. E.g. a question with perl5.10 tag would not need perl tag due to "contained in" relationship between them, and thus the latter will be either automatically removed or suggested to be removed (I suggest automated + notification).
- NOTE One additional technical detail - for Google's permission, the ancestor tags of question need to be included (hidden or visible?) on the question.
Main benefit is, of course, much simplified searching and filtering as illustrated in use cases.
Some of the 35000 tags (especially composite ones) could be removed, as their only reason for existence is to denote hierarchy. Examples. perlre, boost-regex, vc++-project-file.
Overly-specialized rarely used tags - visual-studio-2008-sp1 - which are the main candidates for being something's child - no longer pose a problem for either searches/filtering, OR for finding them when tagging a question.
Misspelled and differently-spelled tags pose less of a problem as well. Much easier to find assorted
dotnetrelated tags if they are all under a single
[.net]hierarchy. I realize that tag-synonyms may solve this particular problem, but it's still a problem meanwhile.
A problem of finding tags that are related to some topic but have nothing at all common as far as wording is removed. autovivification is a lot easier to find in a list of perl's children than on its own, especially for people who don't necessarily remember that word.
Potential issues and resolutions
Issue: Tags that apply to multiple domains creating logistical nightmare. For example, C++/memory-allocations, Java/memory-allocation, etc...
Solution: memory-allocations should be independent of either of those 2. The examples above should be 2 independent tags each.
Issue: It is very hard to organize 35000 tags into a tree, never mind do it correctly.
Solution: The 3 components of "hard" assessment are sheer volume, the fact that most tags are really independent of each other, and the difficulty in placing certain tags under just one parent.
First of all, a vast majority of the tags will not be in a single tree. There will be many top level tags and the forest would for most part be rather flat and sparse.
Second, this is meant to be a gradual incremental process. 10 tags related first day. 5 tags second day. All Perl tags within 2 weeks (I'll volunteer :). etc...
Third, while rare, multiple parents are allowed, thus eliminating "but it belongs to both!" headaches. Although, as noted, most cases when it appears that the tag belongs to multiple parents, either it should be independent in the first place, or it is a composite tag which should be split into two.
Issue: This proposal is too bloody long
Solution: I should get more sleep.
NOTE: this question is NOT a duplicate of this one, since it contains detailed proposal instead of a general idea. Some of the ideas are based on results of this discussion.