Introduction of a partial inheritance relation into the set of tags on SO, in order to allow more efficient question organization/filtering/tagging (especially via "Interesting"/"Ignored" mechanism).
The proposal is designed to achieve its goals while adhering to two principles:
As much backwards compatible as possible - e.g., all work-flows/GUIs not related to tags should remain 100% identical; and any work-flows related to tags should remain very similar with exception of new tag creation for lower-rep users.
As far as data population, this is intended to be a gradual incremental process - e.g. instead of one-time massive hierarchical classification, it is intended as an incremental addition/improvement of relationships.
Why bother? (aka use cases)
As a use case, take a fictitious user "Larry Ceiling" . He is a Perl expert, and would like to ensure that every Perl question has good, complete and accurate answer. To do so, he will need to either use "Interesting Tags" mechanism, or to employ a search by tags.
Currently, "Larry Ceiling" needs to use between 15 frequent Perl-related tags to ensure "good enough" coverage and at least 40 all perl-related tags to ensure full Perl coverage (may be more, as some Perl-related tags might not contain the word "Perl"). In addition, due to new tag additions, this list needs to be maintained by virtue of monitoring for new Perl-related tags.
What Larry wishes is a way to simply say that he's interested in
[Perl] tag, and achieve the correct result.
A mirror use case is user named "Jon Biatlon", who would like to never see another C#, or .net question for whatever reason. I estimate the amount of C#/.net tags far exceeds Perl ones, and "Jon" will need to add every single one of them to "Ignored Tags" list or exclude from searches.
Additional use cases are implied by examples in Benefits section.
Add an ability for a many-to-many transitive relation "contained in" (aka "sub-tag") between some tags.
This relation would only be applied to completely contained tags. For example,
[STL]tag would be contained in
[C++]tags would be completely independent.
[garbage-collection]would be contained in
[memory-management]but both would be independent of
The relation could be many-to-many, for example visual-c++ tag would be contained in both
[C++], and, if such exists,
[visual-ms-languages]tag. However, I don't envision too many cases where multiple ancestors would exist - see next bullet point for the reason.
Some composite tags should be split up. For example, all of
[apache-regex]need to be changed to
[regex]+ specialized tag (Perl, boost or apache). This is the reason I believe multiple ancestors would be rare.
Tag relationships can be specified by either a creation of a new tag, or via a special "relate tags" management screen.
When tagging a question; the GUI allows "searching" for tags by sub-string via auto-complete drop-down (as we do now), or by browsing a tag hierarchy, e.g. typing "C++>" will bring up a drop-down of all children of
NOTE To increase backwards compatibility, we may want to only enable the "include descendant tags in searches" if the special form of the tag is used in Include/Exclude/Search (e.g.
[perl*]. Personally I don't think this is needed but may address the concerns by people that automated inclusion/exclusion of more questions may be disconcerning).
(The thresholds are kind of random, I didn't put too much thought into exact #s)
Users with <250 rep can not create new tags
Users with 251-3000 rep can create new tags contained in existing tags, subject to later approval; or to propose new "contained in" relationship via a very simple new GUI on TAGs screen. They can not create "top level" tags (e.g. tags without proposed parent), but a tag they create may be turned into "top level" tag by an approver/moderator if it ought to become one.
Please note that only relationships are provisional - the tag creation itself is, as currently, final.
Users with 3000-10k rep can create new "top level" tags (e.g. tags without a proposed parent). However, such a tag's lack of parent also needs approval. The GUI should be the same as <5k, but with "No parent" option.
Users with 10k+ can create any tags and any relationships, and approve the proposed relationships (or lack thereof). I'm also thinking that any multiple-parent relationships are the purview of 10k but not sure. UPDATE Based on discussions below, one possible solution I can see is requiring 3 independent approvals from users who are listed as N top answerers for the tag(s) in question.
There's a set of moderator tools for approving suggested tag relationships as well as verifying newly created top level tags separately from child tags; and a list of tags that have not been classified yet.
Implementation - technical
Add 2 new tables (I don't know exact table structure of SE, so details may be off):
Table 1: *tag_contained* (parent-child relationship) | parent_tag | tag | approved (could be >1 column) |
This table contains the relationship definitions. It is populated either when a new tag is created to be a child of an existing tag, or when a user with sufficiently high rep (say 250, or 3000 for "TOP" parent) proposes a new relationship, with "approved" value of false.
In addition it would contain newly proposed "top level" tags, with a special "TOP" parent tag value - but once approved, such records can probably be removed from the table.
When a user who can approve (10k? moderator?) approves it, "approved" is updated to true value.
This table is also used by the "tag population" GUI, by listing N-level drop-down menu instead of current 1-level auto-complete drop-down (better GUI approaches are welcome!)
Table 2: *tag_contained_expand* | ancestor_tag | tag |
This table contains ALL transitively computed relationships. E.g. if the first table contains A->B, B->C, C->D, this table will contain A->B, A-C, A->D, B->C, B->D, C->D. This table is populated automatically based on all approved rows in the first table, either by a regular job or by a trigger. Since I do not anticipate an unduly high amount of tag relationships, this should not be overly intensive process.
The second table is actually used by search engine, include and ignore mechanisms - namely, a query of a type "where tag in (X,Y,Z)" will be expanded to join with tag_contained_expand.
It will also be used when tagging a question, by eliminating un-needed ancestor tags. E.g. a question with
[perl5.10] tag would not need
[perl] tag due to "contained in" relationship between them, and thus the latter will be either automatically removed or suggested to be removed (I suggest automated + notification).
- NOTE One additional technical detail - for Google's permission, the ancestor tags of question need to be included (hidden or visible?) on the question.
Main benefit is, of course, much simplified searching and filtering as illustrated in use cases.
Some of the 35000 tags (especially composite ones) could be removed, as their only reason for existence is to denote hierarchy. E.g.
Overly-specialized rarely used tags -
[visual-studio-2008-sp1]? - which are the main candidates for being someone's child - no longer pose a problem for either searches/filtering, OR for finding them when tagging a question.
Misspelled and differently-spelled tags pose less of a problem as well. Much easier to find assorted .net and dotnet related tags if they are all under a single
[.net]hierarchy. I realize that tag-synonyms may solve this particular problem, but it's still a problem meanwhile.
A problem of finding tags that are related to some topic but have nothing at all common as far as wording is removed.
[autovivification]is a lot easier to find in a list of perl's children than on its own, especially for people who don't necessarily remember that word.
Potential issues and resolutions
Issue: Tags that apply to multiple domains creating logistical nightmare. For example, C++/memory-allocations, Java/memory-allocation, etc...
Solution: memory-allocations should be independent of either of those 2 (but contained in memory-management). The examples above should be 2 independent tags.
Issue: It is very hard to organize 35000 tags into a tree, never mind do it correctly.
Solution: The 3 components of "hard" are sheer volume, the fact that most tags are really independent of each other, and the difficulty in placing certain tags under just one parent.
First of all, a vast majority of the tags will not be in a single tree. There will be many top level tags and the forest would for most part be rather flat.
Second, this is meant to be a gradual incremental process. 10 tags related first day. 5 tags second day. All Perl tags within 2 weeks (I'll volunteer :). etc...
Third, while rare, multiple parents are allowed, thus eliminating "but it belongs to both!" headaches. Although, as noted, most cases when it appears that the tag belongs to multiple parents, either it should be independent in the first place, or it is a composite tag which should be split into two.
Issue: This proposal is too bloody long
Solution: I should get more sleep.