Part 1 - "Black hat" questions are often exploiting very specific systems or security functions. If a user doesn't bother trying to cover up their intentions, the question usually ends up invalid because it's too narrow of a case to be useful or solvable by an Internet-based audience anyway. If they try too hard to cover up, the question becomes too broad or undefined and again is close-worthy without needing to be labeled as "black hat". Then there's the ones who hit it just right. For those, the question is a clearly defined good fit for the StackExchange format and you probably can't even tell - or at the very least can't definitively prove - that it's even a "black hat" question. So, in many ways, the perceived problem is mostly self-fixing.
Part 2 - The perceived problem isn't even one StackExchange is designed to care about. The criteria by which one could judge a question as "black hat" or "white hat" are entirely irrelevant to determining a solution for it. Any other such immaterial details are considered noise in the question or comment threads on StackExchange and are generally unwelcome for precisely that reason - this should be no different.
Part 3 - From our position, definitively and accurately assessing someone's "hat color" is impossible. We do not have the authority to require a user to provide us the required information or documentation to prove authorization or intent, nor are we able or empowered to reliably authenticate such information if it is provided.
I'm pretty sure a global SE policy doesn't exist particularly for or against malicious hacking. When questions are abstracted enough to fit general SE guidelines in the first place (seeking to have useful Q&A which is valuable to a wide audience), the specific nature of the asker's intent is quite easily obscured even if they don't mean for it to be. Add in intentional obfuscation by the asker to avoid the stigma of being called out as a "black hat", and you quickly end up with a question that demonstrates a very useful problem of interest without being very clear as to what its final goal is.
So, a majority of "black hat" questions will often naturally fall into one of a few categories which are typically not welcome on StackExchange regardless of any malicious intent on part of the asker:
Too narrow: This used to actually be a standard close reason. These questions are too situation-specific to be very useful to a wide audience, and in some cases even so obscure as to be unanswerable by anyone not actually directly in front of the problem.
Too broad: The asker has generalized the question so far that to answer it would require coverage of way too much subject matter to fit the StackExchange format.
Unclear: Again, this can easily happen when a question is abstracted from its actual intent. Here, it's done to the point where nobody really has a solid idea of what a useful answer would look like to the asker.
The issue of accepting "black hat" questions has been hashed over and re-addressed a few times on the IT Security site, (I'll try to toss in some Meta links later) where it is very much expected and in many ways accepted that we will be discussing "black hat" topics from time to time. The majority consensus there, over and over again, is that there are many useful "black hat" questions - perhaps even more so for the IT Security site than others, but that's more due to the specific nature of the site than it is the nature of the questions.
However, StackExchange question quality policies still very much apply. Aside from the aforementioned intent to generate questions that are valuable to a wide audience, these policies also are meant to result in questions that are of interest to people who would have specific expertise in the field targeted by each site. Leveraging that intent, the IT Security site has adopted the following custom off-topic closure option:
Questions asking us to break the security of a specific system for you are off-topic unless they demonstrate an understanding of the concepts involved and clearly identify a specific problem.
While I'm not saying this should be globally applied to all StackExchange communities, it does represent the culmination of a lot of debate and consensus among the one community which by its very nature is expected to include the foremost experts on "black hat"/"white hat" issues within StackExchange. So, it seems that it should stand to reason that the logic applied there should carry some amount of weight.
So, let's measure the supposed "rainbow table" question up against this and see where we land. Do pay close attention though, because absolutely none of this actually has to do with whether or not the question is "black hat" or even whether or not it's related to rainbow tables.
- Is the user asking us to break the security of a specific system for them?
The only details we know for certain are that he has a somewhat large list of MD5 hashes, and another extremely large list of MD5 hashes, and he wants to do a comparison of the two. Further, he's not even asking how to do it. He's just asking if there's any particular language which is better-suited to the task than others. This is a very far cry from asking to be handed "Rainbow Tables for Dummies" by any objective measure.
- Do they demonstrate an understanding of the concepts involved?
Reasonably Enough, IMHO.
If we accept that the problem being presented is simply one of language choice for a specified task, without needing any regard for what that task is or what it is useful for, I think realizing that such a question should be asked in the first place is enough to demonstrate understanding of the fact that certain languages have natural benefits in terms of features and performance over others. He might not have gone into a whole lot of depth on the topic, perhaps leaving him to appear a bit of a newbie, but newbie questions aren't unwelcome just because they're newbie questions either.
- Does the question identify a specific problem?
Yes and No.
And here is where it fails just enough to be close-worthy. Yes, he tells us he's got a big file of hashes and a really big file of hashes and he wants to compare them. Yes, he's identified that the issue he wants addressed in answers is one of language choice with the primary goal being processing speed. He's even gone so far as to specify a language that he thinks might be a good candidate, and provide some justification (albeit perhaps a bit weak) as to why.
What has not been done here, however, is to give a sufficiently defined objective measure by which an answer might be judged acceptable for his case or generally superior. Sure, we have "fastest" to go by. But fastest on what platform? How much more speed above another choice is important for your use case? Are you asking because you're writing it yourself, or because you're trying to choose among programs written by others? If writing it yourself, what languages are you comfortable with? How well-versed are you in writing performance-optimized code in those languages?
There are a large number of variables which could affect software performance in general, and may affect the performance of different software languages attempting the same task. Beyond that, there's a number of other variables which are needed to determine what is right for you and your environment even when the use case is known. None of this is very well-defined in the question, so it's hard to objectively weigh what value an answer might have for the asker unless the answer is written so broadly as to be a marketable book!
Further, in order to disqualify "malicious" or "black hat" questions from this site, we need to first establish what qualifies a question as such.
For the sake of argument, let's accept the claim that "There's no practical purpose for matching large quantities of MD5 hashes except for hacking". It's a reasonable enough assumption to make for this discussion but it still leaves the issue moot. As @Gilles touched on in his answer, not all hacking is malicious. System Administrators routinely, some by company policy or legal requirement, have to hack their own users' passwords as part of security assessments. And there's an entire profession dedicated to authorized hacking of systems to identify and report weaknesses.
The most widely-accepted measure by which anyone can really delineate "black hat" from "white hat" is the intent and outcome of their actions. Both hats need to do password cracking, and in nearly every other way need to do the all of exact same things to accomplish their goals. There's only one other criteria that's ever used by any reputable member of the security community to distinguish the two, and that's authorization.
- Was their activity authorized?
Unauthorized activity is often enough alone for most people to say that something is "black hat". However, some may argue that unauthorized activity resulting in a better security posture for the target system falls into a gray area. In any case, the second criteria is by far the most widely accepted (although in certain respects still debatable) demonstration of someone's nature and intent.
- What did they do with the access/knowledge gained?
Even authorized hacking can be turned against the authorizing party, and unauthorized hacking can still be leveraged for good.
Disclosing a vulnerability to affected parties will usually push you towards the "white hat" line, although it is very much a subject of wide debate as to how that disclosure should properly be handled. Using a vulnerability to exploit systems in unauthorized attacks for your own gain, or selling the fruits of your labor to third parties who intend to use it for purposes other than protecting their own systems will most squarely land you under a "black hat".
However, neither of these are criteria which are at all relevant to solving a given problem. To judge a question based on either of these criteria would require inclusion of details, or additional requests for such detail, that have absolutely no value in determining what sort of solution is required to fulfill the need represented by any question. In every other case, we would call such details or discussion noise. Therefore it it is against the intent and nature of StackExchange communities for us to attempt to establish policies regarding matters which require such information in order to be judged.
Lastly, we are ultimately not in any position to accurately judge the color of someone's hat in the first place. As established above, there are only two criteria which are useful to distinguish "white hats" from "black hats":
As we have no association beyond StackExchange with, nor any outside authority over, the person asking the question we cannot mandate that the asker provide us the information required to properly judge their color of hat. Many times, disclosure of such information is forbidden by legal agreement with a client or discouraged (even for those who might ultimately be judged as "white hats") by its potential to affect future legal proceedings. Even given such information, we have no reliable means or authority by which to verify its authenticity.
With rare exception (if any), that is just the nature of Internet-based public communities. To change that would put extraordinary and undue burden upon the organization and individuals who create, manage, and maintain StackExchange.