As Tinkeringbell discusses, I am seeing things meant to be hidden.
I solve problems like that for a living, so I can speak to how that happened and what to do about it. I discovered this in a general web search engine that is well-behaved, meaning it obeys the rules for directing both web spidering and content inclusion in search engines. Note these are voluntary rules and a blackhat spider may not obey.
How the spider found it
- it found a contemporary link to the page somewhere on StackExchange,
- it found a contemporary link on a third-party web site,
- it already knew about the page's existence from past crawling.
I broke those out into 3 cases so I can address common misconceptions.
The second case is one where a webmaster (i.e. StackExchange) doesn't create any internal links, but an outsider does. Suppose someone's user page links to their Web page at sfu.edu. Unbeknownst to them, SFU publishes aggregate data from their web logs, including Referer. The search engine considers these to be a rich data source, for URL discovery (although not for link relevance aka PageRank). SFU uses rel=nofollow, but that is not a "do not follow the link" directive, that is a "disregard link relevance" directive.
The third case is the likeliest one here. Once a search engine has a URL, it doesn't drop the URL when all links to the URL disappear. That does have an effect on link relevance, but that only affects ranking. If the searcher is specific enough to where only dozens of results exist, even a low-ranking page will surface.
How to hide it from well-behaved engines
... Assuming StackExchange wishes to. There are arguments both ways.
The best option is the "noindex" HTML meta-tag (or X-Robots equivalent in HTTP), as such:
<meta name="robots" content="noindex">
This tag says "do not include this page in search results". The tag can also have elements such as nofollow (which actually does mean "the crawler should disregard links found on this page", unlike rel=).
Note that the search engine has to fetch the page in order to see this tag; that's nothing to worry about. I suppose you could serve it in the X-Robots HTTP field and serve blank page content. However at that point you could just as easily return 403 Forbidden, 404 Not Found, or 410 Gone. Note that a 410 Gone means the URL is planned to be gone forever; this matters to some search engines.
Blocking the engine from crawling a page, e.g. excluding the page in robots.txt, will not work. Robots.txt only directs crawling, not indexing. A search engine may still create a search result for the page, inferring the page content from links and archived data.
