The use of %2F (coding) in the URL hints that the remainder of the URL is also encoded, which isn't true; that causes mode switching and resets the parentheses count.
Test of link in MarkDown rendering. - Use your URL and 🔗 Button above.
Test of link using brackets and parentheses. - Duplicate your example.
Test of link using brackets, parentheses, and backslashes. - Use no URL encoding and 🔗.
Test of link using fully encoded URL (but keeping the protocol selection unencoded).
Using your example (exactly) but using a fully unencoded URL
https://docs.racket-lang.org/reference/Filesystem.html?q=rktl#(idx._(gentag._305._(lib._scribblings/reference/reference..scrbl))) (instead of a partially encoded URL, with %2F's in it) OR a fully encoded URL, excluding the protocol
https://docs.racket-lang.org%2Freference%2FFilesystem.html%3Fq%3Drktl%23%28idx._%28gentag._305._%28lib._scribblings%2Freference%2Freference..scrbl%29%29%29, instead of mixing the input, produces the correct decoding in the renderer.
"... many URI schemes and protocols based on RFCs 1738 and 2396 presume that the data characters will be converted to bytes according to some unspecified character encoding before being represented in a URI by unreserved characters or percent-encoded bytes. If the scheme does not allow the URI to provide a hint as to what encoding was used, or if the encoding conflicts with the use of ASCII to percent-encode reserved and unreserved characters, then the URI cannot be reliably interpreted. Some schemes fail to account for encoding at all, and instead just suggest that data characters map directly to URI characters, which leaves it up to implementations to decide whether and how to percent-encode data characters that are in neither the reserved nor unreserved sets.