How should this hamper readability? Readability is determined by some factors:
What readers are used to – even despite half a century of ASCII and the Internet, curly quotes are used in almost every book, scientific paper, and most other printed media (and at least some parts of the Internet). So this is what people are used to.
Accessible information – Curly quotes allow you to distinguish between opening and closing quotes directly. Of course, there are other means to do so, but readability often relies on redundancy: Removing all capital letters, punctuation, and even vowels still makes most texts decypherable – but not readable. Arguably, on smaller zoom levels, this advantage gets lost, but still, no harm arises from it.
This in particular applies to many languages other than English which have a more prominent distinction between opening and closing quotes.
Typographical Harmony – This applies only to the titles on some sites. Consider this title which I just happened to see on English Language & Usage:

For some reason, the first pair of quotes is curly, while the second one isn’t (and probably because the first one is, SE’s auto-curling did not activate). Since this is an italic font, dominated by round forms, straight quotes are dissonant (and cannot be made more harmonious by the type designer since they may not bent in either direction). See this post of mine on Graphic Design for more examples and more elaboration.
There is also the factor of portability to consider, but since Stack Exchange thankfully supports Unicode rather well, I see no reason not to make use of it (when appropriate).
By the way, at least on some Stack Exchanges, quotes in titles are automatically rendered as curly quotes.
So, if you are willing to spend the time on inserting curly quotes or it’s not really a time issue for you, since they are readily available on your keyboard (like they are on mine), I do not see any reason from keeping you from doing so (given that the edit is substantial otherwise).