The title for the FAQ tab of the questions list on SO is currently "FAQ questions". If you expand that, I believe it makes "frequently asked questions questions".
Is it supposed to be that way or is it just a small grammatical error?
Grammatically, it's not correct. When you expand FAQ outside it's abbreviation, it becomes "Frequently Asked Questions", and in the title element on the FAQ Questions page, in the source, we do see:
While it's a grammatical issue, we all understand what it means, and when it comes to SEO, there may be some benefit to this that we don't see.
Whether it's supposed to be that way, I don't know, but I wouldn't suggest putting this at the top of the list when it comes to bug fixes and new feature development. Most people probably will never notice this or care, as long as they get great answers to their questions.
That is called RAS Syndrome. It also happens when you say PIN number, or ATM machine.
Grammatically, there isn't anything wrong with FAQ questions; the fact you know what FAQ means doesn't make FAQ questions ungrammatical. If you wrote "Frequently Asked Questions" questions, you would be using the same word twice, but that would not make the phrase ungrammatical. (I can think of a perfectly grammatical sentence where the same word is repeated more than twice.)