For computer science questions, including algorithm questions, I recommend asking on Computer Science Stack Exchange. (I am a moderator there.) I don't know anything about computational geometry, but your question looks reasonable, and there are competent people who visit CS.SE. While my understanding is that all algorithm questions are considered on-topic on Math.SE, they tend to be outliers: you have the wrong target audience.
I believe that Math.SE's policy allows migrating computer science questions when they are unanswered and the asker agrees, so you should flag your question for migration to CS.SE. Please do not repost the question, as having the question in multiple places makes answers harder to find. Since you have only 1 reputation on Math.SE, you won't be able to flag; if you agree to the migration, someone else (e.g. me) can flag it for you.
Algorithm questions are allowed by Stack Overflow's FAQ, but they tend to get a poor reception: if you haven't committed to a programming language, SO is probably the wrong place.