I made a major edit to The Fermi paradox then wrote an answer to it. In fact, I rewrote the body of that question entirely. That doesn't seem like such a bad thing to do because the edit didn't become visible until it was reviewed. I actually wrote my answer before the edit became visible and now the other answers were written before the edit to the question by people who probably never would have dreamed of writing that as an answer to the current form of the question. Is it better to write a new question that's related to an existing question but more on topic then mark the original question as a duplicate of it than to make a major edit to the original question? If so, how can I mark it as a duplicate?
If your edit makes existing answers invalid then your edit is too substantial. It can be ok to do substantial (as in entirely re-write) edits to a question, but only in cases where the question is going to be closed otherwise and only within the intentions of the original asker (a lot of site scope can be down to interpretation so editing to make some 'more on-topic' can be a bit of a grey area).
As the help center says, edits should be used (emphasis mine);
- To fix grammar and spelling mistakes
- To clarify the meaning of the post (without changing that meaning)
- To include additional information only found in comments, so all of the information relevant to the post is contained in one place
- To correct minor mistakes or add updates as the post ages
- To add related resources or hyperlinks
In short; don't change the meaning of the question beyond the intent of the asker and don't make substantial edits unless absolutely necessary.
Of course it can be a fine line between inconsequential and too substantial edits. Err on the side of caution and certainly if your edit affects existing answers then you've gone too far.
Is it better to write a new question that's related to an existing question but more on topic then mark the original question as a duplicate of it than to make a major edit to the original question?
You shouldn't intentionally be asking duplicates.
If the current question is OK in meaning and intent but needs editing then edit it (within the guidelines).
If you have enough to add to the question that would make it a new question then ask a new question, but link to the original question for context and explain what more you have to add and why that makes it not a duplicate.