I'm building an application that detects plagiarized answers on Stack Overflow, so I need to retrieve the content of answers programatically.
I was considering just making normal HTTP page requests and scraping those rather than going through the API, but I was wondering if page requests are also rate-limited? For example, if I make 5 page requests a second for 30 seconds, would my IP address start to get significantly rate-limited? Is it possible that my IP address would even incur a permanent ban?
There's also a limit for page requests per IP address per time period which I think must have recently been tightened (saw it twice yesterday) but I don't know what the exact limit is.
but other than that I didn't find anything definitive and authoritative about this. Are the exact details intentionally kept a secret to make it harder for malicious entities to DDoS Stack Exchange?