I'm building an application that detects plagiarized answers on Stack Overflow, so I need to retrieve the content of answers programatically.

I know I can do this using the Stack Exchange API, but the API uses rate-limiting/throttling to prevent abuse.

I was considering just making normal HTTP page requests and scraping those rather than going through the API, but I was wondering if page requests are also rate-limited? For example, if I make 5 page requests a second for 30 seconds, would my IP address start to get significantly rate-limited? Is it possible that my IP address would even incur a permanent ban?

I found the following comment by Martin Smith mentioning page request rate-limiting on The Complete Rate-Limiting Guide,

There's also a limit for page requests per IP address per time period which I think must have recently been tightened (saw it twice yesterday) but I don't know what the exact limit is.

but other than that I didn't find anything definitive and authoritative about this. Are the exact details intentionally kept a secret to make it harder for malicious entities to DDoS Stack Exchange?

I'm building an application that detects plagiarized answers on Stack Overflow, so I need to retrieve the content of answers programatically.

I know I can do this using the Stack Exchange API, but the API uses rate-limiting/throttling to prevent abuse.

I was considering just making normal HTTP page requests and scraping those rather than going through the API, but I was wondering if page requests are also rate-limited? For example, if I make 5 page requests a second for 30 seconds, would my IP address start to get significantly rate-limited? Is it possible that my IP address would even incur a permanent ban?

I found the following comment by Martin Smith mentioning page request rate-limiting on The Complete Rate-Limiting Guide,

There's also a limit for page requests per IP address per time period which I think must have recently been tightened (saw it twice yesterday) but I don't know what the exact limit is.

but other than that I didn't find anything definitive and authoritative about this. Are the exact details intentionally kept a secret to make it harder for malicious entities to DDoS Stack Exchange?