1

When I open ELL.SE in a browser, I get a list of 48 questions. I want to manipulate these questions in Python with a Beautiful Soup script.

But when I download the page with the requests library in Python, the resulting page only has 14 questions, not 48. I'd like to know how to submit the request so that I get the full page back with 48 questions.

Here's my code:

url = 'https://ell.stackexchange.com'
request = requests.get(url)
html_text = request.text
print(html_text.count('s-post-summary    js-post-summary'))
soup = BeautifulSoup(html_text, 'lxml')
print(len(soup.find_all('div', attrs={'data-post-type-id':'1'})))

Result:

14
14
Improve this question
9
  • 9
    Don't scrape. Use the API instead, that's what it's meant for.
    – Shadow Wizard Is Sad And Angry
    Oct 20, 2022 at 15:20
  • 2
    @ShadowTheKidWizard I looked into it. It was too complicated for me to understand. I don't know how to handle JSON or do any of the other things required, and I don't want to learn all that. This is just a useful thing for me.
    – gotube
    Oct 20, 2022 at 15:26
  • 2
    From the tag excerpt of screen-scraping (emphasis mine): "This tag should be used with posts concerning sites using Stack Exchange content without proper attribution". Apparently you aren't looking to do that. Stack Overflow has several tags that might have already "the answer" that you are looking for like web-scraping. Might not be exactly about ell.se but certainly there should be a lot that cover different aspects around web-scraping.
    – Rubén
    Oct 20, 2022 at 15:27
  • 1
    api.stackexchange.com/docs/questions this page allows you to construct the kind of query you're looking for, no auth required. Just change the number of questions you want, and add other parameters, and it will create a query for you.
    – Esther
    Oct 20, 2022 at 15:32
  • You might be interested in the latest blog post -> stackoverflow.blog/2022/10/20/… The "key partner" has an open source project to make available Stack Overflow offline.
    – Rubén
    Oct 20, 2022 at 18:57
  • 4
    Scraping the page is going to be notably more complex than getting the data from the SE API. Look at it as an opportunity to learn.
    – Makyen
    Oct 20, 2022 at 18:58
  • 1
    @Makyen Gotta say it gets frustrating every time I ask a tech question on the Exchange, almost everyone tells me not to do it that way, as if they know better. It happens I just learned how to use BeautifulSoup yesterday and I want to practice. So, no, learning the API is the wrong thing for me to do. I don't want it easy. I don't want a new learning experience. I want the learning experience that I've chosen for myself, which is scraping the page. You can help me like Glorfindel did, or just don't help.
    – gotube
    Oct 20, 2022 at 20:34
  • 4
    @gotube SE has explicitly stated that they don't want people creating scripts to scrape SE pages when the task could be performed through the SE API (that's why the SE API exists). Us giving you the feedback to use the SE API is quite reasonable, as it is the "right" way to do the task. That you have some reason you are wanting to use BeautifulSoup, that you did not disclose in the question, is something we can't have any knowledge about, unless you tell us. Without you telling us that, we see someone doing something everyone writing scripts for SE has been asked not to do.
    – Makyen
    Oct 20, 2022 at 20:55
  • 2
    @gotube but many times that's exactly the case: people with more experience in something do know better. That's how things work. They don't do it out of arrogance, but out of actual desire to help other people, and to guide them how to become better. Same in this specific case: as Makyen said correctly, scraping is the wrong way. It works, yes. It's not forbidden in small scale, yes. But it's not the correct way to collect questions and answers data.
    – Shadow Wizard Is Sad And Angry
    Oct 21, 2022 at 12:33

1 Answer 1

Reset to default
9

If you open the front page in an incognito window, you see that it only lists the top 14 questions. For users who are logged in, it's 48. I didn't know there's a difference either, or why it's not 15 vs. 50 (which are regular values for page sizes here). Anyway, you could try to scrape the 'all active questions page' and set the page size to 50. That's this URL: https://ell.stackexchange.com/questions?tab=active&pagesize=50.

Also, you'll quickly get rate-limited if you scrape pages like that. The API isn't that difficult, and we have several Python libraries which might be helpful.

Improve this answer
3
  • 1
    That's perfect, thanks. I don't think I'll go over my quota as I'm only using it a few times a day to filter the questions I see. I just learned how to use BeautifulSoup yesterday and wanted to apply it to something meaningful.
    – gotube
    Oct 20, 2022 at 16:17
  • 4
    It is 14 questions because that is 6 + 8 .... always check your memes.
    – rene
    Oct 20, 2022 at 17:02
  • 4
    @rene, ah, and 48 because that's 6 * 8 ...
    – Glorfindel Mod
    Oct 20, 2022 at 17:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .