9

Since a couple of weeks back, a chat bot I run is unable to access the site, specifically when run from GitHub Actions.

I sent a question about this using the "contact us" form a couple of weeks ago, but I haven't heard back. I'm posting this here for visibility, and possibly to collect more information from other users who may be affected by the same problem.

Sloshy the Thawman is a simple Python script which attempts to scrape some chat pages using requests (and eventually log in and communicate using ChatExchange; but it never even gets that far). Since a couple of weeks back, even simple curl-like actions fail.

They work fine from my laptop, and they work fine from CircleCI. The failures only happen when the code is run from GitHub Actions.

The traceback from the last link looks like this:

Traceback (most recent call last):
  File "/home/runner/work/sloshy/sloshy/sloshy.py", line 731, in <module>
    main()
  File "/home/runner/work/sloshy/sloshy/sloshy.py", line 727, in main
    me.perform_scan(args.startup_message)
  File "/home/runner/work/sloshy/sloshy/sloshy.py", line 573, in perform_scan
    self.scan_rooms(startup_message)
  File "/home/runner/work/sloshy/sloshy/sloshy.py", line 516, in scan_rooms
    self.startup_notice(homeroom, startup_message)
  File "/home/runner/work/sloshy/sloshy/sloshy.py", line 371, in startup_notice
    self.send_chat_message(
  File "/home/runner/work/sloshy/sloshy/sloshy.py", line 361, in send_chat_message
    room.send_message(message)
  File "/home/runner/work/sloshy/sloshy/sloshy.py", line 180, in send_message
    chat = self.clients.get_client_for_server(self.server)
  File "/home/runner/work/sloshy/sloshy/sloshy.py", line 104, in get_client_for_server
    self.rep_check = self.perform_rep_check()
  File "/home/runner/work/sloshy/sloshy/sloshy.py", line 118, in perform_rep_check
    self.rep_check = RepScrape(21818820)
  File "/home/runner/work/sloshy/sloshy/scrape_chat.py", line 244, in __init__
    soup = self._get_requests_soup(url)
  File "/home/runner/work/sloshy/sloshy/scrape_chat.py", line 42, in _get_requests_soup
    result.raise_for_status()
  File "/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://stackexchange.com/users/21818820/sloshy?tab=accounts

In some more detail, this part of the code attempts to ascertain that Sloshy has enough reputation to speak in chat before attempting to log in, by scraping the bot's profile page and extracting the reputation information. In so many words, it's doing the equivalent of

curl -s 'https://stackexchange.com/users/21818820/sloshy?tab=accounts'

except using a custom HTTP User-Agent: string. This generally worked fine in the past, for several years (though the code to scrape the reputation page is new).

I'm really at loss as to what might be the problem. My primary speculation at this point is that Cloudflare has decided to block GitHub Actions for some reason.

I disabled the GitHub Actions automation over a week ago, but enabled it briefly just now to check whether cooling down for a week would help; but it is still broken.

For additional background, I added the reputation check because I originally thought the recent failures were reputation-related. The failures started on Feb 16 (i.e., the Feb 15 nightly run was still successful).

11
  • At one point I assumed this was related to meta.stackexchange.com/questions/396674/… but the failures are not related to using the API at all.
    – tripleee
    Commented Mar 2 at 8:49
  • 1
    It still is the CloudFlare bot detection.
    – rene
    Commented Mar 2 at 8:51
  • @rene Thanks, so do we think the answer is simply "don't use Github Actions" or "don't attempt to automate"?
    – tripleee
    Commented Mar 2 at 8:52
  • 5
    I've status-reviewed the heck out of these reports and staff has them on their board but by this time I believe that only if you access SE from Google owned cloud services you'll be let in, all others are denied access until they pay up.
    – rene
    Commented Mar 2 at 8:54
  • @rene So should this be closed as a duplicate of ... something?
    – tripleee
    Commented Mar 2 at 8:56
  • I don't know what is best here.
    – rene
    Commented Mar 2 at 8:58
  • @rene For cohesiveness, could you please link to related information? Especially the information about paid access was completely new to me, even though I have scoured related questions recently.
    – tripleee
    Commented Mar 2 at 9:04
  • 3
    The paid access is inferred by me due to the partnership with Google. Similar bug reports: stackapps.com/questions/10513/… stackapps.com/questions/10497/… meta.stackexchange.com/questions/396801/…
    – rene
    Commented Mar 2 at 9:22
  • @rene can't see how the one you linked about traducir is similar? Related, at best, but 1) it's not about Github, and 2) already fixed. Commented Mar 2 at 12:47
  • 1
    @ShadowWizardLoveZelda it was triggered by the same CloudFlare bot detection. That they fixed their own "bots" for accessing endpoints but ignored all other reports for the same issue is telling. That it is fixed is the point that annoys the heck out of me. It matters to be linked here as it shows possible ill-intent here on the SE side.
    – rene
    Commented Mar 2 at 12:58
  • 1
    For the record, I have another bot on AWS which is also unaffected.
    – tripleee
    Commented Mar 2 at 18:13

0

Browse other questions tagged .