Stephen

@zeborah@mastodon.nz @BobLefridge@mastodon.nz one must also assume no monitoring for unusual behaviour. Because systematically exfiltrating docs by guessing document IDs should show up.

@BobLefridge@mastodon.nz @zeborah@mastodon.nz do we know that? If I were doing this, and I were being careful, I'd probably use one of those dodgy proxy services that onsells compromised residential networks, maybe see if I could just use NZ domiciled ones, but at least not use a single IP but a large rotating set. But. It would still stand out in the noise, I would think, to snag that many docs in any limited period.

@aaron @asrg at least a couple of hits on the Spamhaus XBL list (has been exploited) or the PBL list (by policy should not be sending email, ie identified by ISP as residential). Am prepared to bet they're all compromised home PCs etc.

@aaron @asrg I just looked up a handful of IPs from the most recent likely bot run (all had same user-agent designed to loook like a Mac, but different IP numbers, never fetched CSS or other assets) on Spamhaus blocklists and about 70% had at least one listing. So that's a start.

@asrg @aaron @marcusb @mike @Fingel I have been doing something primitive with fail2ban and a "trigger" URL. But. What I see is that the latest in scraping is to use a rotating set of IPs or proxies so requests never seem to come from the same IP number, and with plausible user agents. I'm struggling with this because although I can see the overall behaviour, it's not clear until after a request is made that's part of a scrape session, and blocking that IP number won't block the remaining scrapes. Firms are offering this kind of service commercially and there are plenty of writeups on how to do it.

@SwiftOnSecurity@infosec.exchange my late dad had a little notebook and I tell you what when I got power of attorney because of his dementia it was a bloody good thing.

@adingbatponder@fosstodon.org @markmcb@mas.to I don't know sorry - I use Caddy as a reverse proxy and do it all with the proxy settings.

@markmcb@mas.to If you have RSS or Atom feeds worth making an exception for them too...most newsreaders seem to be on 1.x too.

@markmcb@mas.to I do something very similar to what you do, a bit more complicated. Anyone requesting .php or a number of other paths gets a 403 and firewalled via fail2ban, no matter what the HTTP version (I don't have WP or any PHP on my site, no one should be asking for /admin, etc etc etc). And then for HTTP 1.x I allow requests for atom.xml and rss.xml; allow a list of bots I approve of; serve 418 to the rest, with a little teapot emoji as the body.