and a handful of other ASNs on almost all of my properties. It would be one thing if it was simple scraping, but it's often genuine agentic abuse (just spamming for vulnerabilities* or scraping millions of pages with very little valuable or dynamic content). These clouds must get the abuse happening under their own roof under control.
I banned Alibaba's Cloud two years ago and it still hits one of my domains 1 million times per day, despite every route 4xx-ing during that time.
Even Apple has aggressively ramped up its scraping in the past few months, but thankfully they do respect robots.txt.
It's frustrating because some of my real users route through these clouds via their work VPNs. Also, there are a lot of less-than-reputable AI scrapers using global bot-nets via residential IPs to scrape, making it even harder to detect and ban them without affecting real people.
* like leaked .env routes, brute forcing admin panels, and so on
dspillett 1 minutes ago [-]
> like leaked .env routes, brute forcing admin panels, and so on
That sort of thing was very common long before agent based abuse scanners. Volume would wax and wane but never went away.
Though other scans for vulnerabilities have shot up in frequency now everyone script kiddie and their flee-bitten mutt can ask an AI to do it for them.
pyrale 2 hours ago [-]
> I banned Alibaba's Cloud two years ago and it still hits one of my domains 1 million times per day, despite every route 4xx-ing during that time.
You may as well silently drop the packets if they don't adapt.
herbst 57 minutes ago [-]
Having the same issue with Bing bot. No idea why it's so eager to not behave.
Sharlin 4 hours ago [-]
Application-level banning seems very inefficient, these domains should be banned at the network (IP) level. Just drop any incoming packets from these domains/IPs, send absolutely nothing back that even confirms that you exist.
switz 4 hours ago [-]
They are banned at the edge in Cloudflare (now), they never hit my application.
elp 4 hours ago [-]
From a domain registry operators point of view Cloudflare are the worst of the bunch by orders of magnitude.
Google DNS are busy but its a hand full of IPs and reasonable. Cloudflare is non-stop never ending traffic. I don't know WTF they are trying to get. I've also seen TCP queries from those same Cloudflare IPs so this is not the result of a DDOS attack on them.
esseph 2 hours ago [-]
> Cloudflare is non-stop never ending traffic. I don't know WTF they are trying to get. I've also seen TCP queries from those same Cloudflare IPs
1. Breach router or desktop that hasn't been patched in ages via mass script
2. Establish C&C using cloudflare tunnels
3. Sell these hacked systems as residential proxy endpoints to everybody from state actors to teenagers
4. Some of them will be data scraping services
You are chasing your tail trying to ban by ASN or IP. It may work for another couple of months, at best.
peter_retief 29 minutes ago [-]
Cloudflare do a great job, I noticed moving from another VM.
cube00 4 hours ago [-]
> These clouds must get the abuse happening under their own roof under control.
Similar to ad click fraud and scam ads, the cloud provider gets paid, why would they turn down the money?
eru 3 hours ago [-]
So people accessing the internet through their cloud don't get automatically ip-banned by most services, perhaps?
cube00 2 hours ago [-]
Most big services will just take the junk traffic. They're never going to block whole cloud providers.
Blocking cloud providers is strictly for us dirty self hosters who are sick of our logs becoming unreadable and our visitor metrics useless from all this malicious traffic.
zetanor 3 hours ago [-]
If advertising and tracking was less egregious, I wouldn't feel the need to use an ad blocker. But, here we are.
eru 3 hours ago [-]
Sorry, what's the connection with the topic at hand?
unrented7977 3 hours ago [-]
[dead]
dawnerd 4 hours ago [-]
I’ve had to block similar on clients sites. Their search pages would start to get hammered with obvious agentic search spam. Searching for thing not even related to the site in all combinations of the queries. Some of the user queries contained potentially PII.
dukeyukey 2 hours ago [-]
Meta went absolutely haywire on one of my sites a couple months ago.
fosco 3 hours ago [-]
Is there a handy list of all of this that users contribute to?
sdcfgy 3 hours ago [-]
AWS is terrible. We had another tenant on AWS attacking us on AWS. AWS support took nearly a week of slowly bouncing around a case to do anything.
EGreg 3 hours ago [-]
This is a very smart thing to do! Thanks.
Why allow random requests coming from the clouds?
However, they could just use residential proxy botnets.
I believe the future is static resources with dynamic apps being only offered to people who can authenticate. And even then I’d recommend grandfathered accounts (eg telegram users that had an account for at least a year).
esseph 2 hours ago [-]
> However, they could just use residential proxy botnets.
Agreed
> I believe the future is static resources with dynamic apps being only offered to people who can authenticate.
I don't know about static resources, but I do think that identity will come into play.
EGreg 3 hours ago [-]
This is a very smart thing to do! Thanks.
Why allow random requests coming from the clouds?
However, they could just use residential proxy botnets.
simmerup 4 hours ago [-]
Isn’t scanning for vulnerabilities illegal? Traditionally security researchers had to be very careful about such things
bilekas 4 hours ago [-]
If there's a know CVE in a version of cPanel ir wordpress for example, those platforms broadcast what version they're on, so the matching cve being known is not illegal. You're just correlating known vulnerable versions without doing any probing. Pen testing however, a bit more involved in pulling info out without permission is illegal.
nostrademons 3 hours ago [-]
Can you enforce it? Would you even know who to go after?
Traditionally cybercrimes have been prosecuted by the victim going to their local PD (or the FBI) with the IP address of the offender, and then law enforcement subpoenas the ISP associated with that IP address to figure out the customer associated with that account, and then they have a case for prosecution. It all works as long as cybercriminals are less numerous than law enforcement, fall in the same jurisdiction (or in jurisdictions that have reciprocal extradition treaties), and have a real identity that can be tied to their online activities.
Now, many law enforcement agencies are tied up prosecuting political opponents of their bosses. To the extent that they have bandwidth to investigate and prosecute ordinary crimes, they're going to prioritize the ones that target a.) the government that backs them (of which there are many - you see another headline about the government being hacked every week or so on HN) b.) major corporations that have sophisticated IT systems that can aid prosecution or c.) headlines that will get their boss a PR boost and themselves a promotion. Your personal website does not register among those priorities. You probably can't even prove damages from the intrusion itself. The perpetrator might be a Russian bot that the U.S. doesn't have jurisdiction over, or a Canadian teen that the U.S. previously might've been able to prosecute with reciprocal judicial treaties but now has no leverage over because we've blown up our relationship with Canada. It may be legally a crime, but economically it's not worth prosecution.
Traditionally security researchers had to be very careful because they were doing everything above-board, with their real identity attached to their actions, and they made a very attractive (and PR-worthy) target. Somebody who's port-scanning from Russia through a VPN or residential proxy network is a.) anonymous b.) untraceable c.) out of U.S. jurisdiction and d.) not a headline-worthy target anyway.
A couple years ago I predicted here that a lot of things will break as the U.S. transitions from a high-trust to a low-trust society. This is one of them. Much of our justice system works well when criminals are rare and utterly falls apart when they're ubiquitous.
firefoxd 3 hours ago [-]
Last year, I wrote a post about how I use zipbombs to deal with bot traffic [0]. It went mega viral over the web, and I regret it. My server was completely overwhelmed with bot traffic. After months of dealing with it, I had to find a better strategy then keep my mouth shut.
So it ended up not having a noticeable impact on the bots?
firefoxd 3 hours ago [-]
It worked fine... on a small scale But it's one thing to swat flies versus fighting swarm of yellow jackets. Serving a 1MB file a million times a day on a small digitalocean droplet is not feasible. Let's just say it didn't scale. And AI bots usually come with beefier machines, and recover faster after a bomb.
kazinator 3 hours ago [-]
Last year, I read your post and implemented it. I had no way to tell whether there is any benefit to it, so I discontinued it. Serving the data, even though it is compressed and the compression is precomputed, was still a waste of bandwidth. It could hit the browsers of legit user (false positive).
The main problem on my site is that bots hit the CGIT repo, where visiting files triggers syntax highlighting scripts.
I put in a skill-testing-question type system where you get a maximally long-lived access cookie if you answer right (so legit users are bothered as infrequently as possible).
chicken-stew 36 minutes ago [-]
Can that technique be applied to packet compression?
giwook 3 hours ago [-]
What is that improved strategy? :pray:
seam_carver 5 minutes ago [-]
Oh is that why I keep getting those ai emails about the project I posted earlier?
est 33 minutes ago [-]
> my website is built as a single-page-application
There you go. Ditch it. Make you pages cachable as much as you can.
8by3 3 hours ago [-]
I rolled my own version of Anubis for reasons*, and > 80% of all traffic is bots that fail the challenge, even when white listing legit crawlers.
* I wanted an elixir version so I could run it alongside my existing apps instead of having it sit as a proxy in front of them. This way my existing apps can validate if the caller is already whitelisted and if not redirect to the challenger https://sphinx.8by3.com which sends you back to a site if you pass eg. https://tuhat.net
ben_w 5 hours ago [-]
It's too late to comment on your older posts, but I'd just like to say: it's nice to see someone making oldschool web browsers work properly on the modern web.
I got $10K GCP bill one month because of ClaudeBot. Its likely some users were trying to replicate our software by using Claude code + chrome plugin. If I was spending $10K on traffic, I can only imagine the money cloners must be spending on tokens.
spindump8930 27 minutes ago [-]
Well, potentially very little! It could be subscription usage, or just really inefficient code authored by an agent. It probably costs LESS in tokens to write a poor quality cloner that breaks upstream things.
naet 3 hours ago [-]
I posted a link to an online game I made in the comments here once, and immediately got all kinds of good and bad traffic.
Some people liked my work, found me on LinkedIn and sent a connection with a positive message. Other people tried every which way to SQL inject or attack the site, posted slurs in every available input, etc.
Someone copied the site and hosted their own version with a couple minor changes. I only found that one because they had left one of the links in, which gave me referring traffic from their copy.
All that from one HN comment, very much pre-AI too, not to mention the standard daily slew of requests probing for what I assume are known vulnerabilities in Wordpress or PHP.
The internet is a floodgate of different actors, many of which are automated, and a link posted somewhere like HN opens the valve especially wide.
beybol 3 hours ago [-]
In server logs you can distinguish the type of LLM's traffic. OAI-SearchBot/GPTBot/ClaudeBot is just indexing but ChatGPT-User/Claude-User etc. may be valuable traffic generated by user's conversation with LLM's. So it's worth choosing which ones you block, instead of blocking all of them.
jeroenhd 20 minutes ago [-]
> valuable traffic
What's the value for the website maker in these? There's user value to it (rather than just value for Anthropic/OpenAI) but the whole point of these bots is to make sure nobody ever needs to visit your website.
spindump8930 25 minutes ago [-]
Strong agree, a good way forward is letting user agent once again truly mean an agent on behalf of the user. And speaking anecdotally, I've seen different traffic "shapes" under the same UA, which leads me to suspect that some actors might be disguising traffic as more mainstream labs.
andai 3 hours ago [-]
I'm trying to understand the reasoning behind the entity that was sending presumably hundreds or thousands of requests per minute to the same page. It's some kind of AI news site, so it makes sense they want the freshest content, but it sounds like they were doing many requests per second, so that can't be the reason?
MisterTea 3 hours ago [-]
> I'm trying to understand the reasoning behind the entity that was sending presumably hundreds or thousands of requests per minute to the same page.
The most likely answer is they don't give a crap because it's not their money and no one is going to do anything about it.
andai 2 hours ago [-]
Nice. How do I get a massive swarm of nodes on someone else's money?
robotmay 1 hours ago [-]
I've been posting about this recently but I continue to be bad at either timing my submissions or titling them. I struggle with YouTube thumbnails too so I think it might be me. But anyway:
I experienced the same thing after submitting articles from my small photography website. I did debate using some of the off-the-shelf solutions like Cloudflare, Anubis, or iocaine, but I ended up building my own L7 proxy that utterly fucks the bots up: https://robertmay.photography/journal/bwbach-my-guardian-gob.... I will make it generally available at some point once I get it optimised how I'd like, but currently I'm able to trap about 3000 concurrent crawlers in my tarpit of slow requests on two small Hetzner VMs. And that 3k is really just the limits of the networks hitting my servers right now, I think it'll go significantly higher than that.
I haven't really seen anyone else combat them this way, possibly because many of the standard proxies and servers are set up for fast requests, not serving them as slowly as possible. Deliberately serving page loads for as long as the requester will accept packets turns out to be a really funny way to stop the botnets - when I just rejected the requests they in turn hammered my server even harder. By tanking the request volume with a ridiculous BEAM-based proxy you can reduce their request speed to a crawl and I'm fairly certain a couple of them have blacklisted my server now.
bayganyo 1 hours ago [-]
I like the spirit of this approach; make them suffer! But I'm surprised the bots wouldn't have reasonable timeouts to counteract. How long do they typically wait on these requests?
EDIT: sorry, that was a dumb question before reading your post. You actually learn the timeout they tolerate. Nice!
robotmay 1 hours ago [-]
I should probably put together some actual stats at some point, but from what I see in the live logs you'd be shocked - some of them will happily sit there for over 5 minutes being drip-fed packets!
Looking at you, Meta.
I do still get some requests going up to 15 minutes which is quite ridiculous, though those don't seem to be super common (maybe they're learning).
I think the shortest timeouts I've seen are between 2-5 seconds, but they're surprisingly infrequent and actually that's still a massive inconvenience for them. I did at one point just drop any request I couldn't keep open for more than 5 seconds but it actually caused more server load. I'm essentially inverting the server load by making slow requests more efficient, which is counter to everything I ever learnt about performance for web serving :D
I think the average I see is probably somewhere between 15-30 seconds. I'm guessing many crawlers don't want to go too low because they'll be trying to fetch large files like images, or perhaps they know they'll be slowing servers down and want to ensure they get the response. Most likely, the majority of crawler authors aren't expecting some arsehole to exploit this and they've just left the setting as the default.
pmlnr 41 minutes ago [-]
> 3000 concurrent crawlers
I gave up on tarpits when it got to 130k "waiting" connections.
bix6 5 hours ago [-]
> one specific user. I won't name names
Who is?!
sammy0910 4 hours ago [-]
haha im not naming names
rvz 3 hours ago [-]
Does the country end with an "a" with 5 - 6 characters long?
nekusar 4 hours ago [-]
You shouldnt defend bad actors and abusers. Hiding them is tantamount to them attacking others indiscriminately. Cause youre not the first victim, and definitely not the last.
You see something, you say something, or you do something.
dsfnsldfnsdfnsd 3 hours ago [-]
How else will the author play the victim for some clicks then?
mediumsmart 2 hours ago [-]
if you want the site to be fast, don't render it with javascript and take a look at the google lighthouse check for things to improve:
> I won't name names, but I looked them up, and they describe themselves as a service that listens for changes to a website, and then summarizes them into an AI slop news letter. And by 'listens to changes' I mean query our website many many times per-minute. Blocked.
That sort of behavior would justify a name and shame imo
Inconsiderate shitty services don’t deserve the politeness of protecting their name
yonatan8070 2 hours ago [-]
I wonder if you could just blast them with ever changing content to burn their summarization tokens on BS.
Though they probably use the cheapest possible models they can so the damage from their dumb business model would be limited
xvxvx 2 hours ago [-]
I wonder if this is why I see so many YouTube links posted under ‘new’ that never make it to the FP. Is this a way to get bot scraped and boost views/interactions?
I run a few private MMO servers, we have a website with a support forum, in order to post, you need to register, log in, fill in account data, then create a thread, each of these pages is protected by Turnstile as well as a general bot check for the website, we frequently get automated spam from RMT (real-money trade) services for the OG game. Turnstile is maybe effective against simple bots that just try to call curl/requests/whatever but if you have bots that use a browser driver it's completely useless.
cullenking 3 hours ago [-]
Turnstile is no longer effective. Anyone with a budget uses a real headless chrome instance and gets around it. You only need the spendy chrome instances to get the initial cookie, then you pass that to your workers and they scrape.
It’s really starting to suck. The worst part is these are coming from consumer vpn providers that we can’t just block (lots of real users paying real money on our service are vpn users) or they come from big residential isps. The whole residential relay thing is a hard adversary, since it’s spread across so many real users paying ips that you can’t whackamole it. I’d say 1% of residential ips are participating in a botnet (there’s prob some real numbers out there if I cared to look).
This is just going to push everyone harder into walled gardens. Web input biggest inbound but our content is our mote, we are going to have to turn off non logged in content viewing, which will have real business impacts.
rrr_oh_man 4 hours ago [-]
I did this for alyph.ai, but was recently flooded with so many browser-using, Turnstile-defeating bots from Taiwan that I had to completely turn off sign ups. It's not fun anymore.
Not even third party login worked (Google, Linkedin, ...) to curb this.
They were logging in, scraping the client API for vulnerabilities/possibilities and then spammed erotic dog poems to Claude Fable until the $5 welcome credit got drained.
Fun New World.
firtoz 3 hours ago [-]
remove free signups
$5 minimum to sign up
sammy0910 4 hours ago [-]
not yet, but I think something like this would be a next step
ArgumentMoney81 4 hours ago [-]
[flagged]
nubinetwork 3 hours ago [-]
I get llm-like traffic on domains with zero web content (never was, probably never will be any).
Just merely existing will get you scanned these days.
oneeyedpigeon 4 hours ago [-]
What confused me most about this is that the entire site is, apparently, a single page (there's no sitemap that I can use to verify this; there IS an RSS file which appears to confirm it) containing all posts. Links at the top (headed "Table of Contents") are all in-page links.
At first, I assumed this was a single blog post, especially with a "Table of Contents" at the top. I guess it's following some old-school blogging conventions, but AFAICR, even those sites retained individual pages for separate posts.
sammy0910 4 hours ago [-]
it's org-html-export-to-html babyyy lmao
sammy0910 4 hours ago [-]
I think I could improve the format a bit with something like Jekyll, but I haven't had a good chunk of time to sit down and work on it
kqr 4 hours ago [-]
You can also link between Org documents and org-publish will export the documents with working web links, if you're lucky. That's how https://entropicthoughts.com works.
Jaxan 3 hours ago [-]
Oh, as soon as you add tls to your subdomain, the bots come. It’s instant.
jeroenhd 16 minutes ago [-]
Certificate transparency logs get scraped. However, they cannot scrape wildcards. Get a domain.example + *.domain.example cert and host your blog on blog.domain.example and you can keep the bots from finding you, at least for a little while.
thmorriss 46 minutes ago [-]
love the simple css of this. very tasteful and beautiful, even if its just simple.
andai 3 hours ago [-]
Interesting, thanks for sharing.
>So I took a look at some data. I added top-level logging for every endpoint (how many times was a request made to each url path), and I found that a majority of the requests did not even make an attempt to fetch news articles! For reference, my website is built as a single-page-application, and what this means is that a majority of the requests did not even run the javascript necessary to render the page (!).
So you got hammered by curl requests! But isn't that part of your delivery static anyway? Were you regenerating the same HTML over and over?
>many, many requests per minute?
How many was it?
Also interesting that it was a single "user". (user-agent?) Did it come from a different IP every time?
But hey. Silver lining: free load testing! ;)
sammy0910 3 hours ago [-]
> So you got hammered by curl requests! But isn't that part of your delivery static anyway? Were you regenerating the same HTML over and over?
i dont think this was the cause of the slowdown, it was just sort of the thing that led to the discovery that this was mostly bots.
in general, because the purpose of the website is to allow users to generate news feeds based on custom criteria, it does do a bit of meaningful work on each request (ones that do follow through and load the full page).
> How many was it?
> Also interesting that it was a single "user". (user-agent?) Did it come from a different IP every time?
I'm intentionally leaving some of the details vague here, both because I don't want to be too harsh on the original author (it may have just been accidental/a buggy program/etc.), and because I don't want the traffic generators to know how im flagging them.
steve_adams_86 2 hours ago [-]
I ran into similar problems. My analytics became so useless that I threw them out.
egberts1 2 hours ago [-]
That is primarily why websites only TLS-negotiates for server-side-only ChaCha-Poly.
MaxBarraclough 1 hours ago [-]
How could the specifics of TLS help deter/block bots?
spiderfarmer 4 hours ago [-]
Try posting something viral on Mastodon. The logs clearly show that it isn't the most efficient social network.
dyingkneepad 4 hours ago [-]
Can you elaborate more, please? I'm curious.
advisedwang 3 hours ago [-]
not the parent commentor, but I've previously heard complaints about how each server separately does a fetch of images and html for previews.
mococa 4 hours ago [-]
Same with me, every-time I got 1st page I got a lot of robots
danieltk76 53 minutes ago [-]
happened to me too!
sebastianconcpt 54 minutes ago [-]
Wait until you answer in "Ask HN: Who wants to be hired?" and start receiving hiring phishing emails.
ChrisMarshallNY 4 hours ago [-]
Heh. I posted a GitHub project Show HN, a couple of days ago, and it never even made it out of new, but it got me some bots.
That's a big reason that I never post any of my apps here.
Welcome to hell, kid...
janalsncm 4 hours ago [-]
Ok, on the one hand to have one bot hitting your website to populate some kind of slop newsletter is pretty annoying.
On the other hand, “multiple times per minute” doesn’t seem that high? Even if it was 60x per min that’s fully within the limits of what a 2015 laptop in the closet can handle.
sammy0910 4 hours ago [-]
yeah, it was a wake-up call that I needed to make certain parts of my application faster, and I've been working on it for the past few weeks
Exoristos 3 hours ago [-]
Once every 24 hours seems sufficient for the stated purpose.
HN has become unbearably AI slop oriented over the last couple of years. It’s nice to see something else.
5324_1 2 hours ago [-]
[flagged]
rahibali 5 hours ago [-]
[flagged]
add-sub-mul-div 5 hours ago [-]
[flagged]
retsibsi 5 hours ago [-]
I don't know HN's rationale, but: if you try to ban self-promotion, then in practice you're only banning honest self-promotion. And if you allow it, but try to ban accounts that only exist for that purpose, then in practice you're encouraging them to spam mediocre comments in other threads to dilute the signal. [edit: I see that the guidelines do say not to use HN primarily for promotion. I'm willing to bet this significantly increases the amount of insincere engagement, but obviously the admins/moderators here are more aware of the tradeoffs than I am, and they have decided they are worth accepting.]
IMO the important thing to try to crack down on is vote manipulation; if you have that under control, then you can probably happily allow self-promotion and trust that only the interesting stuff will rise to the top.
unshavedyak 5 hours ago [-]
Any thoughts on what the alternative would be?
Perhaps a contribution requirement per post? Though this proposal just invites bots to actually post comments because that becomes a requirement for posting links.
jaggederest 5 hours ago [-]
The guidelines already cover this:
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
I assume that users who primarily/only post their own stuff and get flagged a lot reap the consequences, there's plenty of spam filtering on here thank goodness.
ThrowawayR2 4 hours ago [-]
Those who slip promo of their own sites among a stream of other legitimate submissions or just space it out enough across multiple accounts can fly under the radar for years. People know how to game HN and already are, e.g. https://news.ycombinator.com/item?id=49517728
That's just one account that unmasked itself and was noticed. Doubtlessly that didn't stop them from moving on to the next one.
jacquesm 2 hours ago [-]
Oh, absolutely. But there are accounts that are active here for months or years, that one at least got killed while still green.
Akronymus 5 hours ago [-]
make interactions with others' posts/post others' stuff a soft req for posting your own stuff? So not following it could be deemed actionable by the hn mod team?
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
vanschelven 5 hours ago [-]
I _also_ posted to HN and made it to #1[0]; the effect this had on my CPU/network traffic on a 5/EUR month VPS over at Hetzner was in the single digit percentages.
As per the wiki page for the C10k problem[1] where it says "_was_ [emphasis mine] the problem of optimizing computer networking stacks to handle a large number of clients at the same time". This was solved over a decade ago...
The c10k problem is a different problem. It relates to simultaneous open TCP connections at a server. It would take a very very large number of crawler bots to force open 10k connections, I suspect.
vanschelven 56 minutes ago [-]
I know it's a different problem but presumably there's at least some relationship/order of magnitude being expressed here...oh well
esseph 1 hours ago [-]
> It would take a very very large number of crawler bots to force open 10k connections, I suspect.
You can do that with any system made in the past 15, maybe 20 years.
From the wikipedia page:
"As of 2025, the problem has long since been solved, with the number of possible connections to a single computer being in the millions."
Rendered at 20:32:24 GMT+0000 (Coordinated Universal Time) with Vercel.
I banned Alibaba's Cloud two years ago and it still hits one of my domains 1 million times per day, despite every route 4xx-ing during that time.
Even Apple has aggressively ramped up its scraping in the past few months, but thankfully they do respect robots.txt.
It's frustrating because some of my real users route through these clouds via their work VPNs. Also, there are a lot of less-than-reputable AI scrapers using global bot-nets via residential IPs to scrape, making it even harder to detect and ban them without affecting real people.
* like leaked .env routes, brute forcing admin panels, and so on
That sort of thing was very common long before agent based abuse scanners. Volume would wax and wane but never went away.
Though other scans for vulnerabilities have shot up in frequency now everyone script kiddie and their flee-bitten mutt can ask an AI to do it for them.
You may as well silently drop the packets if they don't adapt.
Google DNS are busy but its a hand full of IPs and reasonable. Cloudflare is non-stop never ending traffic. I don't know WTF they are trying to get. I've also seen TCP queries from those same Cloudflare IPs so this is not the result of a DDOS attack on them.
1. Breach router or desktop that hasn't been patched in ages via mass script
2. Establish C&C using cloudflare tunnels
3. Sell these hacked systems as residential proxy endpoints to everybody from state actors to teenagers
4. Some of them will be data scraping services
You are chasing your tail trying to ban by ASN or IP. It may work for another couple of months, at best.
Similar to ad click fraud and scam ads, the cloud provider gets paid, why would they turn down the money?
Blocking cloud providers is strictly for us dirty self hosters who are sick of our logs becoming unreadable and our visitor metrics useless from all this malicious traffic.
Why allow random requests coming from the clouds?
However, they could just use residential proxy botnets.
I believe the future is static resources with dynamic apps being only offered to people who can authenticate. And even then I’d recommend grandfathered accounts (eg telegram users that had an account for at least a year).
Agreed
> I believe the future is static resources with dynamic apps being only offered to people who can authenticate.
I don't know about static resources, but I do think that identity will come into play.
Why allow random requests coming from the clouds?
However, they could just use residential proxy botnets.
Traditionally cybercrimes have been prosecuted by the victim going to their local PD (or the FBI) with the IP address of the offender, and then law enforcement subpoenas the ISP associated with that IP address to figure out the customer associated with that account, and then they have a case for prosecution. It all works as long as cybercriminals are less numerous than law enforcement, fall in the same jurisdiction (or in jurisdictions that have reciprocal extradition treaties), and have a real identity that can be tied to their online activities.
Now, many law enforcement agencies are tied up prosecuting political opponents of their bosses. To the extent that they have bandwidth to investigate and prosecute ordinary crimes, they're going to prioritize the ones that target a.) the government that backs them (of which there are many - you see another headline about the government being hacked every week or so on HN) b.) major corporations that have sophisticated IT systems that can aid prosecution or c.) headlines that will get their boss a PR boost and themselves a promotion. Your personal website does not register among those priorities. You probably can't even prove damages from the intrusion itself. The perpetrator might be a Russian bot that the U.S. doesn't have jurisdiction over, or a Canadian teen that the U.S. previously might've been able to prosecute with reciprocal judicial treaties but now has no leverage over because we've blown up our relationship with Canada. It may be legally a crime, but economically it's not worth prosecution.
Traditionally security researchers had to be very careful because they were doing everything above-board, with their real identity attached to their actions, and they made a very attractive (and PR-worthy) target. Somebody who's port-scanning from Russia through a VPN or residential proxy network is a.) anonymous b.) untraceable c.) out of U.S. jurisdiction and d.) not a headline-worthy target anyway.
A couple years ago I predicted here that a lot of things will break as the U.S. transitions from a high-trust to a low-trust society. This is one of them. Much of our justice system works well when criminals are rare and utterly falls apart when they're ubiquitous.
[0]: https://news.ycombinator.com/item?id=43826798
The main problem on my site is that bots hit the CGIT repo, where visiting files triggers syntax highlighting scripts.
I put in a skill-testing-question type system where you get a maximally long-lived access cookie if you answer right (so legit users are bothered as infrequently as possible).
There you go. Ditch it. Make you pages cachable as much as you can.
* I wanted an elixir version so I could run it alongside my existing apps instead of having it sit as a proxy in front of them. This way my existing apps can validate if the caller is already whitelisted and if not redirect to the challenger https://sphinx.8by3.com which sends you back to a site if you pass eg. https://tuhat.net
https://www.sammystraus.com/#using-llms-to-make-the-emacs-we...
Some people liked my work, found me on LinkedIn and sent a connection with a positive message. Other people tried every which way to SQL inject or attack the site, posted slurs in every available input, etc.
Someone copied the site and hosted their own version with a couple minor changes. I only found that one because they had left one of the links in, which gave me referring traffic from their copy.
All that from one HN comment, very much pre-AI too, not to mention the standard daily slew of requests probing for what I assume are known vulnerabilities in Wordpress or PHP.
The internet is a floodgate of different actors, many of which are automated, and a link posted somewhere like HN opens the valve especially wide.
What's the value for the website maker in these? There's user value to it (rather than just value for Anthropic/OpenAI) but the whole point of these bots is to make sure nobody ever needs to visit your website.
The most likely answer is they don't give a crap because it's not their money and no one is going to do anything about it.
I experienced the same thing after submitting articles from my small photography website. I did debate using some of the off-the-shelf solutions like Cloudflare, Anubis, or iocaine, but I ended up building my own L7 proxy that utterly fucks the bots up: https://robertmay.photography/journal/bwbach-my-guardian-gob.... I will make it generally available at some point once I get it optimised how I'd like, but currently I'm able to trap about 3000 concurrent crawlers in my tarpit of slow requests on two small Hetzner VMs. And that 3k is really just the limits of the networks hitting my servers right now, I think it'll go significantly higher than that.
I haven't really seen anyone else combat them this way, possibly because many of the standard proxies and servers are set up for fast requests, not serving them as slowly as possible. Deliberately serving page loads for as long as the requester will accept packets turns out to be a really funny way to stop the botnets - when I just rejected the requests they in turn hammered my server even harder. By tanking the request volume with a ridiculous BEAM-based proxy you can reduce their request speed to a crawl and I'm fairly certain a couple of them have blacklisted my server now.
EDIT: sorry, that was a dumb question before reading your post. You actually learn the timeout they tolerate. Nice!
Looking at you, Meta.
I do still get some requests going up to 15 minutes which is quite ridiculous, though those don't seem to be super common (maybe they're learning).
I think the shortest timeouts I've seen are between 2-5 seconds, but they're surprisingly infrequent and actually that's still a massive inconvenience for them. I did at one point just drop any request I couldn't keep open for more than 5 seconds but it actually caused more server load. I'm essentially inverting the server load by making slow requests more efficient, which is counter to everything I ever learnt about performance for web serving :D
I think the average I see is probably somewhere between 15-30 seconds. I'm guessing many crawlers don't want to go too low because they'll be trying to fetch large files like images, or perhaps they know they'll be slowing servers down and want to ensure they get the response. Most likely, the majority of crawler authors aren't expecting some arsehole to exploit this and they've just left the setting as the default.
I gave up on tarpits when it got to 130k "waiting" connections.
Who is?!
You see something, you say something, or you do something.
https://pagespeed.web.dev/analysis/https-sprinklz-io/bmc4mv1...
That sort of behavior would justify a name and shame imo
Inconsiderate shitty services don’t deserve the politeness of protecting their name
Though they probably use the cheapest possible models they can so the damage from their dumb business model would be limited
It’s really starting to suck. The worst part is these are coming from consumer vpn providers that we can’t just block (lots of real users paying real money on our service are vpn users) or they come from big residential isps. The whole residential relay thing is a hard adversary, since it’s spread across so many real users paying ips that you can’t whackamole it. I’d say 1% of residential ips are participating in a botnet (there’s prob some real numbers out there if I cared to look).
This is just going to push everyone harder into walled gardens. Web input biggest inbound but our content is our mote, we are going to have to turn off non logged in content viewing, which will have real business impacts.
Not even third party login worked (Google, Linkedin, ...) to curb this.
They were logging in, scraping the client API for vulnerabilities/possibilities and then spammed erotic dog poems to Claude Fable until the $5 welcome credit got drained.
Fun New World.
$5 minimum to sign up
Just merely existing will get you scanned these days.
At first, I assumed this was a single blog post, especially with a "Table of Contents" at the top. I guess it's following some old-school blogging conventions, but AFAICR, even those sites retained individual pages for separate posts.
>So I took a look at some data. I added top-level logging for every endpoint (how many times was a request made to each url path), and I found that a majority of the requests did not even make an attempt to fetch news articles! For reference, my website is built as a single-page-application, and what this means is that a majority of the requests did not even run the javascript necessary to render the page (!).
So you got hammered by curl requests! But isn't that part of your delivery static anyway? Were you regenerating the same HTML over and over?
>many, many requests per minute?
How many was it?
Also interesting that it was a single "user". (user-agent?) Did it come from a different IP every time?
But hey. Silver lining: free load testing! ;)
i dont think this was the cause of the slowdown, it was just sort of the thing that led to the discovery that this was mostly bots.
in general, because the purpose of the website is to allow users to generate news feeds based on custom criteria, it does do a bit of meaningful work on each request (ones that do follow through and load the full page).
> How many was it? > Also interesting that it was a single "user". (user-agent?) Did it come from a different IP every time?
I'm intentionally leaving some of the details vague here, both because I don't want to be too harsh on the original author (it may have just been accidental/a buggy program/etc.), and because I don't want the traffic generators to know how im flagging them.
That's a big reason that I never post any of my apps here.
Welcome to hell, kid...
On the other hand, “multiple times per minute” doesn’t seem that high? Even if it was 60x per min that’s fully within the limits of what a 2015 laptop in the closet can handle.
HN has become unbearably AI slop oriented over the last couple of years. It’s nice to see something else.
IMO the important thing to try to crack down on is vote manipulation; if you have that under control, then you can probably happily allow self-promotion and trust that only the interesting stuff will rise to the top.
Perhaps a contribution requirement per post? Though this proposal just invites bots to actually post comments because that becomes a requirement for posting links.
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
https://news.ycombinator.com/newsguidelines.html
I assume that users who primarily/only post their own stuff and get flagged a lot reap the consequences, there's plenty of spam filtering on here thank goodness.
https://news.ycombinator.com/threads?id=FannyJoeStyles
Edit: nvm thats already in the guidelines
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
As per the wiki page for the C10k problem[1] where it says "_was_ [emphasis mine] the problem of optimizing computer networking stacks to handle a large number of clients at the same time". This was solved over a decade ago...
Now get off my lawn!
(TBF I think the author acknowledges this)
[0] https://news.ycombinator.com/item?id=43725815
[1] https://en.wikipedia.org/wiki/C10k_problem
You can do that with any system made in the past 15, maybe 20 years.
From the wikipedia page:
"As of 2025, the problem has long since been solved, with the number of possible connections to a single computer being in the millions."