https://OpenStreetMap.org has been disrupted today.
-
@osm_tech@en.osm.town
Time to deploy Anubis in front of all the URLs that the scrapers are hitting.@gme Anubis is great, but not ideal for us. We have reasonable technical mitigations in place, but it is unfortunately an arms race (and time/resource blackhole)
-
@Don_Clemente Just dumb scrapers. They want our data, so maybe we /should/ redirect them to the latest full history planet file, it is only 245GB
@osm_tech yes, but that's heavy on your resources and bandwidth. A zip file of 10GB Zeros is much more resource-efficient on your side
-
-
@pesasa @ISibboI @osm_tech
If its helpful (if you want to stuff something unpleasant down their throats), I believe @jwz has been making a 'json bomb' for (AIUI) pretty much this purpose:
https://mastodon.social/@jwz/116049057703097965 -
https://OpenStreetMap.org has been disrupted today. We're working to keep the site online while facing extreme load from anonymous scrapers spread across 100,000+ IP addresses. Please be patient while we mitigate and protect the service. #OpenStreetMap #DDoS #Scrapers #AI
@osm_tech @lehtimaeki I wonder if anyone has looked at the various AI browsers out there to see if they're used for active scraping.
-
@Don_Clemente Just dumb scrapers. They want our data, so maybe we /should/ redirect them to the latest full history planet file, it is only 245GB
@osm_tech @Don_Clemente Replacing the actual data with Null Island data that points to planet.osm.org is not a dumb idea IMHO
-
https://OpenStreetMap.org has been disrupted today. We're working to keep the site online while facing extreme load from anonymous scrapers spread across 100,000+ IP addresses. Please be patient while we mitigate and protect the service. #OpenStreetMap #DDoS #Scrapers #AI
@osm_tech Block all read access except from registered users? And be restrictive on whom to grant an account (captcha, verified email address)?
-
https://OpenStreetMap.org has been disrupted today. We're working to keep the site online while facing extreme load from anonymous scrapers spread across 100,000+ IP addresses. Please be patient while we mitigate and protect the service. #OpenStreetMap #DDoS #Scrapers #AI
-
-
@osm_tech yes, but that's heavy on your resources and bandwidth. A zip file of 10GB Zeros is much more resource-efficient on your side
@osm_tech @Don_Clemente Can confirm. >:D
-
@osm_tech Block all read access except from registered users? And be restrictive on whom to grant an account (captcha, verified email address)?
had a similar issue with a much, much smaller wiki server (
) but facing the same anonymous scrapers coming from thousands of random IP's.Couldn't find a way to keep them from crashing the server, so now access is by registration only.
Very disruptive, this "AI" race to the bottom destroyed the open internet
-
https://OpenStreetMap.org has been disrupted today. We're working to keep the site online while facing extreme load from anonymous scrapers spread across 100,000+ IP addresses. Please be patient while we mitigate and protect the service. #OpenStreetMap #DDoS #Scrapers #AI
@osm_tech I went to use OpenStreetMap today to get directions but it asked me to sign up/ log in, so I used Mapquest instead which doesn't.
-
@ISibboI Correct. Not an ounce of brain. It has been going on for months, but it is just getting worse. Waste of their resources/time and ours.
-
@osm_tech I went to use OpenStreetMap today to get directions but it asked me to sign up/ log in, so I used Mapquest instead which doesn't.
@Air_Quotes_Comedian Weird. We don't require login for directions.
-
@Air_Quotes_Comedian Weird. We don't require login for directions.
@osm_tech Hmmm.
Perhaps I'm the only one who is required to log in for directions.
-
https://OpenStreetMap.org has been disrupted today. We're working to keep the site online while facing extreme load from anonymous scrapers spread across 100,000+ IP addresses. Please be patient while we mitigate and protect the service. #OpenStreetMap #DDoS #Scrapers #AI
@osm_tech thank you for your service 🪖
-
@ISibboI Correct. Not an ounce of brain. It has been going on for months, but it is just getting worse. Waste of their resources/time and ours.
-
-
https://OpenStreetMap.org has been disrupted today. We're working to keep the site online while facing extreme load from anonymous scrapers spread across 100,000+ IP addresses. Please be patient while we mitigate and protect the service. #OpenStreetMap #DDoS #Scrapers #AI
@osm_tech I blocked user agents matching "Chrome.139.0.0.0 Safari.537.36" as this combination seems to not exist, except for millions of requests out of china.
-
@geospacedman @ISibboI They are scraping the website pages: /ways/, /nodes/ and /relations/. All this data is published on planet.osm.org already.
@osm_tech
Bloody idiots. I wonder if it's AI platforms?
@geospacedman @ISibboI
