I still have to deal with #WebsiteOutages , and currently I have another massive one which won't go away by purging the Varnish cache.
-
I still have to deal with #WebsiteOutages , and currently I have another massive one which won't go away by purging the Varnish cache. As usual, the outage started between 23:00 and 24:00 UTC, which is _weird_.
I have asked Gandi.net customer service a few weeks ago, but beyond suggesting blocking a random crawler, they said the following:
"Personally, I would check the plugins on my various websites and the scheduled tasks (especially backup plugins)."
I don't remember installing such plugins manually. Does anyone know of such processes that are installed on either #WordPress or #MediaWiki sites by default? (Or perhaps this is done by #SemanticMediaWiki ?)
In three days I will be leaving for a vacation, so this is the last opportunity I have to improve this situation this year.
@juergen_hubert Crawlers and scrapers and fetchers! Oh my! - Dorothy (allegedly)
got dark visitors ? #RobotsTXT #DarkVisitors https://darkvisitors.com/
-
@juergen_hubert Crawlers and scrapers and fetchers! Oh my! - Dorothy (allegedly)
got dark visitors ? #RobotsTXT #DarkVisitors https://darkvisitors.com/
Numerously so, and they certainly massively contribute to website traffic.
However, the real mystery is what process starts between 23:00 and 24:00 UTC that actually triggers the outages. I mean, the crawlers are active around the clock!
-
Numerously so, and they certainly massively contribute to website traffic.
However, the real mystery is what process starts between 23:00 and 24:00 UTC that actually triggers the outages. I mean, the crawlers are active around the clock!
@juergen_hubert @wolfofthewisp
Do you have an access log that includes duration? How long the request took, can be useful (besides the usual URL, IP, User Agent) to find slow/expensive request among many quick ones.
-
I still have to deal with #WebsiteOutages , and currently I have another massive one which won't go away by purging the Varnish cache. As usual, the outage started between 23:00 and 24:00 UTC, which is _weird_.
I have asked Gandi.net customer service a few weeks ago, but beyond suggesting blocking a random crawler, they said the following:
"Personally, I would check the plugins on my various websites and the scheduled tasks (especially backup plugins)."
I don't remember installing such plugins manually. Does anyone know of such processes that are installed on either #WordPress or #MediaWiki sites by default? (Or perhaps this is done by #SemanticMediaWiki ?)
In three days I will be leaving for a vacation, so this is the last opportunity I have to improve this situation this year.
@juergen_hubert do you have insight into the database? It's likely running a weird task killing mediawiki if it's mediawiki. DB cleanups are often annoying -
@juergen_hubert @wolfofthewisp
Do you have an access log that includes duration? How long the request took, can be useful (besides the usual URL, IP, User Agent) to find slow/expensive request among many quick ones.
I do, but so far I haven't found a clear pattern.
-
@juergen_hubert do you have insight into the database? It's likely running a weird task killing mediawiki if it's mediawiki. DB cleanups are often annoying
Admittedly, I wouldn't know where to begin looking.
-
Admittedly, I wouldn't know where to begin looking.
@juergen_hubert does your webhost surface any logs? A lot of hosts tell you its your job and don't show any logs at all. -
@juergen_hubert does your webhost surface any logs? A lot of hosts tell you its your job and don't show any logs at all.
I do have access to the Apache logs.
-
I do have access to the Apache logs.
@juergen_hubert yeah - so you might be able to see it passing things off to mysql - but it's the MySQL side - if it's dropping connections or having worker issues that might be a thing. I dunno how your config is with your host.
If Mediawiki and Wordpress are both going down - the DB is where I'd look first. What's the error? 404? 502? The page just not loading forever?
-
@juergen_hubert yeah - so you might be able to see it passing things off to mysql - but it's the MySQL side - if it's dropping connections or having worker issues that might be a thing. I dunno how your config is with your host.
If Mediawiki and Wordpress are both going down - the DB is where I'd look first. What's the error? 404? 502? The page just not loading forever?
A 504 error. A timeout after 300 s.
