Still trying to figure out my website outages.

To recap: I have two #MediaWiki wikis and one #WordPress site. And there seems to be _some_ background process which starts soon after 23:00 UTC (or 24:00 in France, where the servers are located) which puts such a strain on the website that I get a lot of "504" errors for a few hours. The exact duration is variable - sometimes it lasts for one or two hours, and sometimes it lasts way into the morning. But I cannot identify what causes it from the Apache logs - there is no clear, consistent trigger to be found there.

Last time I asked customer support, they suggested that I check my plugins and background processes. But I don't know enough about website administration to know where to look.

For what it's worth, my WordPress website uses the following plugins (all updated to the latest versions):

ActivityPub
Contact Form 7
Include Mastodon Feed
JM Twitter Cards
Leaflet Map
MC4WP: Mailchimp for WordPress
TablePress
WP DSGVO Tools (GDPR)

The site is here:
https://sunkencastles.com/

The extensions on my wiki can be found here:
https://wiki.sunkencastles.com/wiki/Special:Version

Does _anyone_ have any idea what kinds of background processes on MediaWiki or WordPress might cause this kind of outage, and how I could check it?

#FediHelp

@juergen_hubert How does your crontab look?

Still trying to figure out my website outages.

To recap: I have two #MediaWiki wikis and one #WordPress site. And there seems to be _some_ background process which starts soon after 23:00 UTC (or 24:00 in France, where the servers are located) which puts such a strain on the website that I get a lot of "504" errors for a few hours. The exact duration is variable - sometimes it lasts for one or two hours, and sometimes it lasts way into the morning. But I cannot identify what causes it from the Apache logs - there is no clear, consistent trigger to be found there.

Last time I asked customer support, they suggested that I check my plugins and background processes. But I don't know enough about website administration to know where to look.

For what it's worth, my WordPress website uses the following plugins (all updated to the latest versions):

ActivityPub
Contact Form 7
Include Mastodon Feed
JM Twitter Cards
Leaflet Map
MC4WP: Mailchimp for WordPress
TablePress
WP DSGVO Tools (GDPR)

The site is here:
https://sunkencastles.com/

The extensions on my wiki can be found here:
https://wiki.sunkencastles.com/wiki/Special:Version

Does _anyone_ have any idea what kinds of background processes on MediaWiki or WordPress might cause this kind of outage, and how I could check it?

#FediHelp

@ratsnakegames

# num@period delay(unused) name script...
#
# uncomment to purge files older than 7days in TMP
#1@daily 0 purgetmp find /srv/data/tmp -type f -mtime +7 -delete > /dev/null

The cron logs are empty.

@juergen_hubert Are you occasionally awake and available at that time of day? If so, could you ssh onto your machine during this outage and run `top`?

It will show you a list of all processes. SHIFT+P will sort them by CPU usage, SHIFT+M will sort by RAM usage.

If nothing suspicious shows up immediately, could you please share screenshots of both lists?

@ratsnakegames

Not usually - this is way past my usual bed time. But maybe I should do this at some point.

@juergen_hubert RE: log analysis, have you tried ingesting the logs, especially the access logs, into a tool like OpenSearch / Kibana and plotted them histographically? You can also slice them by endpoint, user agent, IP address, and other things that could show a mis-behaving scraper / bot, or some other site "accidentally" DDOSing you. Might be useful if you can't find a batch job that explains the load.

@juergen_hubert i can probably figure out a command that lets you schedule that stuff, one moment

@juergen_hubert `echo "top -b -n 5 > processes.txt" | at 23:15` (without the backticks!)

You might have to install `at` first (with `apt install at` if you're on Ubuntu) and change the timestamp to 00:15 if your server is running on French time

@juergen_hubert Output should look like this:

@ratsnakegames

Thanks! I'll try this on the weekend.

@ruxbat

I did some scraper analysis last year, and while they are a persistent nuisance, their load tends to vary a lot. The consistency of the time period has led me to suspect that this is an inside problem.

@ratsnakegames

# num@period delay(unused) name script...
#
# uncomment to purge files older than 7days in TMP
#1@daily 0 purgetmp find /srv/data/tmp -type f -mtime +7 -delete > /dev/null

The cron logs are empty.

Still trying to figure out my website outages.

To recap: I have two #MediaWiki wikis and one #WordPress site. And there seems to be _some_ background process which starts soon after 23:00 UTC (or 24:00 in France, where the servers are located) which puts such a strain on the website that I get a lot of "504" errors for a few hours. The exact duration is variable - sometimes it lasts for one or two hours, and sometimes it lasts way into the morning. But I cannot identify what causes it from the Apache logs - there is no clear, consistent trigger to be found there.

Last time I asked customer support, they suggested that I check my plugins and background processes. But I don't know enough about website administration to know where to look.

For what it's worth, my WordPress website uses the following plugins (all updated to the latest versions):

ActivityPub
Contact Form 7
Include Mastodon Feed
JM Twitter Cards
Leaflet Map
MC4WP: Mailchimp for WordPress
TablePress
WP DSGVO Tools (GDPR)

The site is here:
https://sunkencastles.com/

The extensions on my wiki can be found here:
https://wiki.sunkencastles.com/wiki/Special:Version

Does _anyone_ have any idea what kinds of background processes on MediaWiki or WordPress might cause this kind of outage, and how I could check it?

#FediHelp

@juergen_hubert 504 means that the server in front of mediawiki (the http gateway that handles https encryption for instance) cannot reach mediawiki.
Is this shared infra, or isolated ?

@juergen_hubert `echo "top -b -n 5 > processes.txt" | at 23:15` (without the backticks!)

You might have to install `at` first (with `apt install at` if you're on Ubuntu) and change the timestamp to 00:15 if your server is running on French time

@ratsnakegames

Unfortunately, it seems I don't have the permissions to install at.

Maybe I'll bother customer service again.

@juergen_hubert You don't have root access? Does `sudo apt install at` work?