Hubzilla php is eating up all memory

Hans Sandmeier

hans@hub.netzgemeinde.eu

This channel has not added a profile description yet

Mon, 12 Aug 2019 19:53:36 +0200

Hans Sandmeier

hans@hub.netzgemeinde.eu

!Hubzilla Support Forum
my hubzilla regularly eats up all my memory, until there is no memory and then the kernel kills php. I am not 100% sure, but I think its triggered by the periodical calls of Zotlab/Master.php .

So my question: where can I set up how often and when the hubzilla scripts are called? -- I looked in the crontab of hubzilla user, but didn't find there any entry. Where is the call hidden?

show all 11 comments

Mon, 12 Aug 2019 20:02:14 +0200

It's possible to put crontabs into directories at /etc/cron.d, and shell scripts into /etc/cron.{hourly,daily,weekly}. The periodic script I use is in a crontab at /etc/cron.d/hubzilla.

Hope this helps.

Mon, 12 Aug 2019 20:13:30 +0200

Hans Sandmeier

hans@hub.netzgemeinde.eu

These are all the places I looked too... but thanks to your comment, I realized that hubzilla doesn't have a fixed location for that.
So I found it in /etc/crontab :-D

Thank you! :-)

Mon, 12 Aug 2019 20:27:27 +0200

By default Hubzilla invokes procedures ones per 10 minutes. Instead cron manipulations I recommend you tune delivery and poll intervals on /admin/site/. Perhaps a good idea will to use new experintal queueworker addon from @M. Dent.

Mon, 12 Aug 2019 20:31:54 +0200

Hans Sandmeier

hans@hub.netzgemeinde.eu

I just change it in the crontab from every 10 minuets to once a day. But you suggest to modify it in the admin site of hubzilla? -- why should that be better?

Mon, 12 Aug 2019 20:42:54 +0200

@Hans Sandmeier More rare queue invoke increases probability you'll face with memory shortage because of queue size so this is quite bad idea.

Mon, 12 Aug 2019 20:46:39 +0200

Hans Sandmeier

hans@hub.netzgemeinde.eu

hmm... you got a point! :)
Can't I somehow limit the memory usage?

Mon, 12 Aug 2019 21:10:06 +0200

@Hans Sandmeier You can start from memory_limit variable in php.ini.
Check current value first via

# php -i | grep memory

But I believe right delivery and poll intervals should mostly help you.

Mon, 12 Aug 2019 22:22:15 +0200

Hans Sandmeier

hans@hub.netzgemeinde.eu

there is no limit set so far

memory_limit => -1 => -1
Collecting memory statistics => No
opcache.memory_consumption => 64 => 64
opcache.preferred_memory_model => no value => no value
opcache.protect_memory => 0 => 0

Do you have any clue, why hubzilla eats up all my memory? -- because to many messages are processed and for each message a new php process?

Mon, 12 Aug 2019 23:04:31 +0200

@Hans Sandmeier

because to many messages are processed and for each message a new php process?

This is closest to the truth.

Tue, 13 Aug 2019 02:55:30 +0200 zuletzt bearbeitet: Tue, 13 Aug 2019 02:54:43 +0200

@Hans Sandmeier

because to many messages are processed and for each message a new php process?

That's pretty close - though the subsystem we're talking about doesn't just handle messages, it handles image resizing, and a bunch of other background tasks as well.  Some of these can also be resource hogs, but it isn't obvious when it happens.

There are a lot of competing principles in the development of HUBZILLA - one of them is being able to run a personal instance on a typical "shared hosting" environment.  This creates limitations on long-running processes and other techniques that would typically be used for process management. The queueworker addon is intended to address many of these by assuming things like at least medium length processes will be OK.

The vanilla "Daemon" system of Hubzilla is VERY robust and very resiliant, but it can also be very resource intensive under certain circumstances.  It has no mechanism to track processes.  Most processes are quick fire and terminate very shortly after starting - but in certain circumstances (eg, a machine with memory limitations that lead to swapping or other resource bounds or lots of hubs in a non-responsive state), it can make a machine completely unresponsive (Ive had load averages approaching 100!).  And while the resource limitation checks are good - they don't catch everything, so things can still run away unexpectedly.

Enter the queueworker addon.

The DEV branch has some significant (HUGELY SIGNIFICANT) updates/upgrades and fixes.  I'd love for some others to work with it more intensively - I think it's about time to move it from "experimental/alpha" to something closer to "beta" - but it definitely needs more testing.

Give it a try - With it, I'm able to support multiple hubs on a lightly resourced VPS with minimal problems.  WIthout it, I've had serious problems.  As with anything else, your mileage may vary - but I'm very interested in getting it fully tested and being able recommend it as a mainstream solution.

Tue, 13 Aug 2019 07:40:18 +0200

El Dorado

mike@z.macgirvin.com

So my question: where can I set up how often and when the hubzilla scripts are called?

https://hub.netzgemeinde.eu/admin/site

Delivery interval
Deliveries per process
Queue Threshold

There are some other settings which affect performance but the first two are the most important.

Quick setup for a low-end VPS:

Set Deliveries per process to 5.
Set Delivery interval to 2.

Your *maximum* rate of delivery with this configuration is 150 transactions per minute (TPM) if all the servers you're connecting with are up and running.

If your system gets overloaded with these settings, set delivery interval to 3 (100 TPM). If it's still runs consistently overloaded set to 4 (75 TPM). If you have to set it above 5 to make things work you probably need a bigger server because at 60 TPM you're probably going to be generating work faster than the maximum delivery rate unless you only have a couple of users and not a lot of friends.

If your system is working just fine at 2 you can try backing off by one (300TPM).

If delivery interval is 1 and is still humming along nicely, increase the deliveries per process.

Shared hosting and hardware servers are completely different. For a hardware server I'd start with 5 and 1 (300TPM). For a shared host I'd start with 1 and 5 (12TPM). With a shared host you're really talking about a single user site.