Attn: Florian (quan) - Advmailer installation issue resulted in very high server load on VPS :: Zikula CMS

Forum

phaidon antwortet auf »[gelöst] Keine Piwik Statistiken unter 1.3« 09:31 Uhr
trichers antwortet auf »[wieder neu] Pagemaster kein Bildupload - Fehlermeldung von Thumbnail« 01:35 Uhr
papamac antwortet auf »menutree füllt meinen error_log« 24. Mai
FRW eröffnet das Thema »Postcalendar Block Upcoming events« 24. Mai
papamac antwortet auf »[gelöst] Installation Clip 092 in frisches Zikula133 misslingt und brin…« 21. Mai
trichers antwortet auf »Clip & Google Maps« 17. Mai
Roadrunner antwortet auf »[geklärt] Ziku.la« 16. Mai

» zum Forum | » aktuelle Diskussionen

Kommentare

29. Apr | GitHub Block
29. Apr | Danke
26. Apr | Ganz dickes fettes!
21. Apr | 1.3.2 Link
18. Apr | 1.3.2
14. Apr | MedienModul
14. Apr | Clip und weiter …

Attn: Florian (quan) - Advmailer installation issue resulted in very high server load on VPS

Thema ausdrucken

raagv

Rang: gerade reingestolpert

Registriert: 20. Aug 2010

Letzter Besuch: 23. Aug 2010

Beiträge: 4
Verfasst: 20. Aug 2010 - 10:32

Hi,

I am running Zikula 1.1.2 on a VPS by Hostgator (Level 3 - 768MB RAM, 1.13GHz processor). I shifted to this VPS account about 25 days back from a shared hosting environment.

I have around 1500 daily visits, and about 2500 daily page views on an average.

What I did:

About 6 days back, I tried installing advmailer 2.0 module (I ultimately wanted to install mailz). When I clicked on "install" under modules, I got a "ZLanguage" error for one of advmailer's ini file (sorry, don't remember the exact name). I did some research in the forums, and following a solution for a similar situation, deleted the offending line from the ini file.

When I tried to install again, I again got the "ZLanguage" error for another ini file for advmailer (again, don't have the exact name). I gave up, and didn't do anything further. The status of advmailer module still shows as "Not installed".

(On later research, I realized that advmailer 2.0 was not meant for Zikula 1.1.2)

(I had also changed the "pnuserapi.php" file in /system/Mailer directory as asked by the advmailer documentation, but have rolled back the change following the unsuccessful installation)

Here's the problem:

After this happened, my VPS is showing unusually high server loads, and the CPU constantly remains occupied at 98-100%. The server load keeps increasing, and reaches as high as 35-40! At that time, I have to restart it. The load starts low (around 0.3), but keeps increasing.... And I have to repeat the process. I have to bounce the server about 2 times a day.

As I said, the CPU remains occupied at 98-100%, the memory usage varies from 35% to 100%.

When I execute a top command (through "Show Current CPU Usage" in WHM cPanel), I can see multiple instances of index.php running (/usr/bin/php /home//public_html/index.php). The number can be as high as 35-40! (I usually don't have more than 5 concurrent users).

In the public_html directory of my host (the root directory for Zikula), I also see many core. files (e.g. core.14068, core.16318, etc). Each file is about 20MB! Currently, there are 18 such files (Update: Increased to 23 in last 4 days)! I am not sure if this is related, but the number of these files seem to have increased lately....

What I have tried so far:

The advmailer module shows as "not installed", so there is no way I can uninstall it. Bur I have changed its directory name (inside the "modules" directory), but nothing changed.

Assuming that there could have been some change to the Zikula database due to the aborted installation, I also uploaded the DB I backed up before installing advmailer. But again, the problem hasn't gone.

Need help:

I have searched the forums, but couldn't find anything about this... I also raised a ticket with Hostgator, but they have said that this is an application (Zikula) related issue and they can't help me with it.

I have recently left my job to work full time on the website, and this problem has really stumped me! Would really appreciate any and all help from all of you... Thanks!

(Originally posted at Zikula.com - http://communit…opic-58135.htm)
Mr.Montesa

Rang: Support-Team

Registriert: 24. Sep 2006

Letzter Besuch: 24. Mai 2012

Beiträge: 614
Verfasst: 20. Aug 2010 - 11:28

Hey Sir,
I'm not familiar with Flo's mail module but I'm Linux Sysadmin.
The core files are memory dumps when an application crashes. The number at the end is the process id while the process was running. And the size reflects the ammount of memory used in RAM while running.
With the unix command
Code
file core.16318
you should get some info wich application crashed. Can you post this here please?

I would bet that it's an apache/webserver or PHP process, that restarts automatically if it dies. But perhaps the memory isn't freed up again. Thats why you'll see increasing load.
Can you provide me with some additional info in regard to your operating system, patchlevel, etc.
The following output would be interesting:

Code
uname -a
cat /etc/issue
cat /proc/cpuinfo
cat /proc/meminfo
vmstat 1 (needs to be stopped by Ctrl + C)
iostast 1 (needs to be stopped by Ctrl + C)

BR

Gruß Sebastian

Tweet with me
raagv

Rang: gerade reingestolpert

Registriert: 20. Aug 2010

Letzter Besuch: 23. Aug 2010

Beiträge: 4
Verfasst: 20. Aug 2010 - 13:48

Thanks Montesa.

I am not very familiar with Linux, so had to do some research about how to run the commands through SSH - did it through PuTTY. Hope this helps.

btw, the main issue is high CPU load (~100% usage, load upto 40) - the RAM utilization fluctuates from 20%-100%.

Here are the results:

file core.16318:
ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'php'

file core.21897 (latest such file created on Aug 17):
ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'php'

uname -a:
Linux raa.raagvamdatt.com 2.6.18-028stab068.3 #1 SMP Tue Feb 16 20:21:42 MSK 2010 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/issue:

This computer system is for authorized users only. Individuals using this
system without authority or in excess of their authority are subject to
having all their activities on this system monitored and recorded or
examined by any authorized person, including law enforcement, as system
personnel deem appropriate. In the course of monitoring individuals
improperly using the system or in the course of system maintenance, the
activities of authorized users may also be monitored and recorded. Any
material so recorded may be disclosed as appropriate. Anyone using this
system consents to these terms.

cat /proc/cpuinfo:

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5520 @ 2.27GHz
stepping : 5
cpu MHz : 2261.053
cache size : 8192 KB
physical id : 1
siblings : 8
core id : 0
cpu cores : 4
apicid : 16
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips : 4522.10
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: [8]

vmstat 1 (needs to be stopped by Ctrl + C):

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
29 0 0 71140 0 0 0 0 549 369 0 411 72 2 26 0 0
30 0 0 67096 0 0 0 0 48 0 0 6487 93 7 0 0 0
28 0 0 82884 0 0 0 0 0 0 0 7069 94 6 0 0 0
27 0 0 100696 0 0 0 0 0 0 0 5401 97 3 0 0 0
27 0 0 98940 0 0 0 0 0 0 0 5042 99 1 0 0 0
27 0 0 98504 0 0 0 0 0 0 0 4166 96 4 0 0 0
27 0 0 98492 0 0 0 0 0 4 0 4937 100 0 0 0 0
25 0 0 138816 0 0 0 0 0 0 0 5741 98 2 0 0 0
25 0 0 138888 0 0 0 0 0 0 0 4085 100 0 0 0 0
25 0 0 139036 0 0 0 0 0 0 0 5178 99 1 0 0 0
25 0 0 137640 0 0 0 0 0 0 0 4445 100 0 0 0 0
25 0 0 137644 0 0 0 0 0 52 0 3846 100 0 0 0 0
23 0 0 177304 0 0 0 0 0 0 0 3996 98 2 0 0 0
23 0 0 177252 0 0 0 0 0 0 0 3645 100 0 0 0 0
23 0 0 176996 0 0 0 0 0 0 0 3685 99 1 0 0 0
23 0 0 176960 0 0 0 0 0 0 0 4255 100 0 0 0 0
23 0 0 177052 0 0 0 0 0 4 0 4836 100 0 0 0 0
23 0 0 176808 0 0 0 0 0 0 0 4630 98 2 0 0 0
23 0 0 176900 0 0 0 0 0 0 0 4167 100 0 0 0 0
23 0 0 176988 0 0 0 0 0 0 0 4351 100 0 0 0 0
23 0 0 176892 0 0 0 0 0 0 0 3371 99 1 0 0 0
24 0 0 177192 0 0 0 0 0 68 0 4576 99 1 0 0 0
23 0 0 165236 0 0 0 0 0 0 0 3768 96 4 0 0 0
23 0 0 176988 0 0 0 0 0 0 0 4932 100 0 0 0 0
23 0 0 176708 0 0 0 0 0 0 0 4511 99 1 0 0 0
23 0 0 176752 0 0 0 0 0 0 0 5037 100 0 0 0 0
23 0 0 176824 0 0 0 0 0 96 0 4451 100 0 0 0 0
23 0 0 176668 0 0 0 0 0 0 0 4114 100 0 0 0 0
23 0 0 176624 0 0 0 0 0 0 0 4945 100 0 0 0 0
23 0 0 176792 0 0 0 0 0 0 0 4842 100 0 0 0 0

iostat 1 (needs to be stopped by Ctrl + C):

Linux 2.6.18-028stab068.3 (raa.raagvamdatt.com) 08/20/2010

avg-cpu: %user %nice %system %iowait %steal %idle
72.24 0.04 1.57 0.19 0.00 25.96

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

avg-cpu: %user %nice %system %iowait %steal %idle
97.00 0.00 3.00 0.00 0.00 0.00

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

avg-cpu: %user %nice %system %iowait %steal %idle
93.07 0.00 6.93 0.00 0.00 0.00

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

avg-cpu: %user %nice %system %iowait %steal %idle
100.00 0.00 0.00 0.00 0.00 0.00

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

avg-cpu: %user %nice %system %iowait %steal %idle
93.00 0.00 7.00 0.00 0.00 0.00

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

avg-cpu: %user %nice %system %iowait %steal %idle
92.00 0.00 8.00 0.00 0.00 0.00

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

avg-cpu: %user %nice %system %iowait %steal %idle
93.07 0.00 6.93 0.00 0.00 0.00

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
quan

Rang: Profi

Registriert: 16. Dez 2004

Letzter Besuch: 26. Apr 2012

Beiträge: 1144
Verfasst: 20. Aug 2010 - 13:51

Hi,

the only thing that is done while the module is initiated is that it creates a system init hook. maybe there is still something left in the hook tables.

As far as I know the module does not make any troubles... I use the module at multiple sites and one of it is a site with really high (email) traffic.

Maype a system trace (strace) for the process will help you to get some information what's going wrong.
FLo

{NetBiker.de} {individuelle Webapplikationen}
raagv

Rang: gerade reingestolpert

Registriert: 20. Aug 2010

Letzter Besuch: 23. Aug 2010

Beiträge: 4
Verfasst: 21. Aug 2010 - 14:57

Hi FLo,

Thanks for your inputs... I anyway restored the backed up database, so nothing should be there in the hook table. But just to be sure, I checked the hook table - didn't find anything related to advmailer that I could identify (no entry with "pn_tmodule" as advmailer).

I understand the module doesn't make any trouble - but I was foolish to install it on Zikula 1.1.2, which I guess has resulted in the issue

There are multiple instances of index.php running, each with a separate process ID. I tried running strace for a couple of them and tried storing the result in a text file, but each resulted in a blank text file (0 Bytes). (I don't know much about Linux commands - I ran strace by doing research on the net. So, I might not have done it right!)

You think the problem can be something other than advmailer? The hostgator folks also told me that there are some crons running (though I can't see anything listed in my cPanel) - does advmailer add any crob jobs?

Would really appreciate any help, as this thing is killing me (and my server )

Thanks...
raagv

Rang: gerade reingestolpert

Registriert: 20. Aug 2010

Letzter Besuch: 23. Aug 2010

Beiträge: 4
Verfasst: 23. Aug 2010 - 08:08

Hi,

Here's the strace output that the hostgator people gave me:

19885 04:38:27 access("config/languages/eng/global.php", F_OK) = -1 ENOENT (No such file or
directory)
19885 04:38:27 access("config/languages/eng/global.php", F_OK) = -1 ENOENT (No such file or
directory)
19885 04:38:27 access("modules/SecurityCenter/pnuserapi.php", F_OK) = -1 ENOENT (No such fil
e or directory)
19885 04:38:27 access("modules/SecurityCenter/pnuserapi.php", F_OK) = -1 ENOENT (No such fil
e or directory)
19885 04:38:32 stat("config/languages/eng/SecurityCenter/common.php", 0x7fffaff24b50) = -1 E
NOENT (No such file or directory)
19885 04:38:32 stat("system/SecurityCenter/pnlang/eng/common.php", 0x7fffaff24b50) = -1 ENOE
NT (No such file or directory)
19885 04:38:32 stat("config/languages/eng/SecurityCenter/common.php", 0x7fffaff24b50) = -1 E
NOENT (No such file or directory)
19885 04:38:32 stat("system/SecurityCenter/pnlang/eng/common.php", 0x7fffaff24b50) = -1 ENOE
NT (No such file or directory)
19885 04:38:32 stat("config/languages/eng/SecurityCenter/userapi.php", 0x7fffaff24b50) = -1
ENOENT (No such file or directory)

Does this help?

Thanks...
Mr.Montesa

Rang: Support-Team

Registriert: 24. Sep 2006

Letzter Besuch: 24. Mai 2012

Beiträge: 614
Verfasst: 23. Aug 2010 - 21:42

Your hardware analysis showed me, what I already expected.
Your PHP binary is crashing constantly. If you are sure that there havent been any changes on the system (OS/Patch installations/etc), I suggest to re-upload a backup of your Zikula installation to replace all files.
Perhaps while installing advmailer, a file was placed in the folder structure that your environment cant handle.
This could result in system crashes.

The strace output shows several missing files. Can you doublecheck that they are available & readble by the webserver user?
Second, you can try to replace them with the original files from the Download package.

Third, it looks like you dont have root access to you server. Perhaps you can ask your hoster to provide logfiles.
Important ones are
/var/log/messages
/var/log/apache2/error_log
and the output of the command
Code
dmesg

BR Sebastian

Edit:
I did a short resarch on internet in regard to your strace output. It's not nesseccarily bad behavior.
For example if you'll do:
Code
strace which ps

this will walk through the PATH variable and gives "file not found" alerts for many folders. Of corse. The ps binary is only available in one of the folders.

Same applies for our language files. They can be placed in some different folders. In the Theme or in the modules. While walking through the possible places, strace will output errors, if a possible file location is empty.

In fact, I suggest, copy & paste a full strace log to pastebin.ca.

Bearbeitet von Mr.Montesa am 23. Aug 2010 - 21:52 Uhr.

Gruß Sebastian

Tweet with me