Connect. Communicate. Collaborate. Securely.

Home » Kerio User Forums » Kerio Connect » Major Performance Problems, please help!!!
  •  
mzimmermann

Messages: 7
Karma: 0
Send a private message to this user
Hi all!

We have major performance problems running our Kerio MailServer and we get no helpful support by Kerio directly (through the support tickets) or anyone else. So give it a try here:

Here's the whole story (I try to give you as much informations I can, so you know enough to give me advice):

The former CIO of our company startet the Kerio project. The project parameters were as follows:

About 150 Mailboxes:
- 110 Users whitch uses all services (Mail, Contacts, Calendar, WebMail etc.), quota set to 2GB (with 8 exeptions from 2 - 7 GB)
- 10 Mail-only Users (they only use IMAP, POP3S or WebMail), quota set to 2GB
- 10 other Mail-only accounts (like mail<_a.t_> etc.)
- 20 Resources
3 Domains
70% Entourage Users (2004 and 2008)
20% Outlook Users (all using the offline connector)
10% iCal/Mail Users
About 50 - 60 Mobile Phones (at the Moment about 40 iPhones and some Nokias)

He decided to use an "old" Xserve G5 as Server with the following configuration (I think because of the requirements from the KMS product website):

2 x 2.3GHz G5 Processors
6GB RAM
2 x 1TB SATA Harddrives running as a Software RAID 1
Mac OS X 10.5 Server
KMS 6.5
No other Services running on this machine, only KMS

The Migration-Plan was as follows:
1. User by User Integration of the first domain (using POP3 download in the background from the old Mailserver) about 80% of all users, only for Mail-Service at the beginning
2. Switch off the old Mailserver
3. Integrate the other two domains (20% of all users)
3. Train users to use the KMS as Groupware
4. Switch over from MeetingMaker (our former Calendar Solution) to Kerio as Groupware (Calendar etc.)

Everything went fine at the beginning. User by user was successfully integratetet, the performance was great! At some day, he (the former CIO) left the company and passed the project over to our team.

After Step 2 (as the old Mailserver was shut down), about 3 weeks ago, we added the other two domains to our configuration for our 2 Inhouse partner companies (because we have to share our calendars and use the same resources etc.) and startet to integrate these employees (about 20% of the user summary). At the same time, we startet to train the users to use the calendar and we defined a transition week (in fact this week) from MeetingMaker to Kerio, so that until today, all users have to use KMS for their appointments.

From this time on (3 weeks ago), the performance of the server was going rapidly down. The CPU startet to go up to 100%, as soon as the users logged in in the morning and began to work (the 100% CPU load is only comming from the "mailserver" process). After they went home, the CPU went down again. Update to 6.6 doesn't improved the situation. I postet a picture with the CPU curve of the last 7 days on the Xserve (see CPU-Xserve.png). From 9 am to 5 pm, using Entourage, Outlook, WebMail or what ever, was no fun at all! Long respond times, email delivery (even in the same domain) take up to 12 hours. Some Mails where delivered up to 12 Times with "no" reasons, appointments whitch has been accepted had never been confirmed etc. etc. It got worser every day. Still no usefull feedback from the Kerio Support. Yesterday, we had to restart the service twice, because nothing was going anymore.

As we don't knew where the problem is (is it a configuration issue, the G5 Architecture, not enough CPU, to slow disks etc. etc.?), got now useful help from anywhere, users went crazy and lost their confidence towards KMS, no better better server is available in our company and delivery of a better one would take to long at the moment, we decided to move the server this night temporarily to an available Mac Pro with the following configuration:

1 x Intel Xeon Quad Core 2.66
5GB RAM
2 x 1TB SATA Harddrives running as a Software RAID 0
Mac OS X 10.5 Server
KMS 6.6
No other Services running on this machine, only KMS

The result is, that the CPU still is on 100% from 9 am to 5 pm (except lunch time), but the services seems to respond a lot faster (see CPU-MacPro.png). But this can't be the final solution.

Futher informations: we have now 8 users with more than 10'000 objects (from 10'000 to 30'000) and about 17 with 5'000 - 10'000 objects. All other users have less than 5'000 objects in their mailbox.

And here are my questions:

- Could it be a configuration issue or one single user that causes the load?
- Is just the hardware to weak (I think the SATA Disks are to slow, but why is the CPU on 100%)?
- How can I find it out?
- I used the debug mode and following IMAP Message appears over 600 times per second:

[07/Nov/2008 14:18:19][3007229952] {imaps} Command 2649192.127 FETCH 1 (FLAGS UID)
[07/Nov/2008 14:18:19][3007229952] {imaps} Sent 0 fetch responses

is he waiting for Disk I/O or what?

- What would be an adequate server for, lets say, 200 Users in our environement (as described above)? I need some hard facts for my boss (and I have to be shure, that it is not a config or user issue) to order a server that costs more than 10'000$ Wink

Any suggestion would be very appreciated!

Thanks in Advance!

Marco

  •  
Nixs

Messages: 159
Karma: 0
Send a private message to this user
We support over double the users on hardware which has similar specs to yours, but we use SCSI not SATA and Windows not OS X. Quad core Intel 3Ghz, 4GB RAM.

Hardware wise; SATA puts a higher load on the CPU, whereas SCSI offloads a lot of the processing from the CPU to the SCSI card. We have many smaller disks so we have more heads to seek with.

Mailserver rarely goes over 20% during the day.

Anyway, that's our boat. My guess is it's something that is corrupt, or a setting - not hardware. In Windows there are tools which go through the mailstore making sure everything's inline. If you manually mess with the files, you can cause duplicate unique identifiers which KMS doesn't like. There are tools to fix these issues if you go through support. Again, no idea on Mac though.

Also, do you defrag? I know they Mac says you don't need to - but that's not true if you have a lot of files which are > 20MB. OS X HFS only defrags smaller files. Anything larger, and you'll need 3rd party defrag tools.

http://www.kernelthread.com/mac/apme/fragmentation/ Note the part "If the file is less than 20 MB in size"

Defraging on KMS on Windows makes a HUGE difference on stores that haven't been defraged in a long time.

  •  
My IT Indy

Messages: 1262
Karma: 40
Send a private message to this user
I have set up servers double your size for users and CPU utilization barely breaks 10-20% during the day.

Something I would see about is moving the mailstore to another volume, say an external raid array. I don't trust Apple's built-in software RAID at all. Hardware raid is exponentially better for reliability and performance.

[Updated on: Sat, 08 November 2008 00:00]


-
My IT Indy
Kerio Certified Reseller and Hosted Provider
http://www.myitindy.com
  •  
elias

Messages: 114
Karma: 0
Send a private message to this user
I fully agree that he needs to fix his disk subsystem, but I don't think that's the issue here. I think this is what he needs to track down first:

Quote:

- I used the debug mode and following IMAP Message appears over 600 times per second:

[07/Nov/2008 14:18:19][3007229952] {imaps} Command 2649192.127 FETCH 1 (FLAGS UID)
[07/Nov/2008 14:18:19][3007229952] {imaps} Sent 0 fetch responses


Sounds to me like there's a misbehaving client on the network and finding it would be my first priority.

-Elias
  •  
My IT Indy

Messages: 1262
Karma: 40
Send a private message to this user
Running Activity Monitor to see what's causing the spikes would help.

Also, watching the Error log, the Operations log, and Debug log will help.

-
My IT Indy
Kerio Certified Reseller and Hosted Provider
http://www.myitindy.com
  •  
Pavel Dobry (Kerio)

Messages: 5245
Karma: 251
Send a private message to this user
elias wrote on Sat, 08 November 2008 00:11

I fully agree that he needs to fix his disk subsystem, but I don't think that's the issue here. I think this is what he needs to track down first:

Quote:

- I used the debug mode and following IMAP Message appears over 600 times per second:

[07/Nov/2008 14:18:19][3007229952] {imaps} Command 2649192.127 FETCH 1 (FLAGS UID)
[07/Nov/2008 14:18:19][3007229952] {imaps} Sent 0 fetch responses


Sounds to me like there's a misbehaving client on the network and finding it would be my first priority.

-Elias



Sounds like known issue with IMAP client in iPhone. I strongly recommend immediate upgrade to KMS 6.5.2.

Also, besides the number of messages in each mailbox, there is also another important factor: number of folders.
  •  
Pavel Dobry (Kerio)

Messages: 5245
Karma: 251
Send a private message to this user
Kerio_pdobry wrote on Sat, 08 November 2008 00:26


Quote:

- I used the debug mode and following IMAP Message appears over 600 times per second:

[07/Nov/2008 14:18:19][3007229952] {imaps} Command 2649192.127 FETCH 1 (FLAGS UID)
[07/Nov/2008 14:18:19][3007229952] {imaps} Sent 0 fetch responses


Sounds like known issue with IMAP client in iPhone. I strongly recommend immediate upgrade to KMS 6.5.2.



After some investigation it seems to be another issue. I think it is important to find out which IMAP client (and user) is causing such wrong behavior. Please check also all other log files.
  •  
generic_penguin

Messages: 45
Karma: 10
Send a private message to this user
Hardware fine for what you want to do

1: Turn the Mail Server off and delete the spam assassin folder which by default will be located at /usr/local/kerio/mailserver/plugins. Download Kerio nd run a repair install. This will re-install the spam assassin directory

If still issues

2: Check / post all log files. Check connections in graph (anything strange there like 1000's of connections every second on any particular protocal (IMAP HHTPS etc..)

Also make sure no 12.1.3 or 12.1.4 entourage users (Still issues etc..)

3: Drop into terminal $top / or activity monitor and get CPU readings. What process is taking all the CPU

  •  
mzimmermann

Messages: 7
Karma: 0
Send a private message to this user
Hi all!

Thank you for all of your responses!

I finally got a response from the Kerio 2nd level support, I think we are analysing a core-file today and I hope we will find the Problem..

<_a.t_>nix: I didn't found any tools for Mac OS X similar to those you use on Windows. I want to do these checks but don't know how... And, we don't do defrag till today, but our installation is just some month of age, so I don't think that's the problem.

<_a.t_>HoosierMac: I would also like to see that! But I have no such hardware available and I don't think that this is the main problem here and I need a solution whitout hardware change. But I will buy a new hardware as soon as possible - of course with a hardware RAID controller.
,
<_a.t_>elias: I totaly agree! Or a mailbox with corrupt files! But how can I find it out?

<_a.t_>HoosierMac: As I wrote, only the mailserver process is causing this "spikes". And all the Log Files didn't helped me so far, I may just don't know how to use it effectively (especially the debug log with al those modules).

<_a.t_>Kerio_pdobry: As I wrote, we have installed KMS 6.6 about 1.5 Weeks ago and nothing has changed, it even got worse (but not because of the update I think). And yes, there are a lot of folders in the users mailboxes, but I demand that KMS is able to handle so many folders! Exchange doesn't has a problem with this amount of folders and if KMS is a true Exchange alternative, it has to be possible - without a server cluster I mean Smile A know that Exchange is DB based and KMS uses a file structure, but I think this CPU load can't come from to many folders...

<_a.t_>generic_pengui: I switched off SPAM processing some days ago (for troubleshooting reasons), but the CPU load remained, so I don't think it has to do with spam assassin. The connection graph is not strange. I'll keep checking the log files and if I find something usefull, I'll post it here. And we only have 4 Entourage 2008 in use, and as far as I know, they all run Version 12.1.2, but I'll check it today! And again, the process "mailserver" is taking all of the CPU


Does any one know, what kind of hardware would be suitable for our purpose (as soon as the real problem is solved)? I thought of a Xserve (Quad or Eight core), but the problem are the disk drives! The Xserve has only three slots. So if I'll do a RAID 1 with SAS disks (with RAID Controller of course), I only get 300GB, and thats not enough. RAID 5 with 3 disks would end up with 600GB disk space (that's enough), but I think the performance is not that good. So I would have to buy an external RAID, witch is to expensive. Another alternativ would be a Mac Pro, with 4 300GB SAS-Drives, RAID Controller running a RAID 10 which also ends up with 300GB. What do you think?
  •  
sfpete

Messages: 165
Karma: 11
Send a private message to this user
We've had great success and performance deploying a "cheaper" server option for some environments that don't need or can't afford the full Xserve + external RAID.


-Mac Pro tower (single quad)
-4 internal 1TB SATA Hard Drives in RAID5 config with the Apple RAID controller card.

This will net you out about 2.4TB of usable storage.

Buy and install the drives yourself (not from Apple... which are really expensive)
  •  
marook

Messages: 520

Karma: 3
Send a private message to this user
Hi,

We just moved to Kerio a few weeks ago, and here are our setup:

180 Users
All on Mac, some 20-30 iPhones via ActiveSync.
Mail.app, iCal, webmail.. all IMAP (POP is disabled)
Total storage: 210GB incl. OS X Server 10.5.5

Disks is 2x500GB SAS in Mirrored RAID, via software.

Hardware is:
Model Identifier: Xserve1,1
Processor Name: Dual-Core Intel Xeon
Processor Speed: 3 GHz
Number Of Processors: 2
Total Number Of Cores: 4
L2 Cache (per processor): 4 MB
Memory: 5 GB
Bus Speed: 1.33 GHz

CPU is spiking at 10% when the company is really at work.

Ohh, and the same hardware is also running a CommuniGate Pro mailserver (on another IP) with 20 domains and arounf 250 users.

I think you are looking at some file-level corruption, or a very bad-acting client (or more) - or maybe both.


Regards,

Jakob Peterhänsel
Consultant - Humac A/S

Apple Certified Support Professional (ACSP)
Apple Certified Technical Coordinator (ACTC)
AppleSeed/CQF member since 1998
Kerio Messaging Partner
  •  
Pavel Dobry (Kerio)

Messages: 5245
Karma: 251
Send a private message to this user

Quote:

- I used the debug mode and following IMAP Message appears over 600 times per second:

[07/Nov/2008 14:18:19][3007229952] {imaps} Command 2649192.127 FETCH 1 (FLAGS UID)
[07/Nov/2008 14:18:19][3007229952] {imaps} Sent 0 fetch responses



I assume this log comes from KMS 6.6.0. It looks some IMAP client is very confused and tries to get non-existent message over and over. I think it could be caused by bug or improper IMAP implementation in the client. The solution could be disabling autodelete in IMAP. Stop KMS, edit the mailserver.cfg file and change the value 'AutoExpungeOnDelete' to '0'.
  •  
mzimmermann

Messages: 7
Karma: 0
Send a private message to this user
Kerio_pdobry wrote on Tue, 11 November 2008 12:42


Quote:

- I used the debug mode and following IMAP Message appears over 600 times per second:

[07/Nov/2008 14:18:19][3007229952] {imaps} Command 2649192.127 FETCH 1 (FLAGS UID)
[07/Nov/2008 14:18:19][3007229952] {imaps} Sent 0 fetch responses



I assume this log comes from KMS 6.6.0. It looks some IMAP client is very confused and tries to get non-existent message over and over. I think it could be caused by bug or improper IMAP implementation in the client. The solution could be disabling autodelete in IMAP. Stop KMS, edit the mailserver.cfg file and change the value 'AutoExpungeOnDelete' to '0'.


Yes, this log comes from KMS 6.6.0. But it only appears ocassionally, and yesterday, as we had again this high load (yesterday only for a few hours!), this message didn't appeared. It's strange. I really think it is just one or two user who causes this high CPU load, but I didn't found them till today!!! It is hard to control who's working and who's not. So I have to wait till the phenomenon appears and then check the log files etc...
  •  
RHarmsen.nl

Messages: 186

Karma: 0
Send a private message to this user
Do you use the KOFF Outlook plugin?

If so, could it be that some/many of your users are still busy with initial syncing their mail?
mzimmermann

Messages: 7
Karma: 0
Send a private message to this user
RHarmsen.nl wrote on Wed, 12 November 2008 10:57

Do you use the KOFF Outlook plugin?

If so, could it be that some/many of your users are still busy with initial syncing their mail?


Yes, we use KOC Offline for all of our Outlook Users. But I don't think that this is the issue, because we had to resync (clear the cache) from one user on monday, and that was not a problem, the CPU load didn't increase at all.
Previous Topic: Sony Ericsson C510 ActiveSyncRequest Error
Next Topic: Kerio iCal subscriptions in iPhone 3.0
Goto Forum:
  


Disclaimer:
Kerio discussion forums are intended for open communication between forum members and may contain information and material posted by members which may be useful in learning about Kerio products. The discussion forums are not intended to provide technical support for any specific product. Any information implied or expressed in the discussion forums is that of the posting member. Kerio is in no way responsible for the information posted in the forums, or its accuracy. Kerio employees may participate in the discussions, but their postings do not represent an offical position of the company on any issues raised or discussed. Kerio reserves the right to monitor and maintain the forums to promote free and accurate exchange of information.

Current Time: Fri Nov 24 04:45:50 CET 2017

Total time taken to generate the page: 0.00510 seconds
.:: Contact :: Home ::.
Powered by: FUDforum 3.0.4.