Connect. Communicate. Collaborate. Securely.

Home » Kerio User Forums » Kerio Connect » Performance issues with Exchange IMAP in Apple Mail
  •  
miless

Messages: 32
Karma: 2
Send a private message to this user
The short version:

Using Exchange IMAP accounts in Apple Mail with the current version of Kerio Connect can kill performance to the point of unusability. (Non-Exchange) IMAP accounts should be used instead.

The long version:

We're an all-Mac business with about 130 users, all of whom use Snow Leopard, Apple Mail and iCal. We switched from Communigate Pro to Kerio Connect 7 last fall. We decided to set our users up with Exchange IMAP accounts in Mail to get around Mail's tendency to use folders with names different than those used by Kerio webmail (Sent Messages vs. Sent Items, for example).

Performance was fine during initial testing, but as we added users it progressively declined. When we'd transitioned all of our users' accounts, we would see long periods (especially early in the day when our employees were first logging in) characterized by up to multiple-hour delays in loading mailboxes and sending/receiving messages. Users reported frequent 993 timeout errors in Mail. If a user moved or deleted a large quantity of messages, Mail would respond unpredictably - sometimes the messages would be duplicated in both the source and destination mailboxes; sometimes they would reappear in the source mailboxes hours later.

We ameliorated the performance issues somewhat by switching our storage volume from RAID 5 to RAID 10, but performance still tanked early in the day when employees were arriving and whenever any user performed a large move or delete operation.

We'd been working with Kerio support from the start but hadn't made much progress. During our own testing, however, we found that every time a user loaded a mailbox, every single .eml file was being retrieved. (This was confirmed by running fs_usage on our Kerio server and grepping for a username while that user switched mailboxes.) This seemed odd, as we were already aware of Kerio's reliance on index files, which would presumably be the only thing that needed to be referenced upon a mailbox load. We set up a non-Exchange, plain-old IMAP account and did the same test: only index files and new/updated .eml files were loaded.

We went through the laborious process of manually switching all of our users from Exchange IMAP to IMAP, and sure enough, our performance issues disappeared almost entirely. Kerio support later confirmed that this was a known issue:

"The problem with Apple mail Exchange account is that it searches for the content-class header of every message. This is not indexed on Kerio, causing it to check every single mail file. This could cause drastic performance degradation if a large network has deployed exchange mode to all users."

Hopefully this will help people avoid the months-long problems we ran into. I've confirmed that the problem still exists in Kerio Connect 7.2.0; here's hoping it's fixed soon.
  •  
TorW

Messages: 769
Karma: 9
Send a private message to this user
Thanks for the heads up. While we only have about half a dozen Apple machine accessing us via "Exchange IMAP" accounts, this is useful knowledge.

However, I'm unsure what you mean by Quote:
I've confirmed that the problem still exists in Kerio Connect 7.2.0

Isn't the problem in Apple Mail? Would be interesting to know how (or if) an MS Exchange server handles this.

On a side note: is it just me, or is Apple Mail and iCal becoming more and more broken with regard to enterprise use?
  •  
miless

Messages: 32
Karma: 2
Send a private message to this user
Quote:
However, I'm unsure what you mean by
Quote:
I've confirmed that the problem still exists in Kerio Connect 7.2.0

Isn't the problem in Apple Mail? Would be interesting to know how (or if) an MS Exchange server handles this.


Sort of - the issue seems to be that Kerio doesn't currently index the content-class header of messages, which means that it can't just rely on the index files for a mailbox and instead must fall back on checking each and every .eml file. If Kerio started indexing that header, Exchange IMAP accounts would theoretically work without problems. Kerio support tells me that fix is in the works, but they didn't have an ETA.

I hear you regarding Apple Mail, iCal, and the enterprise. I'm crossing my fingers that Lion helps, but Apple's made it quite clear that the enterprise isn't a primary focus. We shall see...
  •  
kevinneal

Messages: 19
Karma: 0
Send a private message to this user
Do you know of any disadvantages of using IMAP instead of Exchange IMAP?
  •  
miless

Messages: 32
Karma: 2
Send a private message to this user
The only Kerio-relevant disadvantage I know of is that IMAP uses Apple's nonstandard names for certain mailboxes:

Sent Messages (should be Sent Items)
Deleted Messages (should be Deleted Items)

That can be fixed on a per-user basis by selecting the correct (xxx Items) mailbox and going to Mailbox menu -> Use This Mailbox For -> Sent/Trash. If not fixed, the account will use xxx Messages when accessed through Mail and xxx Items when accessed through webmail, thus splitting those mailboxes.
  •  
bigmountain

Messages: 116

Karma: 0
Send a private message to this user
I have done extensive testing and reporting to Kerio. They believe they have a way to optimize the Exchange IMAP accounts. They claim the difference between the two is that Exchange IMAP call for a "content class" header. They submitted a request to the developers to add this as cached info into the properties.fld file which they believe will help solve the problem. Aside from this, they said that Exchange IMAP will provide access to notes, to-dos, etc. in addition to the regular mail folders. Also, as stated above, Mail does use a different set of folders (Junk, Trash and Sent), but you can highlight the correct folder to use as default (Sent Items, for example) go to Mailbox in the menu and select Use this Folder for and select the appropriate folder. This way, Mail setup as IMAP will use the same folders as Kerio. Also, here is the settings I have been using for my clients who have had performance issues with Exchange IMAP:

Setup account as IMAP
Make sure IDLE command is checked (in advanced tab in the account settings)
Uncheck the box to disable the account from being included when checking for new mail (in advanced tab)
Select from the drop down (also in advanced tab) to Download the entire message, but omit attachments

I have found this setup to be very effective after a lot of trial and error with both clients and some test accounts I setup. Let me know if you have any questions. Post your experience with this setup.

Preferred Kerio Partner and Cloud Solutions Provider - Offering both shared and dedicated Kerio Connect hosting solutions.
Visit us at http://bigmountainmail.com
  •  
miless

Messages: 32
Karma: 2
Send a private message to this user
bigmountain, thanks for your informative reply!

I tried setting up an account the way you described (I'm assuming you meant "Setup account as Exchange IMAP" when you wrote "Setup account as IMAP"). Using:

fs_usage | grep accountname

I found the same results I discovered earlier: the Exchange IMAP account loads every .eml file in a directory when switching to the corresponding mailbox, while (non-Exchange) IMAP just loads the various .fld files.

I also checked out Notes and To-Do items. It looks like Notes don't work on either type of account (I create a note and press Done, but the note doesn't appear anywhere) and To-Do items work on both types of accounts.

Based on the .eml and .fld behavior, we'll probably stick with IMAP accounts and wait until the content-class header is added to properties.fld before considering Exchange IMAP again.
  •  
Tony Dennis

Messages: 7
Karma: 2
Send a private message to this user
This issue is purportedly corrected in KC 7.2.1. Anyone finding this to be (or not) the case?
  •  
AaronAdams

Messages: 1
Karma: 0
Send a private message to this user
bigmountain wrote on Tue, 07 June 2011 07:56
Aside from this, they said that Exchange IMAP will provide access to notes, to-dos, etc. in addition to the regular mail folders.


I don't mean to hijack this thread, but do you mean that Exchange IMAP provides access to the Notes function of Apple Mail?
  •  
uptimjeff

Messages: 61
Karma: 1
Send a private message to this user
I'm also wondering if this is fixed in 7.2.1

I'm currently faced with switching some users from Exchange-IMAP to standard IMAP.
If 7.2.1 fixes this, I won't have to do make that switch and will continue to use Exchange-IMAP due to its easier/automatic config of folders.

  •  
macinfosys

Messages: 100
Karma: 1
Send a private message to this user
Can anyone confirm if this has been resolved on 7.2.1 or 7.2.2 ? I know 7.3 Beta 2 has just been released and the next question would be if this has been resolved in 7.3 if not in 7.2.1/2.

  •  
miless

Messages: 32
Karma: 2
Send a private message to this user
According to fs_usage, the issue is not fixed in Kerio Connect 7.2.2. Every .eml file in a mailbox is still loaded every time the user switches to a mailbox.
  •  
schwifty

Messages: 32
Karma: 0
Send a private message to this user
I was unable to replicate this in 10.6.8 or 10.7 Mail.app when configured as Exchange IMAP against KC 7.2.2.

Running fs_usage on the mailserver process while grepping the username, then launching, relaunching, and switching mailboxes in Mail.app always only loaded the indices and the occasional new message.

However, in limited testing with Lion 10.7 regular IMAP against KC 7.2.2, I would see mailbox contents download repeatedly in their entirety to the client. In short, I witnessed the exact reverse of what is being described here. Is there a way to get the behavior you are observing to repeat reliably?
  •  
miless

Messages: 32
Karma: 2
Send a private message to this user
That's very strange - I'm not sure why we'd be seeing such different behavior.

I'm currently just testing using Mail in 10.7, but I'm guessing this applies to 10.6 Mail as well. I'll describe in greater detail what I'm seeing and perhaps that will be useful.

IMAP

Switch from Inbox to Misc MIS folder in Mail using miless account
On mail server, as root: fs_usage | grep miless

12:01:34  open              /Volumes/mailvol/Kerio/mail/darkhorse.com/miless/Misc MIS/search.fld             0.000016   mailserver  
12:01:34  open              /Volumes/mailvol/Kerio/mail/darkhorse.com/miless/Misc MIS/search.fld             0.000007   mailserver 


Exchange IMAP

Switch from Inbox to Misc MIS folder in Mail using miless account
On mail server, as root: fs_usage | grep miless

Hundreds of lines like the following:

12:02:13  open              /Volumes/mailvol/Kerio/mail/darkhorse.com/miless/Misc MIS/#msgs/00000b3f.eml     0.000010   mailserver  
12:02:13  open              /Volumes/mailvol/Kerio/mail/darkhorse.com/miless/Misc MIS/#msgs/00000b3e.eml     0.000010   mailserver  


Note that these tests were performed after selecting Mailbox -> Synchronize All Accounts in Mail and waiting for the process to finish as reflected by the Activity window. If messages aren't already cached, their initial caching will make it look like indexes aren't being used.
schwifty

Messages: 32
Karma: 0
Send a private message to this user
I just reconfirmed that it does indeed behave for my 10.7 IMAP Exchange client.

Complete logs from switching from folder Drafts, to Sent, to MacUpdate (a user-created mail folder):

14:18:41  open	/store/mail/example.com/exampleuser/Drafts/search.fld    0.000048   mailserver  
14:18:41  open	/store/mail/example.com/exampleuser/Drafts/search.fld    0.000020   mailserver  
14:18:43  open	/store/mail/example.com/exampleuser/Drafts/sort.fld    0.000050   mailserver  
14:18:43  open	/store/mail/example.com/exampleuser/Drafts/deleted.fld    0.000016   mailserver  
14:18:46  open	/store/mail/example.com/exampleuser/Sent Items/search.fld    0.000044   mailserver  
14:18:46  open	/store/mail/example.com/exampleuser/Sent Items/search.fld    0.000018   mailserver  
14:18:52  open	/store/mail/example.com/exampleuser/Sent Items/sort.fld    0.000055   mailserver  
14:18:52  open	/store/mail/example.com/exampleuser/Sent Items/deleted.fld    0.000017   mailserver  
14:18:52  open	/store/mail/example.com/exampleuser/MacUpdate/search.fld    0.000049   mailserver  
14:18:52  open	/store/mail/example.com/exampleuser/MacUpdate/search.fld    0.000018   mailserver


Have you tried killing your local Envelope Index in Mail.app? There are three:

rm ~/Library/Mail/V2/MailData/Envelope*

[Updated on: Tue, 16 August 2011 21:27]

Previous Topic: Uninstall AD Extensions
Next Topic: bayesian filter reset to 0
Goto Forum:
  


Disclaimer:
Kerio discussion forums are intended for open communication between forum members and may contain information and material posted by members which may be useful in learning about Kerio products. The discussion forums are not intended to provide technical support for any specific product. Any information implied or expressed in the discussion forums is that of the posting member. Kerio is in no way responsible for the information posted in the forums, or its accuracy. Kerio employees may participate in the discussions, but their postings do not represent an offical position of the company on any issues raised or discussed. Kerio reserves the right to monitor and maintain the forums to promote free and accurate exchange of information.

Current Time: Tue Oct 24 09:36:10 CEST 2017

Total time taken to generate the page: 0.00535 seconds
.:: Contact :: Home ::.
Powered by: FUDforum 3.0.4.