Connect. Communicate. Collaborate. Securely.

Home » Kerio User Forums » Kerio Connect » Differential Backups Seem to Miss Emails
  •  
jcooper

Messages: 65
Karma: 3
Send a private message to this user
Hello All,

I'd like a reality check before I submit this as a rather serious bug. This weekend I moved our KMS to a new server. I did the migration using "Method 1" from here:

http://kb.kerio.com/product/kerio-connect/server-configurati on/export-and-migration/transferring-an-installation-of-keri o-connect-to-another-server-or-operating-system-360.html

A full backup takes about 15 hours and it's hardly practical to shut down my mail server for that long. So, I did a full backup earlier in the week (Tuesday). My plan was to do a Differential backup the morning of the move, then use kmsrecover to pull the data into the new server.

The live server was on during the full backup earlier in the week. Prior to the differential backup, I turned off all the services to effectively "stop" emails but keep the admin UI working so I could schedule the backup (our gateway held the incoming messages while KMS was off).

To recover the data, I used >
./kmsrecover /volumes/mailbackup/backup


I did a couple test runs prior to the move and it all seemed to go well; it took about 90 minutes to recover the data after which I turned the server on and all seemed fine. A test email I sent to myself just prior to the differential backup appeared on the new server. I should have looked closer.

No sooner had I completed the actual migration that I started getting calls from users who were missing emails. A comparison of the two servers indicated that most (but not all) emails for the two or three days prior to the migration did not migrate.

We only have 50 or so users, and we're on Mac, so I was able to mount the old server as a share and use Carbon Copy Cloner to manually copy the missing contents of each user's store, reindex the mailbox, and they came back (iOS users had to turn their mail accounts off then on again for the missing emails to show up). "Only" took a few hours.

In the end the migration worked, albeit with a few panic attacks. But I think this problem means there's something seriously wrong with Kerio's backup system.

Like most places, we rely on these backups in case of a server meltdown, fire, etc. I have the full backup off-site and take the differentials off-site regularly. My assumption was if the server room exploded we'd have all our emails up to the last backup. But now it seems we have our emails except for the couple days leading up to the last backup.

I did find a couple threads alluding to others having similar issues, but with no clear resolution. Did I forget a needed option for kmsrecover or to check a box somewhere?

Thanks,

Jeff



  •  
greg_m

Messages: 65
Karma: 0
Send a private message to this user
Hi, did you get to the bottom of this? I'm seeing the same thing. We're trying to migrate from an old Xserve to a new HP Prolient server and after a test run I found that some, but strangely not all emails, were missing from the few days before the backup was run.

I've spoken to Kerio support and they admit that the recovery process is not 100% reliable, and say we will just have to accept the loss of the emails.

This has seriously dented my confidence in Kerio's backup and recovery tools. We are now considering migrating to hosted Exchange or Google as we can't trust KC anymore, but I just wanted to see if I'd missed anything.

Thanks
  •  
Pavel Dobry (Kerio)

Messages: 5245
Karma: 251
Send a private message to this user
"Recovery process is not reliable"? It is reliable and we are not aware of any problem with missing emails.
Backup always contains all the data from a time when the backup was created.
  •  
jcooper

Messages: 65
Karma: 3
Send a private message to this user
Didn't get a resolution. Moved on once I was able to manually migrate the missing messages. It's been a while so I was re-reading my post a bit more objectively (as if someone else had written it) and it now occurs to me perhaps the restore command doesn't work as I thought it did (or how I read it in the docs). With differential backups, I'm wondering if they need to be restored independently of base full backup and/or explicitly selected. Perhaps my backup command only grabbed the last full backup and if there are differential they need to be restored by running the command again, but explicitly specifying the D...zip file.

Just a theory and I just thought of it five minutes ago; other projects took over space in my brain following the migration. So I hadn't really thought of it since.

If I get some time in the coming weeks I may run a test to see what happens. If and when I do, I'll post the results.

Thanks,

Jeff
  •  
Mark Price (Kerio)

Messages: 136
Karma: 17
Send a private message to this user
Hi There Jeff

to be able to do a complete recovery, you must have both the last full and last differential in the same folder. KMSrecover then uses both of these to recover from.

I hope this helps.

Mark Price
Kerio technical support.

Log Support Incidents here: http://www.kerio.com/support
Also, please use our KB: http://kb.kerio.com
  •  
greg_m

Messages: 65
Karma: 0
Send a private message to this user
Hi Jeff, thanks for getting back to me, I appreciate it was sometime ago. As I understand it pointing kmsrecover to the backup folder will cause it to use the latest full plus any diffs associated with it. You can specify a particular backup set by using the filename in the command.

The command I used was;

kmsrecover -v -s D:\store E:\[backup directory]

This reported a successful recovery. However, there were messages missing. The important thing that I noticed, and you noticed too, was that NOT ALL emails were missing. If the full backup was performed on day 1, the incremental on day 2 and the recovery on day 3, then if the incremental had not been included in the restore, than you would expect to see only emails from day 1 on the restored server. We saw some emails from day 2, but not all. For instance, one user had emails from 9am in the morning, but some from 11am were missing. This is after I had reindexed all mailboxes, so the messages reported by the server should be correct.

I took screen shots of the user list showing the total item counts before and after the reindex (attached) and there were differences of up to several hundred files.

Mark/Pavel

Your confidence in the backup/recovery process is admirable and comforting, so perhaps you can help me, as my experience (and perhaps Jeff's) seems to contradict your claims? We can't reliably migrate from an old server to a new one right now. This is understandably causing us a headache. I've tried first line support and the advice given is that there isn't much I can do about these missing files, so if the wider Kerio userbase or second line support have any good ideas as to how I can fix this, I'm all ears!

Having spent £6,000 on a nice shiny new HP server, I'm not looking forward to telling my MD we can't move away from the 7 year old server that is failing.
  •  
greg_m

Messages: 65
Karma: 0
Send a private message to this user
It has been pointed out by Scott Riegelhaupt-Herzig (Kerio's Technical Director in the US) that I have misrepresented Kerio's UK support team on this issue and I have to agree with him, to a point. Kudos to him for writing to me directly to clarify the matter.

I made the assertion that UK support had told me that the recovery process was not 100% reliable, a statement that I must retract. What the support representative actually said was "Sadly there is no real way to determine how an index has become corrupted after a migration like this. Sometimes it can be to do with the change in OS and how the OS is reading the files, if it locks the files, if there are too many emails in the folder the index belongs too etc... But there is no 100% answer as to what causes this issue.". So not quite the same thing, but still enough for me to be concerned about.

And this still doesn't alter my position that I am unable to complete a reliable recovery from a backup whilst migrating from a Mac to a Windows server. Perhaps there are factors relating to my environment (and possibly Jeff Cooper's) that have caused this, but the essence of the support response was that there was little they could do to help. Hence the reason for coming to these forums for further help and insight.

I wish Kerio would invest some money into bringing the recovery tool into the management console and expanding its capabilities. Using the command line tool has required a lot of work, searching and support tickets to explain various errors. Real time replication to a second backup server would be great. None of this old zip file nonsense. I could then put my KC server in the cloud and keep a second instance running to cover me in case of outage or corruption. Kerio have consistently told me I should never use any other backup tool on the live mail store, otherwise I'd use Veeam or something similar.
Previous Topic: Drafts folder not syncing
Next Topic: Problem with Kerio client web mail
Goto Forum:
  


Disclaimer:
Kerio discussion forums are intended for open communication between forum members and may contain information and material posted by members which may be useful in learning about Kerio products. The discussion forums are not intended to provide technical support for any specific product. Any information implied or expressed in the discussion forums is that of the posting member. Kerio is in no way responsible for the information posted in the forums, or its accuracy. Kerio employees may participate in the discussions, but their postings do not represent an offical position of the company on any issues raised or discussed. Kerio reserves the right to monitor and maintain the forums to promote free and accurate exchange of information.

Current Time: Mon Aug 21 10:14:30 CEST 2017

Total time taken to generate the page: 0.00588 seconds
.:: Contact :: Home ::.
Powered by: FUDforum 3.0.4.