Connect. Communicate. Collaborate. Securely.

Home » Kerio User Forums » Kerio Connect » Backups for Large servers?
  •  
mcholdings

Messages: 24
Karma: 0
Send a private message to this user
Hi all,

We have a LOT of mail. 285 mailboxes, 846G of mail (as of this morning) over 2.7 million files.

Kerio backup can't deal with it - takes over 2 days to complete a backup, and mostly fails.

Rsync is very slow - I can make it work (takes me 5 nights to get a baseline...), but, very slow.

I tried a few custom systems and not succeeding. Anyone else in this position? What are you using? i'm looking to see what others are doing as well as looking in the market.

I've recently moved to a physical machine with the mail stored on raid 10 disks (virtual had too many disk bottlenecks). I am wondering if we have grown too far for the solution.... We are still single server (we only have a single head office\data centre with 20 small branches), but it's pretty fast now (dual Xeon Dell with raid 1 main disk, raid 10 data store, 24G ram which is more than enough...)
  •  
My IT Indy

Messages: 1262
Karma: 40
Send a private message to this user
What are you backing up to where it takes 2 days to complete?

-
My IT Indy
Kerio Certified Reseller and Hosted Provider
http://www.myitindy.com
  •  
mcholdings

Messages: 24
Karma: 0
Send a private message to this user
I backup to a dedicated NAS. It doesn't seem to be the actual volume of data, but the number of individual files. I have an identical NAS that does the backups for my other servers, and it takes 100's of gigs a day no problem. The issue seems to be writing 2.7 million files....
  •  
My IT Indy

Messages: 1262
Karma: 40
Send a private message to this user
You might consider external storage for your backups. That would be faster than over the network.

-
My IT Indy
Kerio Certified Reseller and Hosted Provider
http://www.myitindy.com
  •  
Machete

Messages: 262
Karma: 5
Send a private message to this user
I've seen the problem mcholdings is describing - transferring/saving large amounts individual files is much slower than fewer large files. I've never found a good solution when I have the problem and just attribute it to read/write the start/end of each file. But I'm no expert to know why it's slower.

Taking HoosierMac's idea - can you connect your NAS direct to the server? I'm pretty sure the most recent Xeon box I got from Dell had a SATA channel. Some NAS devices have USB3.0 as well on them for direct connectivity.
  •  
sfpete

Messages: 165
Karma: 11
Send a private message to this user
Agreed x3.

You'll want your local backup going to fast direct attached storage to maximize speed and minimize the window.

And then of course back _that_ built-in kerio backup data to your preferred backup system.

  •  
sim72

Messages: 6
Karma: 0
Send a private message to this user
Maybe it is not the worst idea. What if you use more than one rsync task?
You split the mail boxes. From A to H one rsync, from I to P another rsync, ....
  •  
dpswatson

Messages: 3
Karma: 0
Send a private message to this user
I'm kind of in the same boat with a similarly-sized message store. I gave up on the built-in Kerio backup method because it took too long and caused too much of a performance hit (plus the only way to interrupt it is to restart the Kerio server)

I'm currently using an SSH-based backup method from another linux VM over a private LAN link, which seems to get good backups most of the time (and it doesn't kill performance), but it still takes far too long, IMHO.

So, after you get your rsync baseline, does it stay in sync OK? I'm thinking of switching to using rsync between two servers running on different boxes, and backing up the second box only. That would also make the second box viable as an offline spare, eliminating the need to restore if the primary server failed.
  •  
mcholdings

Messages: 24
Karma: 0
Send a private message to this user
Now I have my baseline, it is staying in sync (almost gigabit link to a remote site via 24Ghz Wireless). It did take 3 days to get the baseline right though.

I think the issue is 'files', rather than space. I can transfer a terrabyte over the link no problems in a short time - it is the simple volume of files being stored.

I'm looking also at a disk based imaging backup - which in theory removes the files barrier. I don't think local storage would be much faster than rsync - I'm not touching my bandwidth limits, rather disk throughput.

A note though - I was only raid 1 before - now I am raid 10 it is much better for this size store.
  •  
RoadKingRick

Messages: 24
Karma: -1
Send a private message to this user
I would like to weigh in here. If it helps one person.......
When our store was just about 1 TERABYTE we were backing up via KMS to a single external eSATA 3GB/s drive and the full backup was taking well over two days.
We switched to a 6GB/s internal eSATA card with an external fast eSATA RAID5 and the first full backup took 23 hours! Less than HALF the time.
Our store has crept up to 1.5 TB and the backup is taking longer, well over a day, but the fast eSATA might be a very viable alternative to those with a smaller store.

I can't stress the difference the 6GB/s controller card and fast external RAID5 made!

I have not tried USB3 yet, but I (think) I read somewhere that eSATA better handles the particular type of data that KMS backup produces (repeated,2 GB chunks).

Has anyone timed backing up the store to an external SAS LTO tape drive?
  •  
giobbi

Messages: 90
Karma: 0
Send a private message to this user
RoadKingRick; Could you post your setup?

I'm in the same position, same amount of files. Since mail is the single most important in any company maybe i should look at more proper hardware with blockbased backup like Netapp (snapshots). This kind of backup only backup the new blocks witch is lightspeed fast. If im not mistaking - new software (DSM 5) from synology has this now (with iscsi?) for a fraction of cost..

  •  
MarkK

Messages: 454
Karma: 46
Send a private message to this user
Are you running a complete backup every time? Or are you doing a base plus incremental?

You may want to take a look at Retrospect's backup product. It is easy to use, the latest version does block level backups, does file level deduplication, can be set to run at any interval (nightly, weekly, every X minutes, etc), and will backup to just about anything. There is a Windows and Mac version, both can backup clients running Windows, Mac, or Linux. Cost is actually very reasonable too.

As others have said, the time factor is complicated by the fact that you are backing up thousands of small files that add up to a lot of data, rather than a few large files. So the fastest drive write connection is going to be of great benefit, whether that be a SATA or eSATA or USB3 connected storage.
Previous Topic: how to edit error messages
Next Topic: Mac Outlook 2011. Identity recover on every reboot!
Goto Forum:
  


Disclaimer:
Kerio discussion forums are intended for open communication between forum members and may contain information and material posted by members which may be useful in learning about Kerio products. The discussion forums are not intended to provide technical support for any specific product. Any information implied or expressed in the discussion forums is that of the posting member. Kerio is in no way responsible for the information posted in the forums, or its accuracy. Kerio employees may participate in the discussions, but their postings do not represent an offical position of the company on any issues raised or discussed. Kerio reserves the right to monitor and maintain the forums to promote free and accurate exchange of information.

Current Time: Sun Oct 22 08:29:00 CEST 2017

Total time taken to generate the page: 0.00542 seconds
.:: Contact :: Home ::.
Powered by: FUDforum 3.0.4.