Connect. Communicate. Collaborate. Securely.

Home » Kerio User Forums » Kerio Connect » Negative TOTAL_SCORE in Junk Filter (getting too much spam)
  •  
Adam Wunn

Messages: 6
Karma: 0
Send a private message to this user
We are getting more and more spam. When I spoke to Kerio Tech support about it last year, they suggested that because our "total" scores are negative that I should reset the bayes filter. So I did so and that worked like a charm, but fast forward a year and I repeated the same process, yet it isn't working this time.

This is an example of a good email that we received today. The total score is still a negative. If what I understood from the tech support rep, then this is not good. However, I've sifted through a lot of email and most of the good ones have a negative score.

X-Spam-Status: No, hits=0.0 required=4.1
tests=AWL: 0.000,BAYES_00: -1.665,HTML_MESSAGE: 0.001,
RDNS_NONE: 0,TOTAL_SCORE: -1.664,autolearn=ham

This is an example of a spam that wasn't picked up, but was moved manually to the Junk Email folder. Same story.

X-Spam-Status: No, hits=0.0 required=4.1
tests=AWL: 0.102,BAYES_00: -1.665,RDNS_NONE: 0,
TOTAL_SCORE: -1.563,autolearn=ham

The final example is a spam that was very obvious but while it received a score much higher it still isn't high enough.

X-Spam-Status: No, hits=4.0 required=4.1
tests=BAYES_99: 4.07,HTML_MESSAGE: 0.001,MIME_HTML_ONLY: 0.001,RDNS_NONE: 0,TOTAL_SCORE: 4.072,autolearn=no

I know that there is a training period with the bayes filter, but I am also using the black lists (added several of my own) to increase the score in most cases. I also use spam repellent and rdns checks. I also added some custom filters to look for certain phrases. My Spam rating is now set to 4.4. I can lower it more, but I was concerned that it would pick up more legitimate email, but if most legitimate emails have a negative score then, it might make some sense if that is the word.

Some help would be appreciated on the question of whether a negative score is acceptable as a TOTAL_SCORE value and what to do about it if that is an issue.

We are running Kerio Connect 7.4.1 on Mac OS X.
  •  
Pavel Dobry (Kerio)

Messages: 5245
Karma: 251
Send a private message to this user
It seems that neither DNS blacklists nor SURBL tests have been applied to these messages. This can happen when the server is behind a firewall with port mapping or behind another SMTP frontend server, which hides real sender's IP address or when DNS is not working properly. I recommend to check debug logs from Spam Filter and SMTP server.
  •  
MacLab

Messages: 216
Karma: 15
Send a private message to this user
Pavel, can you clarify if a negative total score in itself is a problem? I've seen this happen consistently and the AWL is adding the negative score. Here's an example from debug.

[31/May/2012 08:51:53][2955440128] {spamassassin} Perl_stderr: [17367] dbg: auto-whitelist: db-based jsmith<_at_>gmail.com|ip=74.125 scores 0/0
[31/May/2012 08:51:53][2955440128] {spamassassin} Perl_stderr: [17367] dbg: auto-whitelist: db-based jsmith<_at_>gmail.com|ip=none scores 0/0
[31/May/2012 08:51:53][2955440128] {spamassassin} Perl_stderr: [17367] dbg: auto-whitelist: AWL active, pre-score: -1.665, autolearn score: -1.665, mean: undef, IP: 74.125.82.175
[31/May/2012 08:51:53][2955440128] {spamassassin} Perl_stderr: [17367] dbg: auto-whitelist: add_score: new count: 1, new totscore: -1.665

[Updated on: Sun, 03 June 2012 14:02]


MacLab, Inc.
Kerio Certified Partner, Reseller, Hosting Provider, Kerio Connect Certified.
http://maclaboratory.com
  •  
MarkK

Messages: 454
Karma: 46
Send a private message to this user
After suffering what I thought was too much spam getting through, looking at the SpamAssissin items that were hit and scored, and researching what those hits meant, I ended up editing my SpamAssissin Local.cf file and "fixing" some of the scores that were way too low.

Some of my scores:
RDNS_NONE 0.5
HTML_MESSAGE 0.5
MIME_HTML_ONLY 1.5

So by my rules, one of your emails would of scored another 2.5. I'm using a tag score of 5 and a block of 8. You have to be careful in that you don't overrate some of the hits and start causing good emails to be marked as bad.

I haven't upgraded to KC 7.4.x yet, so I don't know if my local.cf is compatible with the upgraded SpamAssissin. Also, you have to make sure that you keep a copy of your edited local.cf file elsewhere, since Kerio upgrades will overwrite it with a default one since Kerio does not endorse the use of an edited local.cf, even though that IS what it is for.
  •  
Adam Wunn

Messages: 6
Karma: 0
Send a private message to this user
Pavel,

We are indeed behind a firewall. We are not using another SMTP server. I checked the debug logs and see nothing unusual. It lists basic mundane information about what Connect is doing (like timezone checks). I see no seemingly relevant information.

The DNS not being right is a possibility. We run our own DNS server locally on the network, but our DNS and RDNS names resolve correctly. The only difference is that on the outside dns our public IP is referenced and internally the DNS uses the non-routable LAN IP. Could that have something to do with it?
  •  
Adam Wunn

Messages: 6
Karma: 0
Send a private message to this user
I have something else I should have added earlier. I have my mailstore is a custom location. I updated the config file per the instructions in the knowledge base for the mailstore folder. Is there a path I need to change to point spamassasin to the correct folder for the bayes database? Currently I can see the file timestamp for the bayes database is being updated as mail comes in so it seems to be updating.
  •  
MacLab

Messages: 216
Karma: 15
Send a private message to this user
No location config is necessary for spam assassin. And having a local IP in itself shouldn't be a problem. If you look in the security log are you seeing connection rejections based on the RBL lists?

MacLab, Inc.
Kerio Certified Partner, Reseller, Hosting Provider, Kerio Connect Certified.
http://maclaboratory.com
  •  
Adam Wunn

Messages: 6
Karma: 0
Send a private message to this user
Yes I can indeed see many entries referring to the lists for rejected email based upon ip address.

Here is a sampling (the actual log has thousands of entries):

[11/Jun/2012 16:13:57] IP address 186.54.139.104 found in DNS blacklist SpamHaus SBL-XBL, mail from <bounce-154904-9600@isp-equipment.com> to <info<_at_>xxxxxxx.com> rejected
[11/Jun/2012 16:14:03] IP address 178.151.48.217 found in DNS blacklist SpamCop, mail from <brenzetti@centermorichesfd.com> to <karie<_at_>xxxxxxx.com> rejected
[12/Jun/2012 21:48:09] IP address 98.138.90.253 found in DNS blacklist Unsubscore, mail from <uptonjulie@yahoo.com> to <karen<_at_>xxxxxxx.com>
[12/Jun/2012 22:01:51] IP address 121.247.162.119 found in DNS blacklist SpamCop, mail from <x.gcvhvx@yahoo.com> to <kate<_at_>xxxxxxx.com> rejected
[14/Jun/2012 07:12:21] IP address 208.67.182.42 found in DNS blacklist SpamHaus SBL-XBL, mail from <dennie@dfw-reviews.com> to <elaine<_at_>xxxxxxx.com> rejected
[14/Jun/2012 07:12:48] IP address 204.45.103.101 found in DNS blacklist Surriel, mail from <bounce-9686-4142696138-karie=xxxxxxx.com@classylogo.com> to <karie<_at_>xxxxxxx.com> rejected
[15/Jun/2012 07:29:42] IP address 61.106.202.120 found in DNS blacklist DNSBL 1, mail from <SariahBiemer@sunline.net> to <george<_at_>xxxxxxx.com>
[15/Jun/2012 13:16:07] IP address 209.223.236.140 found in DNS blacklist SpamHaus SBL-XBL, mail from <Alyss@mta.plasticshawaii.net> to <bob<_at_>xxxxxxx.com> rejected
[15/Jun/2012 13:16:27] IP address 69.175.77.221 found in DNS blacklist Surriel, mail from <findinfo@lssbetul.org> to <bob<_at_>xxxxxxx.com> rejected
[15/Jun/2012 13:17:13] IP address 77.122.158.143 found in DNS blacklist SpamCop, mail from <JamariSuares@hense.freeserve.co.uk> to <kate<_at_>xxxxxxx.com> rejected
[16/Jun/2012 08:52:38] IP address 87.14.81.147 found in DNS blacklist SpamHaus SBL-XBL, mail from <anestheticc84@ultimauk.com> to <karie<_at_>xxxxxxx.com> rejected
[16/Jun/2012 08:53:19] IP address 114.79.141.205 found in DNS blacklist SpamCop, mail from <charitably6@4dmsillustrations.com> to <bob<_at_>xxxxxxx.com> rejected
  •  
MarkK

Messages: 454
Karma: 46
Send a private message to this user
"The total score is still a negative. If what I understood from the tech support rep, then this is not good. However, I've sifted through a lot of email and most of the good ones have a negative score."

I don't agree with the comment about the scores should not be negative. I don't think the tech understands how Bayes works. The structure of the Bayes rating starts with rating an email that it finds to not be spam (regardless if it is spam in the real world or not (see below)) is a BAYES_00 rating giving the email score of -1.665. The way Bayes works, this means that after examining the words in the email, it really thinks this is a true good email and that whoever setup the scoring level thinks that the Bayes score should override some of the other spam scoring that will happen.

So with a BAYES_00 and a score of -1.665, it thinks it is a good email. With a BAYES_99 and a score of 4.07, it is sure it is spam.

Good email:
"Did you pickup my prescription Joe's pharmacy"

Spam:
"Pickup your prescriptions at Joe's pharmacy"

Here is where I personally think Bayes falls short. The key words listed in those will be prescription and pharmacy. To a real person, one is a good email and one is a spam. After watching our Bayes spam ratings on both good and spam emails, I have seen where because of the way spam is being built today that the Bayes rates an obvious spam as being good. The Spammer uses proper sentence structure and doesn't always use the obvious spam phrases.

This is why I suggested earlier to look at the email's individual spam test that were applied and what each individual score was. Then look up on Spam Assassin's web site to see what triggers that filter hit, and adjust the Spam Assassin's score for that item in the local.cf file.

To me, the default Spam Assassin scoring that has scores of 0.001 are worthless. I think you have a spam rating limit of 4.4. That means that it would take 4,400 of these low rated tests to hit that limit. That would be one fantastic spam to hit that many tests; I would want to see that spam.

HTML_MESSAGE: 0.001
The email has HTML included in message. Typically people do not send HTML emails. I type an email and it is sent either text or RTF. HMTL emails are typically sent by companies trying to deliver ads. I gave this a rating of .5 versus the default .001

MIME_HTML_ONLY: 0.001,
Format used to combine resources that are typically represented by external links (such as images, Flash animations, Java applets, audio files) together with HTML code into a single file, BUT lacks the plain text alternative part. After watching our emails, I only saw this happen on real spams. I gave this a rating of 1.5 (high), versus the default .001

RDNS_NONE: 0
This test checks to see if there is a reverse DNS entry for the last untrusted relay. Valid email senders will have reverse DNS entries setup for their domain. But after watching our emails, it is pretty common on spams but is not 100% true with valid emails, some valid ones would get hit by this test. I gave this a rating of .5 versus the default 0

http://wiki.apache.org/spamassassin/AdjustRuleScore
According to Spam Assassin themselves, you should monitor the tests and scores that were applied to the emails and adjust accordingly.

As I mentioned before, if you make changes in the local.cf file to create your custom levels of ratings, keep a backup copy of your changed local.cf. The Kerio Connect upgrade will overwrite the local.cf file with a default one at every upgrade, wiping out the changes you have made. If you have a backup one, you can simply put it back in place and restart Kerio.

You can also turn on the debug logging for spam processing and see what is happening.

Just my (long) .02¢ opinion...

[Updated on: Sat, 16 June 2012 18:48]

  •  
Pavel Dobry (Kerio)

Messages: 5245
Karma: 251
Send a private message to this user
MarkK wrote on Sat, 16 June 2012 18:45

HTML_MESSAGE: 0.001
The email has HTML included in message. Typically people do not send HTML emails. I type an email and it is sent either text or RTF. HMTL emails are typically sent by companies trying to deliver ads. I gave this a rating of .5 versus the default .001


Actually, almost all emails use HTML in these days (usually with plaintext alternative). RTF is very rare, So, increasing score for this test will significantly increase probability of false positive result.
  •  
MarkK

Messages: 454
Karma: 46
Send a private message to this user
I made that decision based on looking at both good and spam emails. My threshold is set high enough that the .5 rating on a good html email will not hurt it, but with the other hits on spams it will help catch them.
  •  
Adam Wunn

Messages: 6
Karma: 0
Send a private message to this user
I am sorry it took so long to respond. MarkK was very important to help me understand what is happening. Updating the local.cf file is genius and I totally agree with the changes he suggests. Mark bless you for your "long 2¢". It was very helpful to me.
  •  
MarkK

Messages: 454
Karma: 46
Send a private message to this user
Glad to help! I hate spam. Every couple of months I will catch emails from my users that get that the most spam and look to see what the current trends are and if I can make more custom adjustments.

DON'T FORGET, keep a current backup copy of your custom local.cf, otherwise all of those needed changes will disappear the next time you update. I had read on a posting here that supposedly you could just create a new .cf file in that folder and it would be used, but I don't think that was the case when I tried to do that. So I keep a copy of the local.cf in another folder, make my changes to that one, and then copy it to the correct folder and restart Kerio Connect.
  •  
Adam Wunn

Messages: 6
Karma: 0
Send a private message to this user
MarkK wrote on Sat, 16 June 2012 18:45


HTML_MESSAGE: 0.001
The email has HTML included in message. Typically people do not send HTML emails. I type an email and it is sent either text or RTF. HMTL emails are typically sent by companies trying to deliver ads. I gave this a rating of .5 versus the default .001...


MarkK,

I wasn't able to find this option in the local.cf file in Kerio 7.4.x. Could that have been removed now as an option?
  •  
MarkK

Messages: 454
Karma: 46
Send a private message to this user
It is located in one of the other rule files. Local.cf is processed after the other rules. All you need to do its add the new value to the bottom section of the file. Just copy user the same format as the other value lines, and add your new override.

This is why I suggest looking at the spams and seeing which triggers were rated from the other rule files. If you learn spamassassin language, youcould add your own evaluation rules and scores.
Previous Topic: Sharing calendars with wives
Next Topic: AD logins fail
Goto Forum:
  


Disclaimer:
Kerio discussion forums are intended for open communication between forum members and may contain information and material posted by members which may be useful in learning about Kerio products. The discussion forums are not intended to provide technical support for any specific product. Any information implied or expressed in the discussion forums is that of the posting member. Kerio is in no way responsible for the information posted in the forums, or its accuracy. Kerio employees may participate in the discussions, but their postings do not represent an offical position of the company on any issues raised or discussed. Kerio reserves the right to monitor and maintain the forums to promote free and accurate exchange of information.

Current Time: Thu Aug 24 05:17:29 CEST 2017

Total time taken to generate the page: 0.00559 seconds
.:: Contact :: Home ::.
Powered by: FUDforum 3.0.4.