--------------------------------------------------------------------------- June 29, 2008 amavisd-new-2.6.1 release notes BUG FIXES - avoid a bounce-killer's false positive when a message is multipart/mixed with an attached message/rfc822 (looking like a qmail or a MSN bounce) and having attached a message with a foreign Message-ID - by restricting the check to messages with an empty sender address or a 'postmaster' or 'MAILER-DAEMON' author address; - privileges were dropped too early when chrooting, causing chroot to fail (a workaround was to specify a jail directory through a command line option -R); reported by Helmut Schneider; - fix unwarranted 'run_av error: Exceeded allowed time' error when using a virus scanned Mail::ClamAV; reported by Chaminda Indrajith; - fix a bug in helper-progs/amavis-milter.c where atoi could be reading from a non-null terminated string which could result in wrong milter return status, or even cause a read-access violation; reported by Shin-ichi Nagamura; - dsn_cutoff_level was ignored if SpamAssassin was not invoked (e.g. on large messages) even if recip_score_boost was nonzero, causing a DSN not to be suppressed for internally generated large score values; reported by Bernd Probst; - add back the 'Ok, id=..., from MTA(...):' prefix to a MTA status responses on forwarded mail when generating own SMTP status response (it was lost in code transition from 2.5.4 to 2.6.0); reported by Thomas Gelf; - replaced '-ErrFile=>*STDOUT' with '-ErrFile=>\*STDOUT' in a call to BerkeleyDB::Env::new in amavisd-nanny and amavisd-agent; seems it was failing in some setups (even though it was in accordance with a BerkeleyDB module documentation); reported by Leo Baltus; - README.sql-mysql: fixed a SQL data type mismatch between maddr.id (used as a foreign key) and msgs.sid & msgrcpt.rid; they all should be of the same type, either integer unsigned or bigint unsigned; a schema as published in README.sql-mysql could not be built because of a conflict in a data type; reported by Leonardo Rodrigues Magalhães and Zhang Huangbin; NEW FEATURES - recognize an additional place-holder %P in a template used to build a file name in file-based quarantining, for example: $spam_quarantine_method = 'local:Week%P/spam/%m.gz'; A %P is replaced by a current partition tag, which makes it easier to better organize a file-based quarantine by including a partition tag (e.g. an ISO week number) in a file name or a file path. For the record, here is a complete list of place-holders currently recognized in filename templates: %P => $msginfo->partition_tag %b => $msginfo->body_digest %m => $msginfo->mail_id %n => $msginfo->log_id %i => iso8601 timestamp of a message reception time by amavisd %% => % The following example organizes spam quarantine into weekly subdirectories: cd /var/virusmails mkdir -p W23/spam W24/spam W25/spam ... (weeks 01..53) chown -R vscan:vscan W23 W24 W25 ... (weeks 01..53) amavisd.conf: $spam_quarantine_method = 'local:W%P/spam/%m.gz'; $sql_partition_tag = sub { my($msginfo)=@_; sprintf("%02d",iso8601_week($msginfo->rx_time)) }; - add a macro %P as a synonym for a macro 'partition_tag', mainly for completeness with the added place-holder %P in a file name template; OTHER - disabled a do_ascii decoder in the default @decoders list: # ['asc', \&Amavis::Unpackers::do_ascii], # ['uue', \&Amavis::Unpackers::do_ascii], # ['hqx', \&Amavis::Unpackers::do_ascii], # ['ync', \&Amavis::Unpackers::do_ascii], The do_ascii is invoking a module Convert::UUlib which in turn calls a troublesome library uulib, which has a history of security problems and on occasion misinterprets a text file as some encoded text, causing false positives (e.g. making it look like an executable); recent false positive on base64-decoding reported by Jeffrey Arbuckle; recent DoS (looping in uulib) reported by Thomas Ritterbach; - added a rule into $map_full_type_to_short_type_re to cope with another example of misclassification by a file(1) utility, where a plain text file is considered a DOS executable: [qr/^DOS executable \(COM\)/ => 'asc'], # misclassified? An example was provided by Leonardo Rodrigues Magalhães; - until the issue is better understood, revert the use of 'my_require' and go back to the standard but less informative 'require'; some people were reporting problems with my_require (loading of some Perl modules can fail, apparently depending on a current directory where amavisd is started from); reports by Tuomo Soini, Max Matslofva, Bill Landry; - use the $myproduct_name value in generated Received header field instead of a hard-wired 'amavisd-new'; suggested by Thomas Gelf; - added missing required header fields to some test mail messages in a directory test-messages to quench down a complaint about a bad header; - changed SQL default clauses in %sql_clause (upd_msg, sel_quar, sel_penpals) to always join tables using both the partition_tag and the mail_id fields, previously just the mail_id field was used in a join. The change has no particular effect (and is not really necessary) on existing 2.6.0 databases where a primary key is mail_id (it is just a redundant extra condition), but saves a day when a primary key is a composite: (partition_tag,mail_id), which may be a requirement of a SQL partitioning mechanism. Thanks to Thomas Gelf for his testing of MySQL partitioning, reporting deficiency in amavisd SQL schema (primary keys) which did not meet MySQL requirements for partitioning, and suggestions; - an AM.PDP release request can specify an additional optional attribute: partition_tag=xx where a requester can supply a partition_tag value of a message to be released. This helps to uniquely identify a message in case where an SQL database did not enforce a mail_id field to be unique (as may be necessary with some partitioning schemes). If a partition_tag information is readily available to a requester, it is advised that the attribute is included in a request even if mail_id is known to be unique. This may expedite a search and provide a double check to a validity of a request. For backwards compatibility amavisd performs a query on msgs.mail_id for a partition_tag value if it is missing form a request, the query uses an SQL clause in a new entry $sql_clause{'sel_msg'}. If exactly one record matches, then everything is fine, and releasing may proceed. If multiple records with the same mail_id exist the release request is aborted with a message asking user to supply a disambiguating partition_tag=xx attribute; - a quarantine id for an SQL-quarantined message as logged in a main log entry is changed from: quarantine: aX3C4f6btXgX to: quarantine: aX3C4f6btXgX[25] i.e. a partition_tag in brackets is appended to mail_id. Correspondingly the amavisd-release is also changed to be able to parse 'aX3C4f6btXgX[25]', splitting it into mail_id and partition_tag, and providing each as a separate attribute in an AM.PDP release request; - README.sql-mysql: changed SQL datatype VARCHAR into VARBINARY for data fields mail_id, secret_id and quar_loc, and CHAR into BINARY for msgs.content and msgs.quar_type to preserve case sensitivity on string comparison operators; suggested by Thomas Gelf; The same change should eventually be done on README.sql-pg too, but as PostgreSQL is more picky than MySQL on matching a field data type to a supplied data value, the change of a data type would need to be reflected in SQL calls in amavisd. This will have to wait until some future version of amavisd-new, having to undergo more testing than I have available before the 2.6.1 release. Background information on UNIQUE constraint in table SQL msgs Amavisd does not know and need not be aware of what is a primary key or what are UNIQUE constraints in SQL table msgs. When generating a mail_id for a message being processed, amavisd tries to INSERT a record with a randomly generated mail_id into table msgs (using SQL clause in $sql_clause{'ins_msg'}). If the operation fails, another mail_id is generated and attempt repeated, until it eventually succeeds. Thus it depends entirely on SQL's decision whether a particular record is allowed or would break some UNIQUE constraint. So, by only changing a declaration on table msgs (PRIMARY KEY or adding a CONSTRAINT), it changes what keys amavisd will be allowed to insert and what kind of duplicates would be allowed. Classically the msgs.mail_id is a PRIMARY KEY and as such it is unique. This was a requirement for versions of amavisd up to and including 2.6.0. Starting with 2.6.1 the JOINs have been tightened to include a partition_tag besides mail_id in a relation, which makes it possible to loosen a unique requirement on msgs.mail_id and only require a pair (partition_tag,mail_id) to be unique. In other words, this way the mail_id is only needed to be unique within each partition tag value. This change allows a partitioning scheme to meet requirements on MySQL partitioning. For non-partitioned databases the change shouldn't make any difference, and one is free to choose between having mail_id unique across the entire table or just within each partition_tag value. Changing a primary key to (partition_tag,mail_id) brings consequences to quarantining, in particular to releasing from a SQL quarantine, where it no longer suffices to specify mail_id=xxx in AM.PDP request, but may be necessary to specify also a partition_tag=xx to distinguish between SQL-quarantined messages which happen to have the same mail_id. --------------------------------------------------------------------------- April 23, 2008 amavisd-new-2.6.0 release notes MAIN NEW FEATURES SUMMARY - integrated DKIM signing and verification; see section A QUICK START TO DKIM SIGNING by the end of this release note; - loading of policy banks based on valid DKIM-signed author's address can be used for reliable whitelisting, for bypassing banned checks, etc. - bounce killer feature: uses a pen pals SQL lookup to check inbound DSN; - SQL logging and quarantining tables have a new field 'partition_tag'; - captures SpamAssassin logging, more flexibility specifying SA log areas; - collects and logs SpamAssassin timing breakdown report (requires SA 3.3); - releasing from a quarantine can push a released message to an attachment; - new experimental code for abuse reporting using formats: ARF/attach/plain; - TLS support on the SMTP client and server side; - connection caching by a SMTP client; - amavisd-nanny and amavisd-agent now re-open a database on amavisd restarts; - amavisd-nanny and amavisd-agent new command line option: -c count; - updated p0f-analyzer.pl to support source port number in queries; - amavisd can send queries either to p0f-analyzer.pl or directly to p0f; COMPATIBILITY WITH 2.5.4 - when using SQL for logging (e.g. for a pen pals feature) or for quarantining, SQL tables tables maddr, msgs, msgrcpt and quarantine need to be extended by a new field 'partition_tag'; see below for details; - when SQL logging (pen pals) or SQL lookups are used, one can choose a binary or a character data type for fields users.email, mailaddr.email, and maddr.email; now may be a good opportunity to change a data type to binary (string of bytes); see below for details; - when using SQL for logging, a default for $sql_clause{'upd_msg'} has changed, so if a configuration file replaces this SQL clause by a non-default setting, it needs to be updated; - perl module Mail::DKIM is now required when DKIM verification or signing is enabled or when spam checking by SpamAssassin is used and a DKIM plugin is enabled; a required version of this module is 0.31 (or later); - because privileges are now dropped sooner, pid and lock files as generated by Net::Server can no longer be located in a directory which is not writable by UID under which amavisd is running (e.g. /var/run). A location of these files is controlled by $pid_file and $lock_file settings, and by default are placed in $MYHOME, which still satisfies the new requirement; - white and blacklisting now takes into account both the SMTP envelope sender address, as well as the author address from a header section (address(es) in a 'From:' header field). Note that whitelisting based only on a sender-specified address is mostly useless nowadays. For a reliable whitelisting see @author_to_policy_bank_maps below, as well as a set of whitelisting possibilities in SpamAssassin (based on DKIM, SPF, or on Received header fields); - if using custom hooks, some of the internal functions have changed, in particular the semantics of a method orig_header_fields - use new functions get_header_field() or get_header_field_body() instead; see updated sample code amavisd-custom.conf, and see entries labeled 'internal' below; - a configuration variable $append_header_fields_to_bottom is now obsolete; the variable is still declared for compatibility with old configuration files, but its value is ignored: new header fields are always prepended, i.e. added to the top of a header section; - semantics of a command line option 'debug-sa' has changed due to a merge of SpamAssassin logging with a mainstream amavisd logging mechanism. A command 'amavisd debug-sa' is now equivalent to 'amavisd -d all' with an implied redirection of all logging to stderr. Previously it only rerouted SpamAssassin logging to stderr but did not affect normal amavisd logging, which still followed the usual $DO_SYSLOG and $LOGFILE settings. Also, a SpamAssassin log level 'info' is now turned on by default (as was previously achievable by a command line option '-d info'), and shows merged with a normal amavisd logging at level 1 or higher. The following table shows mapping of SpamAssassin log levels to amavisd log levels, and for completeness also shows mapping of amavisd log levels to syslog priorities (which has not changed since previous version): SA amavisd syslog ----- ------- ----------- -3 LOG_CRIT -2 LOG_ERR error -1 LOG_WARNING warn 0 LOG_NOTICE info 1 LOG_INFO 2 LOG_INFO dbg 3 LOG_DEBUG 4 LOG_DEBUG 5 LOG_DEBUG - an additional requirement for loading a policy bank 'MYUSERS' is that 'originating' flag must be on, which typically means that mail must be coming from internal networks or from authenticated roaming users to be able to load a policy bank 'MYUSERS'; BUG FIXES - run_av: limit the number of filenames given as arguments to a command line scanner to stay within a safe (POSIX) program argument space limit, run a command line scanner multiple times if necessary. This required a larger change in the program (run_av, ask_av) which is why the fix was listed for a long time on a TODO list and not implemented so far. The problem affected command line virus scanners which are unable to traverse a directory by themselves and need a list of filenames as arguments (such as KasperskyLab's aveclient and kavscanner, MkS_Vir mks, and CyberSoft VFind). Actual problem reported by Danny Richter; NEW FEATURES - DKIM signing and verification - see below: A QUICK START TO DKIM SIGNING. Not to forget upgrading Mail::DKIM to 0.31 (or later) and adding the following to amavisd.conf; $enable_dkim_verification = 1; $enable_dkim_signing = 1; - SQL tables tables maddr, msgs, msgrcpt and quarantine are extended by a new field 'partition_tag'. When amavisd creates new records in these tables, a current value of a configuration variable $sql_partition_tag (or its value from policy banks) is written into 'partition_tag' fields. An undefined value translates to 0. The 'partition_tag' field is usually declared in a schema as an integer, but in principle could be any data type, such as a string. A value of 'partition_tag' field may be used to speed up purging of old records by using partitioned tables (MySQL 5.1 +, PostgreSQL 8.1 +). A sensible value is a week number of a year, or some other slowly changing value, allowing to quickly drop old table partitions without wasting time on deleting individual records. Records in all tables carrying the 'partition_tag' field are self-contained within each value of a field. In other words, foreign keys never reference a record in a subordinate table with a value of a 'partition_tag' field different from the referencing record. Consequently, mail addresses in table maddr are also self-contained within a partition tag, implying that the same mail address may appear in more than one maddr partition (using different 'id's), and that tables msgs and msgrcpt are guaranteed to reference a maddr.id within their own partition tag. Too fine a granularity of partition tags (e.g. changing a value daily) wastes space in table maddr by storing multiple copies of the same mail address. The $sql_partition_tag may be a scalar (usually an integer or a string), or a reference to a subroutine, which will be called with an object of type Amavis::In::Message as argument (giving access to information about a message being processed), and its result will be used as a partition tag value. Possible/typical usage (in amavisd.conf): $sql_partition_tag = sub { my($msginfo)=@_; iso8601_week($msginfo->rx_time) }; yields an ISO 8601 week number (1..53) corresponding to a mail reception timestamp in a local time zone. Another possible use of 'partition_tag' field is to let a policy bank set its specific value (a fixed value or a subroutine) for $sql_partition_tag. This would allow for example labeling of SQL records for mail originating from inside with a different partition_tag value, compared to entries for incoming mail, and consequently let them be stored in a separate partition if desired. Amavisd process itself does not use the 'partition_tag' field for its own purposes, all records regardless of their 'partition_tag' value are available for example to pen pals lookups, as before. The field is provided only as a convenience to SQL database maintenance, and can be ignored by smaller sites where current practice of database maintenance is fast enough. If SQL partitioning is not in use (or not intended to be used in a near future), it is more economical to use a fixed value (such as 0, which is a default) for the $sql_partition_tag. Using week numbers as partition tags adds about 50 % to the number of records in table maddr, the exact number depends on retention period and a ratio of regular vs. infrequent mail addresses observed. To convert tables of an existing database, please use ALTER command. Here is a conversion example (MySQL or PostgreSQL, probably others): ALTER TABLE maddr ADD partition_tag integer DEFAULT 0; ALTER TABLE msgs ADD partition_tag integer DEFAULT 0; ALTER TABLE msgrcpt ADD partition_tag integer DEFAULT 0; ALTER TABLE quarantine ADD partition_tag integer DEFAULT 0; As the maddr.email is no longer guaranteed to be unique, but a pair of (maddr.partition_tag, maddr.email) is unique, the constraint and an associated index needs to be changed: => PostgreSQL: ALTER TABLE maddr DROP CONSTRAINT maddr_email_key, ADD CONSTRAINT maddr_email_key UNIQUE (partition_tag,email); => MySQL: ALTER TABLE maddr DROP KEY email, ADD UNIQUE KEY part_email (partition_tag,email); Should a need arise to revert to amavisd-new-2.5.4 while keeping the new partition_tag field, the 'SELECT id FROM maddr ...' may become slow due to dropped index on a field maddr.email, which is replaced by an index on a pair (maddr.partition_tag, maddr.email). The following change to amavisd 2.5.4 solves the problem: @@ -901,2 +901,2 @@ 'sel_adr' => - 'SELECT id FROM maddr WHERE email=?', + 'SELECT id FROM maddr WHERE partition_tag=0 AND email=?', The use of partitioned tables to speed up purging of old records was suggested by Robert Pelletier. - when SQL logging (pen pals) or SQL lookups are used, one can choose a binary or a character data type for fields users.email, mailaddr.email, and maddr.email; now may be a good opportunity to change a data type to binary (string of arbitrary bytes, no character set associated). Background: values of these fields come from SMTP envelope or from a mail header section of processed mail. Even though RFC 2821 and RFC 2822 restrict these addresses to 7-bit ASCII, there is nothing preventing a malicious or misguided sender from supplying any 8-bit byte values. If SQL fields are declared as VARCHAR or CHAR, a character set is associated with data and its rules apply, e.g. control characters may not be permitted, or UTF-8 byte sequences are validated, or a restriction to codes below 128 apply. Depending on strictness of an SQL server on validating data, a violation of character set rules may lead to aborting an SQL operation and failing of mail processing. Even though new standards for e-mail addresses are being negotiated allowing for UTF-8 encoding, an actual e-mail address may still supply arbitrary bytes, which may violate UTF-8 byte sequence rules. A new configuration variable $sql_allow_8bit_address now controls how amavisd passes e-mail addresses to SQL. If a value is true, then it is expected that SQL tables will accept strings of arbitrary bytes for these fields, without associating a character set with data. No data sanitation is done by amavisd. An appropriate SQL data type is 'VARBINARY' or with PostgreSQL a 'BYTEA'. If a value of $sql_allow_8bit_address is false (which is a default for compatibility) then amavisd performs sanitation before passing data to SQL: control characters and characters with codes above 127 are converted to '?', which brings strings within ASCII character set restrictions. A suitable SQL data type is VARCHAR or CHAR. Note that some information is lost in this case. The following clauses can convert pre-2.6.0 tables into the now preferred and more universal form: MySQL: ALTER TABLE users CHANGE email email varbinary(255); ALTER TABLE mailaddr CHANGE email email varbinary(255); ALTER TABLE maddr CHANGE email email varbinary(255); PostgreSQL: ALTER TABLE users ALTER email TYPE bytea USING decode(email,'escape'); ALTER TABLE mailaddr ALTER email TYPE bytea USING decode(email,'escape'); ALTER TABLE maddr ALTER email TYPE bytea USING decode(email,'escape'); If a binary data type is chosen for these three fields, the setting $sql_allow_8bit_address MUST be set to true to let the amavisd program use the appropriate data type in SQL commands, otherwise PostgreSQL will complain with: 'types bytea and character varying cannot be matched' when amavisd tries to execute SQL commands. MySQL is more forgiving and does not complain about a data type mismatch, so one may get away with a mismatch, although it is appropriate to eventually make it right. If a change of a data type of these fields is chosen while using some third-party management interface to SQL data set (e.g. MailZu), make sure the management interface supports the changed data type. This is primarily a concern with PostgreSQL which is more strict in requiring a match between field data types in tables and data in SQL clauses. The need for a change was pointed out by Xavier Romero, reporting that PostgreSQL SQL lookups with pre-2.6.0 versions of amavisd can fail when 8-bit data appears in SMTP envelope addresses: lookup_sql: sql exec: err=7, 22021, DBD::Pg::st execute failed: ERROR: invalid byte sequence for encoding "UTF8" - bounce killer feature: uses a pen pals SQL lookup to check inbound DSN, attempting to match it with a previous outbound message. If a Message-ID found in an attachment of the inbound DSN matches a Message-ID of a message previously sent from our system by a current recipient of the DSN, the DSN message is spared, otherwise it receives $bounce_killer_score spam score points (0 by default, i.e. disabled) and can be blocked as spam (although technically it is just a misdirected bounce, not spam). A received delivery status notifications is parsed looking for attached header section of an original message in an attempt to find a Message-ID. A standard DSN structure (RFC 3462, RFC 3464) is recognized, as well as a few nonstandard but common formats. Other automatic reports and bounces with unknown structure and no attached header section are ignored for this purpose (are subject to other regular checks). Unfortunately there are still many nonstandard mailers around (12+ years after DSN format standardization) and many ad-hoc filtering solutions which do not supply the essential information. If a Message-ID can be found in an SQL log database matching a previous message sent by a local user (which is now a recipient of a DSN), using a normal pen pals lookup (no extra SQL operations are necessary), or if domain part of the Message-ID is one of local domains, then the DSN message is considered a genuine bounce, is unaffected by this check and passes normally (subject to other checks). On the other hand, if the attached DSN header does supply a Message-ID but but it does not meet the above pen pals matching criteria, then it is assumed that the message is a backscatter to a faked address belonging to our local domains, and $bounce_killer_score spam score points are added, so the message can be treated as spam (subject to spam kill level and other spam settings). The only user-configurable setting is $bounce_killer_score (also member of policy banks), its default value is 0. To activate the bounce killer feature set the $bounce_killer_score to a positive number, e.g. 100. A pre-requisite is a working SQL logging database (pen pals). A couple of SNMP-like counters are added to facilitate assessing effectiveness of the feature (e.g. viewed by amavisd-agent utility): InMsgsBounce 21310 333/h 9.9 % (InMsgs) InMsgsBounceKilled 19967 312/h 93.7 % (InMsgsBounce) InMsgsBounceRescuedByDomain 7 0/h 0.0 % (InMsgsBounce) InMsgsBounceRescuedByOriginating 242 4/h 1.1 % (InMsgsBounce) InMsgsBounceRescuedByPenPals 67 1/h 0.3 % (InMsgsBounce) InMsgsBounceUnverifiable 1027 16/h 4.8 % (InMsgsBounce) More information on operations can be obtained from a log, search for: inspect_dsn: bounce killed bounce rescued by penpals bounce rescued by domain bounce unverifiable The feature was suggested by Scott F. Crosby. See also http://www.postfix.org/BACKSCATTER_README.html, http://wiki.apache.org/spamassassin/VBounceRuleset and a SpamAssassin man page Mail::SpamAssassin::Plugin::VBounce for additional ideas on fighting joe-jobbed backscatter mail. - a new configuration variable @author_to_policy_bank_maps (also a member of policy banks) is a list of lookup tables (typically only a hash-type lookup table is used), which maps author addresses(es) (each address in a 'From:' header field - typically only one) in a mail header section to one or more policy bank names (a comma-separated list of names). A match can only occur if a valid DKIM author signature or a valid DKIM third-party signature is found, so in as much as one can trust the signing domain, loading of arbitrary policy banks can be safe, offering a flexibility of whitelisting against spam (absolute or just contributing score points), bypassing of checks (banned, virus, bad-header), using less restrictive banned rules for certain senders, by-sender routing, turning quarantining/archiving on/off, and other tricks offered by the existing policy bank loading mechanisms. When a message has a valid DKIM (or DomainKeys) author signature (i.e. when a 'From:' address matches a signing identity according to DKIM (RFC 4871) or DomainKeys (RFC 4870) rules), a lookup key is an unchanged author address and the usual lookup rules apply (README.lookups - hash lookups). When a valid third-party signature is found, a lookup key is extended by a '/@' and a lowercased signing domain, as shown in the example below. The semantics is very similar to a whitelist_from_dkim feature in SpamAssassin, but is more flexible as is allows any dynamic amavisd setting to be changed depending on author address, not just skipping of spam checks. A few examples of a SpamAssassin's whitelist_from_dkim (as in local.cf) along with equivalent amavisd @author_to_policy_bank_maps entries follow. To whitelist any From address with a domain example.com when a message has a valid author signature (i.e. a signature by the same domain): SA: whitelist_from_dkim *@example.com am: 'example.com' => 'WHITELIST', which is equivalent to a lengthy but redundant: SA: whitelist_from_dkim *@example.com example.com am: 'example.com/@example.com' => 'WHITELIST', Similar to above, but applies to subdomains of example.com carrying a valid author signature (i.e. signature BY THE SAME SUBDOMAIN): SA: whitelist_from_dkim *@*.example.com am: '.example.com' => 'WHITELIST', Note that in amavisd hash lookups a '.example.com' implies a parent domain 'example.com' too, while in SpamAssassin and in Postfix maps a parent domain needs its own entry if desired. To whitelist From addresses from subdomains of example.com which carry a valid third-party signature of its parent domain: SA: whitelist_from_dkim *@*.example.com example.com am: '.example.com/@example.com' => 'WHITELIST', To whitelist any From address as long as a message has a valid DKIM or DomainKeys signature by example.com, i.e. a third-party signature. Typical for mailing lists or discussion groups which sign postings. SA: whitelist_from_dkim *@* example.com am: './@example.com' => 'WHITELIST', Here is a complete example to be included in amavisd.conf: @author_to_policy_bank_maps = ( { # 'friends.example.net' => 'WHITELIST,NOBANNEDCHECK', # 'user1@cust.example.net' => 'WHITELIST,NOBANNEDCHECK', '.ebay.com' => 'WHITELIST', '.ebay.co.uk' => 'WHITELIST', 'ebay.at' => 'WHITELIST', 'ebay.ca' => 'WHITELIST', 'ebay.de' => 'WHITELIST', 'ebay.fr' => 'WHITELIST', '.paypal.co.uk' => 'WHITELIST', '.paypal.com' => 'WHITELIST', # author signatures './@paypal.com' => 'WHITELIST', # 3rd-party sign. by paypal.com 'alert.bankofamerica.com' => 'WHITELIST', 'amazon.com' => 'WHITELIST', 'cisco.com' => 'WHITELIST', '.cnn.com' => 'WHITELIST', 'skype.net' => 'WHITELIST', 'welcome.skype.com' => 'WHITELIST', 'cc.yahoo-inc.com' => 'WHITELIST', 'cc.yahoo-inc.com/@yahoo-inc.com' => 'WHITELIST', 'google.com' => 'MILD_WHITELIST', 'googlemail.com' => 'MILD_WHITELIST', './@googlegroups.com' => 'MILD_WHITELIST', './@yahoogroups.com' => 'MILD_WHITELIST', './@yahoogroups.co.uk' => 'MILD_WHITELIST', './@yahoogroupes.fr' => 'MILD_WHITELIST', 'yousendit.com' => 'MILD_WHITELIST', 'meetup.com' => 'MILD_WHITELIST', 'dailyhoroscope@astrology.com' => 'MILD_WHITELIST', } ); $policy_bank{'MILD_WHITELIST'} = { score_sender_maps => [ { '.' => [-1.8] } ], }; $policy_bank{'WHITELIST'} = { bypass_spam_checks_maps => [1], spam_lovers_maps => [1], }; $policy_bank{'NOVIRUSCHECK'} = { bypass_decode_parts => 1, bypass_virus_checks_maps => [1], virus_lovers_maps => [1], }; $policy_bank{'NOBANNEDCHECK'} = { bypass_banned_checks_maps => [1], banned_files_lovers_maps => [1], }; - smtp client connection caching is a new feature which allows smtp client code in amavisd to keep a SMTP session to MTA open after forwarding a message or a notification, so that a next mail message that needs to be sent by this child process can avoid re-establishing a session and the initial greeting/EHLO (and TLS) handshake. A current value of a global settings $smtp_connection_cache_enable controls whether a session will be retained after forwarding a message or not. Its default initial value is true. A global setting $smtp_connection_cache_on_demand controls whether amavisd is allowed to dynamically change the $smtp_connection_cache_enable setting according to its estimate of the message frequency. The heuristics is currently very simple: if time interval between a previous task completion by this child process and the arrival of a current message is 5 seconds or less, the $smtp_connection_cache_enable is turned on (which will affect the next message); if the interval is 15 seconds or more, it is turned off. The default value of the $smtp_connection_cache_on_demand is true, thus enabling the adaptive behaviour. On a busy server the connection caching can save some processing time. Savings are substantial if client-side TLS is enabled, otherwise just a few milliseconds are saved. On an idle server the feature may unnecessarily keep sessions to MTA open (until MTA times them out), so one can disable the feature by setting both controls to false (to 0 or undef). To monitor the connection caching effectiveness, some SNMP-like counters were added, so amavisd-agent may display something like: OutConnNew 2764 319/h 98.2 % (OutMsgs) OutConnQuit 2521 291/h 89.5 % (OutMsgs) OutConnReuseFail 7 1/h 0.2 % (OutMsgs) OutConnReuseRecent 21 2/h 0.7 % (OutMsgs) OutConnReuseRefreshed 31 4/h 1.1 % (OutMsgs) OutConnTransact 2816 325/h 100.0 % (OutMsgs) - client-side TLS support is added, i.e. on forwarding a passed mail back to MTA. Currently only encryption is supported, no client certificates are offered. A $tls_security_level_out is a per-policy-bank setting which controls client-side TLS, its value is either undefined (default), or a string: undef ... client-side TLS is disabled (a default setting); 'may' ... TLS is used if MTA offers a STARTTLS capability (RFC 3207), otherwise a plain text SMTP session is established; 'encrypt' TLS is used if MTA offers a STARTTLS capability, otherwise amavisd refuses to forward a message. The client-side TLS imposes some performance penalty on passing a message back to MTA, although it is still reasonably fast: a benchmark indicates a drop in transfer rate by about a factor of 2, from 22 MB/s (no TLS) to 9 MB/s (with TLS). The smtp client connection caching (see previous item) should preferably be left enabled (permanently or opportunistically), as TLS negotiation adds significantly to the initial SMTP handshake time. - server-side TLS support is added, i.e. on accepting mail from MTA. Encryption is supported, server (i.e. amavisd) offers its certificate, but client certificates are not verified. A $tls_security_level_in is a per-policy-bank setting which controls server-side TLS, its value is either undefined (default), or a string: undef ... server-side TLS is disabled, STARTTLS capability is not offered; 'may' ... STARTTLS capability is offered by amavisd, but client is not required to enter TLS, plain text sessions are permitted; 'encrypt' STARTTLS capability is offered and enforced by amavisd, any SMTP command other than STARTTLS, NOOP, EHLO or QUIT is rejected. If $tls_security_level_in is enabled (any value other than undef or 'none'), amavisd offers a certificate to a connecting client requesting TLS, so a path to a certificate and to its private key must be provided through two global settings: $smtpd_tls_cert_file and $smtpd_tls_key_file, e.g.: $smtpd_tls_cert_file = "$MYHOME/cert/amavisd-cert.pem"; $smtpd_tls_key_file = "$MYHOME/cert/amavisd-key.pem"; The private key should be guarded as secret (not world-readable). A self-signed certificate is acceptable by most mailers. Server-side TLS imposes a significant performance penalty on accepting a message from MTA. A benchmark indicates a drop in transfer rate by a factor of 10, from about 10 MB/s (no TLS) to 1 MB/s (using TLS), so it should only be enabled with a good reason or for experimentation. - enhanced a subroutine delivery_status_notification (along with dispatch_from_quarantine and msg_from_quarantine) to produce a message in one of several formats (derived from a message being processed, or from a quarantined message). Its new arguments can be strings as follows: $request_type: dsn, release, requeue, report $msg_format: dsn, arf, attach, plain, resend $feedback_type: abuse, fraud, miscategorized, not-spam, opt-out, opt-out-list, virus, other (according to ARF draft) Per-policy settings $report_format and $release_format control the format of a generated message. Their value can be one of the following strings, although not all combinations make sense: 'arf' .... an abuse report is generated according to draft-shafranovich-feedback-report-04: "An Extensible Format for Email Feedback Reports"; a plain-text part contains text from a template; 'attach'.. generates a report message as plain text according to a template, with an original message attached; 'plain'... generates a simple (flat) mail with an only MIME part containing a text from a template, followed inline by original message (some service providers can't handle abuse reports with attachments, e.g. Yahoo!); 'resend'.. original message is forwarded unchanged, except for an addition of header fields Resent-From, Resent-Sender, Resent-To, Resent-Date and Resent-Message-ID; 'dsn' .... (for internal use) a delivery status notification is generated according to rfc3462, rfc3464 and rfc3461; When a request_type is 'release' or 'requeue', the format of a generated message is governed by a per-policy setting $release_format according to the table above. Only the 'attach', 'plain' and 'resend' values are useful. A default setting is: $release_format = 'resend'; # with alternatives: attach, plain, resend A plain-text part (if used) is taken from a $notify_release_templ template and a sending address is obtained from %hdrfrom_notify_release_by_ccat. When a request_type is 'report', the format of a generated message is governed by a per-policy setting $report_format according to the table above. Only the following settings are useful: arf, attach, plain, resend. A default setting is: $report_format = 'arf'; # alternatives: arf, attach, plain, resend A plain-text part (if used) is obtained from a $notify_report_templ template, and a sending address from %hdrfrom_notify_report_by_ccat. It is possible to automatically generate abuse reports from custom hooks by calling delivery_status_notification() and mail_dispatch(). Extreme care must be taken to only produce legitimate abuse reports (about true fraud and true spam), sent only to parties that are truly responsible for a message being reported. Non-repudiation is a key factor here - trust only header fields covered by a valid DKIM signature, or generated by your own MX MTA (such as an IP address of the last hop), and only report messages received from a network which officially belongs to the party (according to whois). Rate-limiting should be used, and abuse reports on the same abuser should only be sent once in a time interval of several hours. A SQL database can be used to maintain a list of recently reported abusers, thus preventing excessive reports. - introduced a variation of a message release from a quarantine, allowing a releaser to send an abuse report based on a quarantined message. It is implemented by: * enhancing a subroutine delivery_status_notification as described in the previous item; * extending AM.PDP protocol with a 'request=report' attribute which can be used in place of a 'request=release', * enhancing the 'amavisd-release' utility program to allow sending an attribute 'request=release' or 'request=requeue' or 'request=report' based on its program name. By making a soft or hard link named 'amavisd-report' linking to 'amavisd-release', the utility will send a 'request=report' in place of the usual 'request=release', e.g.: # ln -s amavisd-release amavisd-report # ln -s amavisd-release amavisd-requeue $ amavisd-report spam/k/kg2P0rP9Lpu3.gz '' abuse@example.com - releasing from a quarantine can push a released message to an attachment (Content-Type: message/rfc822), with a configurable template for a header section and the plain-text part; select by: $release_format='attach'; suggested by Patrick Ben Koetter; - detect and save a new attribute SOURCE from an XFORWARD smtp command; the value is also accepted as AM.PDP protocol attribute 'client_source'. Possible values are: 'LOCAL', 'REMOTE', or '[UNAVAILABLE]', the information corresponds to 'local_header_rewrite_clients' postfix setting and is not supposed to be used for security decisions according to Postfix documentation (which makes it less interesting for our purposes); - added client and server support for a PORT attribute of an XFORWARD command, allowing MTA to pass a TCP port number of a remote client to a content filter (and back if necessary); the PORT attribute is made available with Postfix version 2.5 (20071004); a source port number is also accepted as an AM.PDP protocol attribute 'client_port'; - updated p0f-analyzer.pl now supports a source port number information in queries while preserving backwards compatibility with previous versions of amavisd-new. Version 2.6.0 of amavisd requires a new version of p0f-analyzer.pl (supplied in the 2.6.0 distribution) if operating system fingerprinting is enabled. A source port number information in a query allows p0f-analyzer.pl to locate a matching entry in its cache faster and also more accurately when multiple connections are present from clients behind NAT using the same IP address. The source port number is made available to a content filter since Postfix version 2.5 (20071004); - besides the ability to send queries to p0f-analyzer.pl, amavisd now also supports sending queries directly to a p0f program over a Unix socket using a p0f query protocol. There is a bug in p0f-2.0.8 (and probably in earlier versions) which makes it send back incorrect results at times, i.e. results belonging to some other unrelated session, so a patch to p0f-2.0.8 MUST be applied in order to use a direct querying mechanism - author has been notified. The patch is supplied: p0f-patch. There are currently no advantages (and some disadvantages) in choosing direct queries to p0f, compared to sending queries to p0f-analyzer.pl, so this new method is not currently recommended. Disadvantages are: * p0f uses a linear search over its list of recent sessions, whereas p0f-analyzer.pl uses a fast hash lookup method; * p0f keeps a relatively small list of recent sessions which is limited by the number of slots (size can be specified on a command line, but is limited by a linear search time), whereas p0f-analyzer.pl expires old entries according to time since entered and is thus independent of a current mail rate; * a direct p0f query protocol uses packed binary data and its on-the-wire representation may depend on a compiler used, so it may be incompatible with queries sent by amavisd, whereas the p0f-analyzer.pl queries and replies use a more environment-independent textual representation. To let amavisd sent queries directly to p0f, specify a p0f socket path: $os_fingerprint_method = 'p0f:/var/amavis/home/p0f.sock'; and specify an IP address and a port number on which MTA is listening: $os_fingerprint_dst_ip_and_port = '[192.0.2.3]:25'; because p0f requires this information in a query and the information is not made available to a content filter via XFORWARD command (the p0f-analyzer.pl does not need this information). To send queries to p0f-analyzer.pl (traditional and recommended), use: $os_fingerprint_method = 'p0f:127.0.0.1:2345'; as before. The $os_fingerprint_dst_ip_and_port in this case is not needed and is ignored. - usually a sending address in spam messages is faked and it is desirable to suppress most if not all bounces by keeping $sa_dsn_cutoff_level low, but sometimes it may be possible to be more certain of the validity of a sending address, and when such mail is considered spam, it may still be desirable to send a non-delivery notification, knowing that a notification will most likely be addressed to a genuine sender. Two new settings are provided for this purpose: @spam_crediblefrom_dsn_cutoff_level_bysender_maps and @spam_crediblefrom_dsn_cutoff_level_maps (with their default being $sa_crediblefrom_dsn_cutoff_level), complementing the existing @spam_dsn_cutoff_level_bysender_maps and @spam_dsn_cutoff_level_maps. It is expected that $sa_crediblefrom_dsn_cutoff_level would be set somewhat higher than $sa_dsn_cutoff_level, allowing for more bounces to be generated for spam from likely-to-be-genuine senders (possibly false positives). The choice between taking a cutoff value from one or the other pair of settings depends on an attribute $msginfo->sender_credible - when it is true (e.g. some nonempty string) the *spam_crediblefrom_* settings will be used instead of the baseline @spam_dsn_cutoff_level_*maps. An initial value of a sender_credible attribute as provided by amavisd is true if either the 'originating' flag is true (e.g. mail from inside), or if dkim_envsender_sig attribute is true, e.g. a domain of a valid DKIM signature matches envelope sender address, otherwise it is false. A user-provided custom hook code is free to change the value of sender_credible attribute. An exact value does not matter (it is only interpreted as a boolean), but serves for logging purposes. Heuristics may be based on some tests provided by SpamAssassin, on DKIM signatures, on p0f results, on policy banks, etc. Here is one complete example of a custom hook, which turns on the sender_credible attribute based on some criteria. Note that some of the referenced SpamAssassin tests may not yet be available in the last officially released version of SpamAssassin. added to amavisd.conf: include_config_files('/etc/amavisd-custom.conf'); /etc/amavisd-custom.conf : package Amavis::Custom; use strict; sub new { my($class,$conn,$msginfo) = @_; bless {}, $class } sub after_send { my($self,$conn,$msginfo) = @_; if ($msginfo->sender ne '') { my(@cred); local($1); my($tests) = $msginfo->supplementary_info('TESTS'); $tests = '' if !defined($tests) || $tests eq 'none'; push(@cred,'orig') if $msginfo->originating; push(@cred,$1) if $tests =~ /\b(RCVD_IN_DNSWL_HI)\b/; push(@cred,$1) if $tests =~ /\b(RCVD_IN_DNSWL_MED)\b/; push(@cred,$1) if $tests =~ /\b(RP_MATCHES_RCVD)\b/; my($os_fingerprint) = $msginfo->client_os_fingerprint; if ($os_fingerprint !~ /^Windows XP(?![^(]*\b2000 SP)/) { push(@cred,'dkim') if $msginfo->dkim_envsender_sig; push(@cred,$1) if $tests =~ /\b(SPF_PASS)\b/; } $msginfo->sender_credible(join(",",@cred)) if @cred; } } 1; # insure a defined return - a new setting $reputation_factor (also a member of policy banks) with a value between 0 and 1 (default 0.2), controlling an amount of 'bending' of a calculated spam score towards a fixed score assigned to a signer identity (i.e. its 'reputation') through @signer_reputation_maps; the formula is: adjusted_spam_score = f*reputation + (1-f)*spam_score; which has the same semantics as auto_whitelist_factor in SpamAssassin AWL; - a new setting @signer_reputation_maps (also a member of policy banks) may contain a list of lookup tables (typically just one hash lookup table), mapping a signing identity to a score, which is typically a long term average spam score of all messages signed by this signing identity. Based on a lookup result and a formula given above ($reputation_factor), the resulting value (positive or negative) is added to the spam score. Here is an example setting: @signer_reputation_maps = ( { 'ebay.fr' => -10.95, 'ebay.ca' => -9.57, 'ebay.co.uk' => -8.59, 'ebay.com' => -8.03, 'ebay.at' => -3.59, 'ebay.de' => -3.38, 'reply3.ebay.com' => -4.57, 'reply.ebay.com' => -3.20, 'paypal.com' => -6.66, 'intl.paypal.com' => -3.70, 'email.paypal.co.uk' => -0.67, 'alert.bankofamerica.com' => 1.35, 'ucsd.edu' => -7.89, 'izb.knu.ac.kr' => -7.51, 'ijs.si' => -4.44, 'tu-graz.ac.at' => -5.44, 'tugraz.at' => -4.03, 'aitech.ac.jp' => -3.03, 'univie.ac.at' => -2.99, 'uni-bremen.de' => -2.80, 'uu.se' => -2.54, 'univ-tours.fr' => -2.06, 'phys.huji.ac.il' => -2.01, 'cern.ch' => -1.69, 'prime.gushi.org' => -9.85, 'dostech.ca' => -9.31, 'resistor.net' => -9.18, 'kitterman.com' => -9.05, 'schetterer.org' => -9.04, 'hege.li' => -8.96, 'fouter.net' => -8.84, 'inetmsg.com' => -8.81, 'charite.de' => -8.81, 'porcupine.org' => -8.75, 'secnap.net' => -7.99, 'netoyen.net' => -7.58, 'state-of-mind.de' => -6.93, 'gmurray.org.uk' => -5.39, 'mtcc.com' => -4.93, 'messiah.edu' => -4.91, 'consulintel.es' => -2.16, 'delphij.net' => 1.06, 'channing-bete.com' => -9.11, 'megan.vbhcs.org' => -8.64, 'scent-team.com' => -8.32, 'suedfactoring.de' => -8.15, 'sendmail.net' => -5.14, 'cisco.com' => -4.95, 'hermes-softlab.com' => -3.77, 'altn.com' => -2.16, 'amazon.com' => 0.09, 'eurescom.eu' => -0.63, 'skype.net' => -1.50, 'welcome.skype.com' => -0.34, 'newsdesk.world-nuclear-news.org'=> -0.74, 'youtube.com' => 1.11, 'email.innocentive.com' => 1.98, 'update.hallmark.com' => 2.26, 'newsletters.trendmicro.com'=> 3.02, 'mail.communications.sun.com'=> 3.73, 'alerts.hp.com' => 0.51, 'email.greenpeace.org' => 2.04, 'avaaz.org' => 3.51, 'mail.cnn.com' => 4.12, 'm-w.com' => 4.81, 'medcompare.com' => 2.46, 'biocompare.com' => 3.48, 'dentalcompare.com' => 4.58, 'news.biomedcentral.com' => 2.89, 'yahoo.com' => -0.53, 'yahoo.se' => -1.48, 'yahoo.de' => -1.33, 'yahoo.co.uk' => 0.85, 'yahoo.ca' => 1.13, 'yahoo.no' => 1.22, 'yahoo.es' => 2.04, 'yahoo.ie' => 4.48, 'yahoo.fr' => 6.35, 'yahoo.it' => 7.09, 'yahoo.dk' => 7.10, 'yahoo.co.in' => 5.34, 'yahoo.com.cn' => 4.34, 'yahoo.com.ar' => 5.48, 'yahoo.co.jp' => 9.25, 'yahoo.com.au' => 9.30, 'yahoo.com.hk' => 13.97, 'yahoo.cn' => 21.94, 'yahoo-inc.com' => -1.57, 'yahoogroups.com' => -1.76, # 'yahoogroups.co.uk' => 0.10, # 'yahoogroupes.fr' => 5.99, 'yahoogroups.de' => 19.96, 'yahoogrupos.com.br' => 17.41, 'gmail.com' => -3.52, 'googlegroups.com' => -3.10, 'googlemail.com' => 3.42, 'google.com' => 1.47, 'prodigy.net' => -8.91, 'amis.net' => -2.38, 'earthlink.net' => -2.27, 'btinternet.com' => -1.96, 'pacbell.net' => -0.96, 'rogers.com' => 2.05, 'ipost.com' => 2.40, 'incertum.net' => -9.80, 'yousendit.com' => -4.70, 'news.yousendit.com' => 3.38, 'abv.bg' => -4.57, 'uclouvain.be' => -4.48, 'birthdayalarm.com' => -4.32, 'geni.com' => -3.50, 'mail120.subscribermail.com'=> -2.58, 'mail6.subscribermail.com' => -1.97, 'spock.com' => -11.67, 'meetup.com' => -1.94, '123greetings.com' => -1.92, 'rocketmail.com' => -1.91, 'arcamax.com' => -0.81, 'skynet.be' => -0.10, 'news.virtualtourist.com' => 3.58, 'bighip.com' => 10.53, 'investorsinsight.com' => 12.19, 'lspromos2007online.com' => 12.41, 'postmaster-direct.com' => 12.71, 'smsacfriends.com' => 12.88, 'specialtyofficial.com' => 13.03, 'usafisnews.org' => 13.82, 'freelotto.com' => 15.59, 'partnershopping.net' => 19.46, 'muffinlocate.com' => 19.49, 'stallrust.com' => 19.71, 'sugargrowth.com' => 19.75, 'washusual.com' => 20.01, 'poursmock.com' => 20.02, 'overseeinvest.com' => 20.19, 'waveholer.com' => 20.27, 'headattomic.com' => 20.32, 'moneysuffer.com' => 20.84, 'ibnyes.com' => 20.95, 'honorfamous.com' => 20.96, 'trapdull.com' => 20.96, 'solvehigh.com' => 21.57, 'tracerope.com' => 21.68, 'yesalumni.com' => 21.71, 'internetpromotional.net' => 23.16, 'domaingln11track.com' => 23.69, 'news-central-99.com' => 24.00, 'sports-bobble-heads.com' => 24.69, 'dans-fishing-adventure.com'=> 26.39, 'myhottdeals.net' => 26.65, 'nitroda.com' => 28.75, 'spaninns.com' => 29.08, 'pinaycamgirls.net' => 29.24, 'aspsbulletins.com' => 13.06, 'latestsbulletins.com' => 13.35, 'bulletinshops.com' => 13.40, 'academicsbulletins.com' => 13.49, 'tigersbulletins.com' => 14.13, 'nakedsbulletins.com' => 14.17, 'paintedsbulletins.com' => 14.33, 'opensbulletins.com' => 14.58, 'domainsbulletins.com' => 14.61, 'seniorssbulletins.com' => 14.72, 'dodgesbulletins.com' => 15.23, 'mindsbulletins.com' => 15.96, 'researchsbulletins.com' => 15.98, 'salessbulletins.com' => 16.12, 'netsbulletins.com' => 16.83, 'virtualsbulletins.com' => 17.14, 'petbulletins.com' => 17.30, 'citysbulletins.com' => 18.46, 'wearesurveys.net' => 23.36, 'takeoursurveys.net' => 24.50, 'thankyousurveys.net' => 24.62, 'greetingssurveys.info' => 24.63, 'allaboutsurveys.net' => 25.71, 'internetssurvey.info' => 25.98, 'theinternetsurveys.net' => 26.14, 'netsurveysnet.net' => 26.85, 'surveydept.info' => 23.39, 'survysample.info' => 25.82, 'gosurrvey.info' => 26.02, 'surrvytime.info' => 26.41, 'thedailyinfo.info' => 21.96, 'thenetinfo.info' => 23.27, 'nettinfo.info' => 27.41, 'theinfoguide.info' => 29.91, 'alloursamples.info' => 22.71, 'officeofferonline.info' => 24.77, 'cashsuggestion.info' => 26.84, 'insaneofferdeal.info' => 26.87, 'rewardcenterrs.info' => 30.04, 'gifttsgroup.info' => 31.48, 'ourgiftworld.info' => 32.46, 'giftwinninngs.info' => 34.11, 'mygiftwinninngs.net' => 31.12, 'gifttsgroup.net' => 33.05, 'giftwinninngs.net' => 34.95, }); - drop privileges sooner if possible - right after reading config file and before forking; - allow inserting X-Quarantine-ID header field into passed (and quarantined) mail for local recipients only; remote recipients should not be made aware that we may have a copy in a quarantine; reported by Robert Fitzpatrick; - check for multiple occurrences of RFC 2045 and RFC 2183 MIME header fields (in addition to checks on RFC 2822 header fields): MIME-Version Content-Type Content-Transfer-Encoding Content-ID Content-Description Content-Disposition and the RFC 3834 header field: Auto-Submitted - capture SpamAssassin logging and integrate it into the usual amavisd log; suggested by Jeff Moss; - when parsing SpamAssassin log areas/facilities, recognize negations (a name prefixed by 'no') and remove duplicates, last entry wins, e.g. amavisd -d rules,noall,norules,dcc,norules,rules debug-sa is equivalent to: amavisd -d noall,dcc,rules debug-sa - a SpamAssassin debug level 'info' is now implicitly prepend to a list of SpamAssassin facilities; it may be overridden by explicitly negating it, e.g. 'amavisd -d noinfo'; - when a command line argument 'debug-sa' is present, or if $sa_debug is true, a SpamAssassin debug level 'all' is prepend to a list of SpamAssassin facilities, which would bring SpamAssassin log level to 'dbg'; it may be overridden by negating it, e.g. 'amavisd -d noall,plugins,rules debug-sa'; - include milliseconds in a log timestamp when logging is directed to stderr; - create a custom hook object sooner, so that loading a policy bank from a custom hook becomes useful (but not soon enough to influence partition_tag); - added a custom hook after_send, which may suppress DSN, send reports, quarantine, etc; - fetch additional information (tag) 'TIMING' from SpamAssassin, making it available through macro 'supplementary_info' (if a version of SpamAssassin in use provides it - available since 3.3.0); - a SpamAssassin TIMING-SA report (timing breakdown by sections) is now collected and logged at log level 2 when available; the information is available since version 3.3.0 of SpamAssassin (currently in development and available through SVN); - add a global configuration variable $listen_queue_size (undef by default) which is passed as an option 'listen' to Net::Server, which in turn passes it on to listen(2) as a 'backlog' parameter. The Net::Server provides a default value of SOMAXCONN in the absence of a valid integer in $listen_queue_size (e.g. 128, but on Solaris it defaults to 5, which is too small for some purposes). Suggested by David Schweikert. A workaround for a small SOMAXCONN default on Solaris is provided by Net::Server 0.98 (?). - in the absence of a smtp client's IP address (normally received by XFORWARD smtp command from Postfix, or in the 'client_address' attribute of AM.PDP), parse the topmost one or two Received header fields and use the first valid IP address found there; based on a suggestion by Richard Bishop; - new macros (useful in notifications and $log_template): - 'week_iso8601' returns an ISO 8601 week number (between 1 and 53); - 'partition_tag' returns a current value of a $sql_partition_tag variable; - 'dkim' reports various DKIM verification results; - 'report_format' gives one of the: dsn, arf, attach, plain, resend - 'feedback_type' is expected to give one of the ARF (draft) strings: abuse, fraud, miscategorized, not-spam, opt-out, opt-out-list, virus, other - 'rfc2822_from' an e-mail address from a From header field; - 'rfc2822_sender' an e-mail address from a Sender header field, or empty; - 'tls_in' returns TLS ciphers in use by a SMTP session if mail came to amavisd through a TLS-encrypted session, otherwise empty - 'limit' takes two arguments: a string size limit and some string, returning the string from the second argument unchanged if its size is below the limit, or cropped to the size limit, with '[...]' appended; For details see README.customize. - new configuration setting $allow_fixing_long_header_lines, also member of policy banks, defaults to true - provides control over truncation of header section lines longer than 998 characters as limited by RFC 2822. The $allow_fixing_improper_header must also be true for fixing to take place. Previously it was only possible to turn off all header fixes, but not specifically just the long header truncation; - strip X-Spam-* headers and other prepended header fields when releasing a quarantined message; - turn on message part attributes 'C' (crypted) and 'U' (undecipherable) if a MIME Content-Type of that body part is /encrypted; (note: if $defang_undecipherable is turned on, this would push a received PGP/GPG-encrypted MIME top-part into an attachment, just as with other password-protected archives); - fetch additional information (tags) DCCB and DCCR from a SpamAssassin DCC plugin, making them available through a macro 'supplementary_info'; - amavisd-nanny and amavisd-agent now reopen a database if/when the underlying database file is re-created (i.e. when its inode changes), as is the case when amavisd restarts; based on a patch by Rob Foehl; - amavisd-nanny and amavisd-agent got a new command line option: -c count, which can restrict the number of iterations for display; usability deficiency pointed out by John Evans; - amavisd-nanny and amavisd-agent will use a value from an environment variable AMAVISD_DB_HOME (if it exists) as a database home directory, otherwise fall back to a built-in default '/var/amavis/db'; suggested by Leo Baltus; OTHER - added an AV entry for a new version of a ESET File Security (for Linux and FreeBSD) command line scanner (ESET Software ESETS Command Line Interface v 2.71.12); the nod32cli utility has been replaced by esets_cli; update provided by Willi Gruber; - keep time of a message reception (or creation) in rx_time as a floating point value as provided by Time::HiRes::time in order to avoid discrepancy in SNMP-like counters showing elapsed times; suggested by David Schweikert; - store additional attributes in per-message and per-recipient objects, reducing a need for repeated lookups; - to save time the: tag, tag2, tag3 and kill levels are not looked up when a recipient is bypassing spam checks; - no longer reverse-resolve an IP address obtained from a Received header field on infected mail: it does not work for IPv6, it can be stuck for prolonged periods waiting for a response from a non-responsive DNS server, and is only used for logging/notifications (macros %e and %o) on infected mail, so it is not too bad to drop it; - obey the always_bcc_by_ccat even when releasing from a quarantine; thanks to Tomas Horacek; - for consistency with other quarantine methods: store recipient address (e.g. a quarantine mailbox e-mail address) as a quarantine location even if a quarantine method is 'smtp:' or 'lmtp:'; - a quarantined message now receives one additional header field: an X-Envelope-To-Blocked. An X-Envelope-To still holds a complete list of envelope recipients, but the X-Envelope-To-Blocked only lists its subset (in the same order), where only those recipients are listed which did not receive a message (e.g. being blocked by virus/spam/ banning... rules). This facilitates a release of a multi-recipient message from a quarantine in case where some recipients had a message delivered (e.g. spam lovers) and some had it blocked; - a release request now takes its default recipients list from a header field X-Envelope-To-Blocked, no longer from X-Envelope-To. This avoids releasing a message to recipients which have already received it in the first place, e.g. spam lovers. For backwards compatibility, if X-Envelope-To-Blocked header field is not found in a quarantined message, the recipients list defaults to X-Envelope-To as before. A release request can still provide its explicit list of recipients to override a default, like before. Loosely based on suggestions by Christer, by Paolo Schiro and others; - when mail is received from a helper program such as a milter, update a true mail size ($msginfo->msg_size) according to size definition in RFC 1870, adjusting for CRLF line terminators; - updated SMTP enhanced status response codes to an AUTH command according to RFC 4954 (SMTP Service Extension for Authentication); - quote_rfc2821_local: within qcontent all non-qtext characters should be represented as quoted-pairs (RFC 2822); in addition to " and \ (which were already handled), this includes NUL, CR, LF, HT, SP and 8-bit characters; - macros T, C and B now bring e-mail addresses in quoted form and in angle brackets, notification templates no longer add angle brackets around %T and %C; - when defanging mail body no longer insert our own Sender header field on a pretense that it helps with DKIM resigning - according to ADSP/SSP drafts the DKIM/ADSP does not care for the Sender header field (unlike a historical DomainKeys); - always provide X-Amavis-PolicyBank header field in a copy of a mail as submitted to SpamAssassin, even if a policy bank path is empty - this allows for simpler SpamAssassin rules to avoid being tricked by a presence of such header field inserted by third parties; - add missing installs of amavisd-nanny and amavisd-agent utilities to amavisd-new.spec; suggested by Eddy Beliveau; - internal: original header section is no longer kept in two copies (in orig_header and in orig_header_fields). Instead, orig_header now holds an array of header *fields* (previously: array of header *lines*), and orig_header_fields now stores a by- header-field-name hash of indices into orig_header (previously it held copies of header field bodies). To facilitate access to individual header fields and allow for top-down and bottom-up search for a n-th occurrence of a header field, two new access methods are provided: get_header_field and get_header_field_body; these are optimized for a quick access to the *last* header field (previously the *first* one was kept easily accessible); - internal: when specific header fields are looked up in an original header section (such as From, To, Subject, Message-ID, ...) now the *last* (the most to the bottom) occurrence of a header field (instead of the first) is used for better compatibility with DKIM which searches for header fields bottom-up; - internal: when a message is received, a current setting of a boolean attribute $originating (global or from current policy banks) is now copied to an Amavis::In::Message object and becomes a property of a message; this allows for other message objects (like notifications, quarantine) to have their own individual setting of this attribute, for example notifications are always flagged as originating and as such are eligible for DKIM signing; - internal: provide an Amavis::MIME::Body::OnOpenFh package which acts as a MIME::Body -type object (read-only) with an underlying representation in an existing (permanently open) temporary mail file, avoiding a need to open it by file name on a separate file handle. It is useful (simpler, faster) when defanging (pushing original mail to an attachment), or when generating notifications or reports which contain an original mail; - internal: new subroutines init_child and rundown_child in the interface module Amavis::SpamControl::SpamAssassin, which call SpamAssassin plugin methods "spamd_child_init" and "spamd_child_post_connection_close"; this is required for correct operation of some SpamAssassin plugins such as the Mail::SpamAssassin::Plugin::DBI; thanks to Michael Scheidell for pointing out that plugin; - internal: renamed a subroutine string_to_mime_entity into build_mime_entity and generalize its functionality to make it useful for generating reports; - internal: generalized subroutine delivery_status_notification which can now also prepare message in a format of a feedback report (ARF) or just as a plain included (inline) original mail; - internal: a subroutine passed to edit_header() may return undef, in which case a header field will be deleted instead of being replaced/edited; - internal: sub lookup_sql incompatible change to the order of arguments; - internal: a small speedup in receiving mail contents over SMTP or LMTP; - internal: abandon package Amavis::Lock, do the four calls to flock directly; - internal: move subroutines dealing with processes from module Amavis::Util into a new module Amavis::ProcControl; - internal: reduce buffer sizes in some copying loops from 64kB back to 32kB, larger buffers do not perform any better; - documentation: amavisd.conf-default: add a list of legacy dynamic configuration variables which can be used in policy banks; thanks to Gary V; - terminology: use a term 'header section' (in comments, log entries and templates) instead of a 'header' to go along with the 2822upd draft; A QUICK START TO DKIM VERIFICATION Starting with 2.6.0, verification of DKIM signatures (and historical DomainKeys signatures) is provided directly by amavisd (not only by a SpamAssassin plugin DKIM). A required version of a perl module Mail::DKIM is 0.31 or later. Signature verification is sufficiently fast so there is no need for concern about extra processing load. To turn on DKIM (and historical DomainKeys) signature verification, please add the following line to amavisd.conf (if not already there): $enable_dkim_verification = 1; Benefits - Whitelisting of banned checks or spam on messages carrying valid DKIM or DomainKeys signatures from trustworthy signers is possible through the @author_to_policy_bank_maps list of lookup tables. The mechanism uses loading of policy banks based on author's e-mail address (addresses in a 'From:' header field) and a signing domain, so a full flexibility of per-policy-bank settings is available. See description of a new configuration variable @author_to_policy_bank_maps earlier in this release notes. - To each message passed to local recipients amavisd inserts a header field Authentication-Results (according to draft-kucherawy-sender-auth-header) for each signature found in a message, reporting a verification result. These header fields can reliably tell a recipient or his MUA what domains claimed responsibility for a message, or can be used for troubleshooting DKIM signing, verification and tracking mail transformations. - Can adjust spam score based on some metrics on a signing domain's reputation for valid signatures found in a message. A useful reputation metric is an average long term spam score for past messages signed by a domain, which can currently be provided manually by @signer_reputation_maps in a configuration file (see example earlier in this release notes). A spam score is shifted towards this reputation score by a configurable factor $reputation_factor (value between 0 and 1, default is 0.2) using a formula: adjusted_spam_score = f * reputation + (1-f) * spam_score Semantics of a $reputation_factor is equivalent to auto_whitelist_factor in a SpamAssassin's AWL plugin, which shifts spam score towards a long term spam score average of a sender. - Notifications and bounces show a "(dkim:AUTHOR)" next to a From address, and a "(dkim:SENDER)" next to a Sender address if these header fields were signed and their domain corresponds to a signer's domain identity. - A valid DKIM or DomainKeys signature turns on a 'sender_credible' attribute which serves to choose one of the two DSN cutoff levels, so that delivery status notifications can be restricted to or preferred for likely-to-be-valid sending addresses, and bounces to possibly fake addresses can be minimized. More information on the 'sender_credible' attribute can be found earlier in this release notes. Currently the ADSP (Author Domain Signing Practices, formerly SSP) draft is not implemented, neither by amavisd, nor by SpamAssassin's plugin DKIM. Work on a draft is still in progress, and until it settles one needs to resort to SpamAssassin rules to block fake mail with no valid signature from domains which are known to be signing all their mail, such as PayPal, eBay, alert.bankofamerica.com, and others. In essence, the ADSP information (usually inferred, or actually published (quite rare today)) from such domains needs to be encoded into SpamAssassin rules. A QUICK START TO DKIM SIGNING 1. Generate one or more keys to be used for signing and enable signing code by adding the following line to amavisd.conf (if not already there): $enable_dkim_signing = 1; # loads DKIM signing code Signing keys must be made available to amavisd, each private key in a separate file in PEM format. Customarily such keys would be generated and kept in a dedicated directory such as /var/db/dkim or /var/lib/dkim, preferably owned by root. Private keys can be generated by a 'openssl genrsa' command (see RFC 4871 Appendix C, also in README_FILES directory), or by an amavisd equivalent. Commonly one key per signing domain or one key per signing host is used, but other choices are possible. If such keys were already prepared for some other DKIM-signing solution, they can be reused by amavisd. # amavisd genrsa /var/db/dkim/a.key.pem # amavisd genrsa /var/db/dkim/b.key.pem 786 # amavisd genrsa /var/db/dkim/sel-example-com.key.pem # amavisd genrsa /var/db/dkim/g-guest-ex-com.key.pem # amavisd genrsa /var/db/dkim/notif-mail.key.pem 512 Amavisd already ensures the generated files are only readable by owner, a manual procedure may require explicitly setting of file permissions. Private keys must be protected from unauthorized access, only the signing software such as amavisd should have access. Amavisd loads these files on startup before dropping privileges, so if amavisd is started as root it is not necessary that these key files are readable by uid under which amavisd is running. 2. Add commands to amavisd.conf to load private keys, associate them with signing domains and selectors, and describe constraints (tags) to be published with public keys. For example: # Load all available private keys and supply their public key RR constraints. # Arguments are a domain, a selector, a key (a file name of a private key in # PEM format), followed by optional attributes/constraints (tags, represented # here as Perl hash key/value pairs) which are allowed by RFC 4871 in a public # key resource record (v, g, h, k, n, s, t), of which only g, h, k, s and t # are considered to be constraints limiting the choice of a signing key. # A command 'amavisd showkeys' can be used for displaying corresponding # public keys in a format directly suitable for inclusion into DNS zone files. # # signing domain selector private key options # ------------- -------- ---------------------- ---------- dkim_key('example.org', 'abc', '/var/db/dkim/a.key.pem'); dkim_key('example.org', 'yyy', '/var/db/dkim/b.key.pem', t=>'s'); dkim_key('example.org', 'zzz', '/var/db/dkim/b.key.pem', h=>'sha256'); dkim_key('example.com', 'sel-2008', '/var/db/dkim/sel-example-com.key.pem', t=>'s:y', g=>'*', k=>'rsa', h=>'sha256:sha1', s=>'email', n=>'testing; 1, 2'); dkim_key('guest.example.com', 'g', '/var/db/dkim/g-guest-ex-com.key.pem'); dkim_key('mail.example.com', 'notif', '/var/db/dkim/notif-mail.key.pem'); A selector paired with a domain name uniquely identifies a key, both for a signer as well as for a recipient. There may be multiple keys for each domain as long as each one has its own selector. A selector with a domain name will be used by a receiving mailer in assembling a DNS query (selector._domainkey.signingdomain) to fetch a public key from a signing domain's DNS server when verifying signature validity. A selector with a domain name will also be used by a signing amavisd when choosing a key applicable to signing, meeting constraints on its public key (tags, RFC 4871 section 3.6) as given by optional arguments. The optional arguments serve as documentation, may help amavisd choose between multiple choices, and supply additional information for step 3. For a list of options (tags) see RFC 4871 section 3.6. Amavisd does not check the syntax of tag values, except for performing qp-section encoding of a tag 'n'. Note the Perl syntax of key/value pairs, e.g. t => 's:y' will end up as "t=s:y", and n => 'testing; 1, 2' will end up encoded as "n=testing=3B 1, 2". 3. Prepare and publish public keys. Public keys can be extracted from generated key files (which contain both a private and a public key). To publish public keys they need to be edited into a format suitable for inclusion in a DNS server's zone file for each signing domain, either by following a procedure in RFC 4871 Appendix C, or if step 2 was completed, by asking amavisd to do so: # amavisd showkeys or more selectively, e.g.: # amavisd showkeys .org example.com This step is not needed if public keys were already prepared and published earlier for some other DKIM-signing solution. 4. Edit zone files in master DNS server(s) for each signing domain, adding the just prepared TXT resource records, not forgetting to bump up the serial number in a SOA record. Optionally add a TXT record with ADSP information (formerly SSP) if a default Author Domain Signing Practices is not appropriate. Then reload zone(s) or restart DNS server(s). 5. Test published public keys. Similar to 'showkeys', a 'testkeys' command walks through available signing keys, generates a test message signed with each key, and validates it by fetching a corresponding public key from a DNS server. # amavisd testkeys or more selectively, e.g.: # amavisd testkeys .org example.com 6. Restart amavisd, watch the log at log level 2, searching for " dkim: ". Note that signing could be started (amavisd reload) right after completing step 2, but mail recipients would not be able to verify validity of signatures until public keys are made available by a signing domain through its DNS. Recipients are supposed to treat mail with signatures which fail verification exactly the same as mail with no signatures, so there is usually no harm done with a premature start of signing, but there is no benefit either. 7. Optional: to override default values for signature tags, one may specify by-sender signature tags through @dkim_signature_options_bysender_maps, e.g.: # @dkim_signature_options_bysender_maps maps author/sender addresses or # domains to signature tags/requirements; possible signature tags according # to RFC 4871 are: (v), a, (b), (bh), c, d, (h), i, l, q, s, (t), x, z; # of which the following are determined implicitly: v, b, bh, h, t # (tag h is controlled by %signed_header_fields); currently ignored tags # are l and z; instead of an absolute expiration time (tag x) one may use # a pseudo tag 'ttl' to specify a relative expiration time in seconds, which # is converted to an absolute expiration time prior to signing: x = t + ttl; # a built-in default is provided for each tag if no better match is found # @dkim_signature_options_bysender_maps = ( { 'postmaster@mail.example.com' => { a => 'rsa-sha1', ttl => 7*24*3600 }, 'spam-reporter@example.com' => { a => 'rsa-sha1', ttl => 7*24*3600 }, 'mail.example.com' => { a => 'rsa-sha1', ttl => 10*24*3600 }, # explicit 'd' forces a third-party signature on foreign (hosted) domains 'guest.example' => { d => 'guest.example.com' }, '.example.com' => { d => 'example.com' }, # catchall defaults '.' => { a => 'rsa-sha256', c => 'relaxed/simple', ttl => 30*24*3600 }, # 'd' defaults to a domain of an author/sender address, # 's' defaults to whatever selector is offered by a matching key } ); The result of a by-sender lookup into @dkim_signature_options_bysender_maps is a hash (a set) of DKIM signing requirements (tags), i.e. canonicalization method, hashing algorithm, domain, identity, selector and expiration time. Resulting tags are then used to choose the most appropriate signing key from a set of keys as declared by calls to dkim_key. Main selection criterium is a match on tags d (domain) and s (selector), but other signature requirements must also meet the constraints of a public key (e.g. subdomain matching flag, granularity, hashing algorithm, key type). If a lookup does not find a signing key which meets requirements, no signing takes place. Also, only mail with 'originating' flag is eligible for signing. A lookup is based on either the From (header field), the Sender (header field), or on a mail_from address from the envelope, whichever yields a useful result first. This results either in an author signature (i.e. a first-party signature, based on a From), or a third-party signature (based on Sender or mail_from or forced through d tag). An associative array %signed_header_fields controls which header fields are to be signed. By default it contains a standard (RFC 4871) set of header field names, augmented by some additional header field names considered appropriate at the time of a release (RFC 4021, RFC 3834). In addition a 'Sender' header field is excluded because it is frequently replaced by a mailing list, and as the RFC 2821 mandates there can only be one such header field the original one is dropped, invalidating a signature. Also the 'To' and 'Cc' are excluded from a default set because sendmail mailers are known to gratuitously reformat the list, invalidating a signature. The default set of header fields to be signed can be controlled by setting %signed_header_fields elements to true (to sign) or to false (not to sign). Keys must be in lowercase, e.g.: $signed_header_fields{'received'} = 0; # turn off signing of Received $signed_header_fields{'to'} = 1; # turn on signing of To --------------------------------------------------------------------------- March 12, 2008 amavisd-new-2.5.4 release notes BUG FIXES - simplify regular expressions in parse_quoted_rfc2821() to avoid perl crashing on a long degenerated e-mail address; reported by Sébastien Aveline; - further simplify (split in two) regular expressions in parse_address_list() to avoid perl crashing on long degenerated e-mail addresses in From, To, and Cc header fields, also reported and sample provided by Tomi Lukkarinen; - incorrect parsing of header fields could let a header field to be ignored when preparing notification templates, or when adding a spam tag to a Subject header field, causing a second Subject header field to be inserted; reported by Mike Cisar; - untaint a policy bank name when it comes from an AM.PDP protocol request; symptom was a failure to insert a pen pals SQL record in a milter setup; reported by Peter Huetmannsberger; - smtp client code inappropriately concluded there is no progress being made when forwarding a message back to MTA, and exited a rw_loop when sysread returned status EAGAIN despite a 'select' setting an input-ready flag; the problem was detected on Solaris, although it could be more general; thanks to Aleksandr for a detailed problem report; - limit the number of filenames given as arguments to a file(1) utility to stay within a safe program argument space limit, run file(1) multiple times if necessary; - change the sprintf format for conversion of 64-bit SNMP-like counter values into a string (replaced %020d by %020.0f) to properly convert large values (beyond 32 bits) into strings on versions of Perl which are not compiled with support for 64-bit integers (Solaris?); reported by David Schweikert; OTHER - invoke unrar and rar without option 'av-', which is no longer valid since version of unrar 3.7.5, and was previously apparently just ignored; the rar documentation states that option 'av' is only available with registered versions, so it appears the 'av-' is redundant with rar as well. Unrar aborting with 'unknown option' error was reported by Piotr Meyer; - a new AV entry for fpscan 6.x, (F-PROT Antivirus for UNIX command-line scanner) based on information from Erik Slooff, Bruno Friedmann, and Steve); - a new AV entry for fpscand 6.x (F-PROT Antivirus for Linux/BSD/Solaris), based on initial research by Alexander Wirt, Henrik K, and F-Prot documentation, thanks also to Haukur Valgeirsson of F-Prot International; - ask_daemon_internal now keeps persistent connection to an F-PROT fpscand daemon, in addition to Sophie and Trophie daemons; - new AV entry 'bdscan' for a new version of BitDefender Antivirus Scanner for Linux and FreeBSD; thanks to Gary V; - updated README.postfix, thanks to Chris Pepper and Patrick Ben Koetter; - updated amavisd-new-docs.html; --------------------------------------------------------------------------- December 12, 2007 amavisd-new-2.5.3 release notes BUG FIXES - fix parsing a SMTP status response from MTA when releasing from a quarantine, when a MTA response did not include an enhanced status code (RFC 3463) (such as with old versions of Postfix); a parsing failure resulted in attribute "setreply=450 4.5.0 Unexpected:..." in an AM.PDP protocol response, even though a release was successful; reported by Ron Miller, John M. Kupski, investigated by Tony Caduto and Jeremy Fowler; - change parsing of addresses in From, To, and Cc header fields, avoiding complex Perl regular expressions which could crash a process on certain degenerate cases of these header fields; thanks for detailed problem reports to Carsten Lührs and Attila Nagy; - completely rewritten parsing of Received header field to work around a Perl regular expression problem which could crash a process on certain degenerate cases of mail header fields; problem reported by Thomas Gelf; - harden to some extent regular expressions in parse_message_id to cope better with degenerate cases of header fields carrying message-id; - sanitize 8-bit characters in In-Reply-To and References header fields before using them in Pen Pals SQL lookups to avoid UTF-8 errors like: penpals_check FAILED: sql exec: err=7, 22021, DBD::Pg::st execute failed: ERROR: invalid byte sequence for encoding "UTF8": 0xd864 - when turning an infection report into a spam report, avoid adding newly discovered virus names (i.e. fraud names) to a cached list if these names are already listed; previously the list would just grow on each passage through a cache, leading to unsightly long lists of spam tests in a report; based on a patch by Henrik Krohns; - fix diagnostics when an invalid command line argument is given; OTHER - reduce log clutter when certain Perl modules are loaded late, i.e. after chrooting and daemonizing, but still before a fork; now only issue one log entry by a parent process: "extra modules loaded after daemonizing: "; - slightly relax mail address syntax in subroutine split_address; - fetch additional information (tags) from SpamAssassin: TESTS, ASN, ASNCIDR, DKIMDOMAIN and DKIMIDENTITY, making them available through a macro 'supplementary_info' (if a version of SpamAssassin in use provides them); - updated DKIM section in amavisd-new-docs.html, removing the historical DomainKeys milter from examples; - declared a dummy subroutine dkim_key() and new dummy configuration variables @dkim_signature_options_bysender_maps, %signed_header_fields, $reputation_factor, @signer_reputation_maps and $sql_partition_tag, members of policy banks, in preparation for 2.6.0 - declared now for improved downgrade compatibility of 2.6.0 configuration files, if need arises. --------------------------------------------------------------------------- June 27, 2007 amavisd-new-2.5.2 release notes BUG FIXES - in a milter setup log_id was left undefined, which resulted in log lines without id, and a SQL constraint violation "Column 'am_id' cannot be null" when logging to SQL was enabled. The bug was introduced in 2.5.1; problem reported by Martin Svensson; - suppress a second quarantining attempt if the message also needs to be archived to the same location (same sql key or same local filename); reported by Wazir Shpoon; - adjust $socketname in amavisd-release to match its default counterpart in amavisd (i.e. /var/amavis/amavisd.sock); reported by Stanley Appel; NEW FEATURES - add snmp-like counters for PenPalsSavedFromKill, PenPalsSavedFromTag3 and PenPalsSavedFromTag2, which correspond to the number of messages since a program (re)start in which spam level would have exceeded a corresponding level had there not been for (negative) score points contributed by pen pals lookups. Note that for any message only one of the three counters could increment, the one corresponding to the highest level crossed. To find more information about rescued mail messages, search the log for a string 'PenPalsSavedFrom' (available at log level 2 or higher). Practical value: mail saved by pen pals from being blocked often indicate false positives by SpamAssassin; examining rules which contributed significantly to the score may indicate which rules need adjustment; - when preparing a SQL SELECT clause in lookup_sql, provide an additional placeholder %a in a clause template, which is much like the existing %k, but evaluates to an exact mail address (i.e. the same as the first entry in the %k list), which makes it suitable for SQL pattern matching; suggested by Daniel Duerr; - macro supplementary_info can supply information on two additional SpamAssassin tags: AUTOLEARNSCORE and LANGUAGES if corresponding plugins are enabled in SpamAssassin; see README.customize for the complete list; - provide two new subroutines available for calling from config files: include_config_files() and include_optional_config_files(), each take a list of filenames as arguments, and reads & evaluates them just like normal configuration files specified on a command line (option -c or a default amavisd.conf). This provides a simplified and uniform mechanism for 'including' additional configuration files, which formerly could be invoked through a perl do() function. The only difference between include_config_files and include_optional_config_files is that the former aborts if some specified file does not exist, while the later silently ignores specified but missing files. Both/each subroutine may be called multiple times, recursion is allowed (but some sanity limit to recursion is provided); based on a suggestion by Gary V. Example line in amavisd.conf: include_config_files('/etc/amavisd-custom.conf'); OTHER - provide a workaround for a crashing altermime by removing its leftover temporary file which would otherwise cause a temporary failure: TempDir::check: Unexpected file problem reported by Dennis A. Kanevsky; - add a mapping to 'doc' for a result 'Microsoft Installer' from a file(1) utility; it seems like versions 4.20 and 4.21 of file(1) (possibly earlier versions too) misclassify MS Word, Excel, and PowerPoint documents as 'Microsoft Installer'; problem investigated and a workaround suggested by Noel Jones, Mike Cappella and Michael Scheidell; - add a mapping to 'asc' for a result 'COM executable for DOS' from a file(1) utility; it seems like later versions of file(1) can misclassify a text in a GB2312 character set as a COM file; reported by Daniel J McDonald; - updated AV entry for ESET NOD32 Linux Mail Server again - command line interface (nod32cli): added a status 3 (e.g. corrupted archive) back to the list of clean statuses; the 3 was removed in 2.5.1 as the entry was substituted with the one from a NOD32 documentation; reported by Tamás Gregorics; - updated AV entry for 'F-Secure Antivirus for Linux servers' to cope with version 5.5 and later; a new entry provided by Peter Bieringer; - when a command line option -g requests changing of group ID, do so by calling POSIX::setgid, after also attempting to assign to perl variables $( and $), which may not work correctly on systems where group ID can be negative (like group 'nobody' being -2 on Mac OS X); follows a SpamAssassin problem report 3994, investigated by Sidney Markowitz; - when an AUTH command parameter (RFC 2554, now RFC 4954) is supplied on a MAIL FROM SMTP command but AUTH support has not been previously offered (like when authentication is disabled by an empty @auth_mech_avail), no longer treat the situation as a fatal error: 503 5.7.4 Error: authentication disabled but mercifully ignore the parameter and just log an informational message. This is a deviation from RFC 2554, but makes it friendlier for those insisting on running amavisd as a Postfix pre-queue smtp proxy; suggested by Alexander 'Leo' Bergolth; - adjust the list of pre-loaded perl modules required by SpamAssassin; - internal: pass a mail message to SpamAssassin as a GLOB instead of an array reference, saving one in-memory copy of a message during a SA call; - internal: make it slightly easier to switch message digest from MD5 to a Digest::SHA family by turning a hard-wired key length into a parameter (admittedly it is still ugly, requiring a change in three places for switching); also pave a transition from Digest::SHA1 to Digest::SHA; - documentation: updated files README.postfix and README.postfix.html now include a section 'Advanced Postfix and amavisd-new configuration' explaining a multiple cleanup service architecture; thanks to Patrick Ben Koetter; retired file: README.postfix.old - documentation: updated README.sql-pg to include a faster alternative to purging a SQL logging database: the alternative 'DELETE FROM maddr' on PostgreSQL runs faster by a factor of 1.5 to 2 from the one previously suggested; - suggestion: when using SpamAssassin plugin Rule2XSBody (available in more recent versions of SA), adding an entry like: Mail::SpamAssassin::CompiledRegexps::body_0 to the @additional_perl_modules list allows preloading of compiled rules. Adding the following two lines to amavisd.conf adds the directory name containing modules with compiled rules to Perl modules search path and allows Perl to find the listed module(s): my($sa_instdir) = '/var/lib/spamassassin/compiled/3.002001'; unshift(@INC, $sa_instdir, $sa_instdir.'/auto'); --------------------------------------------------------------------------- May 31, 2007 amavisd-new-2.5.1 release notes COMPATIBILITY WITH 2.5.0 - setting $bypass_decode_parts to true now also disables MIME decoding (see below); SECURITY - provides checking the number of archive members against $MAXFILES quota even when just listing an archive directory, providing some additional protection (besides a time limit) against runaway dearchivers (such as a recent Zoo archiver DoS); - please use the most recent versions of file(1) utility (currently 4.21) and recent versions of external dearchivers/decoders to avoid known security vulnerabilities in them; NEW FEATURES - introduced a variation of a message release from a quarantine, allowing a releaser to choose between forwarding a message to the back-end MTA port as usual (avoiding re-checking of a message), or to send it to MTA on its incoming port (normally 25) and let the message be rescanned, which might be useful after adjusting spam rules or antivirus database. It is implemented by: * adding a configuration variable $requeue_method (also a member of policy banks), with a default value: 'smtp:[127.0.0.1]:25' * extending the AM.PDP protocol with a 'request=requeue' attribute which can be used in place of a 'request=release', * enhancing the 'amavisd-release' utility program to choose between sending 'request=release' and 'request=requeue' based on its program name, i.e. by making a soft or hard link to amavisd-release (or its copy) named 'amavisd-requeue', the utility will send a 'request=requeue' in place of the usual 'request=release', e.g.: # ln -s amavisd-release amavisd-requeue $ amavisd-requeue spam/k/kg2P0rP9Lpu3.gz * enhancing amavisd daemon to choose between forwarding a released message either to $release_method or to $requeue_method destination based on a 'request' attribute value in an AM.PDP request; - new AV entry: ArcaVir for Linux and Unix, see below for links; - a new macro 'supplementary_info' gives access to some additional information provided by content scanners, such as a provided by SpamAssassin API routine get_tag. The macro takes two arguments, the first is a tag name (a name of some attribute which is expected to provide an associated value), the second argument is a sprintf format string and is optional, if missing a %s is assumed. Currently the only available attributes are AUTOLEARN, SC, SCRULE, SCTYPE, and RELAYCOUNTRY. These are nonempty only when an associated SpamAssassin plugin or function is enabled. BUG FIXES - fixed quarantining to a SQL database of messages with a null envelope sender address (broken in 2.5.0, causing such messages to tempfail); reported by Markus Edholm, Vahur Jõesalu and Michael Scheidell; - fixed parsing of certain broken 'From' header fields, which would result in a temporary failure and the following logged error: check_init2 FAILED: parse_address_list PANIC1 53 at /usr/local/sbin/amavisd line 3292 reported by Michael Scheidell; - avoid encoding nonprintable characters in X-Envelope-From and X-Envelope-To header fields in a quarantined message even if envelope mail addresses contain such invalid characters, so that a quarantine release is possible; (RFC 2047 allows encoding of a 'phrase' in From, To, and similar headers, as well as in comments, but not in the address specification); - avoid unnecessarily RFC 2047 -encoding of 8-bit characters in those lines of inserted X-Spam-Report (and similar) multiline header fields which only contain ASCII characters; also avoid encoding of newlines; reported by Anant Nitya; - sanitize 8-bit characters in SpamAssassin report before inserting it into an X-Spam-Report header field; - properly recognize PostgreSQL error code 'S8006' and reconnect to a disconnected server right away; thanks to Brian Wong; - call $mail_obj->finish after a SA call to allow for garbage collection and removal of SA temporary files; see: http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5444 - avoid nonstandard SMTP status code 254 on discarded malware; on discarding turn status 554 into a 250 instead; violation of a SHOULD in RFC 2822 pointed out by Alexander 'Leo' Bergolth; - an informational log message was reported inappropriately: INFO: truncated ... header line(s) longer than 998 characters it didn't reflect reality, it was always reported together with the: INFO: unfolded 1 illegal all-whitespace continuation lines - when a SMTP option BODY=8BITMIME (RFC 1652) is not given on mail reception, avoid turning it on while forwarding, even if mail body contains 8-bit characters; following a garbage-in-garbage-out principle, this doesn't break anything that isn't already broken, but might prevent later conversion to 7-bit quoted-printable MIME by some downstream MTA, invalidating signatures (DKIM, S/MIME, PGP, ...) - at a risk that some overzealous firewall might block a mail transfer; - fixed a couple of documentation typos/bugs in README.customize, thanks to Mike Cappella; OTHER - modified code for checking each eval {} status: it turns out that eval is able to capture certain error conditions (e.g. certain I/O errors) but without setting the $@ variable, leaving it empty; use new idiom throughout for proper error handling and more informative reporting, showing errno in such cases; - setting $bypass_decode_parts to true now also disables MIME decoding, not just decoders/dearchivers listed in a @decoders list, and also implicitly retains full original message for virus checking, equivalent to having a regular expression /^MAIL$/ in a @keep_decoded_original_maps list; prompted by Bill Landry; - new AV entry: ArcaVir for Linux and Unix, see: http://www.arcabit.pl/ http://www.arcabit.com/download_product.html?product=ArcaVirLinux2007 http://www.arcabit.com/products_arcavir_for_unix_2006.html the entry was kindly provided by Michal Seremak; - updated AV entry for ESET NOD32 Linux Mail Server - command line interface (nod32cli), version 2.7, thanks to Simon; - updated AV entry for Sophos sweep, adding options -mime and -oe ; - avoid repeatedly reporting the same set of modules by a log entry 'extra modules loaded:', only report it on changes to the list; repeated reports could be misinterpreted that modules were loaded with each mail task, where actually missing modules were only loaded once within each child process; - avoid reporting 'BOUNCE' in a SMTP response text when a bounce (i.e. a nondelivery status notification) was actually suppressed, such as is usually the case with infected mail or when spam score exceeds spam_dsn_cutoff_level. Previously the SMTP response text only reflected the setting of a final_*_destiny, which could mislead mail administrators into believing that excessive unconditional backscatter was being generated. The new text looks like: 250 2.5.0 Ok, id=67685-15, DISCARD(bounce.suppressed) instead of previous: 250 2.5.0 Ok, id=67685-15, BOUNCE A general note worth reiterating: to reduce backscatter pollution (sending of bounces to innocent sender addresses), please either: * set $final_virus_destiny and $final_spam_destiny to D_DISCARD or to D_PASS (_not_ to D_REJECT or D_BOUNCE), or: * carefully configure virus and spam bounce suppression by: . configuring @viruses_that_fake_sender_maps correctly (the default is fine, it suppresses all bounces to infected mail), this way one may safely set $final_virus_destiny to D_BOUNCE, it is equivalent to D_DISCARD for all infected mail containing malware matching the @viruses_that_fake_sender_maps; . and: configuring @spam_dsn_cutoff_level_maps and @spam_dsn_cutoff_level_bysender_maps, keeping levels just slightly over a kill level, have a well maintained SpamAssassin with network tests enabled and updated rules - then one may set $final_spam_destiny to D_BOUNCE, which will produce bounces for mail with spam score between kill level and cutoff level, and suppress bounces above a suppress level; some domains may still consider such practice abusive, so do not take this choice lightly; . to monitor bounces generated by amavisd, one may assign some dedicated monitoring e-mail address to $dsn_bcc, which will then receive a copy of all delivery status notifications sent out by amavisd; - dspam options changed with version 3.8.0, replacing option --feature with --tokenizer; reported by Jim Knuth; - modified syslog writing to check errno after calling Unix::Syslog::syslog, and to informatively attempt to log status when unsuccessful; an unsuccessful status is just informational, as syslog(3) routine does its own retries and leaves an unsuccessful status of a previous attempt in errno even if a subsequent logging attempt did succeed; unfortunately the system routine syslog(3) returns no value according to documentation (and according to its source code), so its completion status can not be tested; a problem of a loss of logging on a syslogd restart on OS X was reported by Paul Walker, but unfortunately can not be solved on the application side; - uncomment some debugging printouts in p0f-analyzer.pl and land them under control of a $debug variable; --------------------------------------------------------------------------- April 23, 2007 amavisd-new-2.5.0 release notes COMPATIBILITY WITH 2.4.5 The 2.5.0 is upwards compatible with 2.4.* versions, except for the following: Default notification and logging templates are enhanced to take advantage of new macros and new concepts, so it is prudent to update templates if defaults are overridden, e.g. $log_templ, $notify_*_admin_templ, ... A client-side AUTH (rfc2554: SMTP Service Extension for Authentication) is currently not available. The last version with this feature working is amavisd-new-2.5.0-pre4. The feature may be restored in a future version if sufficient interest is demonstrated. A workaround for a qmail bug (which complains when CR and LF are split across a TCP segment boundary) is no longer available, as the program has no control over IP packet splitting done by the TCP/IP stack. When pen pals feature is in use, it is worth creating an index on a msgs.message_id field. NEW FEATURES AT A GLANCE - new concept: blocking contents category; - true per-recipient defanging/sanitation of a mail body (previously a true per-recipient handling was available for mail header edits, but not for mail body modifications); - added interface code to invoke Anomy Sanitizer or the 'altermime' program allows defanging or adding disclaimers by external utilities on a per-recipient basis; - rewritten SMTP client code: get rid of the troublesome module Net::SMTP; new code now supports pipelining, client-side LMTP, IPv6, Unix sockets, more reliable error detection and handling, passes on ENVID parameter unchanged, is bare-CR-clean, tidier code (no more workarounds for rough corners in Net::SMTP), fewer context switches (handshake handovers) due to pipelining if pipelining is offered by MTA (which usually is); - makes available pedantically parsed addresses from a mail header: From, Sender, To, Cc. Addresses from mail header may be needed for deciding on inserting disclaimers, signing mail (DKIM), custom hooks (like 'vacation'-type applications), and other future applications. Get rid of inexact parsing by module Mail::Address, provide own parser; - phishing fraud as returned by ClamAV is now treated as spam, no longer as a virus; - compatible with SpamAssassin 3.2.0; - enhancements to amavisd-nanny: shows more detailed states of processes; - enhancements to amavisd-agent: shows average processing times per message; - extended AM.PDP protocol with an attribute 'policy_bank' which may be used in a client's request to require loading additional policy banks; - add support for 7-Zip archives if external utility 7z is available; - custom hooks allow custom code to be called at few strategic places; - penpals can now also match replies which reference previous outgoing mail by its Message-Id (taking into account References or In-Reply-To header field); - new key 'originating' in policy banks generalizes a MYNETS policy bank; - a documentation rewrite for setting up amavisd-new with Postfix by Patrick Ben Koetter (one of the two authors of The Book of Postfix). Previous documentation has been renamed to README.postfix.old and will be removed in the next version; the new documentation is README.postfix.html, and its automatically converted plain text version is README.postfix. A big thanks to Patrick for his efforts! BUG FIXES - if a sender is both white- and black-listed at the same time, then inserted X-Spam-* header fields were inconsistent, e.g. X-Spam-Level, X-Spam-Flag and X-Spam-Status reflected a whitelisted status (no asterisks, not a spam), while X-Spam-Score showed 64 points; now whitelisting prevails in all X-Spam-* header fields; - relax argument parsing in amavisd-release to allow releasing of quarantine id containing a body hash in a name (%b in template); reported by Ron Rademaker; - skip a SQL-logging database operation if an associated clause in %sql_clause is disabled, e.g. set to undef or ''; this allows for example to selectively disable SQL logging based on a policy bank; thanks to Riaan Kok; - let LHA decoder (do_lha) recognize also other listing formats, e.g. MS-DOS, symlinks, not just plain Unix archives; problem reported by Ryuhei Funatsu; OTHER - catch and log uncaught '__DIE__' and '__WARN__' Perl pseudo-signals which would otherwise go to stderr and not be noticed with a daemonized process; patch by Alexander 'Leo' Bergolth; - insert 'X-Spam-Flag: NO' if spam level is above tag level but below tag2 level, previously it was not inserted at all (it is still redundant, but some may appreciate an explicit statement nevertheless); - drop support for Archive::Tar; main drawback of this module is: it either loads an entire tar into memory (horrors!), or when using extract_archive() it does not relativize absolute paths (which makes it possible to store members in any directory writable by uid), and does not provide a way to capture contents of members with the same name. Use pax program instead! - abandon the use of libnet (modules Net::SMTP and Net::Cmd), replaced by own code to implement client-side SMTP and LMTP protocol support, with a full support for pipelining and IPv6. Some of my issues with Net::SMTP go back to year 2002, but the one that broke camel's back is a 3+ months status quo in not fixing a serious misfeature introduced in 1.20, which mangles 8-bit characters in mail, causing a series of support questions, and even affecting some larger service providers without them realizing there is a problem. Here are some relevant bug reports: http://rt.cpan.org/Public/Bug/Display.html?id=24835 http://rt.cpan.org/Public/Bug/Display.html?id=2608 http://rt.cpan.org/Public/Bug/Display.html?id=2607 http://rt.cpan.org/Public/Bug/Display.html?id=14875 http://rt.cpan.org/Public/Bug/Display.html?id=9394 P.S. libnet-1.21 eventually fixed the UTF8 encoding problem and added support for ENVID and AUTH options; other issues are still open (e.g. split code/text in smtp status, no pipelining support, no LMTP); - represent score in X-Spam-Status header field as a single numeric field instead of the previous explicit sum of a SA score and a score boost, which some MUAs don't know how to interpret and label mail spaminess incorrectly; - recipient notifications (in their default template) now keep the original To and Cc header fields, instead of placing a recipient envelope address into a To header field; suggested by Jorgen Lundman; - no longer removes an X-Amavis-Alert header field (as inserted by some foreign MTA), it does not hurt to keep it; - kavscanner AV entry (Kaspersky Antivirus): added a new entry to the search list of paths to a binary, kavscanner changed its default installation location again; provided by Gary V; - added a rule into a $map_full_type_to_short_type_re list, mapping a file(1) type 'RIFF...animated cursor' into '.ani'; a patch by Eric; - example rules (commented-out) to block animated cursors (ANI) and icons are added to configuration files amavisd.conf and amavisd.conf-sample; - suppress bounces not only for Precedence: (bulk|list|junk) or null return path (as before), but also for mail with a 2822.From header field matching one of: *-request | *-owner | *-relay | *-bounces | owner-* | postmaster | mailer-daemon | mailer | uucp, and for mail containing a header field List-Id (RFC 2919). Note that this new additional rule practically never triggers in practice, true bounces and common mailing list traffic is already covered by previous checks; - make available a list of detected virus names in an Amavis::In::Message object (virusnames), suggested by Tom Sommer; - split SQL documentation into three files: README.sql general SQL considerations and some examples README.sql-mysql MySQL-specific notes and schema README.sql-pg PostgreSQL-specific notes and schema (also SQLite) - README.sql-pg now adds CHECK (x >= 0) for fields that are supposed to contain unsigned integers; suggested by Hanne Moa; - repurpose/rename a contents category CC_TEMPFAIL to CC_MTA; it is turned on when MTA rejects or tempfails a mail when amavisd attempts to pass it back after checking it; (this should not normally happen, MTA should be doing most of its checks (e.g. recipient validation) before passing mail to a content filter, not after); NEW FEATURES - custom hooks allow custom code to be called at few strategic places: * during child process initialization: allows initialization of custom code, including establishing an additional SQL session or similar; * after built-in checks: allows custom code to inspect and/or modify results of checks; * before sending, allows for additional quarantining and for sending additional notifications; * at the end of message processing: allows inspecting results of checks and status of mail forwarding (e.g. for statistics or logging purposes). Mail processing sequence: child process initialization *custom hook: new() loop for each mail: receive mail mail checking and collecting results *custom hook: checks() - may inspect or modify checking results deciding mail fate (lookup on *_lovers, thresholds, ...) quarantining sending notifications (to admin and recip) *custom hook: before_send() - may send additional notifications, additional quarantining, may modify mail forwarding (unless blocked) sending delivery status notification (if needed) issue main log entry, manage statistics (timing, counters, nanny) *custom hook: mail_done() - may inspect results endloop after $max_requests or earlier If Amavis::Custom::new returns undef then no further custom calls are made. Otherwise, this method is supposed to return an object, thus enabling other custom hooks, which receive this value as their first argument. See amavisd-custom.conf for an example, and the example invocation of amavisd-custom.conf at the end of file amavisd.conf-sample; thanks to Kasscie (Yohanna Monsalvez); - formerly penpals could only match replies to previous outgoing mail where envelope sender and recipient addresses are exactly reversed. Now, in addition to this, penpals can also match replies which reference previous outgoing mail by its 'Message-ID' (taking into account the 'References' or 'In-Reply-To' header fields), even if the envelope sender address of the reply is null or does not match a recipient address of a previous outgoing mail. This covers for incoming replies to mailing list postings, incoming message disposition notifications (MDN, RFC 3798) and incoming replies from alias or role addresses. A query on a message-id is fast compared to matching on recipient id, and if it succeeds, the later one is skipped. Based on a suggestion and a patch by Alexander 'Leo' Bergolth; The %sql_clause now contains one additional SQL SELECT clause under 'sel_penpals_msgid' key, which is used in place of 'sel_penpals' when a list of references in a reply is nonempty. It is worth creating an index on a msgs.message_id field to speed up SQL lookups: CREATE INDEX msgs_idx_mess_id ON msgs (message_id); Note that a spammer can gain a (small) advantage by including a reference to a recent outgoing message in his message. With private correspondence this information is hard to come by efficiently in a timely manner, and can effectively be exploited only if generated Message-IDs of outgoing mail is very easy to guess (e.g. generated by a very poor random number generator). With postings to public mailing lists this information is more readily available. Setting $sql_clause{'sel_penpals_msgid'} to undef or to empty disables matching on message-id, if it turns out the mechanism is being actively exploited. - added penpals snmp-like counters, which are displayed by amavisd-agent like the following: PenPalsAttempts 10222 438/h 18.3 % (InMsgsRecipsInboundOrInt) PenPalsHits 2957 127/h 48.9 % (ContentCleanMsgsInboundOrInt) PenPalsHitsMid 88 4/h 3.0 % (PenPalsHits) PenPalsHitsMidNullRPath 26 1/h 0.9 % (PenPalsHits) PenPalsHitsMidRid 917 39/h 31.0 % (PenPalsHits) PenPalsHitsRid 1926 82/h 65.1 % (PenPalsHits) Some comments on the above figures: - PenPalsAttempts shows that SQL lookups are performed only on a fraction of all incoming and internal mail (and not at all on outgoing mail) because a lookup is not performed on a high score spam (score above $penpals_threshold_high, which was 8 in this example, typically the same as kill level); - PenPalsHits shows that more than half of incoming clean messages can be associated with previous outgoing mail and can benefit from pen pals soft-whitelisting, such message is either a reply directly referencing a previous mail, or a new conversation between a sender/recipient pair which had previous conversation on a different topic; - PenPalsHitsMidRid shows that 1/3 of incoming matching replies match a previous outgoing mail by both the Message-ID as well as the exact recipient address; these are true replies to previous outgoing messages; - PenPalsHitsRid shows that 2/3 of incoming matching replies match a a previous outgoing mail by recipient address, but not by a Message-ID; these messages usually correspond to new topics among previous correspondents; - PenPalsHitsMid match only in reference to a previous Message-ID, but not by sender/recipient addresses; these are usually mailing list replies to a previous posting by a local user; - PenPalsHitsMidNullRPath are messages with null return path which are matching some previous outgoing message by Message-ID; these are usually incoming message disposition notifications (MDN, RFC 3798), or certain bounces which specify In-Reply-To or References in their header; - configuration variable %defang_by_ccat is renamed to %defang_maps_by_ccat and may now contain a list of by-recipient lookup tables (or a boolean as before for compatibility); this allows defanging/mangling to be selected on per-recipient basis; compatibility is retained by making the old variable %defang_by_ccat an alias for %defang_maps_by_ccat; - provided interface code to allow mangling/defanging/sanitation to be performed by an external utility, either by directly calling a Perl module Anomy Sanitizer (within the same process, avoiding startup cost), or by invoking a program 'altermime' (or by internal defanging code as before). Mail body defanging is only allowed for local recipients (those matching @local_domains_maps), i.e. for inbound and internal-to-internal mail. If there is more than one mangling code option available, the result of a %defang_maps_by_ccat can choose between them by returning one of the following strings, the selection can depend on mail content type and on by-recipient lookups if needed: 'anomy' chooses Anomy Sanitizer (if $enable_anomy_sanitizer is true); 'altermime' chooses a program whose path is $altermime (if found); 'attach' chooses the traditional amavisd-new defanging method which pushes an original mail message to an attachment; 'null' for testing purposes - doesn't modify mail body, but pretends it does (in logging and mail header); other non-empty and non-zero value automatically choose one of the above options depending on what is available; at least the 'attach' is always available; an empty, zero or undef value disables mail body modifications; Controls: $enable_anomy_sanitizer, @anomy_sanitizer_args, and: $altermime, @altermime_args_defang; Typical use: # with altermime: $altermime = '/usr/local/bin/altermime'; @altermime_args_defang = qw(--verbose --removeall); # with Anomy Sanitizer: $enable_anomy_sanitizer = 1; @anomy_sanitizer_args = qw( /usr/local/etc/sanitizer.cfg ); $defang_spam = 1; # old style, applies the first available mangler # to all spam-loving local recipients # unnecessarily complicated example of selective choices: $defang_maps_by_ccat{+CC_BANNED} = [ 'altermime', # use altermime for everybody (a 'constant' lookup table) ]; $defang_maps_by_ccat{+CC_SPAM} = [ { # a per-recipient hash lookup table 'user@example.com' => 1, # old style, auto-selects a mangler 'user-a@example.com' => 'anomy', 'user-m@example.com' => 'altermime', 'user-t@example.com' => 'attach', '.example.net' => 0, # no mangling }, $defang_spam, # fallback to old style setting if no match above ]; - a special case of mangling is adding a disclaimer, by invoking an external program 'altermime' (if available and enabled). This differs from mangling inbound mail in two details: * uses a separately configurable list of arguments to altermime: @altermime_args_disclaimer; and * it applies only to mail submitted from internal networks or roaming users (as recognized through a policy bank which sets: allow_disclaimers => 1), and where any of the following addresses matches local domains: author (2822.From) or sender (2822.Sender) or return path (2821.mail_from); Typically the $allow_disclaimers should be set by a policy bank which also sets the $originating flag. In addition to strings that may be returned by %defang_maps_by_ccat as described above, there are two more, only taken into account when $allow_disclaimers is true: 'disclaimer' invokes $altermime program for outgoing mail with arguments as given in @altermime_args_disclaimer; 'nulldisclaimer' for testing purposes - doesn't modify mail body, but pretends it does (in logging and mail header); Typical use: $altermime = '/usr/local/bin/altermime'; @altermime_args_disclaimer = qw(--verbose --disclaimer=/etc/altermime-disclaimer.txt); $defang_maps_by_ccat{+CC_CATCHALL} = [ 'disclaimer' ]; @mynetworks = qw( ... ); $policy_bank{'MYNETS'} = { # mail originating from our networks originating => 1, allow_disclaimers => 1, } For the moment there is one limitation: there can only be one mangler in effect at a time, it is not currently possible to both defang and to append a disclaimer on the same message: for internal-to-internal mail inserting a disclaimer takes precedence. To make it possible to provide different disclaimer texts when hosting multiple domains, there is an experimental additional configuration variable available: the @disclaimer_options_bysender_maps. It is a list of lookup tables, looked up by a sender address. The sender address is chosen from the following list, first match wins: * 'Sender:' header field, if its domain matches @local_domains_maps; * 'From:' header field, if its domain matches @local_domains_maps; * envelope sender address, if its domain matches @local_domains_maps; We already know that at least one of the above will match, otherwise adding disclaimers would be skipped at an earlier stage. The result of lookups should be one simple string, which replaces a string '_OPTION_' anywhere in @altermime_args_disclaimer elements. Typical use: @altermime_args_disclaimer = qw(--disclaimer=/etc/_OPTION_.txt); @disclaimer_options_bysender_maps = ( { 'host1.example.com' => 'altermime-disclaimer-host1', 'boss@example.net' => 'altermime-disclaimer-boss', '.example.net' => 'altermime-disclaimer-net', '.' => 'altermime-disclaimer-default' }, ); It is currently not possible to disable adding disclaimers through @disclaimer_options_bysender_maps results. This needs to be improved. The exact interpretation of the @disclaimer_options_bysender_maps lookup result may change in the future (which is why I call it 'experimental'). Note that disclaimers are pretty much useless legally. If you can help it at all, please avoid the pollution. See: http://www.goldmark.org/jeff/stupid-disclaimers/ - as mentioned above, the new SMTP/LMTP client code now supports a LMTP protocol too. This allows amavisd-new to act as a LMTP-to-LMTP content filter, possibly being inserted between MTA and a LMTP-based mail delivery agent such as Cyrus (if checking of outgoing mail is not needed). LMTP is selected when the first field of a $*_method (such as $forward_method, $notify_method, $resend_method, $release_method, $*_quarantine_method) is a 'lmtp:'. Possible uses: $forward_method = 'lmtp:/var/imap/socket/lmtp'; # over a Unix socket or: $forward_method = 'lmtp:[127.0.0.1]:24'; # over IPv4 or: $forward_method = 'lmtp:[::1]:24'; # over IPv6 If a Postfix 'lmtp' service is used to feed amavisd (instead of the more usual content filter feed through a service named 'amavisfeed' or 'smtp-amavis'), make sure not to forget to limit the number of concurrent feeds to amavisd (e.g. lmtp_destination_concurrency_limit=15) to a value same (or less) than $max_servers, or limit the maxproc field in master.cf such as: 'lmtp unix - - n - 15 lmtp' . Note that invoking amavisd as a LMTP delivery agent has a disadvantage that outgoing mail is not being checked, so infected internal hosts are able to pollute the world. Also the pen pals feature is no longer useful, as it requires the information on previous outgoing mail to be present in a SQL database. - a new command line option -i, it takes one argument which can be any string (an instance/personality name), which is then made available to amavisd.conf in a variable $instance_name (intended to be read-only); code in amavisd does not assign any semantics to this argument and does not use it for any purpose, it is purely intended for administrator's use in amavisd.conf if desired; this simple mechanism may facilitate running multiple instances of amavisd using a single configuration file, or to choose at startup time between amavisd personalities using the same config file; A possible usage is to start a test instance of amavisd while a production amavisd is still running, and letting a test instance listen on its dedicated TCP port number. Each server instance needs its own pid and lock files, its own TCP port number or socket name, and its own $db_home (nanny, cache, agent) unless bdb usage is disabled. A working directory may be shared or kept separate. An example to put by the end of amavisd.conf: if ($instance_name eq 'test') { $log_level = 5; $sa_debug = 1; $max_servers = 1; $TEMPBASE = "$MYHOME/tmp-am2"; $ENV{TMPDIR} = $TEMPBASE; $pid_file = "$MYHOME/home/amavisd2.pid"; $lock_file = "$MYHOME/home/amavisd2.lock"; $enable_db = 0; $inet_socket_port = [8888]; # listen on port 8888 } Start a test instance: # amavisd -i test debug and submit a test message to it, e.g.: $ mini_sendmail -fpostmaster@example.net \ -s127.0.0.1 -p8888 test@example.net <0.msg - policy banks now contain a new key 'originating', which generalizes a previously hard-wired policy bank MYNETS. It is a boolean variable, turned on automatically in the currently loaded policy bank when a smtp client's IP address matches @mynetworks_maps, to retain full compatibility with existing setups. When a new policy bank is loaded over a current one, the new policy bank may also modify the 'originating' key - a typical use is to turn it on by a policy bank activated by mail submission from authenticated roaming users (SASL/AUTH), so that such users are treated as locals (originating ma