Mailfromd NEWS -- history of user-visible changes. 2007-05-12
Copyright (C) 2005, 2006, 2007 Sergey Poznyakoff
See the end of file for copying conditions.

Please send mailfromd bug reports to <bug-mailfromd@gnu.org.ua>


Version 4.0, 2007-05-12

Note for users of 3.1.x: see also the notes for previous alpha
(3.1.9x) versions.

* SIGHUP handling

  SIGHUP instructs `mailfromd' to restart itself.  

* rc.mailfromd reload

  The `reload' option given to `rc.mailfromd' instructs it to send
SIGHUP to the running instance of the program.

* mtasim

  The `mtasim' utility is an MTA simulator for testing and debugging
mailfromd filter scripts.  It supports stdio (-bs) and daemon (-bd)
modes, has GNU readline support and `expect' facility, which makes it
useful in automated test cases.

  See the documentation, chapter `mtasim'.

* `begin'/`end' handlers

  The `begin' and `end' special handlers may be used to
supply startup and cleanup code for the filter program.

  The `begin' special handler is executed once for each
SMTP session, after the connection has been established but
before the first milter handler has been called.  Similarly, an
`end' handler is executed exactly once, after the connection has
been closed.  Neither of handlers takes any arguments.

  See the documentation, section `begin/end'.

* Cache control

  Use function `db_set_active' to enable or disable given cache
database.  E.g.

  # Disable DNS cache:
  db_set_active("dns", 0)
  # Enable it back again: 
  db_set_active("dns", 0)

Similarly, the function `db_get_active' returns a number indicating
whether the given cache database is used or not.   

** Bugfixes
* Fix a long-standing bug in parsing the --predict option argument.  Is there
anybody using this option at all?
  

Version 3.1.91, 2007-04-23

* Non-blocking syslog

This version is shipped with non-blocking syslog implementation by
Simon Kelley.  You may wish to enable it if you noticed that the
number of mailfromd processes grows uncontrollably and the processes
are hung for prolonged amounts of time.  Usually this indicates that
the daemon blocks in syslog() calls.  Read the description of
`--enable-syslog-async' option in chapter `Building' for the detailed
discussion of this (try `info -f doc/mailfromd.info --index-search
syslog-async').

* SPF support

The function check_host() tests the SPF record for the given
identity/host name.  The syntax is:

 number check_host(string ip, string domain, string sender, string helo)

See the documentation, node `SPF functions' for the detailed
description of this function and the related functions and variables.

* next and pass

Use `pass' instead of `next'.

The `next' keyword has changed its semantics: it is now used to
resume the next iteration of the enclosing loop statement (see
below).

For compatibility with the previous versions, its use outside of a
loop statement is still allowed, but a warning is issued.  You are
encouraged to replace all occurrences of `next' in your configuration
scripts with `pass'.

* Loop

The loop statement is implemented.  Its syntax is:

loop [name]
     [for <stmt>,] [while <stmt>,] [<stmt>]
do
     ...
done [while <stmt>]

See the documentation, section `Loop Statements'.

* break and next

The `break' statement exits from the enclosing loop.

The `next' statement resumes the next iteration of the enclosing loop
statement.

Both statements take an optional argument specifying the identifier
(name) of the loop to break from (or continue), this allows to build
complex iterations consisting of nested loops.  For example, in this
code (line numbers added for clarity):

 1   loop outer for set i 1, while %i < %N
 2   do
 3	...
 4	loop for set j 1, while %j < %i
 5	do
 6	   if foo(%j)
 7	     break outer
 8         fi
 9	done
10  done		   		   
11 accept

if the call to `foo' in line 6 returns true, the control is immediately passed
to `accept' in line 11.

* Resizable stack

The runtime stack of the MFL grows automatically as the need arises.
Thus, `#pragma stacksize' sets the initial size of the stack, and
the `Out of stack space' error, which was common in the previous
versions, now can occur only if there is no more virtual memory left.

Whenever the stack gets expanded, mailfromd issues a warning message
to the logs, notifying of the new stack size, e.g.:

    warning: stack segment expanded, new size=8192

You can use these messages to adjust your stack size configuration
settings. 
    
* Runtime stack traces

New command line option --stack-trace enables dumping stack traces
on runtime errors.  This might help localize the source of the error.
The trace looks like:

mailfromd: RUNTIME ERROR near ../mflib/match_cidr.mf:30: invalid CIDR (boo%)
mailfromd: Stack trace:
mailfromd: 0077: test.mf:7: match_cidr
mailfromd: 0096: test.mf:13: bar
mailfromd: 0110: test.mf:18: foo
mailfromd: Stack trace finishes
mailfromd: Execution of the configuration program was not finished

Each trace line describes one stack frame, the lines appear in the
order of most recently called to least recently called.  Each frame
consists of:

1. Value of program counter at the time of its execution
2. Source code location, if available
3. Name of the function called

The same output can be obtained by calling function stack_trace()
in your filter program.

See the documentation, section `Runtime Errors', for the detailed
description and examples.

* connect handler

Connect handler is implemented.

* envfrom and envrcpt

Both handlers take an additional second argument, containing the rest
of the SMTP command line.

* New functions

- string db_name(string fmt)

Return full file name of the database file corresponding to format
`fmt'

- number db_expire_interval(string fmt)

Return the expiration period for db format `fmt'

- string getmx(string domain [, number resolve])

Returns a whitespace-separated list of MX names (if `resolve' is not
given or is 0) or MX IP addresses (if `resolve'==1) for `domain'.  If
`domain' has no MX records, empty string is returned.  If the DNS
query fails, `getmx' raises an appropriate exception.

This interface differs from that of version 3.1.4 in that the calls to
getmx(domain) and getmx(domain,1) can return different number of
entries (see the docs for an example).

* #pragma regex stack

The `#pragma regex' statement can keep a stack of regex flags.  The
stack is maintained using `push' and `pop' commands.  The statement

   #pragma regex push [options]

saves current regex flags on stack and then optionally modifies them
as requested by options.

The statement

   #pragma regex pop [options]

does the opposite: restores the current regex flags from the top of
stack and applies [options] to it.

This statement is useful in include files to avoid disturbing user
regex settings.  E.g.:

   #pragma regex push +extended +icase
    .
    .
    .
   #pragma regex pop

* Optional arguments in user-defined functions

User-defined functions can take optional arguments.  In a declaration,
optional arguments are separated from the mandatory ones by a
semicolon.  For example:

    func foo(number a, number b; string c)

This function is declared with two mandatory arguments (a and b), and
an optional one (c).  Subsequently it can be invoked either as

   foo(x, y, z)      

or

   foo(x, y)

When invoking such functions, any missing arguments are replaced with
default values:

 - 0   for numeric arguments
 - ""  for string arguments

Thus, continuing our previous example, the invocation `foo(x, y)' is
equivalent to `foo(x, y, "")'.

* New statement #include_once

This statement works exactly like `#include' except that it keeps
track of the included files. If the requested file has already been
included, `#include_once' returns silently, while `#include' issues
an error message.

* New statement #require

Requires use of the named module, e.g.:

#require dns

See the documentation, section `Modules', for the description of MFL
module system.

* Internet address manipulation functions

- number ntohl (number N)
- number htonl (number N)
- number ntohs (number N)
- number htons (number N)
- number inet_aton (string S)
- string inet_ntoa (number N)
- number len_to_netmask (number N)
- number netmask_to_len (number N)

* DNS functions

DNS functions are reimplemented in two layers:

1. Primitive calls:

- string primitive_hostname (string IP)
- string primitive_resolve (string S [, string DOMAIN])
- number primitive_hasmx (string DOMAIN)
- number primitive_ismx (string DOMAIN, string IP)

These functions throw an exception if the requested lookup operation
fails.

2. Traditional calls:

- string hostname (string IP)
- string resolve (string S [, string DOMAIN])
- number hasmx (string DOMAIN)
- number ismx (string DOMAIN, string IP)

These are implemented in MFL and work exactly as their predecessors in
3.1.x branch.

To use the traditional calls, add the following statement at the
beginning of your script file:

  #require dns

(see the documentatio, section `Modules' for the description of
#require statement)

* Function `match_cidr'

This function has been reimplemented in MFL.  To use it, add
`#require match_cidr' at the top of your script source (see the
documentatio, section `Modules' for the description of
#require statement)

* Catch arguments

Catch takes two positional arguments: $1 gives the exception code,
$2 is the diagnostic string, explaining what happened in detail.

* New statement `throw'

The `throw' statement throws the given exception.  For example:

    throw invcidr "invalid CIDR (%cidr)"

* New command line option -v (--variable) allows to alter initial value of a
global variable, e.g:

  mailfromd -v ehlo_domain=mydomain.org

The old way of assigning values to the globals from the command line
(by prefixing an assignment with a percent sign) is discontinued.

* Pragmatic options `mailfrom' and `ehlo'

Both options create ambiguities in the language and are therefore
deprecated.  They are still understood but a deprecation warning is
issued when the parser sees them.  To update your scripts:

Change
  #pragma option mailfrom @var{value}
to   
  set mailfrom_address @var{value}

Change  
  #pragma option ehlo @var{value}
to  
  set ehlo_domain @var{value}

* Code generation redone to decrease memory requirements and make compiled
code self-sufficient.

* New statement `const' defines a named constant, e.g.:

  const n 10
  const s "String"

  if $f != s
     ...
  fi

In program text, constants are referred to by their name. In strings,
they are referred to using variable syntax (e.g. "%s").

* It is allowed to initialize variables in declarations.

For example, instead of

    number x
    set x 1

you can write

    number x 1

* Built-in macro __statedir__

New built-in macro `__statedir__' expands to the name of the default
program state directory.

* Changing state directory at run-time

It is possible using `--state-directory' command line option or
`#pragma option state-directory' statement.

* Bugfixes
** Fix incorrect packet length calculation in connect Milter handler.
The bug manifested itself with the following log messages:

- In mailfromd log:

   MailfromFilter: shan_connect: wrong length

- In Sendmail log:

   milter_read(mailfrom): cmd read returned 0, expecting 5
   Milter(mailfrom): to error state
   
** Fix coredumps on printing void returns with --dump-tree
** Fix coredumps on optimizing conditionals like

       if 0
	  do_something
       fi    

** Fix function context checks.

Previous versions bailed out on using context-limited built-ins (like `sa' and
`clamav') in functions. It is fixed. The context limit of the built-in
propagates to the function it is used in, that is defining

func sa_wrapper(string url)
do
	if sa(%url, 3)
	   discard
	fi
done

makes function `sa_wrapper' limited for use in `prog eom' only.	

** Fix passing of string handler arguments between handlers, as in
 
        prog helo
       do
 	  set x $1
        done
 
        prog envfrom
        do
 	  if %x = "dom.ain"
 	  ...	
 
(from 3.1.3)

** If, during sender verification, the remote server replies with
4xx to MAIL FROM command, do not try next sender address, but tempfail
immediately.
           

Version 3.1.90, 2006-12-13

* `==' can be used as well as `=' to test for equality

This is a bit of syntactic sugar for seasoned C programmers, who seem
to type == instinctively (like the author does).

* resolve takes an optional second argument

The argument specifies the domain.  For example, the following calls
are equivalent:

  resolve("puszcza", "gnu.org.ua")        = resolve("puszcza.gnu.org.ua")
  resolve("22.0.120.213", "in-addr.arpa") = hostname("213.130.0.22")  

* listens takes an optional second argument

The second argument specifies the port to use instead of the default 25.

* New functions

- message_header_decode(string TEXT, [string CS])

The TEXT must be encoded as per RFC 2047.  The function decodes it and
returns the resulting string.  Optional argument CS specifies the
character set for the output string.  Default is UTF-8.

- message_header_encode(string TEXT, [string ENC, string CS])

Encodes TEXT according to RFC 2047.  Optional arguments:

 ARG       Meaning          Default value
------+------------------+------------------- 
 ENC       Encoding         quoted-printable
 CS        Character set    UTF-8

Valid values for ENC are quoted-printable, Q, base64, B 

- unfold (string TEXT)

Unfold TEXT as defined in RFC 2822, section 2.2.3.  Return unfolded
string.

* Default logging facility is LOG_MAIL

* While checking sender validity, issue RSET if the previous MAIL
FROM returned 4xx.

Bugfixes:

* `Make install' creates state directory if it does not exist
* `clamav' function could cause coredumps when using socket ports.
* `resolve' function altered its argument if it was a CNAME.
* A typo in gram.y prevented some correct `matches' conditions from
being compiled. In particular, strip_domain_part.mf triggered this
error.


Version 3.1, 2006-12-07 

* Incompatible changes.

For detailed instructions on how to upgrade from version 3.0, please
see http://mailfromd.software.gnu.org.ua/upgrade.html

1. The package refuses to compile without DBM

2. The command line option --config-file (-c) is no longer supported.
To use an alternative filter script, give its name as an argument in the
command line, e.g.

   mailfromd my-script.rc

For backward compatibility, the invocation
`mailfromd --config-file my-script.rc' still works but produces a
warning message.

The semantics of `-c' will change in the next release.
   
3. The function `dbmap' takes an optional third argument.  If it is 1,
then the length of the lookup key will include the terminating null character.

In previous versions dbmap always counted the terminating null
character in the key length.  So, you should add the non-zero third argument
to the calls to dbmap to preserve their functionality.

4. Variables, implicitly declared within a function, are automatic.
Previous versions created them as global.

* Language changes

** Hex and octal constants

Usual C notation (0xNNN for hex and 0NNN for octal) is accepted.

** Bitwise operators: &, |, ^ (logical and, or, xor) and ~ (twos-complement)

** Search path for include files

The `#include' statement handles its argument the same way C
preprocessor does:  the argument is searched in the include file path,
if enclosed in angle brackets (<>), and in the current working
directory first and then in the include file path, if it is enclosed
in double quotes.  The default include file path is

/usr/share/mailfromd/include:/usr/local/share/mailfromd/include

plus $prefix/share/mailfromd/include if $prefix is not `/usr' or
`/usr/local'.

The command line option -I (--include) adds the named directory in
front of the default include path.

** Code optimization

Parse tree is optimized before code generation.  This can be
controlled using -Olevel option, where `level' is the optimization
level.  Currently implemented levels are 0 (no optimization) and 1
(full optimization), which is the default. 

** All variables are now strongly typed.

The declaration of the variable has the form: `TYPE NAME', where TYPE
is one of `string' or `number', and NAME is the variable name.  For
compatibility with the previous versions, the declaration is
optional.  If it is absent, the first assignment to the variable
defines its type.  Subsequent assignments will implicitly cast the
value being assigned to the type of the variable.

** New style of function declarations.  Named parameters.

Functions should be defined as:

func NAME (PARAM-LIST) returns TYPE

where TYPE is as described in the previous paragraph and PARAM-LIST is
a comma-separated list of parameter declarations in the form TYPE
NAME.  Consequently, instead of the positional notation, parameters
can be referenced by their names:

func sum(number a, number b) returns number
do
	return %a + %b
done

Within the function body, the named parameters can be handled the same
way as other variables, in particular they can be assigned new values
using `set' instruction.

For compatibility with the previous version, old type of function
declarations is supported as well.

** Automatic variables

Automatic variables are defined within a function or handler.  Their
scope of visibility ends with the terminating `done' statement.
Automatic variables are declared and referenced the same way as global
ones.  To declare an automatic variable, use `TYPE NAME' notation.
Variable declarations can be intermixed with executable statements.

The following example defines two automatic variables for
the function `foo':

func foo()
do
	number a
	string s
	...

If a variable is declared implicitly within a function or handler, it
is declared automatic.
	
See the documentation for the detailed description and examples.

** New functions:

*** I/O functions: open, close, write, getline
    See http://mailfromd.software.gnu.org.ua/manual/bi/io.html
    
*** Time functions: time, strftime
    See http://mailfromd.software.gnu.org.ua/manual/bi/system.html
    
*** System functions: system
    See http://mailfromd.software.gnu.org.ua/manual/bi/system.html
    
*** DBM functions: dbput, dbdel
    See http://mailfromd.software.gnu.org.ua/manual/bi/db.html

*** String functions: substr, index, rindex
    See http://mailfromd.software.gnu.org.ua/manual/bi/string.html

*** Debugging functions: debug, cancel_debug, program_trace,
    cancel_program_trace
    See http://mailfromd.software.gnu.org.ua/manual/bi/debug.html

*** Mail sending functions: send_mail, send_text, send_dsn
    See http://mailfromd.software.gnu.org.ua/manual/bi/mail.html

** The legacy function numrcpt() has been withdrawn

Use %rcpt_count instead.

** Built-in macros

Built-in macros have names beginning and ending with double underscore.
As their name implies, the macros are expanded to constant values.
The following built-in macros are defined:

1. __file__ expands to the name of the current source file
2. __line__ expands to the number of line in the current source file
3. __function__ expands to the name of the current lexical context,
i.e. the function or handler name.
4. __package__ expands to the string containing package name ("mailfromd")
5. __version__ expands to the textual representation of the program
version (e.g. "3.0.90")
6. __major__ expands to the major version number
7. __minor__ expands to the minor version number
8. __patch__ expands to the version patch level, or 0 if it is not defined.

Built-in macros can be used in variable context.  For example, to use
them within a string or here-document, prepend them with % as if they
were regular variables, e.g.:

   echo "%__file__:%__line__: Checkpoint"

* The envfrom and envrcpt handlers print entire argument array in the
debugging output. 

* New DNS caching scheme.

All DNS lookups are cached on global basis, as opposed to the per-session
basis in previous versions.  The cache is stored in the DBM database
`dns'.  It can be listed and otherwise operated upon using usual
mailfromd commands.

If a lookup gives a positive result, the TTL from the DNS record is
used as the record expiration interval.  For negative lookups, the
default interval of 3600 seconds is used.  It can be altered by the
following pragmatic comment:

  #pragma database dns negative-expire-interval N

* New command line option --xref

Produces a cross-reference listing of global variables.

* Fuller SMTP timeout control

In order to more fully control SMTP transactions, new timeout value
is introduced: initial-response-timeout.  This is the maximum time
to wait for the remote to issue the initial SMTP response.  This value
is especially useful for dealing with the servers that impose a
delay before the initial reply (most notably "CommuniGate Pro"
ones").  The default value is 30 seconds which should be enough for
most normal servers.  See the documentation, node "SMTP Timeouts" for
the detailed discussion.

* No more `retry' options.

The `retry' options and pragmas have been removed.  The new timeout
control scheme warrants that the polling will take at most the given
interval of time.  In particular, that affects:

** Command line option `--retry' 
** Pragma options io-retry and connect-retry

* Bugfixes
** Switch statements without the default branch produced incorrect code
(the very first branch was used as the default one).  This is fixed.
** Fix handling of escape sequences at the beginning of a string and before
the beginning of an interpreted sequence within the string.
** Fix the declarations of the built-in functions `toupper' and `tolower'.
** Fix storing the macro values obtained from Sendmail
** Collect zombie subprocesses as soon as possible
** Fix arithmetical expression syntax in rc.mailfromd
** Fix multiple from address handling
** Fix race condition when using GDBM
  

Version 3.0, 2006-11-05

* The mailfromd binary is now installed in ${prefix}/sbin.  Please,
update your scripts.  You are encouraged to update the startup script
(run `cp etc/rc.mailfromd /wherever-your-startup-lives'), since the new
version contains lots of enhancements (see below).

* The package no longer uses libmilter.

* Several `from' addresses can be specified both with polling functions
and in `#pragma option mailfrom' statement.  In this case the probing
will try each address until either the remote party accepts it or the
list of addresses is exhausted, whichever happens first.  This can
help if a remote host is picky about sender addresses. 

* After discussions with Jan, the final part of the standard poll method
has been redone.  Now the last-resort poll (i.e. querying the domain part
of the sender email, treated as an MX) is done only if the domain has
no MX records.

* New option --dump-macros shows all Sendmail macros used in the
configuration file, by milter states.  It is useful to create Sendmail
`Milter.macros.*' (confMILTER_MACROS_*) statements.

* rc.mailfromd stript two new options:

- rc.mailfromd configtest [FILE]

Checks configuration file syntax.  If FILE is not given, the default
one is assumed.

- rc.mailfromd macros [-c] [FILE]

Generate milter export statements for Sendmail configuration files.
Optional FILE specifies alternative mailfromd configuration file.
By default, `.mc' statements are generated.  Specifying `-c' option
instructs the script to create `.cf' statements instead.

* New pragmatic options `connect-retry' and `connect-timeout' set retry
count and timeout values for initial connections.  The corresponding values 
for I/O operations are set using `io-retry' and `io-timeout' options.
The pragmatic options `retry' and `timeout' are retained for backward
compatibility.  They are synonymous to their `io-' counterparts.  The
default values are:

#pragma option connect-retry 1
#pragma option connect-timeout 30
#pragma option io-retry 3
#pragma option io-timeout 3

* New function `sa' checks the message for spam via SpamAssasin spamd
interface.  It supplies additional data via the global variables
sa_score, sa_threshold, and sa_keywords.

* New function `clamav' checks the message for viruses via ClamAV daemon
interface.  Additional data (the virus name, if found) is stored in
the global variable clamav_virus_name.

* New function `ismx' returns true if the IP address or hostname given by its
second argument is one of the MX records of the domain name given by
the first argument.
 
* New variables `last_poll_host', `last_poll_send', `last_poll_recv' contain
the host name of the lastly polled host, the command sent and the
first line of the reply received.  The variables are set by polling
functions. 

* New variable `cache_used' is set to true if cache data were used instead of
polling, and to false otherwise.  The variable is set by stdpoll and
strictpoll built-in functions (and by `on poll' statement, accordingly).

* New string functions: `length' and `substring'

* Here document syntax expanded.

** Removing leading whitespace

Inserting a single space between the dash and the terminator word in
the beginning of the here-document construct, as in:

<<- WORD
...
WORD

instructs parser to remove leading white space characters
from each line of the document body, including terminating WORD.

** Expansion of macros and variables

Variables and sendmail macros are expanded when used within a
double-quoted string or a here-document body.  The variable expansion
and backslash interpretation is suppressed by quoting the WORD, e.g.:

<<\WORD
...
WORD

or

<<'WORD'
...
WORD

** Numeric escape sequences

Two new kinds of escape sequences are supported:

 \0ooo, where o is any octal digit
 \xhh, where h is any hex digit

** Back-references.

References to the parenthesized subexpressions of the previous regular
expression are expanded both in the code and in double-quoted string
literals.  For example:

 if domainpart $f matches '\(.*\).com'
   set d \1
 fi  

* Bugfixes
** Fix berkeley 4.x support
** Fix expiration of the greylist and rate databases.
** Fix returning multiline replies.  The last line of the reply was not taken
into account unless it ended with a newline. 
** Fix type casting of arguments to user-defined functions
** Fix argument passing in function calls generated for `on poll' statements.


Version 2.0

* Program requires Mailutils version 1.0 or newer
* Support for old DBM and NDBM has been withdrawn.
* Added support for Berkeley DB versions 3.x and 4.x

* INCOMPATIBLE CHANGES

** To use version 1.x configuration files, the following changes should
be applied to them:

 1. The entire code section should be enclosed in the following
    statement:

    prog envfrom
    do
	...
    done

    See the section `Handler declarations' below for the detailed
    description.
 2. Convert any `rate' statements to function calls, e.g.:

        if rate $f 180 / minute

    should be rewritten as

        if rate($f, interval(minute)) > 180

    See the section `rate(key, interval)' below for the detailed description.
    See also section "Special test functions" in the mailfromd documentation.

** Format of cache and rates database has changed: the key field now
includes the trailing nul character, which is also reflected in its
length.  This allows for empty (zero-length) keys.  To convert existing
databases to the new format, run 

   mailfromd --compact --all

after compiling the package.

** The database management options (--list,--delete,--expire) do not take
any argument.  To specify that the option refers to the rate database,
use --format=rate option.

* Language features

** Compiled code

Configuration file handling completely rewritten.  The file is parsed
into a pseudo-code program, which is executed considerably faster than
the parse tree in 1.x branch.

** Sendmail macros

Multiletter Sendmail macros can be used with and without
surrounding curly braces, i.e. ${rcpt_count} and $rcpt_count are both
valid. 

** File inclusion

Configuration file can include other files.  The syntax is:

#include "FILENAME"

Inclusion depth is not limited.

** Adjacent expressions concatenate:

  $f => "gray@gnu.org.ua"
  ${client_addr} => "127.0.0.1"

  $f "-" $client_addr => "gray@gnu.org.ua-127.0.0.1"

** Arithmetical expressions

The four usual arithmetical expressions are now supported.

** Comparisons

In addition to equal (=) and not equal (!=) the following comparison
operators are supported: <, <=, >, >=.  Their precedence and
associativity is the same as in C.

** Type casting

The rules for implicit type casts are:

  1. Both arguments to an arithmetical operation are cast to numeric
     types.
  2. All arguments of the concatenation operation are cast to string
     type. 
  3. Both arguments to `match' or `fnmatch' function are cast to
     string type.
  4. The argument of the unary negation (arithmetical or boolean) is
     cast to numeric type.
  5. Otherwise, if the arguments to a binary operation have different
     types, the right-hand side argument is cast to the type of the
     left-hand side argument.

There is no special syntactic sugar for explicit type casting.  To
cast an expression to string, concatenate an empty string to it:

  %var ""

To cast an expression to numeric, add it to zero:

  %var + 0

** Single-quoted strings

In addition to double-quoted strings, single-quoted ones are also
supported.  The single-quoted strings are not subject to backslash
expansion, thus they are particularly useful for writing regular
expressions: 

  if $f matches '.*\.com'

** Splitting strings between several lines.

Double-quoted strings can be split over several lines, by placing
a backslash immediately before the newline.  For example:

"A very\
 long string"

produces "A very long string". 
  
** Here document syntax and multiline sendmail replies

Mailfromd supports "here document" syntax:

<<[-]WORD
...
WORD

Optional '-' instructs parser to remove leading tabulation characters
from each line of the document body, including terminating WORD.  This
allows for here documents to follow normal program indentation. The
backslash expansion is performed on the contents of the here document,
unless WORD is quoted (i.e. either 'WORD' or \WORD).

This is particularly useful in providing multiline Sendmail replies:

  reject 550 5.0.0 <<-EOT
	 This service is not available now.
	 Please, refer to http://some.site for more information
	 on the subject.
	 EOT	 

** Handler declarations

The program consists of a set of handler declarations.  Each handler
is defined as

    prog STATE
    do
       ...
    done

where `...' represents the actual code, and STATE is the milter state
this handler is defined for. Recognized milter states are: helo,
envfrom, envrcpt, header, eoh, body, eom.

** Positional arguments ($N notation)

Handlers can take several positional arguments, which can be
dereferenced in the program using $N notation, where N is a decimal
number.  The semantics of the positional arguments depends on the state
for which the handler is designed.  For example:

    prog helo
    do
	/* For helo, $1 means helo domain */
	if $1 matches "gnu.org.ua"
	   ...
	fi
    done   

** Internal variables (% notation)

Mailfromd variables can be defined using `set' statement:

  set VAR expr

where VAR is the variable name.  The variables are dereferenced using
%VAR notation.  Their values are retained between handlers, for
example:

    prog helo
    do
	set helo_domain $1
    done

    prog envfrom
    do
	if $f = "gray" and %helo_domain matches "gnu.org.ua"
	   ...
	fi
    done 

The `set' statement can be used both inside and outside of program
handlers or functions.  When used outside them, it declares a global
variable and assigns it an initial value.  The variable will be
initialized to that value at the beginning of each message.
    
** Built-in function syntax

Built-in functions of more than 1 argument are invoked using the usual
C-like syntax, e.g.:

    name(arg1, arg2, arg3)

Built-in functions of one argument can be invoked with or without
parentheses:

    name arg
    name(arg)

** User defined functions

User defined functions are declared using the following syntax:

func NAME ( TYPELIST ) returns TYPE
do
	...
done

where NAME is the name of the function, TYPELIST is a comma-separated
list of parameter types, and TYPE is the return type of the
function. Types are designated by a single letter: 'n' denotes the numeric
type, 's' denotes the string type.

Within the function body, the arguments are denoted using positional
argument notation (see `Positional arguments' above).  The `return'
statement is used to return a value from the function.  For example:

func sum(n,n) returns n
do
	return $1 + $2;
done

** Switch statement

The mailfromd language now has switch statement.  Its syntax is
similar to that of C:

switch EXPR
do
   case VAL [or VAL...] :
	STMT
	.
	.
	.
   default:
	STMT		
done   

where EXPR is any valid mailfromd expression, VAL is a literal value
(numeric or string), STMT is any mailfromd statement or a list of
statements.  For example:

switch %x
do
	case 1 or 2:
	     echo "Accepting mail"
	     accept

	default:
	     reject
done

** Catch statement

The new `catch' statement can be used to handle exceptional conditions
occurring within a function.  An exceptional condition is signalled
when the filter script program encounters a condition it is not able
to handle.  See the documentation, section "Exceptions" for the
details.

The syntax of the `catch' statement is:

catch EXCEPTION-LIST
do
   HANDLER-BODY
done

where EXCEPTION-LIST is the list of exception types, separated by the
word `or'.  Special form `catch *' catches all exceptions.

The HANDLER-BODY is the list of statements comprising the handler body.

** On statement

The support for `on' statement has been entirely rewritten.  The
statement is implemented as a wrapper around `catch'.  Instead of
`poll' you can specify any function call as its selector value.

For example, this statement:

prog envfrom
do
        on poll $f do
        when success:
            accept
        when not_found or failure:
            reject 550 5.1.0 "Sender validity not confirmed"
        when temp_failure:
            tempfail 450 4.1.0 "Try again later"
        done
done

is actually a shortcut for:

function poll_wrapper(s) returns n
do
        catch &success or &not_found or &failure or &temp_failure
        do
              return $1
        done
        return stdpoll($1, %ehlo_domain, %mailfrom_address)
done

prog envfrom
do
        switch poll_wrapper($f)
        do
        case &success:
            accept
        case &not_found or &failure:
            reject 550 5.1.0 "Sender validity not confirmed"
        case &temp_failure:
            tempfail 450 4.1.0 "Try again later"
        done
done

(See also the description of %ehlo_domain and %mailfrom_address variables)

** MX matching

The `matches' and `fnmatches' operations has been extended to allow
operations on MX lists.  The test

   if ${client_addr} mx matches '.*\.gnu\.org'

yields true if the list of MXs for the IP address ${client_addr}
contains a host name that matches the regular expression
'.*\.gnu\.org'.  The left side argument can be either a hostname or IP
address or an RFC 2822 email address.  In the latter case, its domain
part is used to query for the MX records.

The similar 'mx fnmatches' construction is also available.

Both `mx' tests can throw one of the following exceptions: not_found,
failure, temp_failure.

** #pragma database

New pragma `database' controls various aspects of databases used by
the program.  The pragma has four forms:

1. Store the database DBNAME in the given FILENAME

   #pragma database DBNAME file FILENAME

2. Set the expiration interval for DBNAME:
   
   #pragma database DBNAME expire-interval INTERVAL

   INTERVAL can be any valid interval specification (see the next 
   section).

If DBNAME is "cache", two additional forms are understood:
   
3. Set the expiration period for positive cache records:

   #pragma database cache positive-expire-interval INTERVAL

4. Set the expiration period for negative cache records:

   #pragma database cache negative-expire-interval INTERVAL

The forms 3. and 4. are equivalent to

   #pragma option positive-expire-interval INTERVAL
and
   #pragma option negative-expire-interval INTERVAL

correspondingly.           

** Time interval specification

  Time intervals can be specified as an English text, e.g. "1 hour 35
minutes".  The following pragmas take a time interval specification as their
argument:

#pragma option timeout
#pragma option milter-timeout
#pragma option expire-interval
#pragma option negative-expire-interval
#pragma option positive-expire-interval
#pragma option rates-expire-interval
#pragma option lock-retry-timeout

* New built-in functions:

** dbmap(dbname, key)

Returns true if `key' is found in the database file `dbname', E.g.:

   dbmap("/etc/mail/aliases.db", $f)
   
** domainpart(str)

Returns the domain part of `str' if it is a valid email address,
otherwise returns `str' itself. 

** greylist(key, interval)

Returns true if the `key' is found in the greylist database
(controlled by `#pragma database greylist' pragma).  The argument
`interval' gives the greylisting interval in seconds.  The function
sets internal variable %greylist_seconds_left to the number of seconds
left to the end of greylisting period.  Sample usage:

 set gltime 500
 if greylist(${client_addr} "-" $f "-" ${rcpt_addr}, %gltime)
   if %greylist_seconds_left = %gltime
      tempfail 470 "You are greylisted for " %gltime " seconds"
   else   
      tempfail 470 "Still greylisted for " %greylist_seconds_left "seconds"
   fi
 fi		     

** hasmx(host)

Returns true if `host' has any MX records.  The function can throw one
of the following exceptions: not_found, failure, temp_failure.

** interval(string)

Converts its argument, which should be a valid time interval
specification, to seconds

** localpart(str)

Returns the local part of `str' if it is a valid email address,
otherwise returns unchanged `str'. 

** match_cidr(ip, cidr)

Returns true if the IP address `ip' matches the network block `cidr'.
For example:

  match_cidr(${client_addr}, "213.130.0.0/19")

Possible exceptions: invip, if the first parameter is not a valid IP,
and invcidr if the second parameter is not a valid CIDR.
  
** stdpoll(email, domain, mailfrom)

Performs standard poll for `email', using `domain' as EHLO domain and
`mailfrom' as MAIL FROM: address.  Returns 0 or 1 depending on the
result of the test.  Can raise one of the following exceptions:
failure, temp_failure.

In `on' statement context, it is synonymous to `poll' 
without explicit `host'. 

** strictpoll(email, domain, mailfrom, host)

Performs strict poll for `email' on host `host'.  See the description
of stdpoll for the detailed information.

In `on' statement context, it is synonymous to `poll host'. 

** _pollhost(ip, email, domain, mailfrom)

Poll SMTP host `ip' for email address `email', using `domain' as EHLO
domain and `mailfrom' as MAIL FROM: address.  Returns 0 or 1 depending
on the result of the test.  In contrast to strictpoll function, this
function does not use cache database and does not fall back to MX poll
if the poll tempfails.  The function can throw one of the following
exceptions: failure, temp_failure.

** _pollmx(domain, email, domain, mailfrom)

Poll MX-s of the `domain' for email address `email', using `domain' as EHLO
domain and `mailfrom' as MAIL FROM: address.  Returns 0 or 1 depending
on the result of the test.  In contrast to stdpoll function, _pollmx
function does not use cache database and does not fall back to host poll
if the poll fails.  The function can throw one of the following
exceptions: failure, temp_failure.

** tolower(string)

Returns a copy of the string str, with all the upper-case characters
in string translated to their corresponding lower-case counterparts.
Non-alphabetic characters are left unchanged.

** toupper(string)

Returns a copy of the string str, with all the lower-case characters
in string translated to their corresponding upper-case counterparts.
Non-alphabetic characters are left unchanged.

** rate(key, interval)

Returns the mail sending rate for `key' per `interval'.  This function
replaces `rate' statement from 1.x branch.  To convert old rate
statements use the following algorithm:

Old statement: if rate KEY LIMIT '/' EXPR
New statement: if rate(KEY, interval("EXPR")) > LIMIT

For example:

Old statement: rate $f 180 / 1 hour 25 minutes
New statement: if rate($f, interval("1 hour 25 minutes")) > 180

** resolve(host)

Returns the IP address corresponding to `host' or "0" if it cannot be
resolved.

** validuser(user)

Returns true if `user' is a valid local account.  It uses mailutils
authentication mechanisms.

* Global variables

** %ehlo_domain

Name of the domain used by polling functions in EHLO or HELO command.
It is set by `#pragma option ehlo' directive, or via --ehlo command
line option.  

** %mailfrom_address

Email address used by polling functions in 'MAIL FROM' command.  Set
by `#pragma option mailfrom' directive or via --mailfrom command line
option. 

** %rcpt_count

The variable %rcpt_count keeps the number of recipients given so far.
The variable is defined in the envrcpt state. 

* Database expiration

The operation of `mailfromd --expire' has been completely redesigned
to avoid skipping some keys when using GDBM.

* Database compaction

New option --compact starts "database compaction" process,
which removes all expired entries and empty blocks from the database. 
It also converts any obsolete (not nul-terminated) keys to
nul-terminated ones.  During this process the original database file is
locked for writing, so the running mailfromd instance is able to read
entries from it, but cannot write or update it.

The existed database will be replaced with the compacted version only
if there were no errors during the process.  If you wish to ignore any
failed reads (keys that were not retrieved), use the 
--ignore-failed-reads option. 

* Database locking

Before accessing, any database file is locked using kernel locking.
By default, if the first attempt to lock the file fails, two more
attempts are undertaken in 1 second intervals.  If the lock cannot be
acquired after the last attempt, the database file is opened in read-
only mode.  The number of locking attempt and the timeout value are
controlled by command line options --lock-retry-count and
--lock-retry-timeout, or the corresponding pragmas:

#pragma option lock-retry-count
#pragma option lock-retry-timeout

* Selecting the database format and file

** New option -H (--format) specifies which format is the database being
operated upon by any of the database management options.  Recognized
formats are:

  cache        Poll cache database
  rate         Sending rate database
  greylist     Greylisting database (see below)

** New option --all can be used with one of the options --expire or
--compact to apply the operation to all configured databases.  This is
useful to invoke mailfromd as a crontab job.
  
** New option --file allows to explicitely specify the database file name.
Notice, that in contrast to the previous version, the name should
include the suffix.

* Debugging

** New option --lint (-l, --syntax-check): check the configuration file
syntax.  

** The argument to --debug option should be the numeric debug level.  The use 
of characters 'c', 'd', 'l', 'y' is discouraged (although still
supported for a while).  See below for the alternatives.

** New option --dump-code dumps the listing of the assembled code on screen
(similar to the earlier --debug=c)

** New option --dump-tree dumps the parse tree in human-readable form
(earlier --debug=d option)

** New option --dump-grammar-trace prints grammar parser traces while parsing
the configuration file (earlier --debug=g option).

** New option --dump-lex-trace dumps lexical analizer traces
(earlier --debug=l option).

** New option -L (--log-tag) sets the identifier used in syslog messages.

** New option --source-info includes source line information into the
debugging messages.  Previously this information was included by default.

* New option --group (-g) allows to retain the given supplementary group when
switching to user privileges.  By default mailfromd does not retain
any supplementary groups.  The use of this option may be necessary if
your mailfromd script needs to access some databases that have
restrictive access privileges.  For example, if mailfromd runs with
the privileges of user 'mail' (the default) and needs to access
/etc/mail/aliases.db, which is usually owned by root.smmsp and has
access rights 0640, you should run

         mailfromd --group smmsp

* New option --source (-S) sets the source address mailfromd will use for
any TCP connections.  The configuration file equivalent is

         #pragma option source

* To set up the local account validation (see the description of `validuser'
function above) mailfromd uses authentication options from mailutils.
See the mailutils documentation, chapter `Authorization and Authentication
Principles' for the detailed description of these. 	 

* Code generation optimized to avoid unnecessary instructions and to reduce
code size.

* MX lookups no longer recurse to parent domains.  Previously, if
the domain "some.domain.com" had no MX records, mailfromd would lookup for
MXs of "domain.com" and use these instead.  This is no longer the
case.  

* Added testsuite


Version 1.4

* Configuration

Added possibility to link against the forked version of libmilter
(--with-forks). The patch for sendmail-8.13.1 is included
(etc/sendmail-8.13.1.diff).

* Configuration file

** New unary expression `listens' checks if the host listens on port 25.
** Several `#pragma option relay' statements accumulate

* Bugfixes

** Fixed coredump on incorrect libmilter socket specification.
** Fixed `poll for EMAIL as EMAIL'.


Version 1.3

* Rewritten DNS resolver functions in order to take into account CNAMEs.
* Updated Makefiles to allow for compilation with the CVS Mailutils
* Improved documentation.


Version 1.2

* Implemented sending rate control.

This feature allows to impose a limit on the number of messages a
user can send within a given interval. If this number is exceeded, the
connection is refused until enough time passes to keep the rate within
the given limit.


Version 1.1

Mostly bugfixes.

Version 1.0

Lots of major improvements. Implemented two methods of sender address
verification, controlled by a sophisticated configuration file. Sender
domains and emails can also be distinguished basing on POSIX regex or
shell-style globbing patterns.


Version 0.2

First release.


=========================================================================
Copyright information:

Copyright (C) 2005, 2006, 2007 Sergey Poznyakoff

   Permission is granted to anyone to make or distribute verbatim copies
   of this document as received, in any medium, provided that the
   copyright notice and this permission notice are preserved,
   thus giving the recipient permission to redistribute in turn.

   Permission is granted to distribute modified versions
   of this document, or of portions of it,
   under the above conditions, provided also that they
   carry prominent notices stating who last changed them.

Local variables:
mode: outline
paragraph-separate: "[	]*$"
eval: (add-hook 'write-file-hooks 'time-stamp)
time-stamp-start: "changes. "
time-stamp-format: "%:y-%02m-%02d"
time-stamp-end: "\n"
end:
