© Anton Voronin (anton@urc.ac.ru), 2000-2001.
The most recent version of this document can be found at:
TAS is designed to fetch and process the traffic statistics from
PC or Cisco routers (actually, with slight modifications - from any
traffic accounting capable devices) - on IP level and from specific applications
on application level.
The application level is needed because some "intermediate" services
(like http-proxy servers, news servers or mail relays) "hide" actual
user's traffic from IP level. For example, a client requests a large file
from the abroad via your http proxy server. On IP level you can notice
only the traffic between the client and your proxy server. So if you wish
to know all traffic flows initiated by or destinated for your clients
(either for billing, for setting traffic limits or just for estimating your
network usage per each client), you have to account the traffic on application
level as well. TAS can deal with the following applications: squid, sendmail
and MailGate.
What TAS can let you, in a few words:
Introduction
What TAS is not intended for:
Although there are currently so many limitations, there is also some kind of
whish list below.
TAS is written completely in Perl and consists of the following
components:
Design notes
The first four programs collect accounting data picked up from routers or specific applications. AcctMax does a specific processing required for IP data before it is processed by AcctLog. AcctLog builds arbitrary reports according to the rules specified in its configuration. AcctJoin summarizes daily databases into current month databases. Periodic scripts are responsible for running other TAS components, send the reports to operator and archive them.
Accounting data is stored in Berkeley DB tables. I know, it is not very smart idea to use db for this task because it leads to consequent search of the full database when selecting data for building reports. But it is very simple and convinient to summarize the data in hash tables tied to db tables.
After you have unpacked the archive, you'll see the Makefile. You don't need
to build or configure anything before install. To install the TAS just type:
Fetches IP accounting data from routers. Runs via cron every several minutes.
The program accepts the following command line switches:
-c cisco1[,cisco2...[,ciscoN]]
Comma-separated list of Cisco routers from which to fetch the data.
See Obtaining data from Cisco routers for details.
-p pc1[,pc2...[,pcM]]
Comma-separated list of PC routers from which to fetch the data.
See Obtaining data from PC routers for details.
You need to add the following to crontab on your accounting server:
Obtains FTP/HTTP accounting data by analysing squid's log files that should be
directed to its standard input.
hostname argument is the [short] host name of the host running the squid
process whoes log file you supply to standard input. It is then used as
agent host field in accounting database.
Runs once per day from periodic scripts. Squid's access log file
has to be rotated exactly at 00:00. To achieve this you can add the following
to root's crontab on the machine where your squid runs:
Gets SMTP accounting data from sendmail via syslog. The following lines should
be added to /etc/syslog.conf:
Gets FTP/HTTP/NNTP accounting data from MailGate via syslog. The following lines should
be added to /etc/syslog.conf:
It is recommended to gather IP accounting data from all the routers of your
network to account all the data flows in it. However in this case
you'll get duplicated data for flows that go through more than one router.
AcctMax processes the database (specified as db-prefix, the
common part of file name without (from|to).db) and leaves only maximum traffic
value among all routers for each db key. Both the to- and from-sorted databases
are rewritten.
-v command line switch makes it to print out progress information
(the number of records processed).
It is run from daily periodic script before building the report.
Joins the data from source databases (specified as src-db-prefix)
to the data contained in destination databases (specified as dst-db-prefix).
-v command line switch makes it to print out progress information
(the number of records processed).
The main and most complicated component of TAS. It processes the given database
(specified as db-prefix) of specific type of traffic (specified as
traffic_type and creates reports consisting of tables whoes captions,
columns, rows, and selection criterias
are specified in its configuration file
(by default, /usr/local/etc/Scripts/AcctLog.conf).
-f command line switch lets user to specify an alternative configuration file.
-v command line switch makes it to print out progress information while
processing the database (the number of records processed).
-h command line switch tells AcctLog to generate report in HTML.
All tables for each type of traffic are computed in a single pass, which makes
AcctLog robust and efficient.
Administrator's web interface for AcctLog. It is much more convinient
when you need to quickly create a report for some host or a group of hosts
without making a specific configuration file.
To use it, make a symbolic link from /usr/local/share/tas/cgi/AcctLog.cgi
(or wherever you have installed it) to the desired location under your web
server document root (typically to /usr/local/www/cgi-bin).
See the Web interface section for more description.
Client's web interface for AcctLog. Can be used to let the client
hosts to learn their traffic themselves.
To use it, make a symbolic link from /usr/local/share/tas/cgi/Client.cgi
(or wherever you have installed it) to the desired location under your web
server document root (typically to /usr/local/www/cgi-bin).
See the Web interface section for more description.
Converts old-database (of hash type, used prior to v1.2 of TAS) to btree
databases, sorted by sources and destinations.
-v command line switch makes it to print out progress information
(the number of records processed).
Calls AcctSquid (to obtain accounting data from yesterday's squid logs),
AcctSendmail and AcctMailgate (to make sure they have been called before
processing and have rotated their current day databases even if there was no
log record since 00:00), AcctMax (to eliminate data duplications for IP traffic),
AcctLog (to process accounting data), and AcctJoin (to summarize
yesterday's data into the current month databases); archives AcctLog's
reports into /usr/local/www/data/acct and sends a copy to admins.
As this is a shell script it can be easily modified on admin's taste.
If your OS doesn't have periodic system, just call this script from cron
every day at about 2:00.
Calls AcctLog to process accounting data for the past month, archives its
reports into /usr/local/www/data/acct, and sends a copy to admins. Also
archives the past month's database, naming it so that it reflectts the year
and the month number and compresses it. Then removes archives older that one
year.
As this is a shell script it can be easily modified on admin's taste.
If your OS doesn't have periodic system, just call this script from cron
every 1st day of a month at about 5:00.
If you use Cisco routers you need to add the following to their configuration:
If you use PC routers, you need to have on them a script by default called
TrafShowAll and placed in /usr/local/bin. This script should print
accounting data to stdout in the following format:
For example, it may use utilities from trafd package:
Then write the TrafShowAll script like so:
As an alternative (and I think, more correct) solution you can use a simple
and very suitable tool - ipacctd.
Unfortunately it is documented in Russian only. But all you have to do is to
compile your kernel with IPDIVERT option and arrange a startup script for
ipacctd like this:
If you don't like both of these, you can just do something yourself.
Arrange ipfw rule for outbound traffic that would permit all packets
and log them (like in ipacctd example but with the log option).
You can redirect these log messages via syslog to a script that would
append the information about source, destination and packet size to a file.
Your TrafShowAll script should output this file and truncate it on each
call. I'd appreciate if someone sent me the working scripts, to place them
into this documentation page.
By default AcctFetch connects to PC routers using ssh.
If you don't change this behaviour in tas.conf, then you have
to arrange passwordless SSH access from your accounting server to all
of your PC routers. This implies your accounting server is a trusted
and highly protected host. If you don't like this scheme, try to use something
else that would let you fetch the information through the network (I'll be
glad if you then share your ideas with me).
Web interface consists of administrator's component and client's component.
Administrator's component AcctLog.cgi allows you to quickly define and
produce simple reports without making a special configuration file.
It has multi-language support, so you can redefine any menu items, button labels,
or status text to be in your native language (currently only English and Russian
texts are defined). See Configuration section for details.
AcctLog.cgi has a few limitations in comparison to report rules definition
through the configuration file:
TIME_PERIOD should be a part of the database file name that refers to the
time period of the traffic it stores (i.e., "today", "month" or "XXXXYY" where
XXXX is a year, and YY is a month number).
PROXIES parameter should contain ip-addresses of your clients' proxy servers
separated by colons. This may be useful if don't want your clents to see
statistics for their proxy servers instead of their own in case if there isn't
possible to determine the end-client IP-address. They will then just see
the notice about that.
For example, you can provide the following form on your web-page
(or several forms for different report configurations):
TAS uses the following configuration files (by default they are placed in
/usr/local/etc/tas):
tas.conf uses perl(1) syntax of variable
definintion. It has the following parameters:
Defines a directory where the accounting databases reside. Default is:
Specify shell commands that AcctFetch issues to obtain accounting data
from PC routers. It may use parameter $router which will be substituted
with the router name or address. Default is:
Specify shell command that AcctFetch issues to obtain accounting data
from Cisco routers. It may use parameter $router which will be substituted
with the router name or address. Default is:
AcctLog.conf uses perl(1) syntax of variable
definintion. It has the following complex parameters:
List of subnets; only addresses belonging to them can be resolved to names.
Format:
Example:
Defines group lists which you can use later
when defining category expressions in %tables parameter
(see below).
Format:
Be careful not to create loops when nesting lists, there's
no automatic recursion loop detection.
Example:
Table descriptions for each type of traffic.
Format:
A table's category expression defines hosts for which the traffic is
counted. Column's category expression defines the "opposite end" hosts
and so limits the category of the counted traffic.
If the example above is used as a table's category expression, then
traffic will be counted separately for each host in domain sub2.sub1.mydomain.com
(because it is marked with '*'), and in summary for each of subdomains of
mydomain.com (because it's marked with '?') - expect sub1.mydomain.com, of course.
Although this example includes only host groups specified as domains, you
can use host groups of any type in the same category expression.
Except host groups you can use two special words in category expressions
- total and each. They imply all possible hosts, but when used in
a table's category expression, total provides accounting for
all hosts in total, while each makes traffic to be counted for each host
separately.
In a few words, in general you can define a category expression like so:
If this still seems complicated, use only single-operand
category expressions, like
Please also note the following. In the database hosts are stored either as
ip addresses or as domain names, depending on traffic type and resolution
possibility (for ip traffic hosts are stored as ip addresses only, for other
traffic types they are stored in the form they were obtained from applications).
So you should care when to specify hosts groups as ip addresses or subnets
and when as domains, because although ip addresses obtained from a database
are then resolved into names (if they belong to @local_nets of course)
for comparison against a category expression, they are first selected from a database
according to that category expression (this behavior is used since v1.2;
in earlier versions AcctLog iterated through all of the database records
which might consume a huge amount of time for large databases, but let to use
domain-only based category expressions even for hosts stored in database
as ip-addresses).
This means that if you wish to group your hosts in domains in a report table
for ip traffic (i.e., summarize traffic by domains, or by list names
of group lists that contain only host groups specified as domains, not
ip subnets), then you should ensure that inclusive host groups of your
table's category expression specify both domains and ip addresses/subnets
for the hosts you intend to count traffic for (of course, in this case the
groups specified as ip addresses/subnets will also be included into the report
table, so it is advised not to prefix them with '*' - then they will
occupy only a few table rows).
This is not the case for exclusive host groups - they may specify hosts
as domains for ip traffic (if those hosts belong to @local_nets, else
they won't be resolved), because exclusive host groups are used to match
hosts already selected from a database according to inclusive host groups.
accounting.conf uses sh(1) syntax of
variable definintion. It has the following parameters:
Whom to mail the report. Default is "root". Example:
recipient=root
Where to put the reports for archive. This directory must already exist, else
periodic scripts will fail. Default is:
Where the accounting databases reside. Default is:
Where to find squid access log files for previous day (applicable
only when process_squid is enabled - see below). Log files should be
gzipped. Should be specified as list of space-separated pairs
"hostname:/local/path/to/file". Hostname is then used as argument to
AcctSquid (see above).
There is no default value. Example (here we assume the proxy servers
filesystems are mounted to accounting server via nfs):
Whether or not to compress the rotated month databases (YES/NO). Default is:
Number of months for which to keep the data. Default is:
Whether to process databases of specific traffic types (YES/NO). Defaults are:
Applicable for IP traffic only: wether to find daily maximum value among all
routers for each source-destination pair. If disabled, accounting data is
stored for each of the routers you take statistics from. Else data for each
src-dst pair is stored only from that router for which daily traffic value
for that pair was maximum.
The usefullness of this action depends on your network topology.
Especially useful if you have backup routes activated if one of your routers
dies.
Also useful if you have complex network and you don't wish to configure
in AcctLog.conf which router's data to count for each group of hosts (you'll
just put "*" instead of router tag in a table column description).
No less it is useful if you need to count traffic even if it doesn't reach
its destination (e.g. if one of your routers dies but the traffic flow
passes preceding routers on the way).
Enabling this also helps you to save a lot of disk space, because less data
is stored.
Absolutely useless if you have only one router or if you prefer to count data
only from a specific router for each group of hosts.
Sould be YES or NO. Default is:
Whether to compose the report as plaintext or HTML (plaintext/html). Default is:
cgi.conf uses perl(1) syntax of
variable definintion. It defines all text strings used in the cgi
interface, so you can translate them into any other language. All parameters
have self-explanatory names. The distribution contains files for Russian and
English. Generally, cgi.conf is just a symlink pointing to one of them.
I'd appreciate if someone sent me files translated into other languages.
In version 1.2 there have been made significant changes regerding data
storage and configuration (see History of changes for
details). So if you have been using a previous version before,
you need to take the following steps to adapt existing database and
configuration for the new version of TAS.
Here you can find a report example, HTML version
(205 KB). As it is very detailed, it implies quite a complex configuration.
I have particularly changed domains, ip-addresses, peering neighbours
names and client names to some nonexistent.
The report building is quite time consuming operation. TAS uses the following
measures to make it more efficient:
Installation
make install
By default all components are installed under /usr/local. If you want to
use any other prefix (for example, /usr/local/tas), then type:
make PREFIX=/usr/local/tas install
After the files are copied you need to do some installation steps manually
(see the next chapter for each TAS component).
The TAS components
*/5 * * * * /usr/local/sbin/AcctFetch -c cisco1,cisco2,cisco3 \
-p pc1,pc2,pc3
0 0 * * * /usr/local/sbin/squid -k rotate && sleep 30 \
&& gzip -f /var/log/squid/*.log.0 2>/dev/null
If you have caching http/ftp proxy server running on a different machine,
then you need to make it's logs available to accounting server (for example,
via NFS).
!sendmail
*.* |/usr/local/sbin/AcctSendmail
If you have mail relay(s) on different machine(s), then
you have to pass their logs to accounting server's syslog.
!QueryServer
user.* |/usr/local/sbin/AcctMailgate
If you have MailGate runnung on a different machine, then
you have to pass its logs to accounting server's syslog.
Obtaining data from Cisco routers
ip accounting-threshold 32768
ip rcmd remote-host root X.X.X.X root enable
where X.X.X.X is your accounting server's ip.
And for each interface:
ip accounting output-packets
Obtaining data from PC routers
from to packets bytes router protocol status
where router, protocol and status are just tags
which you can later use as filters when extracting traffic records. Actually
you can use any arbitrary strings for these fields, but for uniformity with
databases of other traffic types it is advised to use the router's short domain
name as router, an upper-level protocol name (or just string "IP" if
your router does not supply this information) as protocol and a packet
transition status (something like "passed", "denied", "altered", etc.) if known
and needed, or just symbol "*". For example:
212.192.192.138 205.11.187.54 51 32411 Router1 IP *
On each call it has to show the data accumulated since the previous call.
#! /bin/sh
/usr/local/bin/trafsave ed0 ed1 ed2 ed3
/usr/local/bin/traflog -i ed0 -n -o mycustomformat 2>/dev/null
/usr/local/bin/traflog -i ed1 -n -o mycustomformat 2>/dev/null
cd /var/log && rm trafd.ed0 trafd.ed1 2>/dev/null
Be sure to describe the log format for traflog in
/usr/local/etc/traflog.format file:
mycustomformat {
from:"%s " to:"%s 0 " psize:"%ld RouterName " proto:"%s *\n"
};
NB: Here we use "0" instead of number of packets because trafd
does not count packets.
#!/bin/sh
/usr/local/bin/trafsave ed0 fxp0
/usr/local/bin/traflog -i ed0 -n -o mycustomformat
/usr/local/bin/traflog -i fxp0 -n -o mycustomformat
NB2: Run trafd/trafsave/traflog only for those interfaces between which
you don't have packed forwarding, else traffic will be counted twice - on input
interface and on output interface. In most cases you should take data only from
a single (external) interface. In case of Cisco routers you won't meet
this limitation because they count outbound traffic only.
/sbin/ipfw add 001 divert 10000 ip from any to any via ed* out
ipacctd -v -p 10000 -f /var/log/ipacct
Then your TrafShowAll script will be the following:
#!/bin/sh
/bin/rm /var/log/ipacct
/usr/bin/killall -HUP ipacctd
sleep 3
/usr/bin/awk '{ print $1, $2, $4, $3, "RouterName", "IP", "*" }' </var/log/ipacct
The advantages of ipacctd against trafd are that it can count
outbound-only traffic (and so it is possible to gather it from all interfaces)
and can count number of packets.
Web interface
However, it also has some advantages:
Here are some screenshots that can give you a point of what you can do with
the web interface:
Client's component, Client.cgi is a very simplified version of
AcctLog.cgi, it can be used to let client hosts to learn their traffic
themselves. Traffic is counted only for host which makes HTTP request. The
report table contains only two columns (host name or address and traffic data).
The report parameters are specified as URL parameters and should be the
following:
All except TIME_PERIOD and PROXIES correspond to the table and column definition
fields of %tables parameter in AcctLog.conf configuration file
(see below) and all of them must present.
<FORM NAME="f1" METHOD=POST TARGET="/cgi-bin/Client.cgi">
<INPUT TYPE=HIDDEN NAME="TIME_PERIOD" VALUE="month">
<INPUT TYPE=HIDDEN NAME="TRAFFIC_TYPE" VALUE="ip">
<INPUT TYPE=HIDDEN NAME="TABLE_CAPTION"
VALUE="Incoming traffic for current month (except current day)">
<INPUT TYPE=HIDDEN NAME="COLUMN_CAPTION" VALUE="Traffic (MBytes)">
<INPUT TYPE=HIDDEN NAME="SORT_COLUMN" VALUE="1">
<INPUT TYPE=HIDDEN NAME="RESOLVE_FLAG" VALUE="true">
<INPUT TYPE=HIDDEN NAME="COLUMN_CATEGORY_EXPRESSION" VALUE="total">
<INPUT TYPE=HIDDEN NAME="TRAFFIC_DIRECTION" VALUE="to">
<INPUT TYPE=HIDDEN NAME="MEASUREMENT_UNITS" VALUE="mbytes">
<INPUT TYPE=HIDDEN NAME="AGENT_HOST_LIST" VALUE="*">
<INPUT TYPE=HIDDEN NAME="PROTOCOL_LIST" VALUE="*">
<INPUT TYPE=HIDDEN NAME="STATUS_LIST" VALUE="*">
<INPUT TYPE=HIDDEN NAME="ROUNDING_OPTION" VALUE="nearest">
<INPUT TYPE=HIDDEN NAME="PROXIES" VALUE="192.168.1.2:10.20.30.40">
</FORM>
<A Href="javascript:document.f1.submit();">
IP-traffic for current month
</A>
To have specific access control for AcctLog.cgi and Client.cgi
you can put into .htaccess file in the same directory something like
this:
<Files AcctLog\.cgi>
AuthName 'HostMaster'
require group hostmaster
SSLRequireSSL
</Files>
<Files Client\.cgi>
Order allow,deny
Allow from 20.20.20.0/22
Allow from 30.30.30.0/24
Allow from 40.40.40.240/28
Deny from all
</Files>
Configuration
$prefix='/var/account';
$pc_fetch_command='/usr/bin/ssh $router /usr/local/bin/TrafShowAll';
$cisco_fetch_command='/usr/bin/rsh -t 60 $router clear ip accounting && /usr/bin/rsh -t 60 $router show ip accounting checkpoint';
@local_nets = ( "net", "net", ... );
@local_nets = (
"20.20.20.0/22",
"30.30.30.0/24",
"40.40.40.240/28"
);
%lists = (
"list name" => [ "host group", "host group", ... ],
"list name" => [ "host group", "host group", ... ],
...
);
%lists = (
"Clients" => [
"?my.domain.com",
"my.another-domain.com",
"*my.internal-lan.com",
"*192.168.0.0/28" # My intranet
"200.200.200.0/22", # To aggregate
"300.300.300.0/24", # unresolved hosts
],
"Another ISP" => [
"50.50.50.0/23",
"60.60.60.0/23"
],
"Yet Another ISP" => [
"70.70.70.0/20"
],
"Peering" => [
"Another ISP",
"Yet Another ISP"
]
);
%tables = (
"traffic type" => [ # set of tables
[ # table definition
"table caption",
"category expression",
sort_column,
"resolve flag",
[ # set of columns
[ # column definition
"column caption",
"category expression",
"traffic direction",
"measurement units",
"agent host list",
"protocol list",
"status list"
],
[
...
],
...
],
"rounding option"
],
...
],
"traffic type" => [
...
],
...
);
"?mydomain.com-sub1.mydomain.com+*sub2.sub1.mydomain.com"
It covers all hosts that belong to domain sub2.sub1.mydomain.com and hosts
that belong to domain mydomain.com but not belong to domain sub1.mydomain.com.
('?' and '*' symbols are used according to host group definition syntax).
"total-Backbone+*Foreign Tunnels-192.168.1.0/22+192.168.1.0/21+*backbone.mydomain.com"
Remember that whitespaces are not allowed between signs and operands, but
they are allowed in list names.
"*Clients"
or
"?mydomain.com"
or
"*192.168.1.0/21"
or so forth. This might limit flexibility, but enhance simplicity. After all,
you can achieve the same results either using complex category expressions
or using complex group lists and category expressions consisting of only
a single host group (a list name prefixed with '*').
storage=/usr/local/www/data/acct
prefix=/var/account
squid_logs=proxy1:/net/proxy1/var/log/squid/access.log.0.gz \
proxy2:/net/proxy2/var/log/squid/access.log.0.gz
compress_rotated=NO
keep=12
process_ip=YES
process_squid=YES
process_sendmail=YES
process_mailgate=NO
ip_daily_max=YES
report_type=html
Upgrading from versions before 1.2
Report example
Performance
| Problem | Fix |
|---|---|
| Periodic scripts don't run and I don't see any error in the "daily run output" messages. |
a) Make sure you didn't redefine local_preiodic parameter in
/etc/periodic.conf, and if you did, include "/usr/local/etc/periodic"
into it.
b) If the problem still exists, then enable daily_show_badconfig and monthly_show_badconfig options in your /etc/periodic.conf to see the error output if the script fails to run (there is a chance you just didn't create a storage directory specified in your accounting.conf, and so periodic scripts fail because they can't write into it). |
| The report the TAS produces, contains empty tables. |
a) Make sure that list names you have specified in table category
expression in your AcctLog.conf, are defined in the %lists
block, and they are in the same case. Remember that only total
and each keywords are predefined.
b) If the lists used in a table category, contain only domain names, be sure that IP addresses that correspond to these domain names, are covered by your @local_nets parameter (else they won't be resolved, and so won't match the lists you expect they will). |
|
a) You get messages from cron regarding AcctFetch:
"Bus error. Core dumped".
b) AcctFetch hangs forever and all its instances consequently run, are blocked. c) Traffic databases get corrupted (AcctLog, AcctJoin or AcctMax either report "Inappropriate file type" or dump core or just hang). d) Traffic databases have enormous size. |
Ensure that neither softupdates nor async mount option is set on filesystem where your traffic databases reside (/var/account by default). |
| You use trafd package and noticed that TAS reports twice bigger amount of traffic than you expected. | Ensure that you don't run trafd for more than one interface between which you have packet forwarding. |
| You use Russian language file for web interface, and pressing submit buttons in the menu frame result "Bad parameters" error. |
This means that data posted from the client to server isn't recoded
into the server character set. So you either need to:
a) Use Russian Apache web server. b) Recode the language file cgi.conf into the character set used on your client machine(s) (windows-1251 for Windows, x-mac-cyrillic for Macintosh, iso-8859-5 for Sun, etc...) c) Translate $menu_submit_* parameter values in cgi.conf into English, or just latin transliteration of Russian. |
| After accessing the client web-interface to learn their traffic, the hosts which are under both NAT and HTTP proxy see an empty report or see their intranet ip-address instead of their real ip-address for which the traffic is counted. |
Client.cgi uses HTTP_X_FORWARDED_FOR variable, when available, to learn
the actual client's address. But if the user host is within the intranet
address space, and the real ip-address for which he may like to learn the
traffic, is on his HTTP proxy (probably the same machine as NAT proxy),
then his HTTP proxy will pass his intranet address instead of its own.
Advise users who are under NAT to either turn off using HTTP proxy for your URL in their browsers (if their firewall permits that) to ensure their addresses will be translated by NAT, or to just access directly from their proxy machine. Another solution is to require SSL connection, in which case HTTP_X_FORWARDED_FOR won't be passed to the server (at least, squid proxy server doesn't pass it). For that make your links or form targets to Client.cgi starting from "https://", not "http://". Of course you should be using some sort of SSL-capable web server to make it possible. If you use apache+mod_ssl, you may also wish to use SSLRequireSSL configuration directive to prohibit plain http connection. Using SSL is also advised for security reasons.
|
In the future it is planned to get rid of DNS resolution of addresses into
names and grouping by names when building a report. Instead AcctLog
should connect to a MySQL database
that keeps all the information about clients, find out who owns the given
address, and so be able to aggregate hosts by clients in the report tables
rather than by ip nets or domain names. Of course, DNS resolution
and grouping will be kept as an option.
When running on the periodic basis, the results of accounting data extraction
for each client have to be automatically put into a client database, not only
into the report.
Optional support for timestamps with 5-minute step. May require too lot of disk
space or a reduction algorithm.
Fix monthly accounting script - databases where kept 1 month less than
specified by the "keep" parameter in accounting.conf.
Subject of a report sent by e-mail now contains date for which the
report was built.
AcctConvert utility added to convert old databases into new format.
Syntax of category expressions has been extended and made easier
for understanding, but its semantics was slightly changed.
Info about agent host (from where the accounting data was
fetched), protocol and information unit's status is now stored
in database for any type of traffic instead of a single tag that had
different semantics for each type of traffic.
A user now can specify more units in which to count traffic (except
"bytes" now "kbytes", "mbytes" and "gbytes" are also supported in
table column definitions).
Web interface has been changed to support selection by agent host,
protocol and status and to support new traffic units.
Progress output of web interface made more detailed.
Fixed an error of computation netmask from masklen, which arised
when maxlen was 32, because perl supports shift operations on
4-byte integers only. The different algorythm is now used.
Fixed an error of old files removal when "compress_rotated" parameter
in accounting.conf was set to "NO".
Multiple squid log files are now supported (parameter "squid_log"
in accounting.conf was replaced with "squid_logs" and its syntax was
changed; however "squid_log" is also supported for compatibility).
Periodic scripts now have hardcoded default values for config
parameters, so commenting them out is now safe.
It is now possible to specify the rounding option for units larger
than "bytes": "up", "down" or "nearest" (corresponding optional field
was added to the table definition).
FAQ section was started in the documentation page.
"ip_daily_max" parameter added to accounting.conf, so a user now can
turn the daily traffic maximization off.
Sample TrafShowAll script mentioned in this documentation have been
changed, and a bit more explaination about using TAS with trafd has
been added.
The first release.
Planned enhancements
History of changes
Is now the same as 1.2.1
Bugfix in monthly accounting script - sendmail database did not rotate
because of a misprint.
Fix for a bug with submit buttons in the web interface (AcctLog.cgi)
that was introduced during the code cleanup at the date of previous
change.
Added client-oriented web interface Client.cgi, which can be used to
let the client hosts to learn their traffic themselves.
Fix for a bug in AcctLog that caused to count traffic more than once
for hosts belonging to key ranges (which are compiled from tables'
category expressions) that happened to be adjacent in a database. I.e.,
if you specified a table's category expression like "10.10.0.0/24+10.1.255.0/24"
and there are no addresses between 10.1.255.255 and 10.10.0.0 in your
database, then traffic for hosts belonging to 10.10.0.0./24 would be
counted twice. This bug was introduced together with the new database
format.
Database format has been changed from hash to sorted B-tree for faster
data extraction.
Added web interface for AcctLog.
HTML report example included into the distribution.
AcctLog now checks the list names used in table or column category
expressions in AcctLog.conf to ensure they are defined in %lists
parameter. This can prevent problems caused by user mistprints.
TAS now can generate nice reports in HTML. Added -h switch for AcctLog
and "report_type" parameter to accounting.conf.
"Troubleshooting" section added to this documentation page.
Shell commands issued by AcctFetch to obtain traffic statistics from
routers have been made configureable with tas.conf.
Another "Use of unitialized value" warning has been fixed in AcctLog.
Many typos fixed in this documentation page.
Fixed a very stupid misprint in AcctJoin, made during the changes
of Jan 5, 2001, that caused to store number of packets instead of
number of bytes when adding data to the month database.
Fixed a minor bug in AcctLog that caused "Use of uninitialized value"
warning in some situations.
Fixed typo in AcctFetch.
SEEK_SET definition is now correctly taken from IO::Seekable module.
Removed unused variables from AcctSendmail and AcctMailgate.
Databases are not locked now - a stampfile is now locked instead.
AcctFetch now opens ip-database only after the data was fetched from
a router and then closes it before fetching data from another router.
Fix for bug with renaming current-day databases that were not processed
in time for some reason.
Accounting.conf now lets a user to specify which types of traffic to
process (directives process_ip, process_squid, process_sendmail,
process_mailgate), so there's no need any more to modify periodic
scripts if statistics for some types of traffic is not gathered.
Russian text accidentally remained in periodic scripts, was translated
into English.
Fix for squid's accounting database rotation failure in monthly
periodic script.
Fix for uninitialized value warning in AcctLog.
Configuration directory is now /usr/local/etc/tas instead of
/usr/local/etc/Scripts.
New configuration file tas.conf has been added, it lets to specify
a directory where accounting databases should reside.
Two configuration parameters added to accounting.conf: "compress_rotated"
to specify whether or not to compress rotated month databases and "keep"
to specify number of months to keep the data.
File permissions fixed for executable scripts so that they could be
run by any user. Thanks to Andreas Klemm (andreas@klemm.gtn.com).
The data fetched from a router now is accumulated in memory and
committed to a database only after the fetch is completed. Commitment
during fetching sometimes led to a db table damage (I don't know why).
Credits to
