Today, Usenet has grown to enormous proportions. Sites that carry the
whole of Netnews usually transfer something like a paltry 60 MB a day.[1] Of course, this requires much
more than pushing files around. So let's take a look at the way most Unix
systems handle Usenet news.
News begins when users create and post articles. Each user enters a
message into a special application called a newsreader, which
formats it appropriately for transmission to the local
news server. In Unix environments the newsreader commonly uses the
inews command to transmit articles to the
newsserver using the TCP/IP protocol. But it's also possible to
write the article directly into a file in a special directory called
the news spool. Once the posting is delivered to the local
news server, it takes responsibility for delivering the article to
other news users.
News is distributed through the net by various transports. The medium used to be UUCP, but today the main traffic is carried by Internet
sites. The routing algorithm used is called flooding.
Each site maintains a number of links (news feeds) to
other sites. Any article generated or received by the local news system is
forwarded to them, unless it has already been at that site, in which case it
is discarded. A site may find out about all other sites the article has
already traversed by looking at the Path: header field. This
header contains a list of all systems through which the article has been
forwarded in bang path notation.
To distinguish articles and recognize duplicates, Usenet articles have
to carry a message ID (specified in the Message-Id: header
field), which combines the posting site's name and a serial number into
<serial@site >.
For each article processed, the news system logs this ID into a
history file, against which all newly arrived articles
are checked.
The flow between any two sites may be limited by two criteria. For one,
an article is assigned a distribution (in the Distribution:
header field), which may be used to confine it to a certain group of
sites. On the other hand, the newsgroups exchanged may be limited by
both the sending and receiving systems. The set of newsgroups and
distributions allowed to be transmitted to a site are usually kept in the
sys file.
The sheer number of articles usually requires that improvements be made
to the above scheme. On UUCP networks, systems collect articles over a period
of time and combine them into a single file, which is compressed and sent to
the remote site. This is called
batching.
An alternative technique is the ihave/sendme protocol that
prevents duplicate articles from being transferred,
thus saving net bandwidth. Instead of putting all articles in batch
files and sending them along, only the message IDs of articles are
combined into a giant “ihave” message and sent to the remote
site. The remote site reads this message, compares it to its history file,
and returns the list of articles it wants in a “sendme” message.
Only the requested articles are sent.
Of course, ihave/sendme makes sense only if it involves two big sites
that receive news from several independent feeds each, and that poll each
other often enough for an efficient flow of news.
Sites that are on the Internet generally rely on TCP/IP-based software that
uses the Network News Transfer Protocol (NNTP). NNTP is described in RFC-977;
it is responsible for the transfer of news between news servers and provides
Usenet access to single users on remote hosts.
NNTP knows three different ways to transfer news. One is a real-time version
of ihave/sendme, also referred to as pushing news. The
second technique is called pulling news, in which the
client requests a list of articles in a given newsgroup or hierarchy that have
arrived at the server's site after a specified date, and chooses those it
cannot find in its history file. The third technique is for interactive
newsreading and allows you or your newsreader to retrieve articles from
specified newgroups, as well as post articles with incomplete header
information.
At each site, news is kept in a directory hierarchy below
/var/spool/news, each article in a separate file, and
each newsgroup in a separate directory. The directory name is made up of the
newsgroup name, with the components being the path components. Thus,
comp.os.linux.misc articles are kept
in /var/spool/news/comp/os/linux/misc. The articles in a
newsgroup are assigned numbers in the order they arrive. This number serves as
the file's name. The range of numbers of articles currently online is kept
in a file called active, which at the same time serves
as a list of newsgroups your site knows.
Since disk space is a finite resource, you have to start throwing away
articles after some time.[2] This is called
expiring. Usually, articles from certain groups and
hierarchies are expired at a fixed number of days after they arrive. This
may be overridden by the poster by specifying a date of expiration in the
Expires: field of the article header.
You now have enough information to choose what to read next. UUCP users
should read about C-News in Chapter 21. If you're using
a TCP/IP network, read about NNTP in Chapter 22. If you
need to transfer moderate amounts of news over TCP/IP, the server described
in that chapter may be enough for you. To install a heavy-duty news server
that can handle huge volumes of material, go on to read about InterNet News
in Chapter 23.
