The Interoperability Report


December 21, 1994-10:20 AM                       3
                                
             On the Origins of Internet Mail Species
              by D. Crocker, Brandenburg Consulting
               Copyright © 1994 by Interop Company


Big bang

When the Arpanet was first built, in 1969, there were grand
visions of its possible uses.  The only problem was that most of
the concrete planning for the Arpanet focused on the underlying
packet-switching mechanisms and little was devoted to the
development of application services.  In RFC 1000 [16], Steve
Crocker discusses this early work on the upper layers and
comments, "We had no official charter.  Most of us were graduate
students and we expected that a professional crew would show up
eventually to take over the problems we were dealing with."  But
the professional crew never did not show up and the graduate
students slogged ahead, developing a few basic protocols,
particularly for terminal access and file transfer.


Out of the primordial soup

While electronic mail was a part of the original vision, it was
not a part of the main development work.  Ray Tomlinson, of Bolt
Beranek and Newman (BBN) is credited with modifying BBN's Tenex
operating system mail-sending program, sndmsg, to support an ad
hoc protocol for transferring to other Tenex's1 .  Users
specified the remote host by appending "@hostname" to the
recipient's login name.  The sndmsg program simply appended the
user's message to the end of a recipient's mailbox file.  In
fact, anyone could append any kind of data.  No special
privileges were required; when the cross-net facility was added,
the server just appended whatever it received.   There were no
format standards, although the BBN program did create objects
with a small set of headers of the same type we are used to
seeing today.

Since most of the Arpanet research community used Tenex, the
enhancement propagated quickly, creating a functioning email
service around the net.  But "most" is not the same as "all" and
the community already had the view that protocols needed to be
independent of any particular platform.  During 1971, there were
some discussions about a "mailbox" protocol, but it did not reach
fruition.


Accidental tourist

In fact, the email transfer protocol for the next 10 years of the
Arpanet - the remainder of its existence - was specified as two,
ancillary commands in the File Transfer Protocol [5].  Better
still, it was not specified during the formal working group
discussions for creating FTP.  Apparently, the editor of the FTP
specification was attempting to beat the final draft into
acceptable form one Saturday night at MIT, and a co-worker walked
by and began discussing email.  He suggested adding a couple of
commands to FTP to support the transfer function.  The editor did
just that, without further discussion among the working group.
The specification was circulated for review.  No one in the
community objected; and some implemented it.  Thus did the
Arpanet get its electronic mail protocol.

Besides adding a key function in the Arpanet suite, this
demonstrated an elaboration on the group's general style of doing
protocol development that largely persists:  work is done by a
core group and reviewed by a larger group.  Individual initiative
is often accepted easily, when it causes no special controversy.

The two commands differed in the mechanism they used for sending
the message content.  FTP, then and now, uses two channels (two
transport connections.)  One is for commands and the other is for
data.  The MAIL command sent the message content as part of the
control channel and the MLFL command sent it over a data
connection.  To send data over the control channel, the command
needed a mechanism for indicating the end of the message content
and the convention of having a line containing only a period
(CRLF.CRLF) was specified. While the mechanism for sending
messages via the data channel was implemented and used, the
simplicity of the "inline" approach, using the control channel,
was more appealing and accounts for carrying this mechanism over
to the current email transfer protocol, SMTP.

Both commands allowed specifying only one address at a time.
That is, for each recipient, a separate transfer of the entire
message was required.  (Some implementations allowed referencing
multiple recipients in the command line, but this was not widely
supported.)  While this inefficiency caused a bit of inefficieny
when sending to several recipients - and especially when sending
to a mailing list -  it was not felt to be a major burden for
some years and was finally remedied by SMTP.  Most messages had
short distribution lists.


Getting the format right

It is a measure of the Arpanet's informality that there was
specification and use of a standard for sending email objects
several years before there was any concerted effort to define the
format of the object itself.  In part, this was because the
original BBN sndmsg program generated a simple, reasonable format
that most of the community was using.  It contained an initial
set of headers for specifying author, addressees, subject and
creation data; this was followed by a free-form body with lines
of text.  There was informal documentation of that format, and
most other systems conformed to the same, basic conventions.
Sometimes, however, messages with remarkably strange formatting
would show up in the inbox.  (Unlike today, of course...)

The popularity of Arpanet email led, inevitably, to discussions
about improving it.  As remains true in today's Internet, those
with the motivation to talk the issues were recruited to work on
fixing them.  This led to RFC 733, the first codification of
email object formats [6].  The document pointedly retained the
style of email format that was already in use, rather than trying
to create a radically new and different system.  (There was
considerable debate about this philosophical choice, but the
decision was essential for protecting the installed base.)
Second, the document was the first Arpanet/Internet specification
to declare itself a standard.  In spite of careful coordination
with the ARPA program manager overseeing the effort, the
publication of RFC 733 raised quite a few hackles around the net.
Conformance to specifications was entirely a matter of individual
choice; there was no precedent for being told one had to use a
particular specification.

The model of connectivity on the Arpanet was that every host
could talk directly to every other host.  Since the number of
hosts was relatively small, this meant that a single, "flat" name
space was adequate.  That is, there was one set of host names,
maintained in a single, global table.

The specification used a formal notation, called Augmented Backus-
Naur Form (ABNF) which was relatively popular among the Arpanet
community.  With respect to email format, RFC 733 continued
existing practice for the basic name:value format of headers:
     
     Subject:  email is fun

and the mailbox@host format of addresses:
     
     pogran@mit-multics

It codified such things as multi-line headers:
     
     Subject:  some subject lines just need to be
               longer than fits on one line

 and date formats:
     
     Date: Mon, 19 Dec 94 9:30:51 PST

The message content, or body, was specified to be simply a series
of lines of ASCII text, without structure.

RFC 733 did attempt some innovations.  The specification provided
for:
     
     Distinguishing human names from mailbox names (Dave Crocker
     <dcrocker@udel-relay>),
     
     Distinguishing originator roles (From/Sender/Reply-to),
     
     Source routing (mailbox@host3@host2@host1),
     
     Uninterpreted comments (Paul (DNS guru) Vixie
     <paul@vix.com>) and
     
     Assorted optional fields, such as Message-ID, Keywords and
     References,

It also tried to permit a very wide range of addressing formats,
including naming for group aliases and to reference postal
destinations.  User-defined headers were explicitly allowed.

Source-routing was never useful.  Given the hands-on,
experimental nature of the net, it's suprising that the feature
was such a complete failure, however it forced pursuit of an
unplanned piece of design cleverness, namely the locally-defined
mailbox field, affectionately called the left-hand-side.  Only
the right-hand-side hostname was expected to be interpretable
globally.  Hence, sites could define any extended interpretation
of the left-hand, local side that they wished.  This led to Unix
UUCP use of additional source-routing, with exclamation (bang)
signs:
     
     uwisc!lhl@udel-relay

and the author's own extension for CSNet, with per-cent sign used
to subdivide the left-hand side for one addition email "hop":2
     
     lhl%uwisc@udel-relay

Of the innovative work, the Sender/From/Reply-to distinction
seems to have been the most successful.  It distinguished the
author(s) of the message, in the From field, from the person
posting the message, in the Sender field.  It also let the
originator specify that replies were to go to yet-another
address.  The From/Reply-to distinction is in heavy use today;
however it is rare for the Sender field to be different from
From.  These fields were RFC 733's one major enhancement that
pertained to user-to-user interaction, rather than user-to-mail
system.  In general, Arpanet and Internet email lack much
discussion or mechanism for user-to-user protocols.


Getting serious about scaling

By the time of the 1983 transition from Arpanet to Internet, the
major problem with email was its success.  Traffic had increased
considerably.  Mailing lists were becoming popular, and the
inefficiencies of the FTP-based mail transfer commands were
noticeably irritating.  The number of hosts on the net became
large enough that maintaining a single, centralized table mapping
their names to their addresses was not feasible.  Also, the
lessons from Usenet and CSNet were that not all email hosts were
attached directly to the net, yet there was no integrated methods
of addressing these "detached" hosts, other than the "left-hand-
side" hacks.  Also, then as now, conformance to the email
specifications was highly problematic; some effort was needed to
adjust the specs to match real-world behaviors.

In other words, Arpanet/Internet email needed further
enhancement.  The effort took place along 3 lines, each involving
the usual style of having a core person doing the leadership and
writing, with a collection of discussants providing ideas and
feedback.  The first line of development was the Simple Mail
Transfer Protocol [15], the second was an acronym-free
enhancement to the format specifications in RFC 822 [7]3, and the
third created a mechanism for distributed administration and
query of the host table [11, 12].  This last activity, called the
Domain Name System, was not specific to email activity and
discussion of it is left for a separate article.


SMTP

Prior to the development of SMTP, there were several other
efforts at creating email-specific transport protocols.  Besides
the Usenet and CSNet work, SMTP's author, Jon Postel, earlier led
in an effort to develop specifications for support of multi-media
mail [14, 18].  Interest was slight, although the effort did have
some influence on the later development of X.400.  The later work
on SMTP was assiduous in its efforts to keeps things simple.  For
message submission, it only provided a means of specifying:
     
     Who the message was from,
     
     An address list of who the message was to, and

     The message.

Protocol replies are in the usual form, with a 3-digit status
code, followed by free-form text.  This produces message posting
sequences of the style as shown in an example from the SMTP
specification, between the 'S'ending client and the 'R'eceiving
server:
R: 220 BBN-UNIX.ARPA Simple Mail Transfer Service Ready
S: HELO USC-ISIF.ARPA
R: 250 BBN-UNIX.ARPA
S: MAIL FROM:<Smith@USC-ISIF.ARPA>
R: 250 OK
S: RCPT TO:<Jones@BBN-UNIX.ARPA>
R: 250 OK
S: RCPT TO:<Green@BBN-UNIX.ARPA>
R: 550 No such user here
S: RCPT TO:<Brown@BBN-UNIX.ARPA>
R: 250 OK
   S: DATA
R: 354 Start mail input; end with <CRLF>.<CRLF>
S: Blah blah blah...
S: ...etc. etc. etc.
S: .
R: 250 OK
S: QUIT
R: 221 BBN-UNIX.ARPA Service closing transmission channel

Transfer of the message, itself, is done "inline" rather than
through a separate data channel.  There is also some provision
for basic tracing information, to be added to the message.

SMTP provides commands for verifying the validity of a particular
address and for expanding an address list reference into its
constituent addresses.  Both of these functions are useful, but
they require a) direct Internet access for the client, and b) a
server with direct access to the full address and list details.


RFC 822

Format specifications in RFC822 sought to maintain the working
portions of the earlier RFC733, clean up assorted minor problems,
and add only limited enhancements.  These included:
     
     Structured host domain names (relay.udel.edu)
     
     Tracing information supplied in the Received header,
     
     User-level message re-sending onto another recipient (Resent-
     to/Resent-From/Resent-Reply-to),
     
     Message body privacy (encrypted), and
     
     The Postmaster reserved mailbox address.

Use of host domain names was specified before the Domain Name
Service (DNS) was actually developed, and specification of the
Received header was redundant with the specification in SMTP.  In
both cases, some inconsistencies between specifications has
resulted in awkwardness of service that had to be resolved in the
later Host Requirements work [4]   Given the current interest in
privacy, it's interesting that the RFC822 mechanism for
encryption was not exploited.  On the other hand, the simple
provision for a standard Postmaster address at every site has
been quite helpful for email operations.  It means that problems
can be reported, without having to know any of the staff at the
remote site.


A matter of models

Electronic mail architecture discussions refer to two, primary
components:  User Agent (UA) and Mail Transfer Agent (MTA).
While many in the community may debate the issue,
Arpanet/Internet email does conform to this model and was even
used as as one of the examples during forumulation of the model.

The model is quite simple.  Is says that one part of the system
represents individual users and the other represents the
conveyance service.  What causes confusion, for some, is that
there are implementations which do not provide such a clean
distinction, such as by having the conveyance software fill in
the From and Date fields.  Recent enhancements to the model are
attempting to deal with the reality of gateways, that is, modules
which are used to connect together email services which are
fundamentally different.  This is a topic which remains
problematic.


Getting serious about scaling, again

For nearly ten years, no further work was done to enhance
Internet email.  During that time, however, the Internet grew
enormously; use of mailing lists grew enormously; and
interconnections through email gateways grew enormously.  All of
this stressed the service, once again.  By the beginning of 1991,
increased internationalization of Internet use finally forced the
community to pursue some changes.  Initially, the focus was
strictly upon the requirement to permit carriage of documents
containing text in non-English character sets.  Two technical
views dominated the early discussions.  One view was that the
email object (RFC 822) needed to be enhanced for labeling of non-
ASCII data.  The other view was that SMTP needed to be enhanced,
for carriage of 8-bit data.  Separate working groups were formed;
the first produced MIME and the second produced ESMTP.4  For a
thorough introduction to the technical aspects of modern Internet
mail service, see [17].


MIME

Multi-purpose Internet Message Extensions (MIME) [1, 2, 3, 13]
expanded its original agenda, just a bit.  It developed into a
general method for structuring and labeling content.  The
structuring is permitted to be an arbitrary, nested tree of sub-
components.  The labeling is for a wide range of data, including
different types of text, but also including graphics, audio, and
so forth.  This opened the doors for general, 8-bit data; however
the existing email transport services tended (and still tend) to
expect simple lines of 7-bit data, so MIME also provides a
mechanism for encoding the data to safely traverse these limited
environments.  In particular, MIME does not require changes to
any of the existing transport infrastructure; it only requires
changes to the participating end-user systems.

MIME is really specified as independent of RFC 822.  The latter
focuses on message headers and the former specifies message
content, although it does provide for some additional headers,
specific to MIME usage.  A side-effect of this independence is
that MIME can easily be used in non-email environments, as is
already happening for the World Wide Web.

Message structuring is accomplished through the use of boundary
strings:

From:
To:
Subject:
Mime-version: 1.0
Content-type: multipart/mixed; boundary=unique1;

--unique1

This is some top-level, introductory text

--unique1
Content-type: multipart/mixed; boundary=unique2;

--unique2
Content-type: text/plain; charset=ISO-8859-1
     
This is some nested text using an alternate character set

--unique2

This text is part of the nested set, but is in the default, ASCII
character set

--unique2--

--unique1


The relationship of the data in the above message is:
     
     Headers
     top-level text
          first nested text
          second nested text


Types of content

The MIME header Content-type is used to specify the nature of the
contained data.  Types are specified in two parts, with the first
indicating general category and the second (the sub-type)
specifying the exact detail.  The MIME specification provided for
a basic set of sub-types, in each category, with provision for
private conventions and for publicly registering extensions:

text
     is for prose.  The parameter charset= allows specification
     of other character sets, for use with non-English languages.
     A basic set of alternatives is specified in the original
     MIME document, but more importantly, it can be expanded
     through separate registration of additional sets.

multipart
     is for basic structuring of MIME data; it allows a
     collection of parts to be "stapled" together.  An innovative
     form is multipart/alternative which indicates that the
     different parts are equivalent and that the recipient should
     choose one among them; an example of this function would be
     to send a document which is in its original word processor
     form, a simple text form, and a postscript form.

message
     is primarily for data which is an email message, principally
     in RFC822 format.  However, two, innovative forms are for
     sending long messages and for retrieving data remotely.  The
     first, message/partial indicates that this is one part of
     the message; hence, a sender may break a long message into
     parts for the receiver to reassemble.  The second,
     message/external-body allows reference to files that are
     elsewhere and permits specification of various retrieval
     techniques, such as through email request or through
     Internet FTP.

audio
     is for sounds; the sub-type audio/basic provides initial
     capability.

image
     is for graphics and pictures; the sub-types image/gif and
     image/jpeg provide initial capability.

video
     is for motion pictures; the sub-type video/mpeg provides
     initial capability.

application
     is for data which does not fit into any of the above
     categories; one example is application/postscript.


Binary data in non-binary environments

In order to squeeze the binary objects through limited email
pipes that often support only 7-bit data, MIME's Content-Transfer-
Encoding mechanism allows specification of the "transfer" form of
the data.  The transfer form is independent of the
"representation" form.  The former specifies what is necessary to
get a bundle of bits through a transport pipe.  The latter
specifies the canonical, interoperable, semantics-related nature
of the data.  For example, the basic audio form used in MIME
encodes data as true binary.  To get it through the usual
Internet email service, it would be processed into a line-
formatted, hexadecimal, textual form, as described below.  MIME
permits the following transfer encodings:

7-bit
     is the default and the most heavily used; it means that the
     data naturally conform to a 7-bit, line-oriented
     environment; it is used for ASCII text data.

8-bit
     indicates that the data conform to line-orientation, but
     that the high-bit might be on for some bytes; it is used for
     non-ASCII text data when the transport allows the high bit
     to be on.

quoted-printable
     says that the transmitted form of the data conform to the 7-
     bit, line-oriented constraints but that the conformance was
     achieved by specially encoding some of the data; it is
     primarily intended for text data which has the 8th bit on,
     but which cannot be passed through the email transport
     transparently.

base64
     indicates that the transmitted form of the data conform to
     the 7-bit, line-oriented constraints, but that the
     conformance was achieved by specially coding all of the data
     into a hexadecimal representation; it is used for true,
     binary data.

MIME originally included a pure binary encoding, but it has not
yet received enough implementation and testing experience to be
retained.  It appears that use of MIME for the World Wide Web
will provide the necessary pressure to further develop binary.


ESMTP

SMTP extensions were the result of the effort parallel to MIME,
to enhance the basic email transport service [8, 9, 10].  It
provides for incremental specification of enhancements to SMTP
systems, through a registration and announcement mechanism, so
that a client SMTP system can ask the server what extensions are
supported, before trying to use any of them.  The first
extensions specified for this mechanism were for support of MIME
Content-transfer-encoding: 8-bit, and for determining the maximum
size of message that the server will accept.

From RFC 1651, an example of an extended SMTP start of session,
in which the server supports a range of options could be:

S: <wait for connection on TCP port 25>
C: <open connection to server>
S: 220 dbc.mtview.ca.us SMTP service ready
C: EHLO ymir.claremont.edu
S: 250-dbc.mtview.ca.us says hello
S: 250-EXPN
S: 250-HELP
S: 250-8BITMIME
S: 250-XONE
S: 250 XVRB
...

Note that the EHLO query from the client is a different command
than the HELO query shown in the earlier SMTP example.  The
enhancement was designed to save from adding even one extra round-
trip transaction across the Internet.  Experience has shown that
this can be a significant concern, particularly when the Internet
gets congested.

The extension mechanism is intended to support the gradual
upgrade of the service, rather than for supporting "optional" use
of enhancements.  That is, Internet services generally do not
"negotiate" among a range of services that client and server
might choose to implement.  Mechanisms like the one added to SMTP
and MIME serve to permit graceful adoption of new features, while
still supporting a core set of useful functionality.  The freedom
created by such a mechanism is that new features can be
specified, tested, and deployed incrementally and independently.
In fact, various new features are getting added to MIME and SMTP
regularly.


2001

The email industry has been extremely active, recently.  It is no
longer novel to have an email account.  Internet email has grown
as explosively as the rest of the Internet, particularly as
witnessed by the frequency with which radio, advertising, and
business cards cite an electronic mail address along with a
telephone number.

With the addition of MIME and ESMTP, Internet email begins to
provide a service base that is competitive with other email
technologies.  Some holes remain, such as a fully deployed
security service, but there is considerable email service
enhancement taking place in the Internet standards forum (IETF)
and in the market place.  It appears that every email vendor will
be offering MIME support over the course of 1995, making fully
interoperable, open-systems, multi-media email a realistic goal
for organizations.


References

1.   N. Borenstein, N. Freed. MIME (Multipurpose Internet Mail
     Extensions): Mechanisms for Specifying and Describing the Format
     of Internet Message Bodies; RFC 1341.  June 1992.
2.   N. Borenstein. A User Agent Configuration Mechanism for
Multimedia Mail Format Information; RFC 1343.  June 1992.
3.   N. Borenstein. Implications of MIME for Internet Mail
Gateways; RFC 1344.
4.   R.T. Braden. Requirements for Internet hosts - application
and support; RFC 1123. October, 1989.
5.   A.K. Bhushan. File Transfer Protocol; RFC 354. July, 1972.
6.   D. Crocker, J. Vittal, K.T. Pogran, D.A. Henderson.
Standard for the format of ARPA network text messages., RFC 733.
Nov-21-1977.
7.   D. Crocker, Standard for the format of ARPA Internet text
messages; RFC 822 . Aug-13-1982.
8.   J. Klensin, N. Freed, M. Rose, E. Stefferud & D. Crocker.
SMTP Service Extensions; RFC 1651. July 1994.
9.   Klensin, N. Freed, M. Rose, E. Stefferud & D. Crocker. SMTP
Service Extension for 8bit-MIME transport; RFC 1652 . July 1994.
10.  J. Klensin, N. Freed & K. Moore. SMTP Service Extension for
Message Size Declaration; RFC 1653. July 1994.
11.  P.V. Mockapetris.  Domain names: Concepts and facilities;
RFC 882 . November, 1983.
12.  P.V. Mockapetris. Domain names: Implementation
specification; RFC 883.  November, 1983.
13.  K.  Moore.Representation of Non-ASCII Text in Internet
Message Headers; RFC 1342 .  June 1992.
14.  J. Postel. Structured format for transmission of multi-media
documents, RFC 767.
15.  J. Postel. Simple Mail Transfer Protocol; RFC 821. Aug-01-
1982.
16.  J.K. Reynolds, J. Postel.  Request For Comments reference
guide; RFC 1000. August, 1987.
17.  M. Rose.  The Internet Message:  Closing the Book with
Electronic Mail.  Prentic Hall, 1993.
18.  S. Sluizer, J. Postel. Mail Transfer Protocol; RFC 772 .
Sept. 1980.
_______________________________
1 The author is indebted to Ken Pogran, John Vittal, Einar
Stefferud and Vint Cerf for participating in the spontaneous
review of early email history that we conducted during a
"Fireside Chat" at the Boston, 1994 EMail World conference.
2 The per-cent hack was intended only as a short-term mechanism
until Domain Names were fully operational.  The persistence of
the mechanism into today's Internet shows either that the evil
men do really does live after them, or that it was a brilliant
invention.  Unfortunately, the author believes that the former is
the more likely truth.
3 The author occasionally jokes that his major contribution to
the later MIME development was to convince the MIME authors to
make sure the name of the specification could be expressed as an
acronym, independent of its publication identifier.  This makes
reference to the specification easier and more stable.
4 Most Internet standards experiences with these sorts of basic,
philosophical differences of opinions are that they result in
some sort of technical warfare.  In this case, however, the
result was a set of highly complementary facilities.