doc: Add RFC documents

1 files changed, 1538 insertions, 0 deletions

diff --git a/doc/rfc/rfc957.txt b/doc/rfc/rfc957.txt
new file mode 100644
index 0000000..8a4b365
--- /dev/null
+++ b/doc/rfc/rfc957.txt
@@ -0,0 +1,1538 @@
+
+Network Working Group                                         D.L. Mills
+Request for Comments: 957                               M/A-COM Linkabit
+                                                          September 1985
+
+              Experiments in Network Clock Synchronization
+
+
+Status of this Memo
+
+   This RFC discusses some experiments in clock synchronization in the
+   ARPA-Internet community, and requests discussion and suggestions for
+   improvements.  Distribution of this memo is unlimited.
+
+Table of Contents
+
+   1.      Introduction
+   2.      Design of the Synchronization Algorithm
+   2.1.    The Logical Clock
+   2.2.    Linear Phase Adjustments
+   2.3.    Nonlinear Phase Adjustments
+   3.      Synchronizing Network Clocks
+   3.1.    Reference Clocks and Reference Hosts
+   3.2.    Distribution of Timing Information
+   4.      Experimental Validation of the Design
+   4.1.    Experiment Design
+   4.2.    Experiment Execution
+   4.3.    Discussion of Results
+   4.3.1.  On Power-Grid Clocks
+   4.3.2.  On Clocks Synchronized via Network Links
+   4.3.3.  On the Accuracy of Radio Clocks
+   4.3.3.1. The Spectracom 8170 WWVB Radio Clock
+   4.3.3.2. The True Time 468-DC GOES Radio Clock
+   4.3.3.3. The Heath GC-1000 WWV Radio Clock
+   4.3.4.  On Handling Disruptions
+   4.4.    Additional Experiments
+   5.      Summary and Conclusions
+   6.      References
+
+List of Figures
+
+   Figure 1. Clock Registers
+   Figure 2. Network Configuration
+
+
+
+
+
+
+
+
+
+
+
+Mills                                                           [Page 1]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+List of Tables
+
+   Table 1. Experiment Hosts
+   Table 2. Link Measurements
+   Table 3. First Derivative of Delay
+   Table 4. GOES Radio Clock Offsets
+   Table 5. WWV Radio Clock Offsets
+   Table 6. ISI-MCON-GW Clock Statistics
+   Table 7. LL-GW Clock Statistics
+   Table 8. LL-GW Clock Statistics
+
+1.  Introduction
+
+   One of the services frequently neglected in computer network design
+   is a high-quality, time-of-day clock capable of generating accurate
+   timestamps with small residual errors compared to intrinsic one-way
+   network delays.  Such a service would be useful for tracing the
+   progress of complex transactions, synchronizing cached data bases,
+   monitoring network performance and isolating problems.
+
+   Several mechanisms have been specified in the Internet protocol suite
+   to record and transmit the time at which an event takes place,
+   including the ICMP Timestamp message [6], Time Protocol [7], Daytime
+   protocol [8] and IP Timestamp option [9].  A new Network Time
+   Protocol [12] has been proposed as well.  Additional information on
+   network time synchronization can be found in the References at the
+   end of this document.  Synchronization protocols are described in [3]
+   and [12] and synchronization algorithms in [2], [5], [10] and [11].
+   Experimental results on measured roundtrip delays in the Internet are
+   discussed in [4].  A comprehensive mathematical treatment of clock
+   synchronization can be found in [1].
+
+   Several mechanisms have been specified in the Internet protocol suite
+   to record and transmit the time at which an event takes place,
+   including the ICMP Timestamp message [6], Time protocol [7], Daytime
+   protocol [8] and IP Timestamp option [9].  Issues on time
+   synchronization are discussed in [4] and synchronization algorithms
+   in [2] and [5].  Experimental results on measured roundtrip delays in
+   the Internet are discussed in [2].  A comprehensive mathematical
+   treatment of the subject can be found in [1], while an interesting
+   discussion on mutual-synchonization techniques can be found in [10].
+
+   There are several ways accurate timestamps can be generated.  One is
+   to provide at every service point an accurate, machine-readable clock
+   synchronized to a central reference, such as the National Bureau of
+   Standards (NBS).  Such clocks are readily available in several models
+   ranging in accuracies of a few hundred milliseconds to less than a
+
+
+Mills                                                           [Page 2]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+   millisecond and are typically synchronized to special ground-based or
+   satellite-based radio broadcasts.  While the expense of the clocks
+   themselves, currently in the range $300 to $3000, can often be
+   justified, all require carefully sited antennas well away from
+   computer-generated electromagnetic noise, as well as shielded
+   connections to the clocks.  In addition, these clocks can require a
+   lengthy synchonization period upon power-up, so that a battery-backup
+   power supply is required for reliable service in the event of power
+   interruptions.
+
+   If the propagation delays in the network are stable or can be
+   predicted accurately, timestamps can be generated by a central
+   server, equipped with a clock such as described above, in response to
+   requests from remote service points.  However, there are many
+   instances where the trans-network delay to obtain a timestamp would
+   be intolerable, such as when timestamping a message before
+   transmission.  In addition, propagation delays are usually not
+   predictable with precisions in the order required, due to
+   probabilistic queuing and channel-contention delays.
+
+   In principle, a clock of sufficient accuracy can be provided at each
+   service point using a stable, crystal-controlled clock which is
+   corrected from time to time by messages from a central server.
+   Suitable inexpensive, crystal-controlled clock interfaces are
+   available for virtually any computer.  The interesting problem
+   remaining is the design of the synchronization algorithm and protocol
+   used to transmit the corrections.  In this document one such design
+   will be described and its performance assessed.  This design has been
+   incorprated as an integral part of the network routing and control
+   protocols of the Distributed Computer Network (DCnet) architecture
+   [5], clones of which have been established at several sites in the US
+   and Europe.  These protocols have been in use since 1979 and been
+   continuously tested and refined since then.
+
+2.  Design of the Synchronization Algorithm
+
+   The synchronization algorithm is distributed in nature, with protocol
+   peers maintained in every host on the network.  Peers communicate
+   with each other on a pairwise basis using special control messages,
+   called Hello messages, exchanged periodically over the ordinary data
+   links between them.  The Hello messages contain information necessary
+   for each host to calculate the delay and offset between the local
+   clock of the host and the clock of every other host on the network
+   and are also used to drive the routing algorithm.
+
+   The synchronization algorithm includes several features to improve
+   the accuracy and stability of the local clock in the case of host or
+
+
+Mills                                                           [Page 3]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+   link failures.  In following sections the design of the algorithm is
+   summarized.  Full design details are given in [5] along with a formal
+   description of the Hello protocol.
+
+2.1.  The Logical Clock
+
+   In the DCnet model each service point, or host, is equipped with a
+   hardware clock, usually in the form of an off-the-shelf interface.
+   Using this and software registers, a logical clock is constructed
+   including a 48-bit Clock Register, which increments at a 1000 Hz
+   rate, a 32-bit Clock-Adjust Register, which is used to slew the Clock
+   Register in response to raw corrections received over the net, and a
+   Counter Register, which is used in some interface designs as an
+   auxilliary counter.  The configuration and decimal point of these
+   registers are shown in Figure 1.
+
+           Clock Register                                   
+
+           0               16               32              
+           +---------------+---------------+---------------+
+           |               |               |               |
+           +---------------+---------------+---------------+
+                                           A                
+                                     decimal point          
+
+           Clock-Adjust Register                            
+
+                           0               16               
+                           +---------------+---------------+
+                           |               |               |
+                           +---------------+---------------+
+                                           A                
+                                     decimal point          
+
+           Counter Register                                 
+
+                           0              16                
+                           +---------------+                
+                           |               |                
+                           +---------------+                
+                                           A                
+                                     decimal point          
+
+                       Figure 1. Clock Registers
+
+   The Clock Register and Clock-Adjust Register are implemented in
+   memory.  In typical clock interface designs such as the DEC KMV11-A
+
+
+Mills                                                           [Page 4]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+   the Counter Register is implemented in the interface as a buffered
+   counter driven by a crystal oscillator.  A counter overflow is
+   signalled by an interrupt, which results in an increment of the Clock
+   Register at bit 15 and the propagation of carries as required.  The
+   time of day is determined by reading the Counter Register, which does
+   not disturb its counting process, and adding its value to that of the
+   Clock Register with decimal points aligned.
+
+   In other interface designs such as the simple LSI-11 event-line
+   mechanism, each tick of the clock is signalled by an interrupt at
+   intervals of 10, 16-2/3 or 20 ms, depending on interface and clock
+   source.  When this occurs the appropriate number of milliseconds,
+   expressed to 32 bits in precision, is added to the Clock Register
+   with decimal points aligned.
+
+   It should be noted at this point that great care in operating system
+   design is necessary in order to preserve the full accuracy of
+   timestamps with respect to the application program, which must be
+   protected from pre-emption, excessive device latencies and so forth.
+   In addition, the execution times of all sequences operating with the
+   interrupt system disabled must be strictly limited.  Since the PDP11
+   operating system most often used in the DCnet (the "Fuzzball"
+   operating system) has been constructed with these considerations
+   foremost in mind, it has been especially useful for detailed network
+   performance testing and evaluation.  Other systems, in particular the
+   various Unix systems, have not been found sufficiently accurate for
+   this purpose.
+
+   Left uncorrected, the host logical clock runs at the rate of its
+   intrinsic oscillator, whether derived from a crystal or the power
+   frequency.  The correction mechanism uses the Clock-Adjust Register,
+   which is updated from time to time as raw corrections are received.
+   The corrections are computed using roundtrip delays and offsets
+   derived from the routing algorithm, described later in this document,
+   which are relatively noisy compared to the precision of the logical
+   clock.  A carefully designed smoothing mechansim insures stability,
+   as well as isolation from large transients that occur due to link
+   retransmissions, host reboots and similar disruptions.
+
+
+
+
+
+
+
+
+
+
+
+Mills                                                           [Page 5]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+2.2.  Linear Phase Adjustments
+
+   The correction is introduced as a signed 32-bit integer in
+   milliseconds.  If the magnitude of the correction is less than 128
+   ms, the low-order 16 bits replaces bits 0-15 in the Clock-Adjust
+   register. At suitable intervals, depending on the jitter of the
+   intrinsic oscillator, the value of this register is divided by a
+   fixed value, forming a quotient which is first added to the Clock
+   Register, then subtracted from the Clock-Adjust Register.  This
+   technique has several advantages:
+
+      1.  The clock never runs backwards;  that is, successive
+          timestamps always increase monotonically.
+
+      2.  In the event of loss of correction information, the clock
+          slews to the last correction received.
+
+      3.  The rate of slew is proportional to the magnitude of the last
+          correction.  This allows rapid settling in case of large
+          corrections, but provides high stability in case of small
+          corrections.
+
+      4.  The sequence of computations preserves the highest precision
+          and minimizes the propagation of round-off errors.
+
+   Experience has indicated the choice of 256 as appropriate for the
+   dividend above, which yields a maximum slew-rate magnitude less than
+   0.5 ms per adjustment interval and a granularity of about 2.0
+   microseconds, which is of the same order as the intrinsic tolerance
+   of the crystal oscillators used in typical clock interfaces.  In the
+   case of crystal-derived clocks, an adjustment interval of four
+   seconds has worked well, which yields a maximum slew-rate magnitude
+   of 125 microseconds per second.  In the case of power-frequency
+   clocks or especially noisy links, the greatly increased jitter
+   requires shorter adjustment intervals in the range of 0.5 second,
+   which yields a maximum slew-rate magnitude of 1.0 ms per second.
+
+   In most cases, independent corrections are generated over each link
+   at intervals of 30 seconds or less.  Using the above choices a single
+   sample error of 128 ms causes an error at the next sample interval no
+   greater than about 7.5 ms with the longer adjustment interval and 30
+   ms with the shorter.  The number of adjustment intervals to reduce
+   the residual error by half is about 177, or about 12 minutes with the
+   longer interval and about 1.5 minutes with the shorter.  This
+   completely characterizes the linear dynamics of the mechanism.
+
+
+
+
+Mills                                                           [Page 6]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+2.3.  Nonlinear Phase Adjustments
+
+   When the magnitude of the correction exceeds 128 ms, the possiblity
+   exists that the clock is so far out of synchronization with the
+   reference host that the best action is an immediate and wholesale
+   replacement of Clock Register contents, rather than a graduated
+   slewing as described above.  In practice the necessity to do this is
+   rare and occurs when the local host or reference host is rebooted,
+   for example. This is fortunate, since step changes in the clock can
+   result in the clock apparently running backward, as well as incorrect
+   delay and offset measurements of the synchronization mechanism
+   itself.
+
+   However, it sometimes happens that, due to link retransmissions or
+   occasional host glitches, a single correction sample will be computed
+   with magnitude exceeding 128 ms.  In practice this happens often
+   enough that a special procedure has been incorporated into the
+   design.  If a sample exceeding the limit is received, its value is
+   saved temporarily and does not affect the Clock-Adjust Register.  In
+   addition, a timer is initialized, if not already running, to count
+   down to zero in a specified time, currently 30 seconds.
+
+   If the timer is already running when a new correction sample with
+   magnitude exceeeding 128 ms arrives, its value and the saved sample
+   value are averaged with equal weights to form a new saved sample
+   value. If a new correction sample arrives with magnitude less than
+   128 ms, the timer is stopped and the saved sample value discarded.
+   If the timer counts down to zero, the saved sample value replaces the
+   contents of the Clock Register and the Clock-Adjust Register is set
+   to zero.  This procedure has the effect that occasional spikes in
+   correction values are discarded, but legitimate step changes are
+   prefiltered and then used to reset the clock after no more than a
+   30-second delay.
+
+3.  Synchronizing Network Clocks
+
+   The algorithms described in the previous section are designed to
+   achieve a high degree of accuracy and stability of the logical clocks
+   in each participating host.  In this section algorithms will be
+   described which synchronize these logical clocks to each other and to
+   standard time derived from NBS broadcasts.  These algorithms are
+   designed to minimize the cumulative errors using linear synchronizing
+   techniques. See [10] for a discussion of algorithms using nonlinear
+   techniques.
+
+
+
+
+
+Mills                                                           [Page 7]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+3.1.  Reference Clocks and Reference Hosts
+
+   The accuracy of the entire network of logical clocks depends on the
+   accuracy of the device used as the reference clock.  In the DCnet
+   clones the reference clock takes the form of a precision crystal
+   oscillator which is synchronized via radio or satellite with the NBS
+   standard clocks in Boulder, CO.  The date and time derived from the
+   oscillator can be sent continuously or read upon command via a serial
+   asynchronous line.  Stand-alone units containing the oscillator,
+   synchronizing receiver and controlling microprocessor are available
+   from a number of manufacturers.
+
+   The device driver responsible for the reference clock uses its
+   serial-line protocol to derive both an "on-time" timestamp relative
+   to the logical clock of the reference host and an absolute time
+   encoded in messages sent by the clock.  About once every 30 seconds
+   the difference between these two quantities is calculated and used to
+   correct the logical clock according to the mechanisms described
+   previously.  The corrected logical clock is then used to correct all
+   other logical clocks in the network.  Note the different
+   nomenclature:  The term "reference clock" applies to the physical
+   clock itself, while the term "reference host" applies to the logical
+   clock of the host to which it is connected. Each has an individual
+   address, delay and offset in synchronizing messages.
+
+   There are three different commercial units used as reference clocks
+   in DCnet clones.  One of these uses the LF radio broadcasts on 60 KHz
+   from NBS radio WWVB, another the HF radio broadcasts on 5, 10 and 15
+   MHz from NBS radio WWV or WWVH and the third the UHF broadcasts from
+   a GOES satellite.  The WWVB and GOES clocks claim accuracies in the
+   one-millisecond range.  The WWV clock claims accuracies in the 100-ms
+   range, although actual accuracies have been measured somewhat better
+   than that.
+
+   All three clocks include some kind of quality indication in their
+   messages, although of widely varying detail.  At present this
+   indication is used only to establish whether the clock is
+   synchronized to the NBS clocks and convey this information to the
+   network routing algorithm as described later.  All clocks include
+   some provision to set the local-time offset and propagation delay,
+   although for DCnet use all clocks are set at zero offset relative to
+   Universal Time (UT).  Due to various uncertaincies in propagation
+   delay, serial-line speed and interrupt latencies, the absolute
+   accuracy of timestamps derived from a reference host equipped with a
+   WWVB or GOES reference clock is probably no better than a couple of
+   milliseconds.
+
+
+
+Mills                                                           [Page 8]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+3.2.  Distribution of Timing Information
+
+   The timekeeping accuracy at the various hosts in the network depends
+   critically on the precision whith which corrections can be
+   distributed throughout the network.  In the DCnet design a
+   distributed routing algorithm is used to determine minimum-delay
+   routes between any two hosts in the net.  The algorithms used, which
+   are described in detail in [5] and only in outline form here, provide
+   reachability and delay information, as well as clock offsets, between
+   neighboring hosts by means of periodic exchanges of routing packets
+   called Hello messages. This information is then incorporated into a
+   set of routing tables maintained by each host and spread throughout
+   the network by means of the Hello messages.
+
+   The detailed mechanisms to accomplish these functions have been
+   carefully designed to minimize timing uncertaincies.  For instance,
+   all timestamping is done in the drivers themselves, which are given
+   the highest priority for resource access.  The mechanism to measure
+   roundtrip delays on the individual links is insensitive to the delays
+   inherent in the processing of the Hello message itself, as well as
+   the intervals between transmissions.  Finally, care is taken to
+   isolate and discard transient timing errors that occur when a host is
+   rebooted or a link is restarted.
+
+   The routing algorithm uses a table called the Host Table, which
+   contains for each host in the network the computed roundtrip delay
+   and clock offset, in milliseconds.  In order to separately identify
+   each reference clock, if there is more than one in the network, a
+   separate entry is used for each clock, as well as each host.  The
+   delay and offset fields of the host itself are set to zero, as is the
+   delay field of each attached reference clock.  The offset field of
+   each attached reference clock is recomputed periodically as described
+   above.
+
+   Hello messages containing a copy of the Host Table are sent
+   periodically to each neighbor host via the individual links
+   connecting them.  In the case of broadcast networks the Hello message
+   is broadcast to all hosts sharing the same cable.  The Hello message
+   also contains a timestamp inserted at the time of transmission, as
+   well as information used to accurately compute the roundtrip delay on
+   point-to-point links.
+
+   A host receiving a Hello message processes the message for each host
+   in turn, including those corresponding to the reference clocks.  It
+   adds the delay field in the message to the previously determined
+   roundtrip link delay and compares this with the entry already in its
+   Host Table.  If the sum is greater than the delay field in the Host
+
+
+Mills                                                           [Page 9]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+   Table, nothing further is done.  If the sum is less, an update
+   procedure is executed.  The update procedure, described in detail in
+   [5], causes the new delay to replace the old and the routing to be
+   amended accordingly.
+
+   The update procedure also causes a new correction sample to be
+   computed as the difference between the timestamp in the Hello message
+   and the local clock, which is used to correct the local clock as
+   described above.  In addition, the sum of this correction sample plus
+   the offset field in the Hello message replaces the offset field in
+   the Hello Table.  The effect of these procedures is that the local
+   clock is corrected relative to the neighbor clock only if the
+   neighbor is on the path of least delay relative to the selected
+   reference clock.  If there is no route to the reference clock, as
+   determined by the routing algorithm, no corrections are computable
+   and the local clock free-runs relative to the last received
+   correction.
+
+   As the result of the operation of the routing algorithm, all local
+   clocks in the network will eventually stabilize at a constant offset
+   relative to the reference clock, depending upon the drift rates of
+   the individual oscillators.  For most applications the offset is
+   small and can be neglected.  For the most precise measurements the
+   computed offset in the Host Table entry associated with any host,
+   including the reference clock, can be used to adjust the apparent
+   time relative to the local clock of that host.  However, since the
+   computed offsets are relatively noisy, it is necessary to smooth them
+   using some algorithm depending upon application.  For this reason,
+   the computed offsets are provided separately from the local time.
+
+4.  Experimental Validation of the Design
+
+   The original DCnet was established as a "port expander" net connected
+   to an ARPAnet IMP in 1978.  It was and is intended as an experimental
+   testbed for the development of protocols and measurement of network
+   performance.  Since then the DCnet network-layer protocols have
+   evolved to include multi-level routing, logical addressing,
+   multicasting and time synchronization.  Several DCnet clones have
+   been established in the US and Europe and connected to the DARPA
+   Internet system.  The experiments described below were performed
+   using the DCnet clone at Linkabit East, near Washington, DC, and
+   another at Ford Motor Division, near Detroit, MI.  Other clones at
+   Ford Aerospace and the Universities of Maryland and Michigan were
+   used for calibration and test, while clones at various sites in
+   Norway and Germany were used for occasional tests.  Additional
+
+
+
+
+Mills                                                          [Page 10]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+   ARPANET gateways of the WIDEBAND/EISN satellite system were also
+   included in the experiments in order to determine the feasability of
+   synchronizing clocks across the ARPANET.
+
+   There were four principal issues of interest in the experiments:
+
+      1.  What are the factors affecting accuracy of a network of clocks
+          using the power grid as the basic timing source, together with
+          corrections broadcast from a central point?
+
+      2.  What are the factors affecting accuracy of a network of clocks
+          synchronized via links used also to carry ordinary data.
+
+      3.  How does the accuracy of the various radio clocks - WWVB, GOES
+          and WWV compare?
+
+      4.  What is the best way to handle disruptions, such as a leap
+          second?
+
+   These issues will be discussed in turn after presentation of the
+   experiment design and execution.
+
+4.1.  Experiment Design
+
+   Figure 2 shows the configuration used in a series of tests conducted
+   during late June and early July of 1985.  The tests involved six
+   hosts, three reference clocks and several types of communication
+   links.  The tests were designed to coincide with the insertion of a
+   leap second in the standard time broadcast by NBS, providing an
+   interesting test of system stability in the face of such disruptions.
+   The test was also designed to test the feasability of using the power
+   grid as a reference clock, with corrections broadcast as described
+   above, but not used to adjust the local clock.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Mills                                                          [Page 11]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+      ARPAnet                              |                      
+      - - - - - - - - - - - - - - - - - -  |  - - - - - - - - - - 
+                                       56K |                      
+      +---------+     +---------+     +----+----+ 1.2 +---------+ 
+      |   WWV   | 1.2 |         | 4.8 |         +-----+  WWVB   | 
+      |  radio  +-----+  DCN6   +-----+  DCN1   |async|  radio  | 
+      |  clock  |async|         |DDCMP|         +--+  |  clock  | 
+      +---------+     +---------+     +----+----+  |  +---------+ 
+                       Ethernet            |       |              
+      DCnet     ===o===============o=======o===    | 1822/DH      
+                   |               |               |              
+              +----+----+     +----+----+     +----+----+         
+      power   |         |     |         |     |         |    power
+      freq <--+  DCN3   |     |  DCN5   |     |  DCN7   +--> freq 
+      60 Hz   |         |     |         |     |         |    60 Hz
+              +---------+     +----+----+     +---------+         
+                               9.6 |                              
+      - - - - - - - - - - - - - -  |  - - - - - - - - - - - - - - 
+                                   | DDCMP                        
+                              +----+----+     +---------+         
+                              |         | 1.2 |  GOES   |         
+      FORDnet                 |  FORD1  +-----+satellite|         
+                              |         |async|  clock  |         
+                              +---------+     +---------+         
+
+                    Figure 2. Network Configuration
+
+   Only those hosts and links directly participating in the tests are
+   shown in Figure 2.  All hosts shown operate using the DCnet protocols
+   and timekeeping algorithms summarized in this document and detailed
+   in [5].  The DCnet hosts operate as one self-contained net of the
+   Internet systems, while the FORDnet hosts operate as another with
+   distinct net numbers.  The gateway functions connecting the two nets
+   are distributed in the DCN5 and FORD1 hosts and the link connecting
+   them.  This means that, although the clock offsets of individual
+   DCnet hosts are visible to other DCnet hosts and the clock offsets of
+   individual FORDnet hosts are visible to other FORDnet hosts, only the
+   clock offset of the gateway host on one net is visible to hosts on
+   the other net.
+
+   In Figure 2 the links are labelled with both the intrinsic speed, in
+   kilobits per second, as well as the link protocol type.  The DDCMP
+   links use microprocessor-based DMA interfaces that retransmit in case
+   of message failure.  The 1822/DH link connecting DCN1 and DCN7
+   operates at DMA speeds over a short cable.  The Ethernet link uses
+
+
+
+
+Mills                                                          [Page 12]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+   DMA interfaces that retransmit only in case of collisions.  The
+   asynchronous links are used only to connect the reference clocks to
+   the hosts over a short cable.
+
+   While all hosts and links were carrying normal traffic throughout the
+   test period, the incidence of retransmissions was very low, perhaps
+   no more than a few times per day on any link.  However, the DDCMP
+   link protocol includes the use of short control messages exhanged
+   between the microprocessors about once per second in the absence of
+   link traffic. These messages, together with retransmissions when they
+   occur, cause small uncertaincies in Hello message delays that
+   contribute to the total measurement error.  An additional uncertaincy
+   (less than 0.5 per-cent on average) in Hello message length can be
+   introduced when the link protocol makes use of character-stuffing or
+   bit-stuffing techniques to achieve code transparency, such as with
+   the LAPB link-level protocol of X.25.  However, the particular links
+   used in the tests use a count field in the header, so that no
+   stuffing is required.
+
+   Although the timekeeping algorithms have been carefully designed to
+   be insensitive to traffic levels, it sometimes happens that an
+   intense burst of traffic results in a shortage of memory buffers in
+   the various hosts.  In the case of the Ethernet interfaces, which
+   have internal buffers, this can result in additional delays while the
+   message is held in the interface pending delivery to the host.
+   Conditions where these delays become significant occur perhaps once
+   or twice a day in the present system and were observed occasionally
+   during the tests.  As described above, the correction-sample
+   processing incorporates a filtering procedure that discards the vast
+   majority of glitches due to this and other causes.
+
+4.2.  Experiment Execution
+
+   The series of experiments conducted in late June and early July of
+   1985 involved collecting data on the delays and offsets of the six
+   hosts and three reference clocks shown in Figure 2.  In order to
+   accomplish this, a special program was installed in a Unix 4.2bsd
+   system connected to the Ethernet link but not shown in Figure 2.  The
+   program collected each 128-octet Hello message broadcast from DCN1
+   every 16 seconds and appended it bit-for-bit to the data base.  The
+   total volume of raw data collected amounted to almost 0.7 megabyte
+   per day.
+
+   The raw Hello-message data were processed to extract only the
+   timestamp and measured clock offsets for the hosts shown in Table 1
+   and then reformatted as an ASCII file, one line per Hello message.
+
+
+
+Mills                                                          [Page 13]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+        Host    Clock   Drift   Experiment Use                 
+        Name    ID      (ppm)                                  
+        ------------------------------------------------------ 
+        DCN1    WWVB    -2.5    WWVB reference host            
+        DCN3    -       60-Hz   power-grid (unlocked)          
+        DCN5    DCN1    6.8     Ethernet host                  
+        DCN6    DCN1    -1.7    DDCMP host, WWV reference host 
+        DCN7    DCN1    60-Hz   power-grid (locked)            
+        FORD1   GOES    17.9    GOES reference host            
+        WWV     -       -       WWV reference clock            
+        WWVB    -       -       WWVB reference clock           
+
+                       Table 1. Experiment Hosts
+
+   In Table 1 the Clock ID column shows the reference host selected as
+   the master clock for each host shown.  In this particular
+   configuration host DCN1 was locked to host WWVB, while hosts DCN5,
+   DCN6 and DCN7 were locked to DCN1.  Although the offset of GOES can
+   not be directly determined from the Hello messages exchanged between
+   DCnet and FORDnet hosts, the offset of FORD1 relative to GOES was
+   determined by observation to be in the order of a millisecond, so for
+   all practical purposes the offset of FORD1 represents the offset of
+   GOES.  In addition, since the WWVB clock was considered by experience
+   the most accurate and reliable and the offset of DCN1 relative to
+   WWVB was negligible, DCN1 was considered the reference clock with
+   offset zero relative to the NBS clocks.
+
+   During the setup phase of the experiments the intrinsic drift rates
+   of the crystal oscillators in the four hosts DCN1, DCN5, DCN6 and
+   FORD1 equipped with them was measured as shown in the "Drift" column
+   in Table 1.  The two hosts DCN3 and DCN7 are equipped with
+   line-frequency clocks. For experimental purposes DCN3 was unlocked
+   and allowed to free-run at the line-frequency rate, while DCN7
+   remained locked.
+
+   An ASCII file consisting of about 0.2 megabyte of reformatted data,
+   was collected for each Universal-Time (UT) day of observation
+   beginning on 28 June and continuing through 8 July.  Each file was
+   processed by a program that produces an eight-color display of
+   measured offsets as a function of time of observation.  Since the
+   display technique uses a bit-map display and each observation
+   overwrites the bit-map in an inclusive-OR fashion, the sample
+   dispersion is immediately apparent. Over eight samples per pixel on
+   the time axis are available in a 24-hour collection period.  On the
+   other hand, the fine granularity of almost four samples per minute
+   allows zooming the display to focus on interesting short-term
+   fluctuations, such as in the case of the WWV clock.
+
+
+Mills                                                          [Page 14]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+4.3.  Discussion of Results
+
+   Each of the four previously mentioned issues of interest will be
+   discussed in following subsections.
+
+4.3.1.  On Power-Grid Clocks
+
+   Telephone interviews with operators and supervisors of the Potomac
+   Electric Power Company (PEPCO), the electric utility serving the
+   Washington, DC, area, indicate that there are three major operating
+   regions or grids, one east of the Rockies, a second west of the
+   Rockies and a third in parts of Texas.  The member electric utilities
+   in each grid operate on a synchronous basis, so that clocks anywhere
+   within the grid should keep identical time.  However, in the rare
+   case when a utility drops off the grid, no attempt is made to
+   re-establish correct time upon rejoining the grrd.  In the much more
+   common case when areas within the grid are isolated due to local
+   thunderstorms, for example, clock synchronization is also disrupted.
+
+   The experiments provided an opportunity to measure with exquisite
+   precision the offset between a clock connected to the eastern grid
+   (DCN3) and the NBS clocks.  The results, confirmed by the telephone
+   interviews, show a gradual gain in time of between four and six
+   seconds during the interval from about 1700 local time to 0800 the
+   next morning, followed by a more rapid loss in time between 0800 and
+   1700.  If the time was slewed uniformly throughout these extremes,
+   the rate would be about 100 ppm.
+
+   The actual slewing rates depend on the demand, which itself is a
+   function of weather, day of the week and season of the year.  Similar
+   effects occur in the western and Texas grids, with more extreme
+   variations in the Texas grid due to the smaller inertia of the
+   system, and less extreme variations in the western grid, due to
+   smaller extremes in temperature, less total industrial demand and a
+   larger fraction of hydro-electric generation.
+
+   The uilities consider timekeeping a non-tariffed service provided as
+   a convenience to the customer.  In the eastern grid a control station
+   in Ohio manually establishes the baseline system output, which
+   indirectly affects the clock offset and slewing rate.  The local time
+   is determined at the control station with respect to a WWVB radio
+   clock. The maximum slewing rate is specified as .025 Hz (about 400
+   ppm), which is consistent with the maximum rates observed.  In the
+   western grid the baseline system output is adjusted automatically
+   using a servomechanism driven by measured offsets from the NBS
+   clocks.
+
+
+
+Mills                                                          [Page 15]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+   Given the considerations above, it would seem feasable for hosts to
+   synchronize logical clocks to a particular power grid, but only if
+   corrections were transmitted often enough to maintain the required
+   accuracy and these corrections were delivered to the hosts
+   essentially at the same time.  Assuming a worst-case 400-ppm slewing
+   rate and one minute between correction broadcasts, for example, it
+   would in principle be possible to achieve accuracies in the 20-ms
+   range.  There are a number of prediction and smoothing techniques
+   that could be used to inhance accuracy and reduce the overhead of the
+   broadcasts.
+
+   Host DCN3, which uses a line-frequency clock interface, was unlocked
+   during the experiment period so that the offset between the PEPCO
+   clock, which is locked to the eastern power grid, could be measured
+   with respect to the reference host DCN1.  Host DCN7, which uses the
+   same interface, remained locked to DCN1.  In spite of the previously
+   noted instability of the power grid, DCN7 remained typically within
+   30 ms of DCN1 and only infrequently exceeded 100 ms in the vicinity
+   of large changes in system load that occured near 0800 and 1700 local
+   time. Over the seven-day period from 2 July through 8 July the mean
+   offset was less than a millisecond with standard deviation about 24
+   ms, while the maximum was 79 ms and minimum -116 ms.
+
+   Experiments were also carried out using ICMP Timestamp messages with
+   hosts known to use line-frequency clock interfaces in California,
+   Norway and Germany.  The results indicated that the western power
+   grid is rather more stable than the eastern grid and that the
+   overseas grids are rather less stable.  In the Oslo, Munich and
+   Stuttgart areas, for example, the diurnal variation was observed to
+   exceed ten seconds.
+
+4.3.2.  On Clocks Synchronized via Network Links
+
+   As mentioned previously, all network links used to synchronize the
+   clocks were carrying normal data traffic throughout the experiment
+   period.  It would therefore be of interest to investigate how this
+   affects the accuracy of the individual clocks.
+
+   Table 2 summarizes the mean and standard deviation of the measured
+   offsets between the WWVB radio clock and various hosts as shown in
+   Figure 2.  Measurements were made over the 24-hour period for each of
+   several days during the experimental period.  Each entry shown in
+   Table 2 includes the mean of the statistic over these days, together
+   with the maximum variation.
+
+
+
+
+
+Mills                                                          [Page 16]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+      Host  Mean          Deviation    Link Type and Speed        
+      ----------------------------------------------------------- 
+      DCN1  .08/.02       0.53/.02     WWVB radio clock (1200 bps)
+      DCN5  -13.61/.04    1.1/0.4      Ethernet (10 Mbps)         
+      DCN6  0.27/.18      5.8/1.0      DDCMP (4800 bps)           
+      FORD1 38.5/1.6      2.5/0.5      DDCMP (9600 bps)           
+
+                       Table 2. Link Measurements
+
+   The departure of the mean shown in Table 2 from zero is related to
+   the drift of the crystal oscillator used in the hardware interface
+   (see Table 1).  As described previously, FORD1 was synchonized to the
+   GOES radio clock with neglible offset, so that the mean and standard
+   deviation shown can be accurately interpreted to apply to the GOES
+   radio clock as well.
+
+   The results show that the uncertaincies inherent in the
+   synchronization algorithm and protocols is in the same order as that
+   of the reference clocks and is related to the speed of the links
+   connected the reference hosts to the other hosts in the network.
+   Further discussion on the FORD1/GOES statistics can be found in the
+   next section.
+
+   Further insight into the error process can be seen in Table 3, which
+   shows the first derivative of delay.
+
+                 Host    Dev     Max     Min     Error 
+                 ------------------------------------- 
+                 DCN3    2.3     12      -17     10    
+                 DCN5    1.5     45      -45     5     
+                 DCN6    9       94      -54     40    
+                 DCN7    1.4     6       -7      5     
+                 FORD1   3.4     68      -51     15    
+
+                   Table 3. First Derivative of Delay
+
+   The mean and standard deviation of delay were computed for all hosts
+   on a typical day during the experimental period.  In all cases the
+   magnitude of the mean was less than one.  The standard deviation,
+   maximum and minimum for each link is summarized by host in Table 3.
+   A common characteristic of the distribution in most cases was that
+   only a handful of samples approached the maximum or minimum extrema,
+   while the vast majority of samples were much less than this.  The
+   "Error" colum in Table 3 indicates the magnitude of the estimated
+   error when these extrema are discarded.
+
+
+
+
+Mills                                                          [Page 17]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+   A very interesting feature of the observations was the unexpectedly
+   low standard deviation of DCN3, which was locked to the power grid
+   and thus would be expected to show wide variations.  Upon analysis,
+   this turned out to be a natural consequence of the fact that the
+   Hello messages are generated as the result of interrupts based on the
+   line frequency when the local clock had just been incremented by
+   1/60th of a second.
+
+   The synchronizing protocol and implementation were carefully
+   constructed to minimize the loss of accuracy due to sharing of the
+   network links between data and control traffic, as long as sufficient
+   resources (in particular, packet buffers) are available.  Since the
+   various network links shown in Figure 2 operate over a wide range of
+   rates, it is possible that undisciplined bursts of traffic can swamp
+   a host or gateway and precipitate a condition of buffer starvation.
+
+   While most hosts using paths through the experimental configuration
+   were relatively well-disciplined in their packetization and
+   retransmission policies, some Unix 4.2bsd systems were notorious
+   exceptions.  On occasion these hosts were observed sending floods of
+   packets, with only a small amount of data per packet, together with
+   excessive retransmissions.  As expected, this caused massive
+   congestion, unpredictable link delays and occasional clock
+   synchronizing errors.
+
+   The synchronizing algorithms described above successfully cope with
+   almost all instances of congestion as described, since delay-induced
+   errors tend to be isolated, while inherent anti-spike and smoothing
+   properties of the synchronizing algorithm help to preserve accuracies
+   in any case.  Only one case was found during the ten-day experiment
+   period where a host was mistakenly synchronized outside the
+   linear-tracking window due to congestion.  Even in this case the host
+   was quickly resynchronized to the correct time when the congestion
+   was cleared.
+
+4.3.3.  On the Accuracy of Radio Clocks
+
+   One of the more potent motivations for the experiments was to assess
+   the accuracy of the various radio clocks and to determine whether the
+   WWV radio clock was an appropriate replacement for the expensive WWVB
+   or GOES clocks.  A secondary consideration, discussed further in the
+   next section, was how the various clocks handled disruptions due to
+   power interruptions, leap seconds and so forth.
+
+
+
+
+
+
+Mills                                                          [Page 18]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+4.3.3.1.  The Spectracom 8170 WWVB Radio Clock
+
+   As the result of several years of experience with the WWVB radio
+   clock, which is manufactured by Spectracom Corporation as Model 8170,
+   it was chosen as the reference for comparison for the GOES and WWV
+   radio clocks.  Washington, DC, is near the 100-microvolt/meter
+   countour of the WWVB transmitter at Boulder, CO, well in excess of
+   the 25-microvolt/meter sensitivity of the receiver.  The antenna is
+   located in a favorable location on the roof of a four-storey building
+   in an urban area.
+
+   Using the data from the instruction manual, the propagation delay for
+   the path from Boulder to Washington is about 8 ms, while the
+   intrinsic receiver delay is about 17 ms.  The clock is read via a
+   1200-bps asynchronous line, which introduces an additional delay of
+   about 7 ms between the on-time transition of the first character and
+   the interrupt at the middle of the first stop bit.  Thus, the WWVB
+   radio clock indications should be late by 8 + 17 + 7 = 32 ms relative
+   to NBS standard time.  While it is possible to include this delay
+   directly in the clock indication, this was not done in the
+   experiments.  In order to account for this, 32 ms should be
+   subtracted from all indications derived from this clock.  The
+   uncertaincy in the indication due to all causes is estimated to be a
+   couple of milliseconds.
+
+4.3.3.2.  The True Time 468-DC GOES Radio Clock
+
+   The GOES radio clock is manufactured by True Time Division of
+   Kinemetrics, Incorporated, as Model 468-DC.  It uses the
+   Geosynchronous Orbiting Environmental Satellite (GOES), which
+   includes an NBS-derived clock channel.  Early in the experiment
+   period there was some ambiguity as to the exact longitude of the
+   satellite and also whether the antenna was correctly positioned.
+   This was reflected in the rather low quality-of-signal indications
+   and occasional signal loss reported by the clock and also its
+   apparent offset compared with the other radio clocks.
+
+   Table 4 shows a summary of offset statistics for the GOES radio clock
+   by day (all day numbers refer to July, 1985).
+
+
+
+
+
+
+
+
+
+
+Mills                                                          [Page 19]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+                 Day     Mean    Dev     Max     Min   
+                 ------------------------------------ 
+                 2       31.6    9.4     53      -76  
+                 3       19.8    22.1    53      -64  
+                 4       42.8    17.1    >150    19   
+                 5       39.3    2.2     54      -45  
+                 6       37.8    2.7     53      19   
+                 7       62.2    13.0    89      22   
+                 8       38.2    2.8     90      -7   
+
+                   Table 4. GOES Radio Clock Offsets
+
+    On all days except days 5, 6 and 8 long periods of poor-quality
+   signal reception were evident.  Since the antenna and satellite
+   configuration are known to be marginal, these conditions are not
+   considered representative of the capabilities of the clock.  When the
+   data from these days are discarded, the mean offset is 38.4 ms with
+   standard deviation in the range 2.2 to 2.8.  The maximum offset is 90
+   ms and the minimum is -45 ms;  however, only a very small number of
+   samples are this large - most excursions are limited to 10 ms of the
+   mean.
+
+   In order to compute the discrepancy between the GOES and WWVB clocks,
+   it is necessary to subtract the WWVB clock delay from the mean
+   offsets computed above.  Thus, the GOES clock indications are 38.4 -
+   32 = 6.4 ms late with respect to the WWVB clock indications.  which
+   is probably within the bounds of experiment error.
+
+4.3.3.3.  The Heath GC-1000 WWV Radio Clock
+
+   The WWV radio clock is manufactured by Heath Company as Model
+   GC-1000.  It uses a three-channel scanning WWV/WWVH receiver on 5, 10
+   and 15 MHz together with a microprocessor-based controller.  The
+   receiver is connected to an 80-meter dipole up about 15 meters and
+   located in a quiet suburban location.  Signal reception from the Fort
+   Collins transmitters was average to poor during the experiment period
+   due to low sunspot activity together with a moderate level of
+   geomagnetic disturbances, but was best during periods of darkness
+   over the path.  The clock locked at one of the frequencies for
+   varying periods up to an hour from two to several times a day.
+
+   The propagation delay on the path between Fort Collins and Washington
+   is estimated at about 10 ms and can vary up to a couple of
+   milliseconds over the day and night.  While it is possible to include
+   this delay in the clock indications, which are already corrected for
+
+
+
+
+Mills                                                          [Page 20]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+   the intrinsic receiver delay, this was not done in the experiments.
+   During periods of lock, the clock indications are claimed to be
+   accurate to within 100 ms.
+
+   Table 5 shows a summary of offset statistics for the WWV radio clock
+   by day (all day numbers refer to July, 1985).
+
+                 Day     Mean    Dev     Max     Min   
+                 ------------------------------------  
+                 2       -31     36      110     -119  
+                 3       -42     38      184     -141  
+                 4       -21     38      61      -133  
+                 5       -31     37      114     -136  
+                 6       -48     42      53      -160  
+                 7       -100    80      86      -315  
+                 8       -71     70      115     -339  
+
+                    Table 5. WWV Radio Clock Offsets
+
+   On inspection of the detailed plots of offsets versus time the data
+   reveal an interesting sawtooth variation with period about 25 cycles
+   per hour and amplitude about 90 ms.  Once the clock has locked for
+   some time the variation decreases in frequency and sometimes
+   disappears.  This behavior is precisely what would be expected of a
+   phase-locked oscillator and accounts for the rather large standard
+   deviations in Table 5.
+
+   On inspection of the plots of offsets versus time, it is apparent
+   that by far the best accuracies are obtained at or in the periods of
+   lock, which is most frequent during periods of darkness over the
+   propagation path, which occured roughly between 0800 UT and 1100 UT
+   during the experiment period.  Excluding all data except that
+   collected during this period, the mean offset is -21.3 ms with
+   standard deviation in the range 29-31.  The maximum offset is 59 ms
+   and the minimum is -118 ms.
+
+   In order to compute the discrepancy between the WWV and WWVB clocks,
+   it is necessary to subtract the total of the propagation delay plus
+   WWVB clock delay from the mean offsets computed above.  Thus, the WWV
+   clock indications are -21.3 - 10 - 32 = -72.3 ms late (72.3 ms early)
+   with respect to the WWVB clock indications.  Considering the large
+   standard deviations noted above, it is probably not worthwhile to
+   include this correction in the WWV clock indications.
+
+   On exceptional occasions excursions in offset over 300 ms relative to
+   the WWVB clock were observed.  Close inspection of the data showed
+   that this was due to an extended period (a day or more) in which lock
+
+
+Mills                                                          [Page 21]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+   was not achieved on any frequency.  The master oscillator uses a
+   3.6-MHz crystal oscillator trimmed by a digital/analog converter and
+   register which is loaded by the microprocessor.  The occasional
+   excursions in offset were apparently due to incorrect register values
+   as the result of noisy reception conditions and excessive intervals
+   between lock.  On occasion the oscillator frequency was observed in
+   error over 4 ppm due to this cause, which could result in a
+   cumulative error of almost 400 ms per day if uncorrected.
+
+4.3.4.  On Handling Disruptions
+
+   The experiment period was intentionally selected to coincide with the
+   insertion of a leap second in the worldwide time broadcasts.  The
+   intent was to examine the resulting behavior of the various radio
+   clocks and the synchronization algorithm when an additional second
+   was introduced at 2400 UT on 30 June.
+
+   As it turned out, radio reception conditions at the time of insertion
+   were quite poor on all WWV frequencies, the WWVB frequency and the
+   GOES frequency.  Thus, all three clocks took varying periods up to
+   several hours to resynchonize and correct the indicated time.  In
+   fact, the only time signals heard around the time of interest were
+   those from Canadian radio CHU, but the time code of the Canadian
+   broadcasts is incompatible with the of the US broadcasts.
+
+   As mentioned above, the WWVB clock was used as the master during the
+   experiment period.  About two hours after insertion of the leap
+   second the clock resynchronized and all hosts in the experimental
+   network were corrected shortly afterwards.  Since the magnitude of
+   the correction exceeded 128 ms, the correction was of a step nature,
+   but was not performed simultaneously in all hosts due to the
+   individual timing of the Hello messages.  Thus, if timing-critical
+   network operations happened to take place during the correction
+   process, inconsistent timestamps could result.
+
+   The lesson drawn from this experience is quite clear.  Accurate time
+   synchronization requires by its very nature long integration times,
+   so that epochal events which disrupt the process must be predicted in
+   advance and applied in all hosts independently.  In principle, this
+   would not be hard to do and could even be integrated into the
+   operation of the step-correction procedure described earlier, perhaps
+   in the form of bits included in Hello messages which trigger a
+   one-second correction at the next rollover from 2400 to 0000 hours.
+
+   In order for such an out-of-band correction to be effective, advance
+   notice of the leap second must be available.  At present, this
+   information is not available in the broadcast format and must be
+
+
+Mills                                                          [Page 22]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+   obtained via the news media.  In fact, there are spare bits in the
+   broadcast format that could be adapted for this purpose, but this
+   would require reprogramming both the transmitting and receiving
+   equipment. Nevertheless, this feature should be considered for future
+   systems.
+
+4.4.  Additional Experiments
+
+   A set of experiments was performed using two WIDEBAND/EISN gateways
+   equipped with WWVB radio clocks and connected to the ARPANET.  These
+   experiments were designed to determine the limits of accuracy when
+   comparing these clocks via ARPANET paths.  One of the gateways
+   (ISI-MCON-GW) is located at the Information Sciences Institute near
+   Los Angeles, while the other (LL-GW) is located at Lincoln
+   Laboratories near Boston.  Both gateways consist of PDP11/44
+   computers running the EPOS operating system and clock-interface
+   boards with oscillators phase-locked to the WWVB clock.
+
+   The clock indications of the WIDEBAND/EISN gateways were compared
+   with the DCNet WWVB reference clock using ICMP Timestamp messages
+   [6], which record the individual timestamps with a precision of a
+   millisecond.  This technique is not as accurate as the one described
+   in Section 3, since the protocol implementation involves the
+   user-process level, which can be subject to minor delays due to
+   process scheduling and interprocess-message queueing.  However,
+   calibration measurements made over several of the links shown in
+   Figure 2 indicate that the measurement errors are dominated by the
+   individual link variations and not by the characteristics of the
+   measurement technique itself.
+
+   Measurements were made separately with each gateway by sending an
+   ICMP Timestamp Request message from the ARPANET address of DCN1 to
+   the ARPANET address of the gateway and computing the round-trip delay
+   and clock offset from the ICMP Timestamp Reply message.  This process
+   was continued for 1000 message exchanges, which took about seven
+   minutes. Table 6 shows the statistics obtained with ISI-MCON-GW and
+   Table 7 those with LL-GW (all numbers are milliseconds).
+
+
+
+
+
+
+
+
+
+
+
+
+Mills                                                          [Page 23]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+            ISI-MCON-GW     Mean    Dev     Max     Min    
+             --------------------------------------------  
+             Offset          -16     40      126     -908  
+             Delay           347     59      902     264   
+
+                 Table 6. ISI-MCON-GW Clock Statistics
+
+             LL-GW (a)       Mean    Dev     Max     Min   
+             --------------------------------------------  
+             Offset          -23     15      32      -143  
+             Delay           310     25      536     252   
+
+                    Table 7. LL-GW Clock Statistics
+
+   The smaller values of standard deviation and extreme for LL-GW are
+   probably due to the shorter ARPANET path involved.  The confidence in
+   the mean offset can be estimated by dividing the standard deviation
+   by the square root of the number of samples (1000), which suggests
+   that the mean offsets are accurate to within a couple of miliseconds.
+   The mean offsets of the WIDEBAND/EISN clocks as a group relative to
+   the DCN1 clock may thus indicate a minor discrepancy in the setting
+   of the delay-compensation switches.
+
+   It is well known that ARPANET paths exhibit wide variations in
+   delays, with occasional delays reaching surprising values up to many
+   seconds.  In order to improve the estimates a few samples were
+   removed from both the offset and delay data, including all those with
+   magnitude greater than one second.
+
+   The above experiments involve a burst of activity over a relatively
+   short time during which the ratio of the measurement traffic to other
+   network traffic may be nontrivial.  Another experiment with LL-GW was
+   designed with intervals of ten seconds between ICMP messages and
+   operated over a period of about three hours.  The results are shown
+   in Table 8.
+
+             LL-GW (b)       Mean    Dev     Max     Min  
+             -------------------------------------------- 
+             Offset          -16     93      990     -874 
+             Delay           371     108     977     240  
+
+                    Table 8. LL-GW Clock Statistics
+
+
+
+
+
+
+
+Mills                                                          [Page 24]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+   Note that the standard deviations and extrema are higher than in the
+   previous experiments, but the mean offset is about the same.
+
+   The results of these experiments suggest that time synchronization
+   via ARPANET paths can yield accuracies to the order of a few
+   milliseconds, but only if relatively large numbers of samples are
+   available.  The number of samples can be reduced and the accuracy
+   improved by using the techniques of Section 3 modified for ICMP
+   Timestamp messages and the longer, more noisy paths involved.
+
+5.  Summary and Conclusions
+
+   The experiments described above were designed to verify the correct
+   operation of the DCnet time-synchronization algorithms and protocols
+   under a variety of scenarios, including the use of line-frequency
+   clocks, three types of radio clocks and various types of
+   interprocessor links.  They involved the collection and processing of
+   many megabytes of data collected over a ten-day period that included
+   the insertion of a leap second in the standard NBS time scale.  Among
+   the lessons learned were the following:
+
+      1.  The algorithms and protocols operate as designed, yielding
+          accuracies throughout the experimental net in the order of a
+          few milliseconds to a few tens of milliseconds, depending on
+          the topology and link type.
+
+      2.  Glitches due to congestion, rebooted hosts and link failures
+          are acceptably low, even in the face of massive congestion
+          resulting from inappropriate host implementations elsewhere in
+          the Internet.
+
+      3.  A synchronization scenario where the clocks in all hosts are
+          locked to the line frequency and corrections are broadcast
+          from a central time standard will work only if all hosts are
+          on the same power grid, which is unlikely in the present
+          Internet configuration, but may be appropriate for some
+          applications.
+
+      4.  In spite of the eastern power grid wandering over as much as
+          six seconds in a day, it is possible to achieve accuracies in
+          the 30-ms range using line-frequency interface clocks and
+          corrections broadcast on the local net.
+
+      5.  Radio clocks can vary widely in accuracy depending on signal
+          reception conditions.  Absolute time can be determined to
+          within a couple of milliseconds using WWVB and GOES radio
+          clocks, but only if they are calibrated using an independent
+
+
+Mills                                                          [Page 25]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+          standard such as a portable clock.  The inexpensive WWV clocks
+          perform surprisingly well most of the time, but can be in
+          error up to a significant fraction of a second under some
+          conditions.
+
+      6.  Adjustments in the time scale due to leap seconds must be
+          anticipated before they occur.  The synchronization protocol
+          must include a mechanism to broadcast an adjustment in advance
+          of its occurance, so that it can be incorporated in each host
+          simultaneously.  There is a need to incorporate advance notice
+          of leap seconds in the broadcast time code.
+
+      7.  Time synchronization via ARPANET paths can yield accuracies in
+          the order of a few milliseconds, but only if relatively large
+          numbers of samples are available.  Further work is needed to
+          develop efficient protocols capable of similar accuracies but
+          using smaller numbers of samples.
+
+6.  References
+
+   1.  Lindsay, W.C., and A.V.  Kantak.  Network Synchronization of
+       Random Signals.  IEEE Trans.  Comm.  COM-28, 8 (August 1980),
+       1260-1266.
+
+   2.  Mills, D.L.  Time Synchronization in DCNET Hosts.  DARPA Internet
+       Project Report IEN-173, COMSAT Laboratories, February 1981.
+
+   3.  Mills, D.L.  DCNET Internet Clock Service.  DARPA Network Working
+       Group Report RFC-778, COMSAT Laboratories, April 1981.
+
+   4.  Mills, D.L.  Internet Delay Experiments.  DARPA Network Working
+       Group Report RFC-889, M/A-COM Linkabit, December 1983.
+
+   5.  Mills, D.L.  DCN Local-Network Protocols.  DARPA Network Working
+       Group Report RFC-891, M/A-COM Linkabit, December 1983.
+
+   6.  Postel, J.  Internet Control Message Protocol.  DARPA Network
+       Working Group Report RFC-792, USC Information Sciences Institute,
+       September 1981.
+
+   7.  Postel, J.  Time Protocol.  DARPA Network Working Group Report
+       RFC-868, USC Information Sciences Institute, May 1983.
+
+   8.  Postel, J.  Daytime Protocol.  DARPA Network Working Group Report
+       RFC-867, USC Information Sciences Institute, May 1983.
+
+
+
+
+Mills                                                          [Page 26]
+
+
+
+RFC 957                                                   September 1985
+Experiments in Network Clock Synchronization
+
+
+   9.  Su, Z.  A Specification of the Internet Protocol (IP) Timestamp
+       Option.  DARPA Network Working Group Report RFC-781.  SRI
+       International, May 1981.
+
+   10. Marzullo, K., and S.  Owicki.  Maintaining the Time in a
+       Distributed System.  ACM Operating Systems Review 19, 3 (July
+       1985), 44-54.
+
+   11. Mills, D.L.  Algorithms for Synchronizing Network Clocks.  DARPA
+       Network Working Group Report RFC-956, M/A-COM Linkabit, September
+       1985.
+
+   12. Mills, D.L.  Network Time Protocol (NTP).  DARPA Network Working
+       Group Report RFC-958, M/A-COM Linkabit, September 1985.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Mills                                                          [Page 27]
+