path: root/doc/rfc/rfc3746.txt

Network Working Group                                            L. Yang
Request for Comments: 3746                                   Intel Corp.
Category: Informational                                         R. Dantu
                                                    Univ. of North Texas
                                                             T. Anderson
                                                             Intel Corp.
                                                                R. Gopal
                                                                   Nokia
                                                              April 2004


     Forwarding and Control Element Separation (ForCES) Framework

Status of this Memo

   This memo provides information for the Internet community.  It does
   not specify an Internet standard of any kind.  Distribution of this
   memo is unlimited.

Copyright Notice

   Copyright (C) The Internet Society (2004).  All Rights Reserved.

Abstract

   This document defines the architectural framework for the ForCES
   (Forwarding and Control Element Separation) network elements, and
   identifies the associated entities and their interactions.

Table of Contents

   1.  Definitions. . . . . . . . . . . . . . . . . . . . . . . . . .  2
       1.1. Conventions used in this document . . . . . . . . . . . .  2
       1.2. Terminologies . . . . . . . . . . . . . . . . . . . . . .  3
   2.  Introduction to Forwarding and Control Element Separation
       (ForCES) . . . . . . . . . . . . . . . . . . . . . . . . . . .  5
   3.  Architecture . . . . . . . . . . . . . . . . . . . . . . . . .  8
       3.1. Control Elements and Fr Reference Point . . . . . . . . . 10
       3.2. Forwarding Elements and Fi reference point. . . . . . . . 11
       3.3. CE Managers . . . . . . . . . . . . . . . . . . . . . . . 14
       3.4. FE Managers . . . . . . . . . . . . . . . . . . . . . . . 14
   4.  Operational Phases . . . . . . . . . . . . . . . . . . . . . . 15
       4.1. Pre-association Phase . . . . . . . . . . . . . . . . . . 15
            4.1.1. Fl Reference Point . . . . . . . . . . . . . . . . 15
            4.1.2. Ff Reference Point . . . . . . . . . . . . . . . . 16
            4.1.3. Fc Reference Point . . . . . . . . . . . . . . . . 17
       4.2. Post-association Phase and Fp reference point . . . . . . 17
            4.2.1. Proximity and Interconnect between CEs and FEs . . 18



Yang, et al.                 Informational                      [Page 1]
^L
RFC 3746                    ForCES Framework                  April 2004


            4.2.2. Association Establishment. . . . . . . . . . . . . 18
            4.2.3. Steady-state Communication . . . . . . . . . . . . 19
            4.2.4. Data Packets across Fp reference point . . . . . . 21
            4.2.5. Proxy FE . . . . . . . . . . . . . . . . . . . . . 22
       4.3. Association Re-establishment. . . . . . . . . . . . . . . 22
            4.3.1. CE graceful restart. . . . . . . . . . . . . . . . 23
            4.3.2. FE restart . . . . . . . . . . . . . . . . . . . . 24
   5.  Applicability to RFC 1812. . . . . . . . . . . . . . . . . . . 25
       5.1. General Router Requirements . . . . . . . . . . . . . . . 25
       5.2. Link Layer. . . . . . . . . . . . . . . . . . . . . . . . 26
       5.3. Internet Layer Protocols. . . . . . . . . . . . . . . . . 27
       5.4. Internet Layer Forwarding . . . . . . . . . . . . . . . . 27
       5.5. Transport Layer . . . . . . . . . . . . . . . . . . . . . 28
       5.6. Application Layer -- Routing Protocols. . . . . . . . . . 29
       5.7. Application Layer -- Network Management Protocol. . . . . 29
   6.  Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
   7.  Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 30
   8.  Security Considerations. . . . . . . . . . . . . . . . . . . . 30
       8.1. Analysis of Potential Threats Introduced by ForCES. . . . 31
            8.1.1. "Join" or "Remove" Message Flooding on CEs . . . . 31
            8.1.2. Impersonation Attack . . . . . . . . . . . . . . . 31
            8.1.3. Replay Attack. . . . . . . . . . . . . . . . . . . 31
            8.1.4. Attack during Fail Over. . . . . . . . . . . . . . 32
            8.1.5. Data Integrity . . . . . . . . . . . . . . . . . . 32
            8.1.6. Data Confidentiality . . . . . . . . . . . . . . . 32
            8.1.7. Sharing security parameters. . . . . . . . . . . . 33
            8.1.8. Denial of Service Attack via External Interface. . 33
       8.2. Security Recommendations for ForCES . . . . . . . . . . . 33
            8.2.1. Using TLS with ForCES. . . . . . . . . . . . . . . 34
            8.2.2. Using IPsec with ForCES. . . . . . . . . . . . . . 35
   9.  References . . . . . . . . . . . . . . . . . . . . . . . . . . 37
       9.1. Normative References. . . . . . . . . . . . . . . . . . . 37
       9.2. Informative References. . . . . . . . . . . . . . . . . . 37
   10. Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 39
   11. Full Copyright Statement . . . . . . . . . . . . . . . . . . . 40

1.  Definitions

1.1.  Conventions used in this document

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in BCP 14, RFC 2119 [1].








Yang, et al.                 Informational                      [Page 2]
^L
RFC 3746                    ForCES Framework                  April 2004


1.2.  Terminologies

   A set of terminology associated with the ForCES requirements is
   defined in [4] and we only include the definitions that are most
   relevant to this document here.

   Addressable Entity (AE) - An entity that is directly addressable
   given some interconnect technology.  For example, on IP networks, it
   is a device to which we can communicate using an IP address; on a
   switch fabric, it is a device to which we can communicate using a
   switch fabric port number.

   Physical Forwarding Element (PFE) - An AE that includes hardware used
   to provide per-packet processing and handling.  This hardware may
   consist of (but is not limited to) network processors, ASICs
   (Application-Specific Integrated Circuits), or general purpose
   processors, installed on line cards, daughter boards, mezzanine
   cards, or in stand-alone boxes.

   PFE Partition - A logical partition of a PFE consisting of some
   subset of each of the resources (e.g., ports, memory, forwarding
   table entries) available on the PFE.  This concept is analogous to
   that of the resources assigned to a virtual switching element as
   described in [9].

   Physical Control Element (PCE) - An AE that includes hardware used to
   provide control functionality.  This hardware typically includes a
   general purpose processor.

   PCE Partition - A logical partition of a PCE consisting of some
   subset of each of the resources available on the PCE.

   Forwarding Element (FE) - A logical entity that implements the ForCES
   Protocol.  FEs use the underlying hardware to provide per-packet
   processing and handling as directed by a CE via the ForCES Protocol.
   FEs may happen to be a single blade (or PFE), a partition of a PFE,
   or multiple PFEs.

   Control Element (CE) - A logical entity that implements the ForCES
   Protocol and uses it to instruct one or more FEs on how to process
   packets.  CEs handle functionality such as the execution of control
   and signaling protocols.  CEs may consist of PCE partitions or whole
   PCEs.

   ForCES Network Element (NE) - An entity composed of one or more CEs
   and one or more FEs.  An NE usually hides its internal organization
   from external entities and represents a single point of management to
   entities outside the NE.



Yang, et al.                 Informational                      [Page 3]
^L
RFC 3746                    ForCES Framework                  April 2004


   Pre-association Phase - The period of time during which an FE Manager
   (see below) and a CE Manager (see below) are determining whether an
   FE and a CE should be part of the same network element.  It is
   possible for some elements of the NE to be in pre-association phase
   while other elements are in the post-association phase.

   Post-association Phase - The period of time during which an FE knows
   which CE is to control it and vice versa, including the time during
   which the CE and FE are establishing communication with one another.

   ForCES Protocol - While there may be multiple protocols used within
   the overall ForCES architecture, the term "ForCES Protocol" refers
   only to the ForCES post-association phase protocol (see below).

   ForCES Post-Association Phase Protocol - The protocol used for post-
   association phase communication between CEs and FEs.  This protocol
   does not apply to CE-to-CE communication, FE-to-FE communication, or
   to communication between FE and CE managers.  The ForCES Protocol is
   a master-slave protocol in which FEs are slaves and CEs are masters.
   This protocol includes both the management of the communication
   channel (e.g., connection establishment, heartbeats) and the control
   messages themselves.  This protocol could be a single protocol or
   could consist of multiple protocols working together, and may be
   unicast or multicast based.  A separate protocol document will
   specify this information.

   FE Manager - A logical entity that operates in the pre-association
   phase and is responsible for determining to which CE(s) an FE should
   communicate.  This process is called CE discovery and may involve the
   FE manager learning the capabilities of available CEs.  An FE manager
   may use anything from a static configuration to a pre-association
   phase protocol (see below) to determine which CE(s) to use; however,
   this is currently out of scope.  Being a logical entity, an FE
   manager might be physically combined with any of the other logical
   entities mentioned in this section.

   CE Manager - A logical entity that operates in the pre-association
   phase and is responsible for determining to which FE(s) a CE should
   communicate.  This process is called FE discovery and may involve the
   CE manager learning the capabilities of available FEs.  A CE manager
   may use anything from a static configuration to a pre-association
   phase protocol (see below) to determine which FE to use; however,
   this is currently out of scope.  Being a logical entity, a CE manager
   might be physically combined with any of the other logical entities
   mentioned in this section.






Yang, et al.                 Informational                      [Page 4]
^L
RFC 3746                    ForCES Framework                  April 2004


   Pre-association Phase Protocol - A protocol between FE managers and
   CE managers that is used to determine which CEs or FEs to use.  A
   pre-association phase protocol may include a CE and/or FE capability
   discovery mechanism.  Note that this capability discovery process is
   wholly separate from (and does not replace) that used within the
   ForCES Protocol.  However, the two capability discovery mechanisms
   may utilize the same FE model.

   FE Model - A model that describes the logical processing functions of
   an FE.

   ForCES Protocol Element - An FE or CE.

   Intra-FE topology - Representation of how a single FE is realized by
   combining possibly multiple logical functional blocks along multiple
   data paths.  This is defined by the FE model.

   FE Topology - Representation of how the multiple FEs in a single NE
   are interconnected.  Sometimes it is called inter-FE topology, to be
   distinguished from intra-FE topology used by the FE model.

   Inter-FE topology - See FE Topology.

2.  Introduction to Forwarding and Control Element Separation (ForCES)

   An IP network element (NE) appears to external entities as a
   monolithic piece of network equipment, e.g., a router, NAT, firewall,
   or load balancer.  Internally, however, an IP network element (NE)
   (such as a router) is composed of numerous logically separated
   entities that cooperate to provide a given functionality (such as
   routing).  Two types of network element components exist: control
   element (CE) in control plane and forwarding element (FE) in
   forwarding plane (or data plane).  Forwarding elements are typically
   ASIC, network-processor, or general-purpose processor-based devices
   that handle data path operations for each packet.  Control elements
   are typically based on general-purpose processors that provide
   control functionality, like routing and signaling protocols.

   ForCES aims to define a framework and associated protocol(s) to
   standardize information exchange between the control and forwarding
   plane.  Having standard mechanisms allows CEs and FEs to become
   physically separated standard components.  This physical separation
   accrues several benefits to the ForCES architecture.  Separate
   components would allow component vendors to specialize in one
   component without having to become experts in all components.
   Standard protocol also allows the CEs and FEs from different
   component vendors to interoperate with each other and hence it
   becomes possible for system vendors to integrate together the CEs and



Yang, et al.                 Informational                      [Page 5]
^L
RFC 3746                    ForCES Framework                  April 2004


   FEs from different component suppliers.  This interoperability
   translates into increased design choices and flexibility for the
   system vendors.  Overall, ForCES will enable rapid innovation in both
   the control and forwarding planes while maintaining interoperability.
   Scalability is also easily provided by this architecture in that
   additional forwarding or control capacity can be added to existing
   network elements without the need for forklift upgrades.

      -------------------------       -------------------------
      |  Control Blade A      |       |  Control Blade B      |
      |       (CE)            |       |          (CE)         |
      -------------------------       -------------------------
              ^   |                           ^    |
              |   |                           |    |
              |   V                           |    V
      ---------------------------------------------------------
      |               Switch Fabric Backplane                 |
      ---------------------------------------------------------
             ^  |            ^  |                   ^  |
             |  |            |  |     . . .         |  |
             |  V            |  V                   |  V
         ------------    ------------           ------------
         |Router    |    |Router    |           |Router    |
         |Blade #1  |    |Blade #2  |           |Blade #N  |
         |   (FE)   |    |   (FE)   |           |   (FE)   |
         ------------    ------------           ------------
             ^  |            ^  |                   ^  |
             |  |            |  |     . . .         |  |
             |  V            |  V                   |  V

      Figure 1. A router configuration example with separate blades.

   One example of such physical separation is at the blade level. Figure
   1 shows such an example configuration of a router, with two control
   blades and multiple forwarding blades, all interconnected into a
   switch fabric backplane.  In such a chassis configuration, the
   control blades are the CEs while the router blades are the FEs, and
   the switch fabric backplane provides the physical interconnect for
   all the blades.  Control blade A may be the primary CE while control
   blade B may be the backup CE providing redundancy.  It is also
   possible to have a redundant switch fabric for high availability
   support.  Routers today with this kind of configuration use
   proprietary interfaces for messaging between CEs and FEs.  The goal
   of ForCES is to replace such proprietary interfaces with a standard
   protocol.  With a standard protocol like ForCES implemented on all
   blades, it becomes possible for control blades from vendor X and
   forwarding blades from vendor Y to work seamlessly together in one
   chassis.



Yang, et al.                 Informational                      [Page 6]
^L
RFC 3746                    ForCES Framework                  April 2004


          -------         -------
          | CE1 |         | CE2 |
          -------         -------
             ^               ^
             |               |
             V               V
      ============================================ Ethernet
          ^       ^       . . .   ^
          |       |               |
          V       V               V
       -------  -------         --------
       | FE#1|  | FE#2|         | FE#n |
       -------  -------         --------
         ^  |     ^  |            ^  |
         |  |     |  |            |  |
         |  V     |  V            |  V

      Figure 2. A router configuration example with separate boxes.

   Another level of physical separation between the CEs and FEs can be
   at the box level.  In such a configuration, all the CEs and FEs are
   physically separated boxes, interconnected with some kind of high
   speed LAN connection (like Gigabit Ethernet).  These separated CEs
   and FEs are only one hop away from each other within a local area
   network.  The CEs and FEs communicate to each other by running
   ForCES, and the collection of these CEs and FEs together become one
   routing unit to the external world.  Figure 2 shows such an example.

   In both examples shown here, the same physical interconnect is used
   for both CE-to-FE and FE-to-FE communication.  However, that does not
   have to be the case.  One reason to use different interconnects is
   that the CE-to-FE interconnect does not have to be as fast as the
   FE-to-FE interconnect, so the more faster and more expensive
   connections can be saved for FE-to-FE.  The separate interconnects
   may also provide reliability and redundancy benefits for the NE.

   Some examples of control functions that can be implemented in the CE
   include routing protocols like RIP, OSPF, and BGP, control and
   signaling protocols like RSVP (Resource Reservation Protocol), LDP
   (Label Distribution Protocol) for MPLS, etc.  Examples of forwarding
   functions in the FE include LPM (longest prefix match) forwarder,
   classifiers, traffic shaper, meter, NAT (Network Address
   Translators), etc.  Figure 3 provides example functions in both CE
   and FE.  Any given NE may contain one or many of these CE and FE
   functions in it.  The diagram also shows that the ForCES Protocol is
   used to transport both the control messages for ForCES itself and the





Yang, et al.                 Informational                      [Page 7]
^L
RFC 3746                    ForCES Framework                  April 2004


   data packets that are originated/destined from/to the control
   functions in the CE (e.g., routing packets).  Section 4.2.4 provides
   more detail on this.

      -------------------------------------------------
      |       |       |       |       |       |       |
      |OSPF   |RIP    |BGP    |RSVP   |LDP    |. . .  |
      |       |       |       |       |       |       |
      -------------------------------------------------
      |               ForCES Interface                |
      -------------------------------------------------
                              ^   ^
                      ForCES  |   |data
                      control |   |packets
                      messages|   |(e.g., routing packets)
                              v   v
      -------------------------------------------------
      |               ForCES Interface                |
      -------------------------------------------------
      |       |       |       |       |       |       |
      |LPM Fwd|Meter  |Shaper |NAT    |Classi-|. . .  |
      |       |       |       |       |fier   |       |
      -------------------------------------------------
      |               FE resources                    |
      -------------------------------------------------

           Figure 3. Examples of CE and FE functions.

   A set of requirements for control and forwarding separation is
   identified in [4].  This document describes a ForCES architecture
   that satisfies the architectural requirements of [4] and defines a
   framework for ForCES network elements and the associated entities to
   facilitate protocol definition.  Whenever necessary, this document
   uses many examples to illustrate the issues and/or possible solutions
   in ForCES.  These examples are intended to be just examples, and
   should not be taken as the only or definite ways of doing certain
   things.  It is expected that a separate document will be produced by
   the ForCES working group to specify the ForCES Protocol.

3.  Architecture

   This section defines the ForCES architectural framework and the
   associated logical components.  This ForCES framework defines
   components of ForCES NEs, including several ancillary components.
   These components may be connected in different kinds of topologies
   for flexible packet processing.





Yang, et al.                 Informational                      [Page 8]
^L
RFC 3746                    ForCES Framework                  April 2004


                          ---------------------------------------
                          | ForCES Network Element              |
   --------------   Fc    | --------------      --------------  |
   | CE Manager |---------+-|     CE 1   |------|    CE 2    |  |
   --------------         | |            |  Fr  |            |  |
         |                | --------------      --------------  |
         | Fl             |         |  |    Fp       /          |
         |                |       Fp|  |----------| /           |
         |                |         |             |/            |
         |                |         |             |             |
         |                |         |     Fp     /|----|        |
         |                |         |  /--------/      |        |
   --------------     Ff  | --------------      --------------  |
   | FE Manager |---------+-|     FE 1   |  Fi  |     FE 2   |  |
   --------------         | |            |------|            |  |
                          | --------------      --------------  |
                          |   |  |  |  |          |  |  |  |    |
                          ----+--+--+--+----------+--+--+--+-----
                              |  |  |  |          |  |  |  |
                              |  |  |  |          |  |  |  |
                                Fi/f                   Fi/f

       Fp: CE-FE interface
       Fi: FE-FE interface
       Fr: CE-CE interface
       Fc: Interface between the CE Manager and a CE
       Ff: Interface between the FE Manager and an FE
       Fl: Interface between the CE Manager and the FE Manager
       Fi/f: FE external interface

            Figure 4. ForCES Architectural Diagram

   The diagram in Figure 4 shows the logical components of the ForCES
   architecture and their relationships.  There are two kinds of
   components inside a ForCES network element: control element (CE) and
   forwarding element (FE).  The framework allows multiple instances of
   CE and FE inside one NE.  Each FE contains one or more physical media
   interfaces for receiving and transmitting packets from/to the
   external world.  The aggregation of these FE interfaces becomes the
   NE's external interfaces.  In addition to the external interfaces,
   there must also exist some kind of interconnect within the NE so that
   the CE and FE can communicate with each other, and one FE can forward
   packets to another FE.  The diagram also shows two entities outside
   of the ForCES NE: CE Manager and FE Manager.  These two ancillary
   entities provide configuration to the corresponding CE or FE in the
   pre-association phase (see Section 4.1).





Yang, et al.                 Informational                      [Page 9]
^L
RFC 3746                    ForCES Framework                  April 2004


   For convenience, the logical interactions between these components
   are labeled by reference points Fp, Fc, Ff, Fr, Fl, and Fi, as shown
   in Figure 4.  The FE external interfaces are labeled as Fi/f.  More
   detail is provided in Section 3 and 4 for each of these reference
   points.  All these reference points are important in understanding
   the ForCES architecture, however, the ForCES Protocol is only defined
   over one reference point -- Fp.

   The interface between two ForCES NEs is identical to the interface
   between two conventional routers and these two NEs exchange the
   protocol packets through the external interfaces at Fi/f.  ForCES NEs
   connect to existing routers transparently.

3.1.  Control Elements and Fr Reference Point

   It is not necessary to define any protocols across the Fr reference
   point to enable control and forwarding separation for simple
   configurations like single CE and multiple FEs.  However, this
   architecture permits multiple CEs to be present in a network element.
   In cases where an implementation uses multiple CEs, the invariant
   that the CEs and FEs together appear as a single NE must be
   maintained.

   Multiple CEs may be used for redundancy, load sharing, distributed
   control, or other purposes.  Redundancy is the case where one or more
   CEs are prepared to take over should an active CE fail.  Load sharing
   is the case where two or more CEs are concurrently active and any
   request that can be serviced by one of the CEs can also be serviced
   by any of the other CEs.  For both redundancy and load sharing, the
   CEs involved are equivalently capable.  The only difference between
   these two cases is in terms of how many active CEs there are
   simultaneously.  Distributed control is the case where two or more
   CEs are concurrently active but certain requests can only be serviced
   by certain CEs.

   When multiple CEs are employed in a ForCES NE, their internal
   organization is considered an implementation issue that is beyond the
   scope of ForCES.  CEs are wholly responsible for coordinating amongst
   themselves via the Fr reference point to provide consistency and
   synchronization.  However, ForCES does not define the implementation
   or protocols used between CEs, nor does it define how to distribute
   functionality among CEs.  Nevertheless, ForCES will support
   mechanisms for CE redundancy or fail over, and it is expected that
   vendors will provide redundancy or fail over solutions within this
   framework.






Yang, et al.                 Informational                     [Page 10]
^L
RFC 3746                    ForCES Framework                  April 2004


3.2.  Forwarding Elements and Fi reference point

   An FE is a logical entity that implements the ForCES Protocol and
   uses the underlying hardware to provide per-packet processing and
   handling as directed by a CE.  It is possible to partition one
   physical FE into multiple logical FEs.  It is also possible for one
   FE to use multiple physical FEs.  The mapping between physical FE(s)
   and logical FE(s) is beyond the scope of ForCES.  For example, a
   logical partition of a physical FE can be created by assigning some
   portion of each of the resources (e.g., ports, memory, forwarding
   table entries) available on the ForCES physical FE to each of the
   logical FEs.  Such a concept of FE virtualization is analogous to a
   virtual switching element as described in [9].  If FE virtualization
   occurs only in the pre-association phase, it has no impact on ForCES.
   However, if FE virtualization results in a resource change taken from
   an existing FE (already participating in ForCES post-association
   phase), the ForCES Protocol needs to be able to inform the CE of such
   a change via asynchronous messages (see [4], Section 5, requirement
   #6).

   FEs perform all packet processing functions as directed by CEs.  FEs
   have no initiative of their own.  Instead, FEs are slaves and only do
   as they are told.  FEs may communicate with one or more CEs
   concurrently across reference point Fp.  FEs have no notion of CE
   redundancy, load sharing, or distributed control.  Instead, FEs
   accept commands from any CE authorized to control them, and it is up
   to the CEs to coordinate among themselves to achieve redundancy, load
   sharing, or distributed control.  The idea is to keep FEs as simple
   and dumb as possible so that FEs can focus their resources on the
   packet processing functions.  Unless otherwise configured or
   determined by a ForCEs Protocol exchange, each FE will process
   authorized incoming commands directed at it as it receives them on a
   first come first serve basis.

   For example, in Figure 5, FE1 and FE2 can be configured to accept
   commands from both the primary CE (CE1) and the backup CE (CE2).
   Upon detection of CE1 failure, perhaps across the Fr or Fp reference
   point, CE2 is configured to take over activities of CE1.  This is
   beyond the scope of ForCES and is not discussed further.

   Distributed control can be achieved in a similar fashion, without
   much intelligence on the part of FEs.  For example, FEs can be
   configured to detect RSVP and BGP protocol packets, and forward RSVP
   packets to one CE and BGP packets to another CE.  Hence, FEs may need
   to do packet filtering for forwarding packets to specific CEs.






Yang, et al.                 Informational                     [Page 11]
^L
RFC 3746                    ForCES Framework                  April 2004


      -------   Fr  -------
      | CE1 | ------| CE2 |
      -------       -------
        |   \      /   |
        |    \    /    |
        |     \  /     |
        |      \/Fp    |
        |      /\      |
        |     /  \     |
        |    /    \    |
      -------  Fi   -------
      | FE1 |<----->| FE2 |
      -------       -------

      Figure 5. CE redundancy example.

   This architecture permits multiple FEs to be present in an NE.  [4]
   dictates that the ForCES Protocol must be able to scale to at least
   hundreds of FEs (see [4] Section 5, requirement #11).  Each of these
   FEs may potentially have a different set of packet processing
   functions, with different media interfaces.  FEs are responsible for
   basic maintenance of layer-2 connectivity with other FEs and with
   external entities.  Many layer-2 media include sophisticated control
   protocols.  The FORCES Protocol (over the Fp reference point) will be
   able to carry messages for such protocols so that, in keeping with
   the dumb FE model, the CE can provide appropriate intelligence and
   control over these media.

   When multiple FEs are present, ForCES requires that packets must be
   able to arrive at the NE by one FE and leave the NE via a different
   FE (See [4], Section 5, Requirement #3).  Packets that enter the NE
   via one FE and leave the NE via a different FE are transferred
   between FEs across the Fi reference point.  The Fi reference point
   could be used by FEs to discover their (inter-FE) topology, perhaps
   during the pre-association phase.  The Fi reference point is a
   separate protocol from the Fp reference point and is not currently
   defined by the ForCES Protocol.

   FEs could be connected in different kinds of topologies and packet
   processing may spread across several FEs in the topology.  Hence,
   logical packet flow may be different from physical FE topology.
   Figure 6 provides some topology examples.  When it is necessary to
   forward packets between FEs, the CE needs to understand the FE
   topology.  The FE topology may be queried from the FEs by the CEs via
   the ForCES Protocol, but the FEs are not required to provide that
   information to the CEs.  So, the FE topology information may also be
   gathered by other means outside of the ForCES Protocol (like inter-FE
   topology discovery protocol).



Yang, et al.                 Informational                     [Page 12]
^L
RFC 3746                    ForCES Framework                  April 2004


            -----------------
            |      CE       |
            -----------------
             ^      ^      ^
            /       |       \
           /        v        \
          /      -------      \
         /    +->| FE3 |<-+    \
        /     |  |     |  |     \
       v      |  -------  |      v
     -------  |           |  -------
     | FE1 |<-+           +->| FE2 |
     |     |<--------------->|     |
     -------                 -------
        ^  |                   ^  |
        |  |                   |  |
        |  v                   |  v

    (a) Full mesh among FE1, FE2, and FE3


                -----------
                |   CE    |
                -----------
               ^ ^       ^ ^
              /  |       |  \
       /------   |       |   ------\
       v         v       v          v
   -------   -------   -------   -------
   | FE1 |<->| FE2 |<->| FE3 |<->| FE4 |
   -------   -------   -------   -------
     ^  |     ^  |       ^  |     ^  |
     |  |     |  |       |  |     |  |
     |  v     |  v       |  v     |  v

   (b) Multiple FEs in a daisy chain















Yang, et al.                 Informational                     [Page 13]
^L
RFC 3746                    ForCES Framework                  April 2004


                   ^ |
                   | v
                -----------
                |   FE1   |<-----------------------|
                -----------                        |
                  ^    ^                           |
                 /      \                          |
          | ^   /        \   ^ |                   V
          v |  v          v  | v                ----------
        ---------        ---------              |        |
        | FE2   |        |  FE3  |<------------>|   CE   |
        ---------        ---------              |        |
            ^  ^          ^                     ----------
            |   \        /                        ^  ^
            |    \      /                         |  |
            |    v     v                          |  |
            |   -----------                       |  |
            |   |   FE4   |<----------------------|  |
            |   -----------                          |
            |      |  ^                              |
            |      v  |                              |
            |                                        |
            |----------------------------------------|

        (c) Multiple FEs connected by a ring

        Figure 6. Some examples of FE topology

3.3.  CE Managers

   CE managers are responsible for determining which FEs a CE should
   control.  It is legitimate for CE managers to be hard-coded with the
   knowledge of with which FEs its CEs should communicate with.  A CE
   manager may also be physically embedded into a CE and be implemented
   as a simple keypad or other direct configuration mechanism on the CE.
   Finally, CE managers may be physically and logically separate
   entities that configure the CE with FE information via such
   mechanisms as COPS-PR [7] or SNMP [5].

3.4.  FE Managers

   FE managers are responsible for determining with which CE any
   particular FE should initially communicate.  Like CE managers, no
   restrictions are placed on how an FE manager decides with which CE
   its FEs should communicate, nor are restrictions placed on how FE
   managers are implemented.  Each FE should have one and only one FE





Yang, et al.                 Informational                     [Page 14]
^L
RFC 3746                    ForCES Framework                  April 2004


   manager, while different FEs may have the same or different FE
   manager(s).  Each manager can choose to exist and operate
   independently of other manager.

4.  Operational Phases

   Both FEs and CEs require some configuration to be in place before
   they can start information exchange and function as a coherent
   network element.  Two operational phases are identified in this
   framework: pre-association and post-association.

4.1.  Pre-association Phase

   The Pre-association phase is the period of time during which an FE
   Manager and a CE Manager are determining whether an FE and a CE
   should be part of the same network element.  The protocols used
   during this phase may include all or some of the message exchange
   over Fl, Ff, and Fc reference points.  However, all these may be
   optional and none of this is within the scope of the ForCES Protocol.

4.1.1.  Fl Reference Point

   CE managers and FE managers may communicate across the Fl reference
   point in the pre-association phase in order to determine whether an
   individual CE and FE, or a set of CEs and FEs should be associated.
   Communication across the Fl reference point is optional in this
   architecture.  No requirements are placed on this reference point.

   CE managers and FE managers may be operated by different entities.
   The operator of the CE manager may not want to divulge, except to
   specified FE managers, any characteristics of the CEs it manages.
   Similarly, the operator of the FE manager may not want to divulge FE
   characteristics, except to authorized entities.  As such, CE managers
   and FE managers may need to authenticate one another.  Subsequent
   communication between CE managers and FE managers may require other
   security functions such as privacy, non-repudiation, freshness, and
   integrity.














Yang, et al.                 Informational                     [Page 15]
^L
RFC 3746                    ForCES Framework                  April 2004


   FE Manager      FE               CE Manager     CE
    |              |                 |             |
    |              |                 |             |
    |(security exchange)             |             |
   1|<------------------------------>|             |
    |              |                 |             |
    |(a list of CEs and their attributes)          |
   2|<-------------------------------|             |
    |              |                 |             |
    |(a list of FEs and their attributes)          |
   3|------------------------------->|             |
    |              |                 |             |
    |              |                 |             |
    |<----------------Fl------------>|             |

   Figure 7. An example of a message exchange over the Fl reference
             point

   Once the necessary security functions have been performed, the CE and
   FE managers communicate to determine which CEs and FEs should
   communicate with each other.  At the very minimum, the CE and FE
   managers need to learn of the existence of available FEs and CEs
   respectively.  This discovery process may entail one or both managers
   learning the capabilities of the discovered ForCES protocol elements.
   Figure 7 shows an example of a possible message exchange between the
   CE manager and FE manager over the Fl reference point.

4.1.2.  Ff Reference Point

   The Ff reference point is used to inform forwarding elements of the
   association decisions made by the FE manager in the pre-association
   phase.  Only authorized entities may instruct an FE with respect to
   which CE should control it.  Therefore, privacy, integrity,
   freshness, and authentication are necessary between the FE manager
   and FEs when the FE manager is remote to the FE.  Once the
   appropriate security has been established, the FE manager instructs
   the FEs across this reference point to join a new NE or to disconnect
   from an existing NE.  The FE Manager could also assign unique FE
   identifiers to the FEs using this reference point.  The FE
   identifiers are useful in the post association phase to express FE
   topology.  Figure 8 shows example of a message exchange over the Ff
   reference point.









Yang, et al.                 Informational                     [Page 16]
^L
RFC 3746                    ForCES Framework                  April 2004


   FE Manager      FE               CE Manager     CE
    |              |                |             |
    |              |                |             |
    |(security exchange)            |(security exchange)
   1|<------------>|authentication 1|<----------->|authentication
    |              |                |             |
    |(FE ID, attributes)            |(CE ID, attributes)
   2|<-------------|request        2|<------------|request
    |              |                |             |
   3|------------->|response       3|------------>|response
    |(corresponding CE ID)          |(corresponding FE ID)
    |              |                |             |
    |              |                |             |
    |<-----Ff----->|                |<-----Fc---->|

         Figure 8. Examples of a message exchange
                   over the Ff and Fc reference points

   Note that the FE manager function may be co-located with the FE (such
   as by manual keypad entry of the CE IP address), in which case this
   reference point is reduced to a built-in function.

4.1.3.  Fc Reference Point

   The Fc reference point is used to inform control elements of the
   association decisions made by CE managers in the pre-association
   phase.  When the CE manager is remote, only authorized entities may
   instruct a CE to control certain FEs.  Privacy, integrity, freshness,
   and authentication are also required across this reference point in
   such a configuration.  Once appropriate security has been
   established, the CE manager instructs the CEs as to which FEs they
   should control and how they should control them.  Figure 8 shows
   example of a message exchange over the Fc reference point.

   As with the FE manager and FEs, configurations are possible where the
   CE manager and CE are co-located and no protocol is used for this
   function.

4.2.  Post-association Phase and Fp reference point

   The Post-association phase is the period of time during which an FE
   and CE have been configured with information necessary to contact
   each other and includes both association establishment and steady-
   state communication.  The communication between CE and FE is
   performed across the Fp ("p" meaning protocol) reference point.
   ForCES Protocol is exclusively used for all communication across the
   Fp reference point.




Yang, et al.                 Informational                     [Page 17]
^L
RFC 3746                    ForCES Framework                  April 2004


4.2.1.  Proximity and Interconnect between CEs and FEs

   The ForCES Working Group has made a conscious decision that the first
   version of ForCES will be focused on "very close" CE/FE localities in
   IP networks.  Very Close localities consist of control and forwarding
   elements that are either components in the same physical box, or
   separated at most by one local network hop ([8]).  CEs and FEs can be
   connected by a variety of interconnect technologies, including
   Ethernet connections, backplanes, ATM (cell) fabrics, etc.  ForCES
   should be able to support each of these interconnects (see [4]
   Section 5, requirement #1).  When the CEs and FEs are separated
   beyond a single L3 routing hop, the ForCES Protocol will make use of
   an existing RFC 2914 [3] compliant L4 protocol with adequate
   reliability, security, and congestion control (e.g., TCP, SCTP) for
   transport purposes.

4.2.2.  Association Establishment

                FE                      CE
                |                       |
                |(Security exchange.)   |
               1|<--------------------->|
                |                       |
                |(Let me join the NE please.)
               2|---------------------->|
                |                       |
                |(What kind of FE are you? -- capability query)
               3|<----------------------|
                |                       |
                |(Here is my FE functions/state: use model to
   describe)
               4|---------------------->|
                |                       |
                |(Initial config for FE -- optional)
               5|<----------------------|
                |                       |
                |(I am ready to go. Shall I?)
               6|---------------------->|
                |                       |
                |(Go ahead!)            |
               7|<----------------------|
                |                       |

   Figure 9. Example of a message exchange between CE and FE
             over Fp to establish an NE association






Yang, et al.                 Informational                     [Page 18]
^L
RFC 3746                    ForCES Framework                  April 2004


   As an example, figure 9 shows some of the message exchange that may
   happen before the association between the CE and FE is fully
   established.  Either the CE or FE can initiate the connection.

   Security handshake is necessary to authenticate the two communication
   endpoints to each other before any further message exchange can
   happen.  The security handshake should include mutual authentication
   and authorization between the CE and FE, but the exact details depend
   on the security solution chosen by the ForCES Protocol.
   Authorization can be as simple as checking against the list of
   authorized end points provided by the FE or CE manager during the
   pre-association phase.  Both authentication and authorization must be
   successful before the association can be established.  If either
   authentication or authorization fails, the end point must not be
   allowed to join the NE.  After the successful security handshake,
   message authentication and confidentiality are still necessary for
   the on-going information exchange between the CE and FE, unless some
   form of physical security exists.  Whenever a packet fails
   authentication, it must be dropped and a notification may be sent to
   alert the sender of the potential attack.  Section 8 provides more
   details on the security considerations for ForCES.

   After the successful security handshake, the FE needs to inform the
   CE of its own capability and optionally its topology in relation to
   other FEs.  The capability of the FE shall be represented by the FE
   model, as required in [4] (Section 6, requirement #1).  The model
   would allow an FE to describe what kind of packet processing
   functions it contains, in what order the processing happens, what
   kinds of configurable parameters it allows, what statistics it
   collects, and what events it might throw, etc.  Once such information
   is available to the CE, the CE may choose to send some initial or
   default configuration to the FE so that the FE can start receiving
   and processing packets correctly.  Such initialization may not be
   necessary if the FE already obtains the information from its own
   bootstrap process.  Once the necessary initial information is
   exchanged, the process of association is completed.  Packet
   processing and forwarding at the FE cannot begin until association is
   established.  After the association is established, the CE and FE
   enter steady-state communication.

4.2.3.  Steady-state Communication

   Once an association is established between the CE and FE, the ForCES
   Protocol is used by the CE and FE over the Fp reference point to
   exchange information to facilitate packet processing.






Yang, et al.                 Informational                     [Page 19]
^L
RFC 3746                    ForCES Framework                  April 2004


           FE                      CE
           |                       |
           |(Add these new routes.)|
          1|<----------------------|
           |                       |
           |(Successful.)          |
          2|---------------------->|
           |                       |
           |                       |
           |(Query some stats.)    |
          1|<----------------------|
           |                       |
           |(Reply with stats collected.)
          2|---------------------->|
           |                       |
           |                       |
           |(My port is down, with port #.)
          1|---------------------->|
           |                       |
           |(Here is a new forwarding table)
          2|<----------------------|
           |                       |

   Figure 10. Examples of a message exchange between CE and FE
              over Fp during steady-state communication

   Based on the information acquired through CEs' control processing,
   CEs will frequently need to manipulate the packet-forwarding
   behaviors of their FE(s) by sending instructions to FEs.  For
   example, Figure 10 shows message exchange examples in which the CE
   sends new routes to the FE so that the FE can add them to its
   forwarding table.  The CE may query the FE for statistics collected
   by the FE and the FE may notify the CE of important events such as
   port failure.

















Yang, et al.                 Informational                     [Page 20]
^L
RFC 3746                    ForCES Framework                  April 2004


4.2.4.  Data Packets across Fp reference point

   ---------------------           ----------------------
   |                   |           |                    |
   |    +--------+     |           |     +--------+     |
   |    |CE(BGP) |     |           |     |CE(BGP) |     |
   |    +--------+     |           |     +--------+     |
   |        |          |           |          ^         |
   |        |Fp        |           |          |Fp       |
   |        v          |           |          |         |
   |    +--------+     |           |     +--------+     |
   |    |  FE    |     |           |     |   FE   |     |
   |    +--------+     |           |     +--------+     |
   |        |          |           |          ^         |
   | Router |          |           | Router   |         |
   | A      |          |           | B        |         |
   ---------+-----------           -----------+----------
            v                                 ^
            |                                 |
            |                                 |
            ------------------->---------------

   Figure 11. Example to show data packet flow between two NEs.

   Control plane protocol packets (such as RIP, OSPF messages) addressed
   to any of NE's interfaces are typically redirected by the receiving
   FE to its CE, and CE may originate packets and have its FE deliver
   them to other NEs.  Therefore, the ForCES Protocol over Fp not only
   transports the ForCES Protocol messages between CEs and FEs, but also
   encapsulates the data packets from control plane protocols.
   Moreover, one FE may be controlled by multiple CEs for distributed
   control.  In this configuration, the control protocols supported by
   the FORCES NEs may spread across multiple CEs.  For example, one CE
   may support routing protocols like OSPF and BGP, while a signaling
   and admission control protocol like RSVP is supported in another CE.
   FEs are configured to recognize and filter these protocol packets and
   forward them to the corresponding CE.

   Figure 11 shows one example of how the BGP packets originated by
   router A are passed to router B.  In this example, the ForCES
   Protocol is used to transport the packets from the CE to the FE
   inside router A, and then from the FE to the CE inside router B.  In
   light of the fact that the ForCES Protocol is responsible for
   transporting both the control messages and the data packets between
   the CE and FE over the Fp reference point, it is possible to use
   either a single protocol or multiple protocols.





Yang, et al.                 Informational                     [Page 21]
^L
RFC 3746                    ForCES Framework                  April 2004


4.2.5.  Proxy FE

   In the case where a physical FE cannot implement (e.g., due to the
   lack of a general purpose CPU) the ForCES Protocol directly, a proxy
   FE can be used to terminate the Fp reference point instead of the
   physical FE.  This allows the CE to communicate to the physical FE
   via the proxy by using ForCES, while the proxy manipulates the
   physical FE using some intermediary form of communication (e.g., a
   non-ForCES protocol or DMA).  In such an implementation, the
   combination of the proxy and the physical FE becomes one logical FE
   entity.  It is also possible for one proxy to act on behalf of
   multiple physical FEs.

   One needs to be aware of the security implication introduced by the
   proxy FE.  Since the physical FE is not capable of implementing
   ForCES itself, the security mechanism of ForCES can only secure the
   communication channel between the CE and the proxy FE, but not all
   the way to the physical FE.  It is recommended that other security
   mechanisms (including physical security property) be employed to
   ensure the security between the CE and the physical FE.

4.3.  Association Re-establishment

   FEs and CEs may join and leave NEs dynamically (see [4] Section 5,
   requirements #12).  When an FE or CE leaves the NE, the association
   with the NE is broken.  If the leaving party rejoins an NE later, to
   re-establish the association, it may need to re-enter the pre-
   association phase.  Loss of association can also happen unexpectedly
   due to a loss of connection between the CE and the FE.  Therefore,
   the framework allows the bi-directional transition between these two
   phases, but the ForCES Protocol is only applicable for the post-
   association phase.  However, the protocol should provide mechanisms
   to support association re-establishment.  This includes the ability
   for CEs and FEs to determine when there is a loss of association
   between them, and to restore association and efficient state
   (re)synchronization mechanisms (see [4] Section 5, requirement #7).
   Note that security association and state must also be re-established
   to guarantee the same level of security (including both
   authentication and authorization) exists before and after the
   association re-establishment.

   When an FE leaves or joins an existing NE that is already in
   operation, the CE needs to be aware of the impact on FE topology and
   deal with the change accordingly.







Yang, et al.                 Informational                     [Page 22]
^L
RFC 3746                    ForCES Framework                  April 2004


4.3.1. CE graceful restart

   The failure and restart of the CE in a router can potentially cause
   much stress and disruption on the control plane throughout a network
   because in restarting a CE for any reason, the router loses routing
   adjacencies or sessions with its routing neighbors.  Neighbors who
   detect the lost adjacency normally re-compute new routes and then
   send routing updates to their own neighbors to communicate the lost
   adjacency.  Their neighbors do the same thing to propagate throughout
   the network.  In the meantime, the restarting router cannot receive
   traffic from other routers because the neighbors have stopped using
   the router's previously advertised routes.  When the restarting
   router restores adjacencies, neighbors must once again re-compute new
   routes and send out additional routing updates.  The restarting
   router is unable to forward packets until it has re-established
   routing adjacencies with neighbors, received route updates through
   these adjacencies, and computed new routes.  Until convergence takes
   place throughout the network, packets may be lost in transient black
   holes or forwarding loops.

   A high availability mechanism known as the "graceful restart" has
   been used by the IP routing protocols (OSPF [11], BGP [12], IS-IS
   [13]) and MPLS label distribution protocol (LDP [10]) to help
   minimize the negative effects on routing throughout an entire network
   caused by a restarting router.  Route flap on neighboring routers is
   avoided, and a restarting router can continue to forward packets that
   would otherwise be dropped.

   While the details differ from protocol to protocol, the general idea
   behind the graceful restart mechanism remains the same.  With the
   graceful restart, a restarting router can inform its neighbors when
   it restarts.  The neighbors may detect the lost adjacency but do not
   recompute new routes or send routing updates to their neighbors.  The
   neighbors also hold on to the routes received from the restarting
   router before restart and assume they are still valid for a limited
   time.  By doing so, the restarting router's FEs can also continue to
   receive and forward traffic from other neighbors for a limited time
   by using the routes they already have.  The restarting router then
   re-establishes routing adjacencies, downloads updated routes from all
   its neighbors, recomputes new routes, and uses them to replace the
   older routes it was using.  It then sends these updated routes to its
   neighbors and signals the completion of the graceful restart process.

   Non-stop forwarding is a requirement for graceful restart.  It is
   necessary so a router can continue to forward packets while it is
   downloading routing information and recomputing new routes.  This
   ensures that packets will not be dropped.  As one can see, one of the
   benefits afforded by the separation of CE and FE is exactly the



Yang, et al.                 Informational                     [Page 23]
^L
RFC 3746                    ForCES Framework                  April 2004


   ability of non-stop forwarding in the face of the CE failure and
   restart.  The support of dynamic changes to CE/FE association in
   ForCES also makes it compatible with high availability mechanisms,
   such as graceful restart.

   ForCES should be able to support a CE graceful restart easily.  When
   the association is established the first time, the CE must inform the
   FEs what to do in the case of a CE failure.  If graceful restart is
   not supported, the FEs may be told to stop packet processing all
   together if its CE fails.  If graceful restart is supported, the FEs
   should be told to cache and hold on to its FE state, including the
   forwarding tables across the restarts.  A timer must be included so
   that the timeout causes such a cached state to eventually expire.
   Those timers should be settable by the CE.

4.3.2.  FE restart

   In the same example in Figure 5, assuming CE1 is the working CE for
   the moment, what would happen if one of the FEs, say FE1, leaves the
   NE temporarily?  FE1 may voluntarily decide to leave the association.
   Alternatively, FE1 may stop functioning simply due to unexpected
   failure.  In the former case, CE1 receives a "leave-association
   request" from FE1.  In the latter, CE1 detects the failure of FE1 by
   some other means.  In both cases, CE1 must inform the routing
   protocols of such an event, most likely prompting a reachability and
   SPF (Shortest Path First) recalculation and associated downloading of
   new FIBs from CE1 to the other remaining FEs (only FE2 in this
   example).  Such recalculation and FIB updates will also be propagated
   from CE1 to the NE's neighbors that are affected by the connectivity
   of FE1.

   When FE1 decides to rejoin again, or when it restarts again after the
   failure, FE1 needs to re-discover its master (CE).  This can be
   achieved by several means.  It may re-enter the pre-association phase
   and get that information from its FE manager.  It may retrieve the
   previous CE information from its cache, if it can validate the
   information freshness.  Once it discovers its CE, it starts message
   exchange with the CE to re-establish the association, as outlined in
   Figure 9, with the possible exception that it might be able to bypass
   the transport of the complete initial configuration.  Suppose that
   FE1 still has its routing table and other state information from the
   last association.  Instead of re-sending all the information, it may
   be able to use a more efficient mechanism to re-sync the state with
   its CE, if such a mechanism is supported by the ForCES Protocol.  For
   example, CRC-32 of the state might give a quick indication of whether
   or not the state is in-sync with its CE.  By comparing its state with
   the CE first, it sends an information update




Yang, et al.                 Informational                     [Page 24]
^L
RFC 3746                    ForCES Framework                  April 2004


   only if it is needed.  The ForCES Protocol may choose to implement
   similar optimization  mechanisms, but it may also choose not to, as
   this is not a requirement.

5.  Applicability to RFC 1812

   [4] Section 5, requirement #9 dictates "Any proposed ForCES
   architecture must explain how that architecture supports all of the
   router functions as defined in RFC 1812."  RFC 1812 [2] discusses
   many important requirements for IPv4 routers from the link layer to
   the application layer.  This section addresses the relevant
   requirements in RFC 1812 for implementing IPv4 routers based on
   ForCES architecture and explains how ForCES satisfies these
   requirements by providing guidelines on how to separate the
   functionalities required into the forwarding plane and control plane.

   In general, the forwarding plane carries out the bulk of the per-
   packet processing that is required at line speed, while the control
   plane carries most of the computationally complex operations that are
   typical of the control and signaling protocols.  However, it is
   impossible to draw a rigid line to divide the processing into CEs and
   FEs cleanly and the ForCES architecture should not limit the
   innovative approaches in control and forwarding plane separation.  As
   more and more processing power is available in the FEs, some of the
   control functions that traditionally are performed by CEs may now be
   moved to FEs for better performance and scalability.  Such offloaded
   functions may include part of ICMP or TCP processing, or part of
   routing protocols.  Once off-loaded onto the forwarding plane, such
   CE functions, even though logically belonging to the control plane,
   now become part of the FE functions.  Just like the other logical
   functions performed by FEs, such off-loaded functions must be
   expressed as part of the FE model so that the CEs can decide how to
   best take advantage of these off-loaded functions when present on the
   FEs.

5.1.  General Router Requirements

   Routers have at least two or more logical interfaces.  When CEs and
   FEs are separated by ForCES within a single NE, some additional
   interfaces are needed for intra-NE communications, as illustrated in
   figure 12.  This NE contains one CE and two FEs.  Each FE has four
   interfaces; two of them are used for receiving and transmitting
   packets to the external world, while the other two are for intra-NE
   connections.  CE has two logical interfaces #9 and #10, connected to
   interfaces #3 and #6 from FE1 and FE2, respectively.  Interface #4
   and #5 are connected for FE1-FE2 communication.  Therefore, this
   router NE provides four external interfaces (#1, 2, 7, and 8).




Yang, et al.                 Informational                     [Page 25]
^L
RFC 3746                    ForCES Framework                  April 2004


      ---------------------------------
      |               router NE       |
      |   -----------   -----------   |
      |   |   FE1   |   |   FE2   |   |
      |   -----------   -----------   |
      |   1| 2| 3| 4|   5| 6| 7| 8|   |
      |    |  |  |  |    |  |  |  |   |
      |    |  |  |  +----+  |  |  |   |
      |    |  |  |          |  |  |   |
      |    |  | 9|        10|  |  |   |
      |    |  | -------------- |  |   |
      |    |  | |    CE      | |  |   |
      |    |  | -------------- |  |   |
      |    |  |                |  |   |
      -----+--+----------------+--+----
           |  |                |  |
           |  |                |  |

      Figure 12. A router NE example with four interfaces.

   IPv4 routers must implement IP to support its packet forwarding
   function, which is driven by its FIB (Forwarding Information Base).
   This Internet layer forwarding (see RFC 1812 [2] Section 5)
   functionality naturally belongs to FEs in the ForCES architecture.

   A router may implement transport layer protocols (like TCP and UDP)
   that are required to support application layer protocols (see RFC
   1812 [2] Section 6).  One important class of application protocols is
   routing protocols (see RFC 1812 [2] Section 7).  In the ForCES
   architecture, routing protocols are naturally implemented by CEs.
   Routing protocols require that routers communicate with each other.
   This communication between CEs in different routers is supported in
   ForCES by FEs' ability to redirect data packets addressed to routers
   (i.e., NEs), and the CEs' ability to originate packets and have them
   delivered by their FEs.  This communication occurs across the Fp
   reference point inside each router and between neighboring routers'
   external interfaces, as illustrated in Figure 11.

5.2.  Link Layer

   Since FEs own all the external interfaces for the router, FEs need to
   conform to the link layer requirements in RFC 1812 [2].  Arguably,
   ARP support may be implemented in either CEs or FEs.  As we will see
   later, a number of behaviors that RFC 1812 mandates fall into this
   category -- they may be performed by the FE and may be performed by
   the CE.  A general guideline is needed to ensure interoperability
   between separated control and forwarding planes.  The guideline we
   offer here is that CEs MUST be capable of these kinds of operations



Yang, et al.                 Informational                     [Page 26]
^L
RFC 3746                    ForCES Framework                  April 2004


   while FEs MAY choose to implement them.  The FE model should indicate
   its capabilities in this regard so that CEs can decide where these
   functions are implemented.

   Interface parameters, including MTU, IP address, etc., must be
   configurable by CEs via ForCES.  CEs must be able to determine
   whether a physical interface in an FE is available to send packets or
   not.  FEs must also inform CEs of the status change of the interfaces
   (like link up/down) via ForCES.

5.3.  Internet Layer Protocols

   Both FEs and CEs must implement the IP protocol and all mandatory
   extensions as RFC 1812 specified.  CEs should implement IP options
   like source route and record route while FEs may choose to implement
   those as well.  The timestamp option should be implemented by FEs to
   insert the timestamp most accurately.  The FE must interpret the IP
   options that it understands and preserve the rest unchanged for use
   by CEs.  Both FEs and CEs might choose to silently discard packets
   without sending ICMP errors, but such events should be logged and
   counted.  FEs may report statistics for such events to CEs via
   ForCES.

   When multiple FEs are involved to process packets, the appearance of
   a single NE must be strictly maintained.  For example, Time-To-Live
   (TTL) must be decremented only once within a single NE.  For example,
   it can be always decremented by the last FE with egress function.

   FEs must receive and process normally any packets with a broadcast
   destination address or a multicast destination address that the
   router has asked to receive.  When IP multicast is supported in
   routers, IGMP is implemented in CEs.  CEs are also required of ICMP
   support, while it is optional for FEs to support ICMP.  Such an
   option can be communicated to CEs as part of the FE model. Therefore,
   FEs can always rely upon CEs to send out ICMP error messages, but FEs
   also have the option of generating ICMP error messages themselves.

5.4.  Internet Layer Forwarding

   IP forwarding is implemented by FEs.  When the routing table is
   updated at the CEs, ForCES is used to send the new route entries from
   the CEs to FEs.  Each FE has its own forwarding table and uses this
   table to direct packets to the next hop interface.

   Upon receiving IP packets, the FE verifies the IP header and
   processes most of the IP options.  Some options cannot be processed
   until the routing decision has been made.  The routing decision is
   made after examining the destination IP address.  If the destination



Yang, et al.                 Informational                     [Page 27]
^L
RFC 3746                    ForCES Framework                  April 2004


   address belongs to the router itself, the packets are filtered and
   either processed locally or forwarded to the CE, depending upon the
   instructions set-up by the CE.  Otherwise, the FE determines the next
   hop IP address by looking in its forwarding table.  The FE also
   determines the network interface it uses to send the packets.
   Sometimes an FE may need to forward the packets to another FE before
   packets can be forwarded out to the next hop.  Right before packets
   are forwarded out to the next hop, the FE decrements TTL by 1 and
   processes any IP options that could not be processed before.  The FE
   performs IP fragmentation if necessary, determines the link layer
   address (e.g., by ARP), and encapsulates the IP datagram (or each of
   the fragments thereof) in an appropriate link layer frame and queues
   it for output on the interface selected.

   Other options mentioned in RFC 1812 [2] for IP forwarding may also be
   implemented at FEs, for example, packet filtering.

   FEs typically forward packets destined locally to CEs.  FEs may also
   forward exceptional packets (packets that FEs do not know how to
   handle) to CEs.  CEs are required to handle packets forwarded by FEs
   for whatever reason.  It might be necessary for ForCES to attach some
   meta-data with the packets to indicate the reasons of forwarding from
   FEs to CEs.  Upon receiving packets with meta-data from FEs, CEs can
   decide to either process the packets themselves, or pass the packets
   to the upper layer protocols including routing and management
   protocols.  If CEs are to process the packets by themselves, CEs may
   choose to discard the packets, or modify and re-send the packets.
   CEs may also originate new packets and deliver them to FEs for
   further forwarding.

   Any state change during router operation must also be handled
   correctly according to RFC 1812.  For example, when an FE ceases
   forwarding, the entire NE may continue forwarding packets, but it
   needs to stop advertising routes that are affected by the failed FE.

5.5.  Transport Layer

   The Transport layer is typically implemented at CEs to support higher
   layer application protocols like routing protocols.  In practice,
   this means that most CEs implement both the Transmission Control
   Protocol (TCP) and the User Datagram Protocol (UDP).

   Both CEs and FEs need to implement the ForCES Protocol.  If some
   layer-4 transport is used to support ForCES, then both CEs and FEs
   need to implement the L4 transport and ForCES Protocols.






Yang, et al.                 Informational                     [Page 28]
^L
RFC 3746                    ForCES Framework                  April 2004


5.6.  Application Layer -- Routing Protocols

   Interior and exterior routing protocols are implemented on CEs.  The
   routing packets originated by CEs are forwarded to FEs for delivery.
   The results of such protocols (like forwarding table updates) are
   communicated to FEs via ForCES.

   For performance or scalability reasons, portions of the control plane
   functions that need faster response may be moved from the CEs and
   off-loaded onto the FEs.  For example, in OSPF, the Hello protocol
   packets are generated and processed periodically.  When done at the
   CEs, the inbound Hello packets have to traverse from the external
   interfaces at the FEs to the CEs via the internal CE-FE channel.
   Similarly, the outbound Hello packets have to go from the CEs to the
   FEs and to the external interfaces.  Frequent Hello updates place
   heavy processing overhead on the CEs and can overwhelm the CE-FE
   channel as well.  Since typically there are far more FEs than CEs in
   a router, the off-loaded Hello packets are processed in a much more
   distributed and scalable fashion.  By expressing such off-loaded
   functions in the FE model, we can ensure interoperability.  However,
   the exact description of the off-loaded functionality corresponding
   to the off-loaded functions expressed in the FE model are not part of
   the model itself and will need to be worked out as a separate
   specification.

5.7.  Application Layer -- Network Management Protocol

   RFC 1812 [2] also dictates that "Routers MUST be manageable by SNMP".
   In general, for the post-association phase, most external management
   tasks (including SNMP) should be done through interaction with the CE
   in order to support the appearance of a single functional device.
   Therefore, it is recommended that an SNMP agent be implemented by CEs
   and that the SNMP messages received by FEs be redirected to their
   CEs. AgentX framework defined in RFC 2741 ([6]) may be applied here
   such that CEs act in the role of master agent to process SNMP
   protocol messages while FEs act in the role of subagent to provide
   access to the MIB objects residing on FEs.  AgentX protocol messages
   between the master agent (CE) and the subagent (FE) are encapsulated
   and transported via ForCES, just like data packets from any other
   application layer protocols.

6.  Summary

   This document defines an architectural framework for ForCES.  It
   identifies the relevant components for a ForCES network element,
   including (one or more) FEs, (one or more) CEs, one optional FE
   manager, and one optional CE manager.  It also identifies the
   interaction among these components and discusses all the major



Yang, et al.                 Informational                     [Page 29]
^L
RFC 3746                    ForCES Framework                  April 2004


   reference points.  It is important to point out that, among all the
   reference points, only the Fp interface between CEs and FEs is within
   the scope of ForCES.  ForCES alone may not be enough to support all
   desirable NE configurations.  However, we believe that ForCES over an
   Fp interface is the most important element in realizing physical
   separation and interoperability of CEs and FEs, and hence the first
   interface that ought to be standardized.  Simple and useful
   configurations can still be implemented with only CE-FE interface
   being standardized, e.g., single CE with full-meshed FEs.

7.  Acknowledgements

   Joel M. Halpern gave us many insightful comments and suggestions and
   pointed out several major issues.  T. Sridhar suggested that the
   AgentX protocol could be used with SNMP to manage the ForCES network
   elements.  Susan Hares pointed out the issue of graceful restart with
   ForCES.  Russ Housley, Avri Doria, Jamal Hadi Salim, and many others
   in the ForCES mailing list also provided valuable feedback.

8.  Security Considerations

   The NE administrator has the freedom to determine the exact security
   configuration that is needed for the specific deployment. For
   example, ForCES may be deployed between CEs and FEs connected to each
   other inside a box over a backplane.  In such a scenario, physical
   security of the box ensures that most of the attacks, such as man-
   in-the-middle, snooping, and impersonation, are not possible, and
   hence the ForCES architecture may rely on the physical security of
   the box to defend against these attacks and protocol mechanisms may
   be turned off.  However, it is also shown that denial of service
   attacks via external interfaces as described below in Section 8.1.8
   is still a potential threat, even for such an "all-in-one-box"
   deployment scenario and hence the rate limiting mechanism is still
   necessary.  This is just one example to show that it is important to
   assess the security needs of the ForCES-enabled network elements
   under different deployment scenarios.  It should be possible for the
   administrator to configure the level of security needed for the
   ForCES Protocol.

   In general, the physical separation of two entities usually results
   in a potentially insecure link between the two entities and hence
   much stricter security measurements are required.  For example, we
   pointed out in Section 4.1 that authentication becomes necessary
   between the CE manager and FE manager, between the CE and CE manager,
   and between the FE and FE manager in some configurations.  The
   physical separation of the CE and FE also imposes serious security
   requirements for the ForCES Protocol over the Fp interface.  This
   section first attempts to describe the security threats that may be



Yang, et al.                 Informational                     [Page 30]
^L
RFC 3746                    ForCES Framework                  April 2004


   introduced by the physical separation of the FEs and CEs, and then it
   provides recommendations and guidelines for the secure operation and
   management of the ForCES Protocol over the Fp interface based on
   existing standard security solutions.

8.1.  Analysis of Potential Threats Introduced by ForCES

   This section provides the threat analysis for ForCES, with a focus on
   the Fp interface.  Each threat is described in detail with the
   effects on the ForCES Protocol entities or/and the NE as a whole, and
   the required functionalities that need to be in place to defend the
   threat.

8.1.1.  "Join" or "Remove" Message Flooding on CEs

   Threats:  A malicious node could send a stream of false "join NE" or
   "remove from NE" requests on behalf of a non-existent or unauthorized
   FE to legitimate CEs at a very rapid rate, and thereby creating
   unnecessary state in the CEs.

   Effects: If maintaining state for non-existent or unauthorized FEs, a
   CE may become unavailable for other processing and hence suffer from
   a denial of service (DoS) attack similar to the TCP SYN DoS.  If
   multiple CEs are used, the unnecessary state information may also be
   conveyed to multiple CEs via the Fr interface (e.g., from the active
   CE to the stand-by CE) and hence subject multiple CEs to a DoS
   attack.

   Requirement: A CE that receives a "join" or "remove" request should
   not create any state information until it has authenticated the FE
   endpoint.

8.1.2.  Impersonation Attack

   Threats: A malicious node can impersonate a CE or FE and send out
   false messages.

   Effects: The whole NE could be compromised.

   Requirement: The CE or FE must authenticate the message as having
   come from an FE or CE on the list of the authorized ForCES elements
   (provided by the CE or FE Manager in the pre-association phase)
   before accepting and processing it.

8.1.3.  Replay Attack

   Threat: A malicious node could replay the entire message previously
   sent by an FE or CE entity to get around authentication.



Yang, et al.                 Informational                     [Page 31]
^L
RFC 3746                    ForCES Framework                  April 2004


   Effect: The NE could be compromised.

   Requirement: A replay protection mechanism needs to be part of the
   security solution to defend against this attack.

8.1.4.  Attack during Fail Over

   Threat: A malicious node may exploit the CE fail-over mechanism to
   take over the control of NE.  For example, suppose two CEs, say CE-A
   and CE-B, are controlling several FEs.  CE-A is active and CE-B is
   stand-by.  When CE-A fails, CE-B is taking over the active CE
   position.  The FEs already had a trusted relationship with CE-A, but
   the FEs may not have the same trusted relationship established with
   CE-B prior to the fail-over.  A malicious node can take over as CE-B
   if such a trusted relationship has not been established prior to or
   during the fail-over.

   Effect: The NE may be compromised after such insecure fail-over.

   Requirement: The level of trust between the stand-by CE and the FEs
   must be as strong as the one between the active CE and the FEs.  The
   security association between the FEs and the stand-by CE may be
   established prior to fail-over.  If not already in place, such
   security association must be re-established before the stand-by CE
   takes over.

8.1.5.  Data Integrity

   Threats: A malicious node may inject false messages to a legitimate
   CE or FE.

   Effect: An FE or CE receives the fabricated packet and performs an
   incorrect or catastrophic operation.

   Requirement: Protocol messages require integrity protection.

8.1.6.  Data Confidentiality

   Threat: When FE and CE are physically separated, a malicious node may
   eavesdrop the messages in transit.  Some of the messages are critical
   to the functioning of the whole network, while others may contain
   confidential business data.  Leaking of such information may result
   in compromise even beyond the immediate CE or FE.

   Effect: Sensitive information might be exposed between the CE and FE.

   Requirement: Data confidentiality between the FE and CE must be
   available for sensitive information.



Yang, et al.                 Informational                     [Page 32]
^L
RFC 3746                    ForCES Framework                  April 2004


8.1.7.  Sharing security parameters

   Threat: Consider a scenario where several FEs are communicating to
   the same CE and sharing the same authentication keys for the Fp
   interface.  If any FE or CE is compromised, all other entities are
   compromised.

   Effect: The whole NE is compromised.

   Recommendation: To avoid this side effect, it's better to configure
   different security parameters for each FE-CE communication over the
   Fp interface.

8.1.8.  Denial of Service Attack via External Interface

   Threat: When an FE receives a packet that is destined for its CE, the
   FE forwards the packet over the Fp interface.  A malicious node can
   generate a huge message storm like routing protocol packets etc.
   through the external Fi/f interface so that the FE has to process and
   forward all packets to the CE through the Fp interface.

   Effect: The CE encounters resource exhaustion and bandwidth
   starvation on Fp interface due to an overwhelming number of packets
   from FEs.

   Requirement: Some sort of rate limiting mechanism MUST be in place at
   both the FE and CE.  The Rate Limiter SHOULD be configured at the FE
   for each message type being received through the Fi/f interface.

8.2.  Security Recommendations for ForCES

   The requirements document [4] suggested that the ForCES Protocol
   should support reliability over the Fp interface, but no particular
   transport protocol is yet specified for ForCES.  This framework
   document does not intend to specify the particular transport either,
   and so we only provide recommendations and guidelines based on the
   existing standard security protocols [18] that can work with the
   common transport candidates suitable for ForCES.

   We review two existing security protocol solutions, namely IPsec (IP
   Security) [15] and TLS (Transport Layer Security) [14].  TLS works
   with reliable transports such as TCP or SCTP for unicast, while IPsec
   can be used with any transport (UDP, TCP, SCTP) and supports both
   unicast and multicast.  Both TLS and IPsec can be used potentially to
   satisfy all of the security requirements for the ForCES Protocol.  In
   addition, other approaches that satisfy the requirements can be used
   as well, but are not documented here, including the use of L2
   security mechanisms for a given L2 interconnect technology.



Yang, et al.                 Informational                     [Page 33]
^L
RFC 3746                    ForCES Framework                  April 2004


   When ForCES is deployed between CEs and FEs inside a box or a
   physically secured room, authentication, confidentiality, and
   integrity may be provided by the physical security of the box.  Thus,
   the security mechanisms may be turned off, depending on the
   networking topology and its administration policy.  However, it is
   important to realize that even if the NE is in a single-box, the DoS
   attacks as described in Section 8.1.8 can still be launched through
   the Fi/f interfaces.  Therefore, it is important to have the
   corresponding counter-measurement in place, even for single-box
   deployment.

8.2.1.  Using TLS with ForCES

   TLS [14] can be used if a reliable unicast transport such as TCP or
   SCTP is used for ForCES over the Fp interface.  The TLS handshake
   protocol is used during the association establishment or re-
   establishment phase to negotiate a TLS session between the CE and FE.
   Once the session is in place, the TLS record protocol is used to
   secure ForCES communication messages between the CE and FE.

   A basic outline of how TLS can be used with ForCES is described
   below.  Steps 1) through 7) complete the security handshake as
   illustrated in Figure 9, while step 8) is for all further
   communication between the CE and FE, including the rest of the
   messages after the security handshake shown in Figure 9 and the
   steady-state communication shown in Figure 10.

   1) During the Pre-association phase, all FEs are configured with the
      CEs (including both the active CE and the standby CE).

   2) The FE establishes a TLS connection with the CE (master) and
      negotiates a cipher suite.

   3) The FE (slave) gets the CE certificate, validates the signature,
      checks the expiration date, and checks whether the certificate has
      been revoked.

   4) The CE (master) gets the FE certificate and performs the same
      validation as the FE in step 3).

   5) If any of the checks fail in step 3) or step 4), the endpoint must
      generate an error message and abort.

   6) After successful mutual authentication, a TLS session is
      established between the CE and FE.

   7) The FE sends a "join NE" message to the CE.




Yang, et al.                 Informational                     [Page 34]
^L
RFC 3746                    ForCES Framework                  April 2004


   8) The FE and CE use the TLS session for further communication.

   Note that there are different ways for the CE and FE to validate a
   received certificate.  One way is to configure the FE Manager or CE
   Manager or other central component as CA, so that the CE or FE can
   query this pre-configured CA to validate that the certificate has not
   been revoked.  Another way is to have the CE and FE directly
   configure a list of valid certificates in the pre-association phase.

   In the case of fail-over, it is the responsibility of the active CE
   and the standby CE to synchronize ForCES states, including the TLS
   states to minimize the state re-establishment during fail-over.  Care
   must be taken to ensure that the standby CE is also authenticated in
   the same way as the active CE, either before or during the fail-over.

8.2.2.  Using IPsec with ForCES

   IPsec [15] can be used with any transport protocol, such as UDP,
   SCTP, and TCP, over the Fp interface for ForCES.  When using IPsec,
   we recommend using ESP in the transport mode for ForCES because
   message confidentiality is required for ForCES.

   IPsec can be used with both manual and automated SA and cryptographic
   key management.  But IPsec's replay protection mechanisms are not
   available if manual key management is used.  Hence, automatic key
   management is recommended if replay protection is deemed important.
   Otherwise, manual key management might be sufficient for some
   deployment scenarios, especially when the number of CEs and FEs is
   relatively small.  It is recommended that the keys be changed
   periodically, even for manual key management.

   IPsec can support both unicast and multicast transport.  At the time
   this document was published, the MSEC working group was actively
   working on standardizing protocols to provide multicast security
   [17].  Multicast-based solutions relying on IPsec should specify how
   to meet the security requirements in [4].

   Unlike TLS, IPsec provides security services between the CE and FE at
   IP level, so the security handshake, as illustrated in Figure 9
   amounts to a "no-op" when manual key management is used.  The
   following outlines the steps taken for ForCES in such a case.

   1) During the Pre-association phase, all the FEs are configured with
      CEs (including the active CE and standby CE) and SA parameters
      manually.






Yang, et al.                 Informational                     [Page 35]
^L
RFC 3746                    ForCES Framework                  April 2004


   2) The FE sends a "join NE" message to the CE.  This message and all
      others that follow are afforded security service according to the
      manually configured IPsec SA parameters, but replay protection is
      not available.

   It is up to the administrator to decide whether to share the same key
   across multiple FE-CE communication, but it is recommended that
   different keys be used.  Similarly, it is recommended that different
   keys be used for inbound and outbound traffic.

   If automatic key management is needed, IKE [16] can be used for that
   purpose.  Other automatic key distribution techniques, such as
   Kerberos, may be used as well.  The key exchange process constitutes
   the security handshake as illustrated in Figure 9.  The following
   shows the steps involved in using IKE with IPsec for ForCES.  Steps
   1) to 6) constitute the security handshake in Figure 9.

   1) During the Pre-association phase, all FEs are configured with the
      CEs (including active CE and standby CE), IPsec policy etc.

   2) The FE kicks off the IKE process and tries to establish an IPsec
      SA with the CE (master).  The FE (Slave) gets the CE certificate
      as part of the IKE negotiation.  The FE validates the signature,
      checks the expiration date, and checks whether the certificate has
      been revoked.

   3) The CE (master) gets the FE certificate and performs the same
      check as the FE in step 2).

   4) If any of the checks fail in step 2) or step 3), the endpoint must
      generate an error message and abort.

   5) After successful mutual authentication, the IPsec session is
      established between the CE and FE.

   6) The FE sends a "join NE" message to the CE.  No SADB entry is
      created in FE yet.

   7) The FE and CE use the IPsec session for further communication.

   The FE Manager, CE Manager, or other central component can be used as
   a CA for validating CE and FE certificates during the IKE process.
   Alternatively, during the pre-association phase, the CE and FE can be
   configured directly with the required information, such as
   certificates or passwords etc., depending upon the type of
   authentication that administrator wants to configure.





Yang, et al.                 Informational                     [Page 36]
^L
RFC 3746                    ForCES Framework                  April 2004


   In the case of fail-over, it is the responsibility of the active CE
   and standby CE to synchronize ForCES states and IPsec states to
   minimize the state re-establishment during fail-over.  Alternatively,
   the FE needs to establish a different IPsec SA during the startup
   operation itself with each CE.  This will minimize the periodic state
   transfer across the IPsec layer though the Fr (CE-CE) Interface.

9.  References

9.1.  Normative References

   [1]  Bradner, S., "Key words for use in RFCs to Indicate Requirement
        Levels", BCP 14, RFC 2119, March 1997.

   [2]  Baker, F., Ed., "Requirements for IP Version 4 Routers", RFC
        1812, June 1995.

   [3]  Floyd, S., "Congestion Control Principles", BCP 41, RFC 2914,
        September 2000.

   [4]  Khosravi, H. and Anderson, T., Eds., "Requirements for
        Separation of IP Control and Forwarding", RFC 3654, November
        2003.

9.2.  Informative References

   [5]  Case, J., Mundy, R., Partain, D. and B. Stewart, "Introduction
        and Applicability Statements for Internet Standard Management
        Framework", RFC 3410, December 2002.

   [6]  Daniele, M., Wijnen, B., Ellison, M. and D. Francisco, "Agent
        Extensibility (AgentX) Protocol Version 1", RFC 2741, January
        2000.

   [7]  Chan, K., Seligson, J., Durham, D., Gai, S., McCloghrie, K.,
        Herzog, S., Reichmeyer, F., Yavatkar, R. and A. Smith, "COPS
        Usage for Policy Provisioning (COPS-PR)", RFC 3084, March 2001.

   [8]  Crouch, A. et al., "ForCES Applicability Statement", Work in
        Progress.

   [9]  Anderson, T. and J. Buerkle, "Requirements for the Dynamic
        Partitioning of Switching Elements", RFC 3532, May 2003.

   [10] Leelanivas, M., Rekhter, Y. and R. Aggarwal, "Graceful Restart
        Mechanism for Label Distribution Protocol", RFC 3478, February
        2003.




Yang, et al.                 Informational                     [Page 37]
^L
RFC 3746                    ForCES Framework                  April 2004


   [11] Moy, J., Pillay-Esnault, P. and A. Lindem, "Graceful OSPF
        Restart", RFC 3623, November 2003.

   [12] Sangli, S. et al., "Graceful Restart Mechanism for BGP", Work in
        Progress.

   [13] Shand, M. and L. Ginsberg, "Restart Signaling for IS-IS", Work
        in Progress.

   [14] Dierks, T. and C. Allen, "The TLS Protocol Version 1.0", RFC
        2246, January 1999.

   [15] Kent, S. and R. Atkinson, "Security Architecture for the
        Internet Protocol", RFC 2401, November 1998.

   [16] Harkins, D. and D. Carrel, "The Internet Key Exchange (IKE)",
        RFC 2409, November 1998.

   [17] Hardjono, T. and Weis, B. "The Multicast Group Security
        Architecture", RFC 3740, March 2004.

   [18] Bellovin, S., Schiller, J. and C. Kaufman, Eds., "Security
        Mechanisms for the Internet", RFC 3631, December 2003.




























Yang, et al.                 Informational                     [Page 38]
^L
RFC 3746                    ForCES Framework                  April 2004


10.  Authors' Addresses

   L. Lily Yang
   Intel Corp., MS JF3-206,
   2111 NE 25th Avenue
   Hillsboro, OR 97124, USA

   Phone: +1 503 264 8813
   EMail: lily.l.yang@intel.com

   Ram Dantu
   Department of Computer Science,
   University of North Texas,
   Denton, TX 76203, USA

   Phone: +1 940 565 2822
   EMail: rdantu@unt.edu

   Todd A. Anderson
   Intel Corp.
   2111 NE 25th Avenue
   Hillsboro, OR 97124, USA

   Phone: +1 503 712 1760
   EMail: todd.a.anderson@intel.com

   Ram Gopal
   Nokia Research Center
   5, Wayside Road,
   Burlington, MA 01803, USA

   Phone: +1 781 993 3685
   EMail: ram.gopal@nokia.com


















Yang, et al.                 Informational                     [Page 39]
^L
RFC 3746                    ForCES Framework                  April 2004


11.  Full Copyright Statement

   Copyright (C) The Internet Society (2004).  This document is subject
   to the rights, licenses and restrictions contained in BCP 78, and
   except as set forth therein, the authors retain all their rights.

   This document and the information contained herein are provided on an
   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
   REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE
   INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR
   IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
   THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed
   to pertain to the implementation or use of the technology
   described in this document or the extent to which any license
   under such rights might or might not be available; nor does it
   represent that it has made any independent effort to identify any
   such rights.  Information on the procedures with respect to
   rights in RFC documents can be found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use
   of such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository
   at http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention
   any copyrights, patents or patent applications, or other
   proprietary rights that may cover technology that may be required
   to implement this standard.  Please address the information to the
   IETF at ietf-ipr@ietf.org.

Acknowledgement

   Funding for the RFC Editor function is currently provided by the
   Internet Society.









Yang, et al.                 Informational                     [Page 40]
^L