Executive summary

Cable operators need to transform their infrastructures to meet the new services, features, and capacity demands of the marketplace. The Distributed Access Architecture (DAA), which will enable them to modernize their infrastructures, will require them to fundamentally modernize the analog network that has connected their headends and hubs to the access network. The new network is called a Converged Interconnect Network (CIN) and should ultimately be a digital, IP routed network that will connect the headends and data centers to the access network.

The migration to the CIN is complex and must be undertaken carefully and with methodical planning. Operators have multiple design considerations and parameters to consider and need to take stock of their current and future needs and capabilities in the process. Some operators may elect to deploy a digital version of their analog interconnect. This will enable them to reap cost benefits but may hamper their long-term, new service opportunities.

In this paper we discuss why operators must transform their networks to a Converged Interconnect Network and the benefits they will derive from doing so. In the near term, operators will reap sizable operational cost savings because the new infrastructure will be more automated, less prone to failure, will deliver a higher quality service, and will enable the consolidation of the headends and hubs. However, the true promise of network modernization is the new revenue opportunities it will unlock; it will enable cable operators to backhaul 5G wireless traffic and will be the foundation for future services to business customers.

 

Key findings

  • Cable operators need to modernize their infrastructure by migrating to DAA.
  • A DAA imperative is the modernization of the interconnect network that connects the headends to the access network called the CIN.
  • Designing and implementing the CIN is complex, requires careful planning and considerations. Operators should proceed carefully with a keen understanding of their current and future needs as well as capabilities.
  • The CIN will unlock significant opportunities for operators, including operational and capital expense cost savings and new service opportunities such as 5G backhaul.

 

Introduction

Cable operators need to transform their networks to meet the increasing demand for bandwidth to optimize the service delivery infrastructure and to support new revenue-generating services. Their current plants are no longer suited for the stringent and evolving demands of the marketplace. Although preserving the status quo is not a viable option, given market dynamics, it is unrealistic to rip-and-replace their entire infrastructures. Therefore, operators need to weigh the cost-benefit analysis of how to evolve their networks and with what technology and to strike the right balance between preserving the parts of their plants that are still relevant while modernizing others, all without compromising service delivery. Many operators are looking to leverage technological innovations to preserve the coax that connects the network edge to the customer’s premises and to move the edge closer to the customer while bringing the rest of the network and infrastructure in line with the broader industry by moving away from cablespecific technology, which is expensive, cumbersome, and difficult to modernize.

 

The legacy plant and its limitations

Today, the headends and hubs contain the major network elements, such as the CMTS and edge QAMs, which deliver video, data and video services. The analog nodes at the edge of the network followed by an amplifier cascade connect to the headends via analog optics, convert the signal to RF and transmit it over coax to the subscribers.

The legacy cable plant mostly relies on RF and on analog optics to connect headends and hubs to the fiber nodes in the access network. The equipment in the headends, hubs and the network edge is unique to the industry and is procured from a small group of vendors that charge high prices to recoup their costs given that they are selling into a limited market; the equipment also requires specialized skills that are increasingly difficult to find. The largely analog and RF plant is expensive to operate and is ill-suited for a world that is progressively digital and fast moving.

Figure 1. Cable Legacy Infrastructure (Source: ACG and Cisco)

As the demand for bandwidth grows, cable operators have been increasing the capacity in the distribution network by node splits, which expand the number of nodes in the access network (essentially reducing the number of subscribers served by each node), to increase the bandwidth available per subscriber. However, the additional nodes need more headend equipment, which requires space, powering and cooling. Headends are becoming more constrained.

Capacity increases are also straining the optical fiber distribution network that relies on legacy analog technology, which is expensive to maintain and operate, is prone to environmental degradation, and has distance and capacity limitations. The legacy infrastructure is rigid and therefore incompatible with the digital services the market demands. Because of the signal loss inherent in analog circuits, the signal to the customer could suffer from degradation, leading to a compromised customer experience.

 

Emerging need for a converged interconnect network 

To modernize the cable infrastructure and to enable it to meet the current and future business needs of cable operators, the cable industry has developed the distributed access architecture (DAA). The DAA enables cable operators to preserve the coax distribution network from the access nodes to the customer while evolving the rest of the infrastructure to align it with broader industry standards and technologies.

The DAA transforms the cable infrastructure by moving the RF elements from the headends and hubs to the access nodes. This is done by relocating functions and elements that have been an integral part of the CCAP (which today reside in the headend or hub) to the access network; this alleviates the capacity constraints in the headends and sets the stage for the network elements in the headends to become software based.

There are two main variants of DAA:

  • Remote PHY or R-PHY : The PHY layer is moved from the CCAP to the fiber node. A Remote PHY Device (RPD) is added to the node to handle the PHY functionality. The RPD handles downstream and upstream RF capabilities.

Figure 2. Remote-PHY

  • Remote MACPHY or R-MACPHY : The Media Access Control (MAC) and the PHY layers are moved to the node. A Remote MACPHY Device (RMD) is added to the node, which handles the MAC and PHY functionalities.

Figure 3. Remote-MACPHY

In an R-PHY implementation the PHY layer is separated from the CCAP core and delivered by an RPD in the access node. The RPDs need to connect to multiple CCAP cores in the headends, and CCAP cores need to connect to video cores and management systems. Analog optics, which typically connect an integrated CCAP to multiple fiber nodes, do not provide the capability for many to many interconnections; they lack switching and routing capability and fall short of the requirements for speed, latency and distance in the distributed network. They are also limited in the number of usable wavelengths, which constrains capacity. Therefore, a new network architecture powered by digital optics is required. This network will interconnect the RPDs to the CCAP cores or the MPDs to the packet engines, which will remain in the headend or hubs or ultimately become consolidated in data centers or in super headends. This networkis referred to as Converged Interconnect Network (CIN). The CIN uses industry-standard digital optics, is typically Ethernet based, and supports multiple access technologies.

Figure 4. R-PHY with the Converged Interconnect Network (Source: Cisco and ACG)

 

Evolving to the converged interconnect network 

Although the need for the CIN is clear, it requires a profound transformation of the cable interconnect network and demands a different mindset and requires skills that may reside in other parts of the organization, necessitating that teams that may have had independent responsibilities work together. Therefore, operators need to carefully consider their current and future requirements, their organizational alignment, take stock of their assets, capabilities and operational constraints, and work with trusted partners.

Designing the CIN

When it comes to designing the CIN, there is no one-size-fits-all. Each operator will design its CIN differently depending on existing infrastructure and future goals.

Some of the parameters to consider in designing the CIN:

  • Existing CMTS/CCAP architecture.
  • Core routers and backhaul capabilities.
  • Hub densities and distributions.
  • Hardware capability and availability.
  • Software functionality and port utilization and capacity.
  • Size of the deployment.
  • Amount of fiber the operator owns.
  • Operational complexity.
  • Short-term and long-term service requirements.

There are numerous design considerations for the CIN. They include:

  • Distributed or centralized: This depends on the amount of fiber the operator owns in a specific area and on the service requirements. For example, a centralized architecture is preferable when the operator owns the fiber for backhaul. However, only a distributed architecture can deliver on the full benefits of DAA and is made possible by implementing Layer 2/Layer 3 in the access. If an operator wants to remain centralized, it could programmatically path everything through a central system. This flexibility is one of the key underlying benefits of the CIN: it can be adapted to the specific needs of the operator.
  • Single-layer or spine-leaf: There are benefits and tradeoffs in both cases. Single-layer networks have a simple architecture, require less rack space, and potentially cost less but have scaling limitations. Spine-leaf networks are highly scalable, have predictable and consistent latency, are more scalable but are more complex to operate.

    The operator can start with a single-layer network (suitable for small to medium deployments) and evolve to a spine-leaf architecture for larger deployments. Spine-leaf architectures are typically well-suited for large data-center types of network architecture.

    There is a desire to migrate away from using spanning-tree while still maintaining a loop-free topology yet utilizing all the multiple redundant links. In general, a leaf-spine architecture delivers the most efficient data flows from end to end, especially considering the nonlinearity of network behavior and considering automation, programmability, segment routing, NFVs, 5G mobility, fronthaul/backhaul and others.

Figure 5. Example of a Spine-Leaf Architecture (Source: Cisco)

  • Full-mesh or point-to-point design: This design ensures physical connectivity to and from all CCAP cores and their respective ports to all RPDs. If such flexibility is not required or desired, partial mesh or point-to-point scenarios provide a mechanism to achieve desired connectivity. However, the operational complexities of future migration of cores and remapping of RPDs increase significantly with a partial-mesh or point-to-point design1.
  • Layer 2 or Layer 3: The main difference between Layer 2 and Layer 3 is the routing function. Layer 2 is capable of switching but only works with MAC addresses and does not support IP addressing. For stating MAC addressing, Layer 2 communication is efficient and fast; however, routing needs the capabilities of Layer 3, which uses IP addressing to perform dynamic routing2. Although a Layer 3 network unlocks significant revenue opportunities, it is not a short-term must. The operator can start with Layer 2 and migrate to Layer 3 based on business drivers and operational realities.
  • IPv4 or IPv6: Because this network will operate for decades, it is best to design it as a nextgeneration Ethernet IPv6 network where every port is a routed v6 port. This design allows for globally unique addresses. The operator gets the ubiquity of a Layer 2 network with the scalability of a routed network.

An additional design consideration, which is increasingly available to operators, is feature licensing. In the past operators had to pay for all the features upfront, long before they were deployed and monetized. Today, some vendors are offering the option of feature licensing, where operators can deploy capabilities but only pay for them when they are turned-on. This enables operators to deploy a CIN with rich functionality but not pay for features and capacity they do not use/monetize. With flexible software licensing, operators can build out their CIN and then scale both features and bandwidth capacity via software licensing. For example, when they are ready to enable new Layer 3 services, they can switch them on. When they need additional capacity, they can license it (assuming the hardware will support it).

CIN Design Guidelines3

Although the specific requirements of the CIN will depend on the needs of each operator, to enable the maximum capabilities that can be supported by the CIN, the following are some of the guidelines to consider:

  • Layer-2 Switching and 802.1Q trunk support, BVI support
  • Optical and Distance Support
    • 10G and 100G DWDM optics
    • IPoDWDM capability
  • IPv4 & IPv6 support
  • Basic Layer 3
    • VLAN/BridgeGroup IP assignments with inter-VLAN routing
    • IGP routing support
    • DHCPv4 & v6 relay, DHCP relay over BVI
  • Multicast (IPv4 & IPv6)
    • IGMP snooping (L2 switching), MLD snooping
    • PIM SSM/PIM SM support (L3 switching), IPv4 & IPv6
    • Multicast IPv4 & IPv6 over BVI
  • RPD Connected Switches
    • 10G DWDM/multi-color optic support, fixed and tunable
    • 802.1X, MACsec
  • Timing: G.8275.2 Profile, IEEE 1588v2 boundary clock, SyncE
  • Redundant core/spine
    • VRRP, VRRP over BVI
    • Spanning tree protocol/MST (L2 Switching or L2/L3 hybrid)
  • Hardware requirements
    • Capacity: 10G, 25G, 40G & 100G
    • Low power consumption
  • Form factor and hardening (such as temperature hardened systems for remote placements)

Although the considerations in designing the CIN are many, it is important to keep in mind that not all capabilities need to be available from day one. However, operators need a platform that can be programmed to learn and gain over time and not be limited by proprietary hardware; they can start with the basics and add capabilities as needed.

Architecture Options4

The following are some of the architecture options:

Figure 6. CIN Architecture Options

Option A: Direct Connect

In this configuration, there is direct connectivity between the dense 10 Gbps aggregation router and the 10 Gbps RPD endpoints. The main advantage of this option is its simplicity, but it has limited flexibility.

Option B: Field Aggregation Router

In this configuration, there are two connectivity options that have in common the introduction of an active networking element in the outside plant, a field aggregation router (FAR) that facilitates several packetprocessing functions to its subtending RPDs. The FAR is at the same location where there was once an analog node, which spawned multiple connections to the RPDs. This option, although more complex, brings more flexibility than Option A.

The option where a field router with DWDM is directly connected to the R-PHY core is the simplest option; it is also Ethernet native, supports 100 G and beyond, can go long distances (80 KM/DWDM), and complies with IEEE and CableLabs standards. However, today, the hardware is not price competitive but is coming down, which will make this an attractive option for the CIN architecture in a few years.

Option C: Muxponding

A TDM framing layer is inserted between hub networking gear and the RPHY endpoints. For DAA, the insertion of a transport mechanism makes sense when hubs are collapsed to a more centralized location and the bandwidth needed between the hubs and headends is in the order of many hundreds of Gigabits. However, this approach may require proprietary solutions for hardware and software build-outs, thus limiting future evolution possibilities.

A Common Denominator: Layer 3 Fabric

Although operators may have different options depending on their current network design and their future service requirements, the overarching principle of the CIN to which they should ultimately evolve is a Layer 3 fabric that will connect everything, which will migrate their network to a leaf-spine topology that mimics that of the data center today except that it will be more distributed for the MSOs.

Figure 7. Example of a Spine-Leaf Architecture (Source: Cisco)

Additional Considerations

The CIN requires a nonblocking network architecture that necessitates that aggregation points be provisioned with the maximum bandwidth utilization5. The architecture should also be scalable with predictable and low latency. A nonblocking architecture is essential for enabling FDX in the future.

Timing is essential in designing the CIN for certain capabilities such as 5G backhaul. The CCAP cores and the RPDs should connect to a common 1588 clock. This ensures that the CIN provides symmetric and predictable latency and that end-to-end latency is minimized. This is important, particularly for operators that want to offer 5G backhaul in the future.

The CIN may need full redundancy: fiber redundancy, RPD/MPD and CCAP core link redundancy, and switch redundancy in addition to high availability on the CCAP core level. However, the multiple layers of redundancy should be weighed against the costs and operational complexity. The CIN enables network function virtualization for both the CCAP and other services.

It is worth noting that in a cloud-native world, redundancy is best thought of as resiliency. When capabilities become microservices based, a failure will have limited impact on the microservice, which will stop, restart and rejoin the overall service—unlike the hardware rigid N+1 requirement of yesterday.

 

Migrating to the CIN 

Evolving the network toward the CIN should proceed in steps and should be made an integral part of the overall network transformation strategy.

General Considerations

The CIN migration can be part of the drive to bring fiber deeper into the network and to reduce the size of the service groups. Most operators are decreasing their service group sizes significantly, in some cases going from what used to be an average of 500 customers per service group to an average of 64 and even lower. This will drive the need for additional ports. The existing analog fiber can be converted to digital fiber, which will significantly increase the number of lambdas available to serve the exploding number of nodes without the need to pull additional fiber, a very expensive process.

The change from analog to digital, in most cases, consists of replacing the analog lasers with SFP+ modules. The fiber remains unchanged (unless there is impairment), and typically, Ethernet can run unassisted for 80 kilometers, which should be sufficient for most hub sites to RPDs.

Some operators will initially consider the CIN as a point-to-point replacement of the analog laser. Such an approach will deliver immediate benefits in terms of cost per bit and in terms of modernizing the last mile. Although this approach is a reasonable first step, it does not enable the operator to reap the benefits of Ethernet and IP. It is recommended that the operator put at least an elementary routing capability upfront, if possible, and build up on it later. Such a limited routing capability can be provided via switching.

Case Study: Stofa

Stofa, a cable operator in Denmark providing cable TV, broadband and telephony services to approximately 500 thousand homes, recently went live with an R-PHY deployment.

In designing the CIN, Stofa considered multiple options6. For example, it contemplated upgrading the existing point-to-point fiber network to a DWDM network to connect the RPDs to the MAC aggregation sites, which would enable the consolidation of active equipment in fewer headends. It could not pursue this approach because of limited fiber between existing headends and because of the cost and complexity of ensuring redundancy in the DWDM network.

Another consideration was a single-service network to support the RPDs, a Layer 2 network with low latency and symmetrical traffic flows. However, the Layer 2 network had downsides, because although it would have been well-suited to RPD traffic backhaul, it had significant limitations in supporting future services, and the operator would have ended up with an isolated, single-purpose network.

In the end, Stofa selected a converged IP (Layer 3) network that supports all existing services and fulfills the requirements for R-PHY. Stofa also selected an entirely IPv6 solution, requiring all devices in the CIN to be IPv6 capable. One of the benefits of this approach is to comply with IEEE 1588 timing, which will support 5G backhaul and other future services.

 

Operational benefits of the CIN 

The CIN is an Ethernet network that will connect residential and business customers and is in line with broadly deployed networking worldwide. A significant benefit of the CIN is that it replaces analog fiber, which is difficult to maintain and requires expertise that is increasingly rare, with industry-standard digital optics such as 10 Gb (or higher 100 Gb). Digital optics technologies are mature, widely available, and present the benefit of a declining price curve. They also open the operators to a host of revenuegenerating opportunities.

Extending Fiber Deep into the Network

Replacing analog fiber with digital extends fiber connectivity all the way to the fiber nodes, which are in the neighborhoods close to the customer. The higher throughput provided by digital fiber and node splits enables operators to substantially increase the amount of bandwidth available to subscribers and improves the quality of experiences because digital fiber is less prone to degradation and failure.

Modernizing the Headends and Hubs

Analog fiber requires RF components, such as RF combiners, in the headends and hubs. The distance constraints inherent to analog optics have led to the propagation of hubs in the cable operators’ networks. With digital fiber, analog components are eliminated, and Ethernet or IP based packet networking is used for transmission between the core and the edge. The longer distances, characteristic of digital fiber, will obviate the need for some hubs, which can be either repurposed for packet transport gear or eliminated.

The eventual elimination of analog optics hardware from the headends and hubs and the unbundling of the CCAP will pave the way for the modernization and consolidation of the headends and hubs. The CCAP, now a hardware network element, can be virtualized and run on common-off-the-shelf servers (COTS), which can either remain in the headend or be consolidated in mega headends or data centers. In the long run, operators can completely eliminate hardware from the headends and consolidate them. The process of digitizing the headends is referred to as Headends Re-Architected as Data Centers or HERD and will lead to significant operational and capital cost savings as well as new opportunities for fast service creation and delivery.

Operational Efficiencies and Reduced Costs

The modernized, automated infrastructure will herald significant cost savings:

  • With fiber-deep deployments, the signal coming out of the node is much stronger than it is today with analog nodes. This improves the signal to the premises and enables it to ride over potential impairments within the customers’ premises, leading to an improved experience and reducing costly truck rolls.
  • Legacy nodes do not have the capability to monitor traffic; therefore, they are fully powered irrespective of use. The new nodes have self-monitoring capabilities and can enable ports as needed based on consumption. This leads to reduced power costs.
  • The digital optical circuits are significantly less expensive than the legacy analog ones. 10 Gb/s fiber (and even 100 Gb/s) is widely adopted in the industry, leading to very attractive pricing. The talent needed to manage these circuits is readily available. These factors result in lower capital expense and operation costs.
  • Digital fiber has a higher signal to noise ratio, is less impacted by environmental factors, does not degrade over time, and has lower latency. It benefits from industry-wide adoption, which brings prices down and makes the talent needed to operate it easier to find. These characteristics result in lower total cost of ownership and in better customer satisfaction because of improved operating characteristics.
  • The consolidation of the hubs and headends will have significant savings implications with respect to real estate, power and cooling and ultimately will lead to personnel reduction or reassignment.
  • Virtualization of the CCAP will decrease the dependence of the industry on expensive, customized hardware and will enable it to benefit from the evolution and price curve of COTS.

Service Velocity

A fully digital plant combined with cloud-native network elements will enable cable operators to embrace fast innovation. Instead of the long development cycles that are associated with the legacy plant because of hardware constraints and endless regression testing, a cloud-native infrastructure and digital connectivity will enable operators to quickly introduce new services and features and help align their capabilities with the competitive landscape, which is increasingly defined by web scalers and other industry innovators.

Virtualization and Automation

One key benefit of DAA and the CIN architecture associated with it is that a significant part of operators’ infrastructures will be virtualized, which will enable them to have a single orchestration layer that abstracts the complexity underneath. Using SDN and NFV principles enables operators to automate network operations and to significantly improve resiliency. With automation, routable protocols are the most important. Without Layer 3 capabilities, the benefits of automation are limited.

In addition to virtualization, using the network as a programmable and interconnected fabric is the foundation of automation. In the cable industry today, virtualization is mainly focused on replacing hardware systems with software-based solutions. Going forward, an IP Ethernet CIN makes the physical transport layer irrelevant and provides the fabric that will enable automation, leading to network optimization, automated capacity planning, service integration and coordination, fault resiliency and a digital experience for customers.

In the new distributed network, automation is an essential capability because the distributed network has a significantly larger number of ports, is multidimensional, and much more dynamic. For example, the manual configuration methods operators have relied upon are no longer adequate. Operators need to embrace automation from day one and adopt a programmable network that utilizes state-of-the-art automation capability to configure and manage.

Figure 8. The Automation Cycle (Source: Cisco)

 

Powering New Opportunities

The modernization of the cable infrastructure, specifically the adoption of the CIN, will pave the way for significant, new business opportunities for cable operators.

Coax as Enabler of New Services

As operators reduce the size of service groups and higher reliability digital fiber is pushed deeper in the network, the coax from the node to the customer is shorter and the signal it delivers is much stronger because it benefits from higher MER (typically 43–45 dB versus 38–40 dB for analog). The operator can now deliver a service level agreement over the coax, which allows it to offer business services such as Layer 2 VPN over coax or to use coax as an effective substitute to fiber for business services. This capability will be further enhanced when the industry implements Full Duplex DOCSIS, which enables symmetric 10 Gb/s upstream and downstream bandwidth capabilities. The improved signal quality will also be a boon for customer satisfaction, because the signal will be less sensitive to potential degradation at the customer’s premises.

5G Transport

With DAA, the cable infrastructure becomes a distributed network that is modernized and powered. This makes it an ideal mechanism for transporting 5G small-cells traffic at large scales. The industry is currently engaged with CableLabs to make enhancements to DOCSIS for timing and sync to enable backhaul of 5G traffic, which will be a significant business opportunity for the cable operators. They can backhaul traffic for mobile operators or backhaul their own traffic.

Transport of Other Traffic

The CIN is access agnostic and carries traffic for multiple access technologies simultaneously:

  • DOCSIS
  • 5G
  • PON
  • ETTH
  • PON
  • WIFI Hotspots and Access Points, including IoT Access

Edge Computing Capabilities

The services of the future, such as IoT and autonomous vehicles, will require low latency and intelligence at the edge of the network. The MSOs’ modernized network infrastructures will provide them with a powerful computing capability very close to the customer or even at the customer’s location.

Segment Routing Unlocks More Benefits

Segment routing capability, which enables network slicing and simplified traffic engineering and alleviates the complexity of multiservice networks, can be added to the CIN. With segment routing the operator can specify different routes for different services or applications, which leads to improved quality of experience and quality of service7.

 

Conclusion

Cable operators need to modernize their networks. The DAA, the primary vehicle for this modernization, will drive them to ultimately reinvent the network that has connected their headends to the access network. This evolved network, referred to as a CIN, will require digital optics instead of analog optics and ultimately will become a routed IP network. It will interconnect their future infrastructure, which is based on broad industry standards, and will pave the road for significant operational efficiencies and new revenue opportunities. Migrating to such a network requires meticulous planning, careful consideration of tradeoffs and other parameters. It should be undertaken in steps and should be done with a trusted partner that has the experience, expertise and capacity to shepherd operators along the way.

ACG Research delivers telecom market share/forecast reports, consulting services, business case analysis, product and service message testing. ACG provides you with accurate market share data, strategic and tactical advice, services and products, and timely answers to industry questions so that you can better understand market dynamics and grow your telecom operations more efficiently and profitably. © Copyright 2019 ACG Research. Reproduction is prohibited unless authorized. All rights reserved.

Liliane Offredo-Zreik (@offredo) is a Principal Analyst with ACG Research. Her areas of coverage include the cable industry and software defined services. Prior to her analyst work, she held senior roles in major telecom and cable companies as well as with industry vendors and has been an industry advisor in areas including marketing, strategy, product development and M&A due diligence.

1 https://blogs.cisco.com/sp/remote-phy-why-cin-architectures-matter

2 https://www.wideband.net.au/blog/difference-layer-3-layer-2-networks/

3 Source: Cisco.

4 Capacity and Technology Considerations in DAA Backhaul Deployment Strategies, Fernando X. Villarruel and Martin Mattingly, Cisco Systems, 2018

5 https://blogs.cisco.com/sp/remote-phy-why-cin-architectures-matter

6 It’s ALIVE! Getting to Successful R-PHY Deployment: Do’s and Don’ts,” Tal Laufer and Jeroen Putzeys, Arris.

7 https://blogs.cisco.com/sp/using-segment-routing-to-unlock-the-full-potential-of-the-rphy-converged-interconnect-network