VoWLAN Troubleshooting Overview

Document Purpose and Target Audience

This document is written specifically for systems engineers, customers and Cisco partners who are responsible for the planning, design, implementation, operation and optimization of voice solutions using a Cisco Unified Wireless Network (CUWN) with Cisco 792xG Series wireless IP phones. This document will cover the fundamental aspects of design and deployment, while focusing on actual troubleshooting practices, tools and techniques that are used by the Cisco Technical Assistance Center (TAC) and the Wireless Networking Business Unit (WNBU) Escalation Team.

Assumptions

It is important to have an intermediate to advanced understand of the following topics:

•Familiarity with Cisco's IOS using both routers and switches.

•Understanding of Radio Frequency (RF) propagation as it relates to the 802.11 standards.

•Understanding of protocol level networking at layer 2 and layer 3 and how to review wireless sniffer traces in both Wireshark and Omnipeek.

•A basic understanding of Voice Over IP (VoIP), Call Signaling, Call setup and teardown (SCCP, SIP) and codecs such as G.711 and G.729.

Software Versions

For the purposes of this document, all screen shots will be provided for Wireless LAN Controllers and Wireless Control System (WCS) running 6.x code. The Cisco 792xG Series wireless IP phones will be loaded with firmware version 1.3.(3).

Release Notes

It is very important to understand and review the Release Notes for each version of code that is being used on the Wireless LAN Controller. The release notes define existing bugs and caveats with regard to controller functionality. Please review the release notes if you suspect that a protocol or feature is not working according to design or existing documentation.

Introduction

During the conceptual conversations that took place before the creation of this document, we at Cisco consulted the Cisco Wireless TAC, TAC Escalation, and the Wireless Networking Business Unit Escalation Team to discuss the caveats related to the proper design, deployment and troubleshooting of a Voice Over Wireless LAN (VoWLAN). In an effort to create the most ideal voice troubleshooting document, we also discussed design and deployment with the Cisco Advanced Services Team. The Advanced Services Team is directly responsible for the Design, Deployment and Implementation of the Cisco Unified Solutions Network around the world. Their expertise and knowledge was fundamental in helping craft the sections on RF propagation and site survey best practices.

In its simplest form, Cisco's Voice over Wireless LAN is most often designed and deployed incorrectly due to a few misconceptions, myths or misunderstandings with regard to the fundamentals of RF propagation and user mobility. While a misconfiguration is also a common occurrence, remediation is relatively simple for the most part. In most cases, the remediation may require down time after hours to resolve the problem. On the other hand, remediating issues that pertain to the improper design and deployment as it relates to RF propagation and poor AP placement are often more costly, time consuming and problematic.

Through extensive experience, Cisco's support teams have determined that most Cisco Wireless Networks are deployed for data, without any long term thoughts about deploying a VoWLAN solution in the future. Specifically, we will touch on topics related to performing a thorough Pre- and Post-Site Survey, while also focusing on the importance of proper AP placement as it relates to Cell Edge Design and frequency reuse within the WLAN.

Overview of VoIP and VoWLAN

VoIP refers to a way to carry phone calls over an IP data network, whether on the Internet or your own internal network or both. The primary attraction to VoIP is its ability to help reduce expenses allowing telephone calls to travel over the data network rather than out onto the PSTN for calls that need to be routed outside the company's network. The Cisco Unified Communications Manager uses technologies such as Session Initiation Protocol (SIP) and Skinny Call Control Protocol (SCCP) along with mobility solutions to unify and simplify all forms of voice communications. VoIP utilizes the same physical layer as defined in the IEEE 802.3 standard; however, VoWLAN utilizes an alternate access method referred to as CSMA/CA, using various 802.11 modulations over the air to define the medium. In both VoIP and VoWLAN, call signaling and control protocols are used for call setup and call tear down (SCCP, SIP) and voice codecs (G.711 and G.729) are used to encode speech over the WLAN and IP network.

Across all verticals, whether retail, education, corporate business or medical, the need for user mobility has increased substantially over the last few years. Voice over WLAN has become an integral part of the business need. Keeping users connected enhances a company's ability to communicate and collaborate while maintaining a high level of quality as the user moves throughout the WLAN. As we move into subsequent chapters of this troubleshooting guide, we will touch on the various aspects of troubleshooting as it pertains to the PDIOO (Plan, Design, Implement, Operate, Optimize) model and provide you with the necessary tools needed to be successful when troubleshooting your VoWLAN.

Common VoWLAN Problems

•Choppy Audio / No Audio

•One-Way Audio

•Clipping, Echo

•Gaps in Audio / No Audio when Roaming

In most cases, all of the above symptoms are related to a problem within the RF environment. This can either be due to poor signal, no signal, or asymmetric transmit where the client can hear the AP, but the AP cannot hear the client (one-way audio). In some instances we discover that it might be a misconfiguration or a problem with the physical network, such as Quality of Service (QoS) misconfiguration or a lack of trust as it relates to QoS Differentiated Service Code Point (DSCP) markings, or perhaps a gateway misconfiguration that causes an impedance mismatch resulting in echo when a VoWLAN user makes a call onto the PSTN. This document will place a great deal of emphasis on understanding RF propagation and stress the importance of performing a site survey as it relates to thorough RF planning.

Troubleshooting Methodology

Cisco provides a high level troubleshooting methodology that is used to gather the facts as they pertain to the problem. The purposes of this methodology will help facilitate the appropriate measures to ensure that each problem can be resolved in the quickest and most efficient manner possible.

Troubleshooting Steps

1. Define the problem.

2. Gather facts.

3. Consider possibilities.

a. Redefine the problem, if necessary.

4. Create an action plan.

5. Implement an action plan.

6. Observe results.

7. If resolved, document action items taken to resolve.

8. If not resolved, iterate the process from step 2.

Define the Problem

When gathering facts, it is important to create a clear and concise problem definition. Ensure that you understand the problem definition from an engineering and technical perceptive, rather than the user's individual perspective. While it is important to gather information from a user, a user's perception of the problem is likely to vary significantly.

Gathering Facts

Gathering facts is of vital importance when troubleshooting a VoWLAN. Aside from asking the customer several questions about the symptoms, it will often require a Systems Engineer to implement various tools such as Omnipeek or Wireshark to capture sniffer traces, while also running debug commands on the Wireless LAN Controller and evaluating the WCS to perform configuration or RF audits within the CUWN. Below are examples of questions that a TAC Customer Support Engineer (CSE) might ask when troubleshooting a VoWLAN issue. It is imperative to understand the answers to each of these before opening a Service Request with the Cisco Wireless TAC.

Consider the Possibilities

After a problem has been defined and facts have been gathered about the symptoms, the next logical step is to consider all of the possible causes. VoWLAN connectivity issues can be very difficult to trace, especially when considering RF propagation. In most situations, there are several possible causes for a network error, and the Systems Engineer administrator should be very thorough when identifying each probable cause.

Redefine the Problem

In most situations, the original problem definition may change once facts have been gathered and possibilities have been considered. There may even be a need to iterate the gathering facts phase by gathering additional sniffer traces or debugs from a controller or switch within the network infrastructure. In any case, it is important to define the problem in a clear and concise manner so that resolution can be provided in the most efficient possible manner.

Creating and Implementing an Action Plan

Once the network problem and possible causes have been identified, an action plan needs to be created to mitigate and facilitate resolution. When developing a solution, it is critical to thoroughly analyze the proposed solution and brainstorm with your peers the potential impacts your solution may have.

Important guidelines to follow when implementing a solution:

•Make one change at a time and document each individual change. It is important to also document and understand any problems that were experienced outside the scope of the current problem when making changes. 

Example

If you make a change that creates a different problem, it is important to thoroughly document that change and problem as well, while also keeping your eye on the current task at hand. Making several changes to the environment can create unnecessary havoc, making the problem much worse. It is important to follow this as a cardinal rule when implementing your action plan.

•Make transparent changes first. This means that if there are multiple potential causes for a problem, try to resolve problems that least impact your network and users first.

•Avoid creating security holes or vulnerabilities when implementing your changes. 

Example

Creating an Open SSID and broadcasting that over one or many access points. In some environments, providing open access to the network could potentially violate organizational and other guidelines (i.e., HIPAA).

•Most importantly, always ensure that you can back out of any changes that were made to the WLAN.

Observe Results

After each change is implemented, observe the results. If the problem is not resolved, reevaluate the possibilities to determine if the change that was made should be reverted or remain due to recommended best practices. Please adhere to the VoWLAN checklist and Cisco-recommended best practices with regard to the change implemented.

Intermittent Problems

In a WLAN, it may be important to observe results over a longer period of time, especially when troubleshooting problems that are considered intermittent. If the problem is readily reproducible, then results can be observed and resolution can be determined at a relatively quick rate. On the other hand, an issue that pertains to clipping in an audio stream or occasional gaps in audio may need to be observed for a longer period of time to ensure that the problem is resolved.

If resolved, document the actions taken

This step is fairly straight forward. If the problem is resolved, document the changes that were made on a step-by-step basis.

Troubleshooting Questions

1. What version of code is installed on the Wireless LAN Controller?

2. What is the firmware version installed on the Cisco IP Phone?

3. What kind of Cisco Controller/AP is in use?

4. Is the AP in local or HREAP mode?

5. Has the problem or symptoms been experienced by users before?

6. Were there any recent changes made to the physical network or WLAN?

7. Are calls made from a wired IP Phone to wireless, wireless to wireless or wireless over the PSTN?

Note Understanding the call path is very important when troubleshooting VoWLAN cases. This helps isolate QoS misconfiguration and provides the TAC engineer with an understanding of where wired or wireless sniffer traces need to be taken.

8. In the case of choppy or one-way audio, does the issue happen throughout the entire WLAN or in one particular area?

–If in a particular area, between how many APs?

9. Is the client roaming when the problem is observed or stationary?

–This is sometimes a tricky question to answer. In poorly deployed environments, a voice handset may actually roam several times even when stationary due to RF related problems and an RSSI differential. We will discuss this in greater depth in the section on troubleshooting the 792xG Series wireless IP phones.

Site Survey Questions

1. Did you perform a VoWLAN site survey?

–If yes, please review the documentation and validate that the deployment and AP placement is in alignment with the site survey recommendations while adhering to Cisco's design and deployment best practices. If a Service Request is opened, please provide the site survey documentation to the Cisco TAC.

2. If no in question 1., did you perform a post site survey after the wireless network was deployed?

–If a post deployment was performed, please reevaluate the post deployment survey and AP placement to ensure that it is in alignment with the design and deployment recommendations, it facilitates the appropriate coverage, and it is optimized for voice and user mobility.

–If a post deployment was not performed, review the heat maps in WCS to gauge approximately what the coverage looks like.

Note WCS heat maps are predictive based on antenna selection and direction configurations in WCS. If WCS was not configured with those parameters, it will not provide an accurate representation or prediction of RF propagation on your WLAN.

Validating Controller Configurations

1. Review wired and wireless configurations.

2. Use the Voice Audit tool to validate the voice configuration on the Wireless LAN Controller.

3. Is QoS implemented end to end?

–If yes, move on.

–If no, remediate and ensure that packets are marked and trusted appropriately.

On most new Cisco switches, the command mls qos trust dscp will ensure that QoS is trusted from the 792xG Series wireless IP phone when it transmits using EF with a setting of UP = 6.

Note EF (DSCP 46) is a L3 marking. For L3 to L2 mapping, remember that EF maps to a CoS = 5.

RF Propagation

1. Perform RF Analysis and ensure that uplink packets are queued correctly.

2. Ensure that the client has enough signals to communicate efficiently with the AP.

3. Is RRM enabled?

–If yes, what code version is in use?

4. Did the customer implement Tx power throttling to define a Min and Max transmit power?

–If Tx throttling is not enabled, verify symmetric vs. asymmetric transmit.

Note This means that you should compare the client's transmit capability to the AP's transmit capability.

5. Run WCS Client Association Report (Mandatory).

6. Power and Channel Change Report.

7. How many instances of CHA were run when the problem was experienced?

Summary

Once the WLAN engineer has gathered the appropriate facts, he or she should then be able to consider the possibilities and create and action plan that can be implemented to resolve the problem according to the symptoms discovered. The action plan should be considered the actual steps that will be taken to remediate the problem, not the actions taken to gather facts.

Once the action plan is implemented, it is simply a matter of observing the results. If the problem remains unresolved, there is often a problem related to the facts that were gathered or another problem that went unnoticed. Cisco then recommends that the troubleshooting process be iterated and additional facts should be gathered to remediate the issue based on the symptoms.

In later chapters, this document will provide examples and case studies with regard to how troubleshooting methodologies are applied to each of the verticals mentioned above. We hope to show how Cisco actually troubleshoots voice cases and isolates root causes based on the data gathered according to our troubleshooting methodologies.