Troubleshoot IX5000 Touch 10

Introduction

This document describes different scenarios observed in the field and steps to troubleshoot, isolate and gather information to fix the problem.

IX5000 is a new generation Cisco Telepresence Immersive Endpoint and uses Touch 10 instead of Touch 12 used by the CTS and TX Immersive  systems. It uses a different software for User Interface (UI) than TC endpoints; however; it uses the same Android base.

Common Problems

Touch Panel Does Not Power On/Boot Up Even Though the Entire System Booted Up Successfully

Once the touch panel boots up successfully, the default screen is displayed as shown in the image.

1. Check if the Ethernet cable is connected to the switch port and the switch is powered on. The only way you can power on the Touch 10 is via Power over Ethernet (PoE).
2. Reseat the Ethernet cable. Try another known working Ethernet cable.
3. Try another Ethernet port on switch, If the basic steps have been performed and still the Touch 10 device shows a blank screen, or stays in maintenance mode, a possible cause could be that switch in the table did not receive its configuration from the codec, which can cause the Touch 10 device to not receive PoE from the switch. This condition requires that you reset the switch back to factory defaults and reboot the codec with these steps. 
   1. Press and hold the Mode button for 11 seconds.

      Note: The Mode button is on the front and underside of the switch, on the same side as the power plug. The switch LEDs begin to blink after three seconds, and stops to blink seven seconds later. Then, the switch reboots and loses its configuration.

   2. To restart the codec, log in to the IX5000 administration Graphical User Interface (GUI) and click Restart/Reset. The default IP address for the GUI is 169.254.1.1, and the default username and password is admin/cisco. If you have already configured your codec, the IP address is different. The switch regains its configuration and the Touch 10 devices successfully initializes.

Brand New Touch Fails to Upgrade - Stuck in [Maintenance Mode]

All brand new Touch comes with the package installed from the factory, which is meant for TC based systems need upgrade to IX package immediately when connected to any IX system. Therefore, if an upgrade fails and the UI hangs with the message "maintenance mode...downloading software", the panel never upgrade to the software used by IX system. The IX software would not display the "maintenance mode" text, but instead displays "Downloading/Extracting/Installing". In order to try to recover the panel, reboot or restart  the software.If that does not fix the issue use the factory reset IX8.1.1 code which is same as used by TC endpoints. Lower versions have a different method.

Another option to troubleshoot, if you have a MX/SX based endpoint on SW code TC7.1 or later, the Touch can be connected back to get the TC software and then can be reconnected to IX. As this helps the Touch to recover the default software, the software of IX can be overwritten again when you connect it to IX.

Loss of Connection - [Lost Connection] Displayed in the UI

A loss of connection to the IX5000 is usually displayed in the UI as lost connection. IX software shows this message if the heartbeat is lost to the IX5000. A heartbeat is a command/response sent to the IX every 15 seconds. If two heartbeats are lost (no response after 30 seconds), the Touch unit will no longer be operable through the UI as commands will not go through to the IX nor status updates will be retrieved from the IX - and the message will therefore be displayed. Also, after a boot of the Touch, if a connection/pairing never can be established to the IX, the message is displayed. The Touch will continuously try to establish the connection to the IX in order to reach a normal operable state. When a connection again is established the message will disappear.

When "Lost of Connection" message appears, there is seldom an issue with the Touch device and troubleshooting of the IX and the table switch where The Touche panels are connected must be performed in order to solve the issue.

If all Touches in the room suddenly get this message, but the IX seems to function well, it is a high probability that there is an issue with the table switch. Logs located /nv/log/touch/ folder when the log bundle is extracted.

Example of Heartbeat in Logs

LOG_NOTICE(169.254.1.102) :06-08 12:16:28.683 WARN com.cisco.telepresence.system.SystemService Tag:SocketThread #codec send: xcommand Peripherals HeartBeat ID : "88:43:E1:C6:54:51" Timeout : "30" | resultId="18093" LOG_NOTICE(169.254.1.101) :06-08 12:16:34.785 WARN com.cisco.telepresence.system.SystemService Tag:SocketThread #codec send: xcommand Peripherals HeartBeat ID : "88:43:E1:C6:52:8E" Timeout : "30" | resultId="18476" LOG_NOTICE(169.254.1.102) :06-08 12:16:43.718 WARN com.cisco.telepresence.system.SystemService Tag:SocketThread #codec send: xcommand Peripherals HeartBeat ID : "88:43:E1:C6:54:51" Timeout : "30" | resultId="18094"

Android Crashes - For Example [Phone App Has Stopped]

Whenever there is an unhandled Java exception thrown in a process/app, this would usually be visible to you through a standard message  {the_app has stopped} with a button for confirmation. This does not necessarily hinder normal use of the system and might not have any after-effects. However, they must not happen.

In order to debug why such a crash happened, the logcat output the backtrace as long as the log is retrieved as soon as possible after the crash was reproduced.There can be a number of different types of exceptions thrown, therefore it is best to search for FATAL or Exception in the logs. Remember that every file on the IX needs to be investigated in order to find the crash. These are located in the /nv/log/touch/ folder when the log bundle is extracted.

Here is one example where the dashboard app crashes with the"Dashboard has stopped" message shown in the UI:

2015-07-08 02:21:32.467 - FATAL EXCEPTION: main
2015-07-08 02:21:32.467 - Process: com.cisco.telepresence.dashboard, PID: 6825
2015-07-08 02:21:32.467 - java.lang.NullPointerException
2015-07-08 02:21:32.467 - at com.cisco.telepresence.dashboard.adapter.
MediaChannelListAdapter.<init>(MediaChannelListAdapter.java:37)
2015-07-08 02:21:32.467 - at com.cisco.telepresence.dashboard.fragment.
MediaChannelListFragment.configureAdapter(MediaChannelListFragment.java:76)
2015-07-08 02:21:32.467 - at com.cisco.telepresence.dashboard.fragment.
MediaChannelListFragment.onViewCreated(MediaChannelListFragment.java:30)
2015-07-08 02:21:32.467 - at android.app.FragmentManagerImpl.moveToState(FragmentManager.java:904)
2015-07-08 02:21:32.467 - at android.app.FragmentManagerImpl.moveToState(FragmentManager.java:1062)
2015-07-08 02:21:32.467 - at android.app.BackStackRecord.run(BackStackRecord.java:684)
2015-07-08 02:21:32.467 - at android.app.FragmentManagerImpl.execPendingActions(FragmentManager.java:1447)
2015-07-08 02:21:32.467 - at android.app.Fragment.performStart(Fragment.java:1721)
2015-07-08 02:21:32.467 - at android.app.FragmentManagerImpl.moveToState(FragmentManager.java:918)
2015-07-08 02:21:32.467 - at android.app.FragmentManagerImpl.moveToState(FragmentManager.java:1062)
2015-07-08 02:21:32.467 - at android.app.BackStackRecord.run(BackStackRecord.java:684)
2015-07-08 02:21:32.467 - at android.app.FragmentManagerImpl.execPendingActions(FragmentManager.java:1447)
2015-07-08 02:21:32.467 - at android.app.FragmentManagerImpl$1.run(FragmentManager.java:443)
2015-07-08 02:21:32.467 - at android.os.Handler.handleCallback(Handler.java:733)
2015-07-08 02:21:32.467 - at android.os.Handler.dispatchMessage(Handler.java:95)
2015-07-08 02:21:32.467 - at android.os.Looper.loop(Looper.java:136)
2015-07-08 02:21:32.467 - at android.app.ActivityThread.main(ActivityThread.java:5076)
2015-07-08 02:21:32.467 - at java.lang.reflect.Method.invokeNative(Native Method)
2015-07-08 02:21:32.467 - at java.lang.reflect.Method.invoke(Method.java:515)
2015-07-08 02:21:32.467 - at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:779)
2015-07-08 02:21:32.467 - at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:595)
2015-07-08 02:21:32.467 - at dalvik.system.NativeStart.main(Native Method)
2015-07-08 02:21:33.059 - LOG_NOTICE(169.254.1.8) :07-08 12:21:24.907 WARN 

Inconsistent UI State

If the UI does not reflect the correct state of the system, for example the (Return to call) bar is visible outside of a call or the (Return to call) bar is not visible in a call, this might have been caused by an incomplete status update from the IX. For example if a call goes down abruptly, the IX halts, media issues and so on.

If the state persists, a reboot or a factory reset of the Touch can fix the problem.

Logs from Touch10 Via Serial Connection

It is highly beneficial to extract the logs directly from the problematic Touch itself, especially in cases where there is a boot issue/lost connection/software upgrade issue with the Touch as no logs would then potentially be transferred to the IX. It is possible to extract the logs from the Touch to connect a micro USB cable to the back of the Touch (the one used for charging normal Android-based mobile phones) and from a computer. Open a serial terminal with the settings:

Baud rate: 115200 

Data/par/stop: 8n1

A shell is available. Enter the logcat in to this shell in order to output the complete log. Enter the bugreport  in order to output the log and additional hardware/ network information as well. This must be copied to a file from the terminal or saved. The log will not survive a boot, so it is important to capture it before it does a boot for recovery.

Recovery Procedure

If it has been concluded that Touch panel is the problematic unit, complete a Return Material Authorization(RMA). Try to recover the panel before you complete the RMA.

• Collect the logs from the Touch panel directly as described with a serial connection.
• Reboot the Touch by a power cycle (reconnect the network cable in the back of the Touch).
• Perform the factory reset of the Touch, as described in this document.
• If you have a MX/SX based system that runs software verion TC7.1 or later, the Touch can be connected to recover the factory package. This is transferred over resync instead of HTTP and this might work as a last resort to get the Touch in an operable state. Once recovered it be connected back to the IX5000.

Factory Reset Touch 10 running IX Code

1. Unplug the power/network cable from the back of the Touch10.
2. Hold the Volume up hard button and replug the power/network cable.
3. Wait for the Mute hard button to light up (red) - approx. 10 seconds.
4. Release the Volume up button and click the Mute hard button The factory reset has been performed successfully when a confirmation of green blinks on the Mute button occurs after step 4.

Factory Reset Touch 10 running TC/CE Code

1. Hold down the Mute button for approximately 10 seconds until it starts to blink red.
2. Press the Volume down button twice.
3. The Mute button will turn solid red and reboot with factory default settings