Event tracking and management has traditionally been performed by devices external to the networking device. Embedded Event Manager (EEM) has been designed to offer event management capability directly in Cisco IOS XE devices. The on-device, proactive event management capabilities of EEM are useful because not all event management can be done off router because some problems compromise communication between the router and the external network management device. Capturing the state of the router during such situations can be invaluable in taking immediate recovery actions and gathering information to perform root-cause analysis. Network availability is also improved if automatic recovery actions are performed without the need to fully reboot the routing device.

EEM is a flexible, policy-driven framework that supports in-box monitoring of different components of the system with the help of software agents known as event detectors. The figure below shows the relationship between the EEM server, core event publishers (event detectors), and the event subscribers (policies). Basically, event publishers screen events and publish them when there is a match on an event specification that is provided by the event subscriber. Event detectors notify the EEM server when an event of interest occurs. The EEM policies that are configured using the Cisco IOS XE command-line interface (CLI) then implement recovery on the basis of the current state of the system and the actions specified in the policy for the given event.

EEM offers the ability to monitor events and take informational or corrective action when the monitored events occur or when a threshold is reached. An EEM policy is an entity that defines an event and the actions to be taken when that event occurs. There are two types of EEM policies: an applet or a script. An applet is a simple form of policy that is defined within the CLI configuration. A script is a form of policy that is written in Tool Command Language (Tcl).

Figure 1

Embedded Event Manager Core Event Detectors

EEM 2.0 is supported in Cisco IOS XE releases and introduces some new features. EEM 2.0 introduced the following event detectors:

• Application-Specific--The application-specific event detector allows any Embedded Event Manager policy to publish an event.
• Counter--The counter event detector publishes an event when a named counter crosses a specified threshold.
• Interface Counter--The interface counter event detector publishes an event when a generic Cisco IOS interface counter for a specified interface crosses a defined threshold.
• Timer--The timer event detector publishes events for the following four different types of timers: absolute-time-of-day, countdown, watchdog, and CRON.
• Watchdog System Monitor (IOSWDSysMon)--The Cisco IOS XE watchdog system monitor event detector publishes an event when CPU or memory utilization for a Cisco IOS XE process crosses a threshold.

EEM 2.0 introduced the following actions:

• Setting or modifying a named counter.
• Publishing an application-specific event
• Generating an SNMP trap.

The ability to run a Cisco defined sample policy written using Tool Command Language (Tcl) was introduced. A sample policy was provided that could be stored in the system policy directory.

EEM 2.1 is supported in Cisco IOS XE releases, and introduces some new features. EEM 2.1 introduced the following new event detectors:

• CLI--The CLI event detector screens command-line interface (CLI) commands for a regular expression match.
• None--The none event detector publishes an event when the Cisco IOS XE event manager run CLI command executes an EEM policy.
• OIR--The online insertion and removal (OIR) event detector publishes an event when a particular hardware insertion or removal event occurs.

EEM 2.1 introduced the following actions:

• Executing a Cisco IOS XE CLI command.
• Requesting system information when an event occurs.
• Sending a short e-mail.
• Manually running an EEM policy.

EEM 2.1 also permits multiple concurrent policies to be run using the new event manager scheduler script command. Support for SNMP event detector rate-based events is provided as is the ability to create policies using Tool Command Language (Tcl).

EEM 2.1 (Software Modularity) is supported in Cisco IOS XE releases on Cisco IOS XE Software Modularity images. EEM 2.1 (Software Modularity) introduced the following event detectors:

• GOLD--The Generic Online Diagnostic (GOLD) event detector publishes an event when a GOLD failure event is detected on a specified card and subcard.

Note	GOLD is not supported on the Cisco ASR 1000 Series routers.

• System Manager--The system manager event detector generates events for Cisco IOS XE Software Modularity process start, normal or abnormal stop, and restart events. The events generated by the system manager allows policies to change the default behavior of the process restart.
• Watchdog System Monitor (WDSysMon)--The Cisco IOS XE Software Modularity watchdog system monitor event detector detects infinite loops, deadlocks, and memory leaks in Cisco IOS XE Software Modularity processes.

EEM 2.1 for Software Modularity introduced the ability to display EEM reliability metric data for processes.

Note	EEM 2.1 for Software Modularity images also supports the resource and RF event detectors introduced in EEM 2.2, but it does not support the enhanced object tracking event detector or the actions to read and set tracked objects.

EEM 2.2 is supported in Cisco IOS XE releases, and introduces some new features. EEM 2.2 introduced the following event detectors:

• Enhanced Object Tracking--The enhanced object tracking event detector publishes an event when the tracked object changes. Enhanced object tracking provides complete separation between the objects to be tracked and the action to be taken by a client when a tracked object changes.
• Resource--The resource event detector publishes an event when the Embedded Resource Manager (ERM) reports an event for the specified policy.
• RF--The redundancy framework (RF) event detector publishes an event when one or more RF events occur during synchronization in a dual Route Processor (RP) system. The RF event detector can also detect an event when a dual RP system continuously switches from one RP to another RP (referred to as a ping-pong situation).

EEM 2.2 introduced the following actions:

• Reading the state of a tracked object.
• Setting the state of a tracked object.

EEM 2.3 introduces enhancements to the Generic Online Diagnostics (GOLD) Event Detector on that product.

Note	GOLD is not supported on the Cisco ASR 1000 Series routers.

• The event gold command was enhanced with the addition of the action-notify, testing-type, test-name, test-id, consecutive-failure, platform-action, and maxrun keywords for improved reaction to GOLD test failures and conditions.
• The following platform-wide GOLD Event Detector information can be accessed through new read-only EEM built-in environment variables:
  • Boot-up diagnostic level
  • Card index, name, serial number
  • Port counts
  • Test counts
• The following test-specific GOLD Event Detector information can be accessed through new read-only EEM built-in environment variables (available to EEM applets only):
  • Test name, attribute, total run count
  • Test result per test, port, or device
  • Total failure count, last fail time
  • Error code
  • Occurrence of consecutive failures

These enhancements result in reduced mean time to recovery (MTTR) and higher availability through improved automation and fault detection.

EEM 2.4 introduced the following event detectors:

• SNMP Notification--The SNMP notification event detector provides the ability to intercept SNMP trap and inform messages coming into the router. An SNMP notification event is generated when an incoming SNMP trap or inform message matches specified values or crosses specified thresholds.
• RPC--The remote procedure call (RPC) event detector provides the ability to invoke EEM policies from outside the router over an encrypted connection using Secure Shell (SSH). The RPC event detector uses Simple Object Access Protocol (SOAP) data encoding for exchanging XML-based messages. This event detector can be used to run EEM policies and then receive output in a SOAP XML-formatted reply.

EEM 2.4 added enhancements to the following event detectors:

• Interface counter rate-based trigger--This feature adds the ability for an interface event to be triggered based on a rate of change over a period of time. A rate can be specified both for the entry value and the exit value. This feature copies the rate-based functionality that currently exists for the SNMP event detector.
• SNMP delta value--The difference between the monitored Object Identifier (OID) value at the beginning of the monitored period and the actual OID value when the event is published will be provided in the event_reqinfo data for both the SNMP event detector and the Interface Counter event detector.

EEM 2.4 introduced the following actions:

• Multiple event support--The ability to run multiple events was introduced. and show event manager commands were enhanced to show multiple events.
• Support for parameters--The parameter argument has been added to the event manager run command. A maximum of 15 parameters can be used.
• Display of Job IDs and completion status--Some of the show event manager commands were enhanced to display Job IDs and completion status.
• Bytecode support--Tcl 8 defines a specialized bytecode language (BCL) and includes a just-in-time compiler that translates Tcl scripts to BCL. Byte sequence is executed by a "virtual machine," Tcl_ExecuteByteCode(), or TEBC for short, as often as needed. Currently EEM accepts file extensions, such as *.tcl for user policies and *.tm for system policies. Tcl standard extension for bytecode scripts are *.tbc. Now EEM will accept *.tbc as valid EEM policies.
• Registration substitution enhancement--Supports replacing multiple parameters in the event registration statement lines with a single environment variable.
• Tcl package support

EEM 3.0 introduces the following new event detectors:

• Custom CLI--The custom CLI event detector publishes an event to add and enhance existing CLI command syntax.
• Routing--The Routing event detector publishes an event when route entries change in the Routing Information Base (RIB).
• NetFlow-- The NetFlow event detector publishes an event when a NetFlow event is triggered.
• IP SLA--The IP SLA event detector publishes an event when an IP SLA reaction is triggered.

EEM 3.0 introduces the following features.

• Class-based scheduling--The EEM policies will be assigned a class using the class keyword when they are registered. EEM policies registered without a class will be assigned to the default class.
• High performance Tcl policies--Three new Tcl commands are introduced event_completion, event_wait, and event_completion_with_wait.
• Interactive cli support--The synchronous applets are enhanced to support interaction with the local console (TTY). Two new IOS commands, action gets and action puts, are introduced to allow users to enter and display input directly on the console.
• Variable logic for applets--The Variable Logic for EEM Applets feature adds the ability to apply conditional logic within EEM applets. Conditional logic introduces a control structure that can change the flow of actions within applets depending on conditional expressions.
• Digital signature support--A new API performs digital signature verification for a Tcl script to check it the script is signed by Cisco before execution.
• Support authenticating e-mail servers--The action mailcommand is modified to include an optional username and password.
• SMTP IPv6 support--The keyword sourceaddr is added in Tcl e-mail templates to specify either an IPv6 or IPv4 address.
• SNMP library extensions--The EEM applet action info and Tcl sys_reqinfo_snmpcommands are enhanced to include functionality for SNMP getid, inform, trap, and set-type operations.
• SNMP Notification IPv6 support--IPv6 address is supported for the source and destination IP addresses.
• CLI Library XML-PI support--Provides a programmable interface which encapsulates IOS command-line interface (CLI) show commands in XML format in a consistent way across different Cisco products. Customers using XML-PI will be able to parse IOS show command output from within Tcl scripts using well-known keywords instead of having to depend on the use of regular expression support.

EEM 3.1 introduced one new event detector:

• SNMP Object--The Simple Network Management Protocol (SNMP) object trap event detector provides an extension to replace the value when an SNMP trap with the specified SNMP object ID (OID) is encountered on a specific interface or address.

EEM 3.1 added an enhancement to the following event detector:

• SNMP Notification--The SNMP notification event detector now can wait and intercept the outgoing SNMP traps and informs.

EEM 3.1 added enhancement to the following action:

• Specify facility--The action syslog command has been enhanced to specify syslog facility.

EEM 3.1 introduces the following features:

• Provides the ability to create a short description for the registered policy--A new description command has been introduced to register policies with a brief description in Cisco IOS CLI and Tcl policies. The show event manager policy available command and the show event manager policy registered command have been enhanced to add the description keyword to display the description of the registered applet.
• Enables EEM policies to bypass AAA authorization--The event manager application command has been enhanced to provide authorization and bypass keywords to disable AAA.
• Introduces CLI Library enhancements--Provides two new commands in the CLI library: cli_run and cli_run_interactive.

EEM 3.2 introduced the following new event detectors:

• Neighbor Discovery--Neighbor Discovery event detector provides the ability to publish a policy to respond to automatic neighbor detection when:
  • a Cisco Discovery Protocol (CDP) cache entry is added, deleted or updated.
  • a Link Layer Discovery Protocol (LLDP) cache entry is added, deleted, or updated.
  • an interface link status changes.
  • an interface line status changes.
• Identity--Identity event detector generates an event when AAA authorization and authentication is successful, when failure occurs, or after normal user traffic on the port is allowed to flow.
• Mac-Address-Table--Mac-Address-Table event detector generates an event when a MAC address is learned in the MAC address table.

Note

The Mac-Address-Table event detector is supported only on switch platforms and can be used only on Layer 2 interfaces where MAC addresses are learned. Layer 3 interfaces do not learn addresses and routers do not usually support the mac-address-table infrastructure needed to notify EEM of a learned MAC address.

EEM 3.2 also introduces new CLI commands to support the applets to work with the new event detectors.

EEM uses software programs known as event detectors to determine when an EEM event occurs. Some event detectors are available on every Cisco IOS XE software release, but most event detectors have been introduced in a specific release. Use the table below to determine which event detectors are available in your specific Cisco IOS release. A blank entry (--) indicates that the event detector is not available: the text "Yes" indicates that the event detector is available. The event detectors shown in the table below are supported in later releases of the same Cisco IOS XE software release train. For more details on each event detector, see the Event Detectors.

Embedded Event Manager (EEM) uses software programs known as event detectors to determine when an EEM event occurs. Event detectors are separate systems that provide an interface between the agent being monitored, for example Simple Network Management Protocol (SNMP), and the EEM policies where an action can be implemented. EEM contains the following event detectors.

Watchdog System Monitor (IOSWDSysMon) Event Detector for Cisco IOS XE

The Cisco IOS XE watchdog system monitor event detector publishes an event when one of the following occurs:

• CPU utilization for a Cisco IOS XE task crosses a threshold.
• Memory utilization for a Cisco IOS XE task crosses a threshold.

Note	Cisco IOS XE processes are now referred to as tasks to distinguish them from Cisco IOS XE Software Modularity processes.

Two events may be monitored at the same time, and the event publishing criteria can be specified to require one event or both events to cross their specified thresholds.

The CLI-based corrective actions that are taken when event detectors report events enable a powerful on-device event management mechanism. Some EEM actions are available on every Cisco IOS XE release, but most EEM actions have been introduced in a specific release. For details of which EEM action is supported in each Cisco IOS XE release, see the EEM Actions Available by Cisco IOS XE Release concept in the "Writing Embedded Event Manager Policies Using the Cisco IOS XE CLI" or the "Writing Embedded Event Manager Policies Using Tcl" modules. EEM supports the following actions:

• Executing a Cisco IOS XE command-line interface (CLI) command.
• Setting or modifying a named counter.
• Requesting system information when an event occurs.
• Sending a short e-mail.
• Manually running an EEM policy.
• Publishing an application-specific event.
• Generating an SNMP trap.

EEM action CLI commands contain an EEM action label that is a unique identifier that can be any string value. Actions are sorted and run in ascending alphanumeric (lexicographical) key sequence using the label as the sort key. If you are using numbers as labels be aware that alphanumerical sorting will sort 10.0 after 1.0, but before 2.0, and in this situation we recommend that you use numbers such as 01.0, 02.0, and so on, or use an initial letter followed by numbers.

EEM allows environment variables to be used in EEM policies. Tool Command Language (Tcl) allows global variables to be defined that are known to all procedures within a Tcl script. EEM allows environment variables to be defined using a CLI command, the event manager environment command, for use within an EEM policy. All EEM environment variables are automatically assigned to Tcl global variables before a Tcl script is run. There are three different types of environment variables associated with Embedded Event Manager:

• User-defined--Defined by you if you create an environment variable in a policy that you have written.
• Cisco-defined--Defined by Cisco for a specific sample policy.
• Cisco built-in (available in EEM applets)--Defined by Cisco and can be read only or read/write. The read only variables are set by the system before an applet starts to execute. The single read/write variable, _exit_status, allows you to set the exit status at policy exit for policies triggered from synchronous events.

Cisco-defined environment variables (see the table below) and Cisco system-defined environment variables may apply to one specific event detector or to all event detectors. Environment variables that are user-defined or defined by Cisco in a sample policy are set using the event manager environment command. Variables that are used in the EEM policy must be defined before you register the policy. A Tcl policy contains a section called "Environment Must Define" that can be defined to check that any required environment variables are defined before the policy runs.

Cisco built-in environment variables are a subset of the Cisco-defined environment variables and the built-in variables are available to EEM applets only. The built-in variables can be read-only or can be read and write, and these variables may apply to one specific event detector or to all event detectors. For more details and a table listing the Cisco system-defined variables, see the ""Writing Embedded Event Manager Policies Using the Cisco IOS XE CLI" module.

Note	Cisco-defined environment variables begin with an underscore character (_). We strongly recommend that customers avoid the same naming convention to prevent naming conflicts.

The table below describes the Cisco-defined variables used in the sample EEM policies. Some of the environment variables do not have to be specified for the corresponding sample policy to run and these are marked as optional.

EEM is a policy driven process in which the EEM policy engine receives notifications when faults and other events occur in the Cisco IOS XE software system. Embedded Event Manager policies implement recovery based on the current state of the system and the actions specified in the policy for a given event. Recovery actions are triggered when the policy is run.

Although there are some EEM CLI configuration and show commands, EEM is implemented through the creation of policies. An EEM policy is an entity that defines an event and the actions to be taken when that event occurs. There are two types of EEM policies: an applet or a script. An applet is a simple form of policy that is defined within the CLI configuration. A script is a form of policy that is written in Tcl.

The creation of an EEM policy involves:

• Selecting the event for which the policy is run.
• Defining the event detector options associated with logging and responding to the event.
• Defining the environment variables, if required.
• Choosing the actions to be performed when the event occurs.

There are two ways to create an EEM policy. The first method is to write applets using CLI commands, and the second method is to write Tcl scripts. Cisco provides enhancements to Tcl in the form of Tcl command extensions that facilitate the development of EEM policies. Scripts are defined off the networking device using an ASCII editor. The script is then copied to the networking device and registered with EEM. When a policy is registered with the Embedded Event Manager, the software examines the policy and registers it to be run when the specified event occurs. Policies can be unregistered or suspended. Both types of policies can be used to implement EEM in your network.

For details on writing EEM policies using the Cisco IOS XE CLI, go to "Writing Embedded Event Manager Policies Using the Cisco IOS XE CLI" module.

For details on writing EEM policies using Tcl, go to "Writing Embedded Event Manager Policies Using Tcl" module.