FireSIGHT eStreamer Integration Guide

Establishing an Authenticated Connection

Before a client can request data from eStreamer, the client must initiate an SSL-enabled TCP connection with the eStreamer service. When the client initiates the connection, the eStreamer server responds, initiating an SSL handshake with the client. As part of the SSL handshake, the eStreamer server requests the client’s authentication certificate, and verifies that the certificate is valid (signed by the Internal Certifying Authority [Internal CA] on the eStreamer server).

Note Cisco recommends that you also require your client to verify that the certificate presented by the eStreamer server has been signed by a trusted Certifying Authority. This is the Internal CA certificate included in the PKCS#12 file that Cisco provides when you register a new eStreamer client with the Defense Center or managed device. See “Adding Authentication for eStreamer Clients” section for more information.

After the SSL session is established, the eStreamer server performs an additional post-connection verification of the certificate. This includes verifying that the client connection originates from the host specified in the certificate and that the subject name of the certificate contains the appropriate value. If either post-connection check fails, the eStreamer server closes the connection. If necessary, you can configure the eStreamer service so that it does not perform a client host name check (see “eStreamer Service Options” section for more information).

While the client is not required to perform post-connection verification, Cisco recommends that the client perform this verification step. The authentication certificate contains the following field values in the subject name of the certificate:

After the post-connection verification is finished, the eStreamer server awaits a data request from the client.

Requesting Data from eStreamer

Your client performs the following high-level tasks in managing data requests:

• initializing the request session — see Establishing a Session.
• requesting events from the eStreamer event archive — Using Event Stream Requests and Extended Requests to Initiate Event Streaming.
• requesting host data — see Requesting Host Data.
• changing a request — see Changing a Request.

Establishing a Session

The client establishes a session by sending an initial Event Stream request to the eStreamer service.

In this initial message, you can either include data request flags or submit the data requests in a follow-on message. This initial Event Stream request message itself is a prerequisite for all eStreamer requests, whether for event data or for host data. For information about using the Event Stream request message, see Event Stream Request Message Format.

Using Event Stream Requests and Extended Requests to Initiate Event Streaming

The eStreamer service provides two modes of requests for event streaming. Your request can combine modes. In both modes, your client starts the request with an Event Stream request message but sets the request flag bits differently. For details about the Event Stream message format, see Event Stream Request Message Format.

When eStreamer receives an Event Stream request message, it processes the client request as follows:

• If the request message does not set bit 30 in the request flag field, eStreamer begins streaming any events requested by other set bits in the request flag field. For information, see Submitting Event Stream Requests.
• If bit 30 is set in the Event Stream request, eStreamer provides extended request processing. Extended request flags must be sent if this bit is set. For information, see Submitting Extended Requests. Note that eStreamer resolves any duplicate requests. If you request multiple versions of the same data, either by multiple flags or multiple extended requests, the highest version is used. For example, if eStreamer receives flag requests for discovery events version 1 and 6 and an extended request for version 3, it sends version 6.

Submitting Event Stream Requests

Event stream requests use a simple process:

• Your client sends a request message to the eStreamer service with a start date and time and a request flag field that specifies the events and their version level to be included in the data stream.
• eStreamer streams events beginning at the specified time. For information about the streaming protocol, see Accepting Data from eStreamer.

For information on the format and content of the client’s Event Stream request message, see Event Stream Request Message Format.

For information on the event types and versions of events that the client can request, see Table 2-6.

Submitting Extended Requests

If you set bit 30 in the request flags field of an Event Stream Request message, you initiate an extended request, which starts a negotiation with the server. Extended request flags must be sent if this bit is set. For the event types available by extended request, see Table 2-21.

The steps for extended requests are as follows:

• Your client sends an Event Streaming Request message to eStreamer with the request flags bit 30 set to 1, which signals an extended request. See Event Stream Request Message Format for message format details.
• eStreamer answers with a Streaming Information message that advertises the list of services available to the client. For details about the Streaming Information message, see Streaming Information Message Format.
• The client returns a Streaming Request message that indicates the service it wants to use, with a request list of event types and versions available from that service. The request list corresponds to setting bits in the request flag field when making a standard event stream request. For details about how to use the Streaming Request message to request events, see “Sample Extended Request Messages” section.
• eStreamer processes the client’s Streaming Request message and begins streaming the data at the time specified in the message. For information about the streaming protocol, see Accepting Data from eStreamer.

Requesting Host Data

Once you have established a session, you can submit a request for host data at any time. eStreamer generates information for the requested hosts from the FireSIGHT System network map.

Changing a Request

To change request parameters for an established session, the client must disconnect and request a new session.

Accepting Data from eStreamer

Note The eStreamer server does not keep a history of the events it sends. Your client application must check for duplicate events, which can inadvertently occur for a number of reasons. For example, when starting up a new streaming session, the time specified by the client as the starting point for the new session can have multiple messages, some of which may have been sent in the previous session and some of which were not. eStreamer sends all message that meet the specified request criteria. Your application should detect any resulting duplicates.

During periods of inactivity, eStreamer sends periodic null messages to the client to keep the connection open. If it receives an error message from the client or an intermediate host, it closes the connection.

eStreamer transmits requested data to the client differently, depending on the request mode.

Event Stream Requests

If the client submits an event stream request, eStreamer returns data message by message. It may send multiple messages in a row without waiting for a client acknowledgment. At a certain point, it pauses and waits for the client. The client operating system buffers received data and lets the client process it at its own pace.

If the client request includes a request for metadata, eStreamer sends the metadata first. The client should store it in memory to be available when processing the event records that follow.

Extended Requests

If the client submits an extended request, eStreamer queues up messages and sends them in bundles. eStreamer may send multiple bundles in a row without waiting for a client acknowledgment. At a certain point, it pauses and waits for the client. The client operating system buffers received data and lets the client read it off at its own pace.

The client unpacks each bundle, message by message, and uses the lengths of the records and the blocks to parse each message. The overall message length in each message header can be used to calculate when the end of each message has been reached, and the overall bundle length can be used to know when the end of the bundle is reached. The bundle requires no index of its contents to be correctly parsed.

For information about the message bundling mechanism, see “Message Bundle Format” section.

For information about the null message that the client can use for additional flow control, see “Null Message Format” section.

Terminating Connections

The eStreamer server attempts to send an error message before closing the connection. For information on error messages, see “Error Message Format” section.

The eStreamer server can close a client connection for the following reasons:

• Any time sending a message results in an error. This includes both event data messages and the null keep-alive message eStreamer sends during periods of inactivity.
• An error occurs while processing a client request.
• Client authentication fails (no error message is sent).
• eStreamer service is shutting down (no error message is sent).

Your client can close the connection to eStreamer server at any time and should attempt to use the error message format to notify the eStreamer server of the reason.

eStreamer Message Header

All eStreamer messages start with the message header illustrated in the graphic below. The following table explains the fields.

Initial Timestamp

Note Your client application should use the archival timestamp in the Initial Timestamp field when submitting an event stream request, as explained below. This ensures that you do not inadvertently exclude events. Devices transmit data to the Defense Center using a “store and forward” mechanism with transmission delays. If you request events by the generation timestamp assigned by the device that detects it, delayed events may be missed.

When starting a session, a best practice is to start up from the archival timestamp (also known as the “server timestamp”) of the last record in the previous session. It is not a technical requirement but is strongly recommended. Under certain circumstances, if you use the generation timestamp you can inadvertently exclude events from the new streaming session.

To include the archival timestamp in your streamed events, you must set bit 23 in the request flag field.

Note that only time-based events have archival timestamps. Events that eStreamer generates, such as metadata, have zero in this field when extended event headers have been requested with bit 23 set.

Request Flags

You set bits 0 through 29 in the event data request flag field to select the types of events you want eStreamer to send. You set bit 30 to activate the extended request mode. Setting bit 30 does not directly request any data. Extended request flags must be sent if this bit is set. Your client requests data during the server-client message dialog that follows submission of the Event Stream Request message. For information on extended requests, see Requesting Data from eStreamer.

See Table 2-6 for definitions of the bit settings in the Request Flags field. Different flags request different versions of the event data. For example, to obtain data in FireSIGHT System 4.9 format instead of 4.10 format you set a different flag bit. For specific information on the flags to use when requesting data for particular product versions, see Table 2-7.

Note that you request metadata by version, not by the individual metadata record. For information about each supported version of metadata, see Request Flags.

The following diagram shades the bits in the flags field that are currently used:

For information on each request flag bit, see the following table.

To help you decide which flags to use to request data for a particular version, see the following table. For Version 5.0 and later, see Submitting Extended Requests for more information about using Bit 30.

Caution In all event types, prior to version 5.x, the reference client labels detection engine ID fields as sensor ID.

---

The following example requests intrusion events of type 7 (compatible with FireSIGHT System 3.2+) with both version 1 metadata and packet flags:

To request only data compatible with FireSIGHT System 3.2 (including intrusion events, packets, metadata, impact alerts, policy violation events, and version 2.0 events), use the following:

To request intrusion impact alerts, correlation events, discovery events, connection events, and intrusion events of type 7 with packets and version 3 metadata in Defense Center 4.6.1+ format, use the following:

Understanding the Organization of Event Data Messages

The event data and metadata messages that eStreamer sends contain the following sections:

• eStreamer message header — the standard message header defined at eStreamer Message Header.
• Event-specific sub-headers — sets of fields that vary by event type, with codes that describe additional event details and determine the structure of the payload data that follows.
• Data record — fixed-length fields and a data block.

Note The client should unpack all messages on the basis of field length.

For the event message formats by event type, see the following:

• Intrusion Event and Metadata Message Format for intrusion event data records and all metadata records. These messages have fixed-length fields.
• Discovery Event Message Format for messages with discovery event or user event data. In addition to the standard eStreamer message header and a record header similar to the intrusion event message, discovery messages have a distinctive discovery event header with an event type and subtype field. The data record in discovery event messages is packaged in a series 1 block that can have variable length fields and multiple layers of encapsulated blocks.
• Connection Event Message Format for messages with connection statistics. Their general structure is identical to discovery event messages. Their data block types, however, are specific for connection statistics.
• Correlation Event Message Format for messages with correlation (compliance) event data. The headers in these messages are the same as in intrusion event messages but the data blocks are series 1 blocks.
• Event Extra Data Message Format for a series of messages that deliver intrusion-related record types with variable-length fields and multiple layers of nested data blocks such as intrusion event extra data. See Event Extra Data Message Format for general information on the structure of this message series. See Data Block Header for information about the structures of this series of blocks which are similar to series 1 blocks but numbered separately.

Intrusion Event and Metadata Message Format

The graphic below shows the general structure of intrusion event and metadata messages.

The following graphic shows the details of the record header portion of the intrusion event and metadata message format. The record header fields are shaded. The table that follows defines the fields.

The following table describes each field in the header of intrusion events and metadata messages.

Discovery Event Message Format

The graphic below shows the structure of discovery event messages. The standard eStreamer message header and event record header are followed by a discovery event header used only in discovery and user event messages. The discovery event header section of the message contains the discovery event type and subtype fields, which together form a key to the data block that follows. For the current discovery event types and subtypes, see Table 4-26.

Discovery Event Message Headers

The shaded section in the following graphic shows the fields of the record header in the discovery event data message format, and shows the location of the event header that follows it. The following table defines the fields of the discovery event message headers.

The following table describes the fields in the record header and the event header of the discovery event message.

Connection Event Message Format

Messages with connection statistics have a structure identical to discovery event messages. See Discovery Event Message Format for general message format information. Connection event messages are distinct in terms of the data block types they incorporate.

Correlation Event Message Format

The graphic below shows the general structure of correlation (compliance) event messages. The standard eStreamer message header and record header are followed immediately by a data block in the data record section of the message. Correlation messages use Series 1 data blocks.

Correlation Record Header

The shaded section of the following graphic shows the fields of the record header in correlation event messages. Note that correlation messages use series 1 data blocks; however, they do not have the discovery header that appears in discovery event messages. Their header fields resemble those of intrusion event messages. The table that follows the graphic below defines the record header fields for correlation events.

The following table describes each field in the record header of correlation event messages.

Event Extra Data Message Format

The graphic below shows the structure of event extra data messages. The Intrusion Event Extra Data message is an example of this message group.

Event extra data messages have the same format as correlation event messages, with a data block directly after the record header. Unlike correlation messages, they use series 2 data blocks, not series 1 data blocks, which have a separate numbering sequence. For information about series 2 block types, see Understanding Series 2 Data Blocks.

Event Extra Data Message Record Header

The shaded section of the following graphic shows the fields of the record header in event extra data messages. The table that follows defines the record header fields for event extra data messages.

The following table describes each field in the record header of event extra data messages.

Data Block Header

Series 1 blocks and series 2 blocks have similar structures but distinct numbering. These blocks can appear anywhere in the data portion of a discovery, correlation, connection, or event extra data message. These blocks encapsulate other blocks at multiple levels of nesting.

The data blocks in both the first and second series begin with the header structure shown in the graphic below. The following table provides information about the header fields. The header is followed immediately by the data structure associated with the data block type.