This is the unfiltered summary timeseries data for the current Scope for the entire corpus. The purpose of this component is to allow you to know what date range is being viewed, and easily change that date range by dragging within the component. The data in the chart is there in case it’s useful for deciding which time range to select. You can select different metrics to be shown, by default the count of flow observations is shown.

The Corpus Selector can currently support selecting up to approximately 2 billion flow observations.

This is where you define filters to narrow-down the search results. Click the (?) icon next to the word Filters for all possible dimensions. For any User Labels data, those columns will also be available for the appropriate intervals. This input also supports and, or, not, and parenthesis keywords, use these to express more complex filters. For example, a direction-agnostic filter between IP 1.1.1.1 and 2.2.2.2 can be written:

Consumer Address = 1.1.1.1 and Provider Address = 2.2.2.2 or Consumer Address = 2.2.2.2 and Provider Address = 1.1.1.1

And to additionally filter on Protocol = TCP:

(Consumer Address = 1.1.1.1 and Provider Address = 2.2.2.2 or Consumer Address = 2.2.2.2 and Provider Address = 1.1.1.1) and Protocol = TCP

The filter input also supports “,” and “-” for Port, Consumer Address and Provider Address, by translating “-” into range queries. The following are examples of a valid filter:

Note

Because flow data is labeled with User Labels only at ingestion time, User Labels will not appear immediately after enabling them. It may take a few minutes before the labels start appearing in Flow Search. Also, the available User Labels will be different depending on which part of the Corpus Selector you have selected, since the enabled Labels might have been changed at various times.

This component displays the aggregated totals of various metrics for the interval selected (the selection made in the above Corpus Selector). Use the dropdown to change which metric is displayed.

Further-narrowing of the selected interval can also be done in this component. Click the area of the chart that you’d like to focus on, and the Top N Charts and the data below will all be updated to include only data from that selected interval.

The charts display the Top N values that contribute to the selection in the Filtered Time series chart to the left. Selecting a peak in Flow Observations in the time series chart, and hostnames in the Top N charts, displays the list of hostnames (Consumer and Provider) that contribute the most to those flow observations. Also, if the time series chart is set to display SRTT, then the Top Hostnames display those that contribute most to that selected SRTT.

Click on any of the items in the Top N charts to display a menu that allows you to either Drill-Down or Exclude that value.

• Click Drill-Down to add a filter that confines the results to just that value.

• Click Exclude to add a filter that excludes that value from the results.

Note	After clicking Drill-Down or Exclude, the Filter icon must be pressed for the filter to take effect. This is so that multiple Exclude actions can be taken quickly without having the page repeatedly update in the middle.

This is the list of actual Flow Observations that matches the filters and selections in the page above. By default, 20 will be loaded starting from the beginning of the interval. It’s possible to increase the number that are loaded by using the dropdown. It’s also possible to load a random set of flow observations from the selected interval by using Sampled rather than In order. The Sampled setting is useful for getting a more representative set of flow observations from the selected interval rather than loading them sequentially from the beginning of the interval.

Click Explore Observations to enable a chart view that allows quick exploration of the high-dimensional data (Parallel Coordinates chart). A bit overwhelming at first, this chart is useful when enabling only the dimensions you’re interested in (by unchecking items in the Dimensions dropdown), and when rearranging the order of the dimensions. A single line in this chart represents a single observation, and where that line intersects with the various axes indicates the value of that observation for that dimension. This can become clearer when hovering over the list of observations below the chart to see the highlighted line representing that observation in the chart:

Due to the high-dimensional nature of the flow data, this chart is wide by default, and requires scrolling to the right to see the entire chart. For this reason, it’s useful to disable all but the dimensions you’re interested in.

Sampling vs. In-Order

It’s recommended that Explore Observations be done with sampling enabled, and with a larger number of flows. This allows you to see more of the variety of flows that comprise the selected interval. So, if you’ve selected 2 million flow observations in the time series chart above, loading a sample of 1000 will take uniformly from throughout the interval, whereas loading flows In-order will load the first 1000 flow observations from the very beginning of the interval:

Notice how the Timestamp for all of the in-order observations is from 9:09 and how the observations are evenly distributed through the selected interval in the sampled version.

Filtering

Dragging the cursor along any of the axes create a selection that shows only observations that match that selection. Click again on the axis to remove the selection at any time. Selections can be made on any number of axes at a time. The list of observations updates to show only the selected observations:

Flow direction (client/server or provider/consumer classification) is important for visibility, automatic policy discovery, and enforcement. Every unicast flow has a client and a server classification.

For example, if there are clients (192.168.1.1-192.168.1.3) accessing a web server (192.168.2.1) using https, typically the source port is an ephemeral port in the range 1025-65535 and the destination port is 443.

The accurate client-server direction is:

• Client: 192.168.1.1-3

• Server: 192.168.2.1

• Services: TCP port 443

Policies generated by automatic policy discovery are shown in the figure (with left endpoints grouped):

Now, if the client - server direction decision is reversed (an inaccurate classification), that is:

• Client: 192.168.2.1

• Server: 192.168.1.1-3

• Services: the list of ephemeral ports (45680, 51112, 45553)

Then, in the above inaccurate classification, the policies generated may be as shown in the figure:

This consumes more resources in terms of policy enforcement. In addition, depending on how you enforce the policy, even though 192.168.1.1-3 uses these ephemeral ports, they can’t access 192.168.2.1. For example, if you use Secure Workload software sensor enforcement, the enforcement policy for Client to Web above (ESTAB) doesn’t match with traffic generated by Client destined to Web (NEW, ESTAB).

Timestamps and TCP flags are used in Secure Workload to determine the client-server direction. If there are no TCP flags information (SYN, SYN/ACK) because, for example, the packets could be UDP/ICMP or an HW sensor is used that doesn’t support direction signals, then user-defined override rules, timestamps, and other heuristics are used to infer the flow direction. Heuristics by definition don’t guarantee 100% accuracy. Client-server accuracy is a function of the type of sensor used and the conditions in which sensors are used. You can use Secure Workload’s REST-API (OpenAPI) to insert client-server override rules to identify the server ports for those flow types that Secure Workload gets the direction wrong. Then allow Secure Workload to process new flow data captured with those rules in place, and then generate the policies over the time duration when the flow direction were fixed. For more details on the API to specify override rules, refer to: Client Server configuration. You can also manually define the policies and examine/remove the undesired policies. See Policies.

Secure Workload supports the following flow analysis fidelity modes:

• Detailed Mode: Historically, the Detailed Mode was the only mode available, where every observed flow was reported by the agent along with detailed stats about the observed flow. Stats captured are packet and byte counts, TCP flags, connection stats, network latency, SRTT, and others. While this kind of reporting is desirable in numerous cases, it is computationally intensive to report and process, also, it may not be strictly required when the primary use case is segmentation only.

• Conversations Mode: The Conversations Mode offers a more lightweight alternative to the traditional detailed mode. Agents in conversations mode aim to report conversations as opposed to flows whenever possible (i.e, whenever they are able to make the client-server classification accurately). This is applicable to TCP, UDP, and ICMP flows.

In detailed mode, for TCP/UDP flows, we report 5-tuple flows {source and destination IP, source and destination port, and protocol}. While for conversations mode, the agent omits the source port as they are ephemeral ports {changes on every new connection}, making it a 4-tuple flow.

Note	Detecting a flow as 4-tuple also depends on client-server detection algorithms, which relies on the server/destination port being a well-known port (0–1023).

Thus, if you’re using a custom application which doesn’t use well-known server/destination ports, the OpenAPI interface can be used to punch well-known server ports. These configs are not applied to past flows, and only affect markings on flows from that point on (that is, going forward). To optimize server ports, see Client Server Configuration.

Agent reports in conversations mode contain trimmed down information, full list of omitted fields includes:

• TCP/UDP source port (ephemeral ports)

• Fwd/Rev TCP bottleneck

• TCP handshake bucket

• SRTT(µs)

• Fwd/Rev Packet retransmissions

• SRTT Available

• Fwd/Rev Congestion Window Reduced

• Fwd/Rev MSS Changed

• Fwd/Rev TCP Rcv Window Zero? Fwd/Rev Burst Indicator

• Fwd/Rev Max Burst Size (KB)

To enable conversations mode, see the Flow Visibility config section in: Software Agent Config

Note	The exact benefit gained by changing agents to report in conversation mode may vary due to multiple factors, including, but not limited to percentage of TCP flows, number of services listening on well-known service ports, and memory limitations at the agent.

Note	After turning on conversations mode for some agents, there may be a mixture of conversations and flows in the observations on the flow search page.