URL Filtering

URL Filtering Overview

Use the URL filtering feature to control the websites that users on your network can access:

  • Category and reputation-based URL filtering—With a URL Filtering license, you can control access to websites based on the URL’s general classification (category) and risk level (reputation). This is the recommended option.

  • Manual URL filtering—With any license, you can manually specify individual URLs, groups of URLs, and URL lists and feeds to achieve granular, custom control over web traffic. For more information, see Manual URL Filtering.

See also Blocking Traffic with Security Intelligence, a similar but different feature for blocking malicious URLs, domains, and IP addresses.

About URL Filtering with Category and Reputation

With a URL Filtering license, you can control access to websites based on the category and reputation of requested URLs:

  • Category—A general classification for the URL. For example, ebay.com belongs to the Auctions category, and monster.com belongs to the Job Search category.

    A URL can belong to more than one category.

  • Reputation—How likely the URL is to be used for purposes that might be against your organization’s security policy. Reputations range from High Risk (level 1) to Well Known (level 5).

Benefits of Category and Reputation-Based URL Filtering

URL categories and reputations help you quickly configure URL filtering. For example, you can use access control to block high-risk URLs in the Hacking category. Or, you can use QoS to rate limit traffic from sites in the Streaming Media category. There are also categories for types of threats, such as a Spyware and Adware category.

Using category and reputation data simplifies policy creation and administration. It grants you assurance that the system controls web traffic as expected. Because Cisco continually updates its threat intelligence with new URLs, as well as new categories and risks for existing URLs, the system uses up-to-date information to filter requested URLs. Sites that (for example) represent security threats, or that serve undesirable content, may appear and disappear faster than you can update and deploy new policies.

Some examples of how the system can adapt include:

  • If an access control rule blocks all gaming sites, as new domains get registered and classified as Games, the system can block those sites automatically. Similarly, if a QoS rule rate limits all streaming media sites, the system can automatically limit traffic to new Streaming Media sites.

  • If an access control rule blocks all malware sites and a shopping page gets infected with malware, the system can recategorize the URL from Shopping to Malware Sites and block that site.

  • If an access control rule blocks high-risk social networking sites and somebody posts a link on their profile page that contains links to malicious payloads, the system can change the reputation of that page from Benign Sites to High Risk and block it.

Best Practices for URL Filtering

Keep in mind the following guidelines and limitations for URL filtering:

Filter by Category and Reputation

Follow the instructions in How to Configure URL Filtering with Category and Reputation.

Configure Your Policy to Inspect Packets That Must Pass Before a URL Can Be Identified

The system cannot filter URLs before:

  • A monitored connection is established between a client and server.

  • The system identifies the HTTP or HTTPS application in the session.

  • The system identifies the requested URL (for encrypted sessions, from the ClientHello message or the server certificate).

This identification should occur within 3 to 5 packets, or after the server certificate exchange in the TLS/SSL handshake if the traffic is encrypted.

Important! To ensure that your system examines these initial packets that would otherwise pass, see Inspection of Packets That Pass Before Traffic Is Identified and subtopics.

If early traffic matches all other rule conditions but identification is incomplete, the system allows the packet to pass and the connection to be established (or the TLS/SSL handshake to complete). After the system completes its identification, the system applies the appropriate rule action to the remaining session traffic.

URL Conditions and Rule Order

  • Position URL rules after all other rules that must be hit.

  • URLs can belong to more than one category. It is possible to want to allow one category of websites and block another—whether explicitly or by relying on the default action. In this case, make sure you create and order URL rules so you get the desired effect, depending on whether the allow or the block should take precedence.

For additional guidelines for rules, see the following topics: Best Practices for Access Control Rules and Rule Condition Mechanics.

Uncategorized or Reputationless URLs

When you build a URL rule, you first choose the category you want to match. If you explicitly choose Uncategorized URLs, you cannot further constrain by reputation.

You cannot manually assign categories and reputations to URLs, but in access control and QoS policies, you can manually block specific URLs. See Manual URL Filtering.

URL Filtering for Encrypted Web Traffic

When performing URL filtering on encrypted web traffic, the system:

  • Disregards the encryption protocol; a rule matches both HTTPS and HTTP traffic if the rule has a URL condition but not an application condition that specifies the protocol.

  • Does not use URL lists. You must use URL objects and groups instead.

  • Matches HTTPS traffic based on the subject common name in the public key certificate used to encrypt the traffic, and also evaluates the reputation of any other URLs presented at any time during the transaction, including the post-decryption HTTP URL.

  • Disregards subdomains within the subject common name.

  • Does not display an HTTP response page for encrypted connections blocked by access control rules (or any other configuration); see Limitations to HTTP Response Pages.

HTTP/2

The system can extract HTTP/2 URLs from TLS certificates, but not from a payload.

Manual URL Filtering

  • Specify URLs using a custom Security Intelligence list or feed object. Do not use a URL object or directly enter a URL into the rule. For details, see Manual URL Filtering Options.

  • If you manually filter specific URLs using URL objects or by entering URLs directly into the rule, carefully consider other traffic that might be affected. To determine whether network traffic matches a URL condition, the system performs a simple substring match. If the requested URL matches any part of the string, the URLs are considered to match.

  • If you use manual URL filtering to create exceptions to other rules, position the specific rule with the exceptions above the general rule that would otherwise apply.

Search Query Parameters in URLs

The system does not use search query parameters in the URL to match URL conditions. For example, consider a scenario where you block all shopping traffic. In that case, using a web search to search for amazon.com is not blocked, but browsing to amazon.com is.

URL Filtering in High Availability Deployments

For guidelines for URL filtering with Firepower Management Centers in high availability, see URL Filtering and Security Intelligence.

Memory Limitations for Selected Device Models

  • If you are using NGIPSv, see the Cisco Firepower NGIPSv Quick Start Guide for VMware for information on allocating the correct amount of memory to perform category and reputation-based URL filtering.

  • Device models with less memory store less URL data locally, and the system may therefore check the cloud more frequently to determine category and reputation for sites that are not in the local database.

    Lower-memory devices include:

    • Virtual FTD (FTDv) with 8 GB of RAM

    • ASA 5506-X, ASA 5506H-X, ASA 5506W-X, ASA 5508-X and ASA 5516-X

    • ASA 5512-X, ASA 5515-X and ASA 5525-X

    • 7100 series

Filtering HTTPS Traffic

To filter encrypted traffic, the system determines the requested URL based on information passed during the TLS/SSL handshake: the subject common name in the public key certificate used to encrypt the traffic.

HTTPS filtering, unlike HTTP filtering, disregards subdomains within the subject common name. Do not include subdomain information when manually filtering HTTPS URLs in access control or QoS policies. For example, use example.com rather than www.example.com.

HTTPS filtering also does not support URL lists. You must use URL objects and groups instead.


Tip

In an SSL policy, you can handle and decrypt traffic to specific URLs by defining a distinguished name SSL rule condition. The common name attribute in a certificate’s subject distinguished name contains the site’s URL. Decrypting HTTPS traffic allows access control rules to evaluate the decrypted session, which improves URL filtering.

Controlling Traffic by Encryption Protocol

The system disregards the encryption protocol (HTTP vs HTTPS) when performing URL filtering in access control or QoS policies. This occurs for both manual and reputation-based URL conditions. In other words, URL filtering treats traffic to the following websites identically:

  • http://example.com/

  • https://example.com/

To configure a rule that matches only HTTP or HTTPS traffic, add an application condition to the rule. For example, you could allow HTTPS access to a site while disallowing HTTP access by constructing two access control rules, each with an application and URL condition.

The first rule allows HTTPS traffic to the website:

  • Action: Allow
  • Application: HTTPS
  • URL: example.com

The second rule blocks HTTP access to the same website:

  • Action: Block
  • Application: HTTP
  • URL: example.com

License Requirements for URL Filtering

FTD License

  • Category and reputation filtering—URL Filtering

  • Manual filtering—No additional license.

Classic License

  • Category and reputation filtering—URL Filtering

  • Manual filtering—No additional license.

Requirements and Prerequisites for URL Filtering

Model Support

Any

Supported Domains

Any

User Roles

  • Admin

  • Access Admin

  • Network Admin

How to Configure URL Filtering with Category and Reputation

Do This

More Information

Step

If you will use category and reputation-based URL filtering on an NGIPSv device, allocate the required amount of memory.

Cisco Firepower NGIPSv Quick Start Guide for VMware

Step

Ensure that you have the correct licenses.

Licensing the Firepower System, including: .

Assign the URL Filtering license to each managed device that will filter URLs.

In order to enable the feature, at least one managed device must have a URL Filtering license assigned to it.

Step

Ensure that your Firepower Management Center can communicate with the cloud to obtain URL filtering data.

Internet Access Requirements and Communication Port Requirements.

Step

Understand limitations and guidelines and take any necessary actions.

Best Practices for URL Filtering

Step

Enable the URL Filtering feature.

Enable URL Filtering Using Category and Reputation

Step

Configure rules to filter URLs by category and reputation.

Configuring URL Conditions

For the best protection against malicious sites, you must block sites by reputation AND block URLs in all Threat categories.

(Optional) Supplement or Selectively Override Category and Reputation-Based URL Filtering

Step

(Optional) Allow users to bypass a website block by clicking through a warning page.

HTTP Response Pages and Interactive Blocking

Step

Order your rules so that traffic hits key rules first.

URL Rule Order

Step

(Optional) Modify advanced options related to URL filtering.

Generally, use the defaults unless you have a specific reason to change them.

For information about advanced options, including the following, see Access Control Policy Advanced Settings.

  • Maximum URL characters to store in connection events

  • Allow an Interactive Block to bypass blocking for (seconds)

  • Retry URL cache miss lookup

Step

Deploy your changes.

Deploy Configuration Changes

Step

Be sure you have enabled other Firepower features that protect your network from malicious sites

See Blocking Traffic with Security Intelligence.

Enable URL Filtering Using Category and Reputation

You must be an Admin user to perform this task.

Before you begin

Complete prerequisites described in How to Configure URL Filtering with Category and Reputation.

Procedure

Step 1

Choose System > Integration.

Step 2

Click Cisco CSI.

Step 3

Configure URL Filtering Options.

Step 4

Click Save.

URL Filtering Options

The following options are on the System > Integration page:

Enable URL Filtering

Allows traffic filtering based on a website’s general classification, or category, and risk level, or reputation. Adding a URL Filtering license automatically enables Enable URL Filtering. URL filtering must be enabled before you can choose other URL filtering options.

When you enable URL filtering, depending on how long since URL filtering was last enabled, or if this is the first time you are enabling URL filtering, the Firepower Management Center downloads URL data from Cisco Collective Security Intelligence (Cisco CSI). This process may take some time.

Enable Automatic Updates

Options for updating URL filtering threat data:

  • If you enable the Enable Automatic Updates option on the System > Integration page, the Firepower Management Center checks the cloud every 30 minutes for updates. This option is enabled by default when you add a URL filtering license.

  • If you need strict control over when the system contacts external resources, disable automatic updates on this page and instead create a recurring task using the scheduler. See Automating URL Filtering Updates Using a Scheduled Task.

Update Now

You can perform a one-time, on-demand update by clicking the Update Now button at the top of this dialog box, but you should also either enable automatic updates or create a recurring task using the scheduler. You cannot start an on-demand update if an update is already in progress.

Although daily updates tend to be small, if it has been more than five days since your last update, new URL data may take up to 20 minutes to download, depending on your bandwidth. Then, it may take up to 30 minutes to perform the update itself.

Query Cisco CSI for Unknown URLs

Allows the system to submit URLs to the cloud for threat intelligence evaluation when users browse to a website whose category and reputation are not in the local dataset. Disable this option if you do not want to submit your uncategorized URLs, for example, for privacy reasons.

Connections to uncategorized URLs do not match rules with category or reputation-based URL conditions. You cannot assign categories or reputations to URLs manually.

Configuring URL Conditions

Protect your network by controlling access to sites based on URL category and reputation.


Caution

Adding the first or removing the last URL or Category category/reputation condition in an access control or SSL (but not a QoS) rule restarts the Snort process when you deploy configuration changes, temporarily interrupting traffic inspection. Whether traffic drops during this interruption or passes without further inspection depends on how the target device handles traffic. See Snort® Restart Traffic Behavior for more information.

Procedure

Step 1

In the rule editor, click the following for URL conditions:

  • Access control or QoS—Click URLs.
  • SSL—Click Category.

Step 2

Find and choose the URL categories that you want to control:

In an access control or QoS rule, click Category.

Step 3

(Optional) Constrain URL categories by choosing a Reputation.

Note that if you explicitly match Uncategorized URLs, you cannot further constrain by reputation, because uncategorized URLs do not have reputations. Choosing a reputation level also includes other reputations either more or less severe than the level you choose, depending on the rule action:

  • Includes less severe reputations—If the rule allows or trusts web traffic. For example, if you configure an access control rule to allow Benign Sites (level 4), it also automatically allows Well Known (level 5) sites.

  • Includes more severe reputations—If the rule rate limits, decrypts, blocks, or monitors web traffic. For example, if you configure an access control rule to block Suspicious Sites (level 2), it also blocks High Risk (level 1) sites.

If you change the rule action, the system automatically changes the reputation levels in URL conditions.

Step 4

Click Add to Rule, or drag and drop.

Step 5

Save or continue editing the rule.

Example: URL Condition in an Access Control Rule

The following graphic shows the URL condition for an access control rule that blocks all malware sites, all High Risk sites, and all non-benign social networking sites.

Screenshot of a sample URL condition

The following table summarizes how you build the condition.

Blocked URL

Category

Reputation

Malware sites, regardless of reputation

Malware Sites

Any

Any URL with a high risk (level 1)

Any

1 - High Risk

Social networking sites with a risk greater than benign (levels 1 through 3)

Social Network

3 - Benign sites with security risks

What to do next

Rules with URL Conditions

The following table lists rules that support URL conditions, and the types of filtering that each rule type supports.

Rule Type

Supports Category and Reputation Filtering?

Supports Manual Filtering?

Access control

Yes

Yes

SSL

Yes

No; use distinguished name conditions instead

QoS

Yes

Yes

URL Rule Order

For the most effective URL matching, place rules that include URL conditions before other rules, particularly if the URL rules are block rules and the other rules meet both of the following criteria:

  • They include application conditions.

  • The traffic to be inspected is encrypted.

If you configure exceptions to a rule, put the exception above the other rule.

Manual URL Filtering

In access control and QoS rules, you can supplement or selectively override category and reputation-based URL filtering by manually filtering individual URLs, groups of URLs, or URL lists and feeds.

For example, you might use access control to block a category of websites that are not appropriate for your organization. However, if the category contains a website that is appropriate, and to which you want to provide access, you can create a manual Allow rule for that site and place it before the Block rule for the category.

You can perform this type of URL filtering without a special license.

Manual URL filtering is not supported in SSL rules; instead, use distinguished name conditions.


Caution

Depending on how you implement manual URL filtering, URL matching may not be what you intend. See Manual URL Filtering Options.

Manual URL Filtering Options

There are several ways to specify URLs for manual URL filtering:

Figure 1. Manual URL Filtering Options in an Access Control Rule
manual-url-filtering-options-in-AC-rule

Option

Description

(Best practice)

Use custom Security Intelligence URL list or feed objects.

This is the New URL List option on the rule page in the web interface.

This is the recommended method for manual URL filtering.

You can create a new list or feed, or choose an existing one from the URLs sub-tab of the URLs tab in an access control or QoS rule.

For more information, see Custom Security Intelligence Lists and Feeds and subtopics.

Use URL objects, individually or as groups. (URL objects are described at URL Objects.)

Or

Enter URLs directly into the access control rule. (The Enter URL option on the rule page in the web interface.)

If you do not include a path (that is, there is no / character in the URL), the match is based on the server’s hostname only. The hostname is considered a match if it comes after the :// separator, or after any dot in the hostname. For example, ign.com matches ign.com and www.ign.com, but it does not match verisign.com.

If you include one or more / character, the entire URL string is used for a substring match, including the server name, path, and any query parameters. However, we recommend that you do not use manual URL filtering to block or allow individual web pages or parts of sites, as servers can be reorganized and pages moved to new paths. Substring matching can also lead to unexpected matches, where the string you include in the URL object also matches paths on unintended servers or strings within query parameters.

The Enter URL option does not support wildcards.

Supplement or Selectively Override Category and Reputation-Based URL Filtering

In access control or QoS rules, you can use Security Intelligence URL lists and feeds to supplement, or to specify exceptions to, your category and reputation-based URL filtering rules.

(In SSL rules, use distinguished name conditions to serve this purpose.)

Before you begin

Procedure

Step 1

Navigate to the access control or QoS policy in which you will define your rule.

Step 2

Create or edit the rule in which you will add your new condition:

  • If you are supplementing a category- or reputation-based URL filtering rule, edit the existing rule.
  • If you are overriding or creating exceptions to a category- or reputation-based URL filtering rule, create a new rule.

Step 3

If you are creating a new rule, configure the rule name, position, action, and other options at the top of the rule.

Important! If the list or feed you are configuring in this procedure contains exceptions to category- or reputation-based rules, put this rule above those rules in the rule order.

Step 4

Click URLs.

Step 5

Click URLs (beside the Category tab.)

Step 6

Select the list or feed you created in the prerequisite to this task.

Step 7

Click Add to Rule.

Step 8

Click Add or continue editing the rule.

What to do next

(Optional) In SSL rules, use distinguished name conditions to configure parallel behavior.

Troubleshoot URL Filtering

Expected URL Category is Missing from the Categories List

The URL filtering feature uses a different set of categories than the Security Intelligence feature; the category that you expect to see may be a Security Intelligence category. To see those categories, look at the URLs tab on the Security Intelligence tab in an access control policy.

Initial Packets Are Passing Uninspected

See Inspection of Packets That Pass Before Traffic Is Identified and subtopics.

How can I find the category and reputation of a particular URL?

Do a manual lookup. See Finding URL Category and Reputation.

Error when attempting a manual lookup: "Cloud Lookup Failure for <URL>"

Make sure the feature is properly enabled. See the prerequisites in Finding URL Category and Reputation.

URL appears to be incorrectly handled based on its URL category and reputation

Problem: The system does not handle the URL correctly based on its URL category and reputation.

Solutions:

  • Verify that the URL category and reputation associated with the URL are what you think they are. See Finding URL Category and Reputation.

  • The following issues may be addressed by settings described in URL Filtering Options, accessible using Enable URL Filtering Using Category and Reputation.

    • The URL cache may hold stale information. Contact Cisco TAC.

    • The local data set may not be updated with current information from the cloud. See information about the Enable Automatic Updates setting in URL Filtering Options.

    • The system may be configured to not check the cloud for current data. See information about the Query Cisco CSI for Unknown URLs setting in URL Filtering Options.

  • Your access control policy may be configured to pass traffic to the URL without checking the cloud. See information about the Retry URL cache miss lookup setting in Access Control Policy Advanced Settings.

  • See also Best Practices for URL Filtering.

  • If the URL is processed using an SSL rule, see SSL Rule Order

  • Verify that the URL is being handled using the access control rule that you think it is being handled by, and that the rule does what you think it does. Consider rule order.

  • Verify that the local URL category and reputation database on the Firepower Management Center is successfully being updated from the cloud and that managed devices are successfully being updated from the Firepower Management Center.

    Status of these processes are reported in the Health Monitor, in the URL Filtering Monitor module. For details, see Health Monitoring.

If you want to immediately update the local URL category and reputation database, go to System > Integration, click Cisco CSI, then click Update Now. For more information, see URL Filtering Options.

A URL category or reputation is not correct

For access control or QoS rules: Use manual filtering, paying careful attention to rule order. See Manual URL Filtering and Configuring URL Conditions.

For SSL rules: Manual filtering is not supported. Instead, use distinguished name conditions.

Events Do Not Include URL Category and Reputation

  • Make sure you have included applicable URL rules in an access control policy, the rules are active, and the policies have been deployed to the relevant devices.

  • URL category and reputation do not appear in an event if the connection is processed before it matches a URL rule.

  • The rule that handles the connection must be configured for URL category and reputation.

  • Even if you have configured URL categories in the Categories tab in an SSL rule, you must also configure the URLs tab in a rule in your access control policy.