Cisco provides CloudFormation Templates and scripts for deploying an auto-scaling group of ASAv firewalls using several AWS services, including Lambda, auto scaling groups, Elastic Load Balancing (ELB), Amazon S3 Buckets, SNS, and CloudWatch.

The ASAv auto scale in AWS is a complete serverless implementation (i.e. no helper VMs involved in the automation of this feature) that adds horizontal auto scaling capability to ASAv instances in the AWS environment. Starting from version 6.4, the auto scale solution is supported on managed by FMC.

The ASAv auto scale solution is a CloudFormation template-based deployment that provides:

• Completely automated configuration automatically applied to scaled-out ASAv instances.

• Support for Load Balancers and multi-availability zones.

• Support for enabling and disabling the auto scale feature.

The Use Case for this ASAv AWS auto scale Solution is shown in the use case diagram. Because the AWS Load Balancer allows only Inbound-initiated connections, only externally generated traffic is allowed to pass inside via the ASAv firewall.

Note	Secured ports need an SSL/TLS certificate, as described SSL Server Certificate in the Prerequisites.

The Internet-facing load balancer can be a Network Load Balancer or an Application Load Balancer. All of the AWS requirements and conditions hold true for either case. As indicated in the Use Case diagram, the right side of the dotted line is deployed via the ASAv templates. The left side is completely user-defined.

Note	Application-initiated outbound traffic will not go through the ASAv.

Port-based bifurcation for traffic is possible. This can be achieved via NAT rules. For example, traffic on Internet-facing LB DNS, Port: 80 can be routed to Application-1; Port: 88 traffic can be routed to Application-2.

To scale the ASAv instances in and out, an external entity called the Auto Scale Manager monitors metrics, commands an auto scale group to add or delete the ASAv instances, and configures the ASAv instances.

The Auto Scale Manager is implemented using AWS Serverless architecture and communicates with AWS resources and the ASAv. We provide CloudFormation templates to automate the deployment of Auto Scale Manager components. The template also deploys other resources required for complete solution to work.

Note	Serverless auto scale scripts are only invoked by CloudWatch events, hence they only run when an instance is launched.

The following components make up the auto scale solution.

CloudFormation Template

The CloudFormation template is used to deploy resources required by auto scale solution in AWS. The template consists of:

• Auto Scale Group, Load Balancer, Security Groups, and other miscellaneous components.

• The template takes user input to customize the deployment.

  Note	The template has limitations in validating user input, hence it is the user’s responsibility to validate input during deployment.

Lambda Functions

The auto scale solution is a set of Lambda functions developed in Python, which gets triggered from Lifecycle hooks, SNS, CloudWatch event/alarm events. The basic functionality includes:

• Add/Remove Gig0/0, and Gig 0/1 interfaces to instance.

• Register Gig0/1 interface to Load Balancer’s Target Groups.

• Configure and deploy a new ASAv with the ASA configuration file.

Lambda Functions are delivered to customer in the form of a Python package.

Lifecycle Hooks

• Lifecycle hooks are used to get lifecycle change notification about an instance.

• In the case of instance launch, a Lifecycle hook is used to trigger a Lambda function which can add interfaces to an ASAv instance, and register outside interface IPs to target groups.

• In the case of instance termination, a Lifecycle hook is used to trigger a Lambda function to deregister an ASAv instance from the target group.

Simple Notification Service (SNS)

• Simple Notification Service (SNS) from AWS is used to generate events.

• Due to the limitation that there is no suitable orchestrator for Serverless Lambda functions in AWS, the solution uses SNS as a kind of function chaining to orchestrate Lambda functions based on events.

Download the files required to launch the ASAv auto scale for AWS solution. Deployment scripts and templates for your ASA version are available in the GitHub repository.

Attention

Note that Cisco-provided deployment scripts and templates for auto scale are provided as open source examples, and are not covered within the regular Cisco TAC support scope. Check GitHub regularly for updates and ReadMe instructions.

In a cloned/downloaded GitHub repository, the infrastructure.yaml file can be found in the template folder. This CFT can be used to deploy VPCs, subnets, routes, ACLs, security groups, VPC end-points, and S3 buckets with bucket policies. This CFT can be modified to fit your requirements.

The following sections provide more information about these resources and their use in auto scale. You can manually deploy these resources and also use them in auto scale.

Note	The infrastructure.yaml template deploys VPCs, subnets, ACLs, security groups, S3 buckets, and VPC end-points only. It does not create the SSL certificate, Lambda layer, or KMS key resources.

You should create the VPC as required for your application requirements. It is expected that the VPC have an Internet gateway with at least one subnet attached with a route to the Internet. Refer to the appropriate sections for the requirements for Security Groups, Subnets, etc.

Subnets can be created as needed for the requirements of the application. The ASAv machine requires 3 subnets for operation as shown in the Use Case.

Note	If multiple availability zone support is needed, then subnets are needed for each zone as subnets are zonal properties within the AWS Cloud

Outside Subnet

The Outside subnet should have a default route with '0.0.0.0/0' to the Internet gateway. This will contain the Outside interface of the ASAv, and also the Internet-facing NLB will be in this subnet.

Inside Subnet

This can be similar to the Application subnets, with or without NAT/Internet gateway. Please note that for the ASAv health probes, it should be possible to reach the AWS Metadata Server (169.254.169.254) via port 80.

Note

In this AutoScale solution, Load Balancer health probes are redirected to the AWS Metadata Server via inside/Gig0/0 interface. However, you can change this with your own application serving the health probe connections sent to the ASAv from the Load Balancer. In that case, you need to replace the AWS Metadata Server object to the respective application IP address to provide the health probes response.

Management Subnet

This subnet includes the ASAv Management interface. It’s optional for you to have a default route.

Lambda Subnets

The AWS Lambda function requires two subnets having the NAT gateway as the default gateway. This makes the Lambda function private to the VPC. Lambda subnets do not need to be as wide as other subnets. Please refer to AWS documentation for best practices on Lambda subnets.

Application Subnets

There is no restriction imposed on this subnet from the auto scale solution, but in case the application needs Outbound connections outside the VPC, there should be respective routes configured on the subnet. This is because outbound-initiated traffic does not pass through Load Balancers. See the AWS Elastic Load Balancing User Guide.

All connections are allowed in the provided Auto Scale Group template. You need only the following connections for the auto scale Solution to work.

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. You can place all the required files for both the firewall template and the application template in the S3 bucket.

When templates are deployed, Lambda functions get created referencing Zip files in the S3 bucket. Hence the S3 bucket should be accessible to the user account.

If the Internet-facing Load Balancer has to support TLS/SSL, a Certificate ARN is required. Refer to the following links for more information:

• Working with Server Certificates

• Create a Private Key and Self-Signed Certificate for Testing

• Create AWS ELB with Self-Signed SSL Cert (Third-party link)

Example of ARN: arn:aws:iam::[AWS Account]:server-certificate/[Certificate Name]

The autoscale_layer.zip can be created in a Linux environment, such as Ubuntu 18.04 with Python 3.9 installed.

The resultant autoscale_layer.zip file should be copied to the lambda-python-files folder.

This is required if the ASAv passwords are in encrypted format. Otherwise this component is not required. Passwords should be encrypted using only the KMS provided here. If KMS ARN is entered on CFT, then passwords have to be encrypted. Otherwise passwords should be plain text.

For more information about master keys and encryption, see the AWS document Creating keys and the AWS CLI Command Reference about password encryption and KMS.

Example:

$ aws kms encrypt --key-id <KMS-ARN> --plaintext 'MyC0mplIc@tedProtect1oN'
{
    "KeyId": "KMS-ARN", 
    "CiphertextBlob": 
"AQICAHgcQFAGtz/hvaxMtJvY/x/rfHnKI3clFPpSXUU7HQRnCAFwfXhXHJAHL8tcVmDqurALAAAAajBoBgkqhki
G9w0BBwagWzBZAgEAMFQGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM45AIkTqjSekX2mniAgEQgCcOav6Hhol
+wxpWKtXY4y1Z1d0z1P4fx0jTdosfCbPnUExmNJ4zdx8="
}
$

The value of CiphertextBlob key should be used as a password.

A make.py file can be found in the cloned repository top directory. This will Zip the python files into a Zip file and copy to a target folder. In order to do these tasks, the Python 3 environment should be available.

After all of the prerequisites are completed for deployment, you can create the AWS CloudFormation stack.

Use the deploy_autoscale.yaml file in the target directory.

Use the deploy_ngfw_autoscale_with_gwlb.yaml file in the target directory for Geneve Autoscale.

Note

Before you deploy deploy_ngfw_autoscale_with_gwlb.yaml file, you must deploy infrastructure_gwlb.yaml file for AWS GWLB auto scale solution. You must create the Gateway Loadbalancer Endpoint (GWLB-E) by choosing the GWLB that is created during deploy_autoscale_with_gwlb.yaml template deployment. After creating the GWLBe, you must update the default route to use GWLBe for Application Subnet and default Route table. For more information, see https://docs.amazonaws.cn/en_us/vpc/latest/privatelink/create-endpoint-service-gwlbe.html.

Provide the parameters as collected in Input Parameters.

Once the template deployment is successful, you should validate that the Lambda functions and the CloudWatch events are created. By default, the Auto Scale Group has the minimum and maximum number of instances as zero. You should edit the Auto Scale group in the AWS EC2 console with how many instances you want. This will trigger the new ASAv instances.

We recommend that you launch only one instance and check its workflow and validate its behavior as to whether it is working as expected. Post that actual requirements of the ASAv can be deployed, they can also be verified for the behavior. The minimum number of ASAv instances can be marked as Scale-In protected to avoid their removal by AWS Scaling policies.

This topic explains how to suspend and then resume one or more of the scaling processes for your Auto Scale group.

Start and Stop Scale Actions

To start and stop scale out/in actions, follow these steps.

• For AWS Dynamic Scaling—Refer to the following link for information to enable or disable scale out actions:

  Suspending and Resuming Scaling Processes

Every 60 minutes, a CloudWatch Cron job triggers the Auto Scale Manager Lambda for the Health Doctor module:

• If there are unhealthy IPs which belong to a valid ASAv VM, that instance gets deleted if the ASAv is more than an hour old.

• If those IPs are not from a valid ASAv machine, then only IPs are removed from the Target Group.

Disable Health Monitor

To disable a health monitor, in constant.py make the constant as “True”.

Enable Health Monitor

To enable a health monitor, in constant.py make the constant as “False”.

In the unlikely event that Lifecycle hook needs to be disabled, if disabled it won’t add additional interfaces to Instances. It can also cause a series of failed deployment of the ASAv instances.

To disable Auto Scale Manager, respective CloudWatch Events “notify-instance-launch” and “notify-instance-terminate” should be disabled. Disabling this won’t trigger Lambda for any new events. But already executing Lambda actions will continue. There is no abrupt stop of Auto Scale Manager. Trying abrupt stopping by stack deletion or deleting resources can cause an indefinite state.

Because the AWS Load Balancer does not allow instance-type targets for instances having more than one network interface, the Gigabit0/1 interface IP is configured as a target on Target Groups. As of now however, the AWS Auto Scale health checks work only for instance-type targets, not IPs. Also, these IPs are not automatically added or removed from target groups. Hence our Auto Scale solution programmatically handles both of these tasks. But in the case of maintenance or troubleshooting, there could be a situation demanding manual effort to do so.

Register a Target to a Target Group

To register the ASAv instance to the Load Balancer, its Gigabit0/1 instance IP (outside subnet) should be added as a target in Target Group(s). See Register or Deregister Targets by IP Address.

Deregister a Target from a Target Group

To deregister the ASAv instance to the Load Balancer, its Gigabit0/1 instance IP (outside subnet) should be deleted as a target in Target Group(s). See Register or Deregister Targets by IP Address.

AWS does not allow instance reboot in the Auto Scale group, but it does allow a user to put an instance in Stand-by and perform such actions. However, this works best when the Load Balancer targets are instance-type. However, the ASAv machines cannot be configured as instance-type targets, because of multiple network interfaces.

Put an Instance in Stand-by

If an instance is put into stand-by, its IP in Target Groups will still continue to be in the same state until the health probes fail. Because of this, it is recommended to deregister respective IPs from the Target Group before putting the instance into stand-by state; see Deregister a Target from a Target Group for more information.

Once the IPs are removed, see Temporarily Removing Instances from Your Auto Scaling Group.

Remove an Instance from Stand-by

Similarly you can move an instance from stand-by to running state. After removal from stand-by state, the instance's IP should be registered to Target Group targets. See Register a Target to a Target Group.

For more information about how to put instances into stand-by state for troubleshooting or maintenance, see the AWS News Blog.

Remove/Detach Instance from Auto Scale Group

To remove an instance from the Auto Scale group, first it should be moved to stand-by state. See "Put Instances on Stand-by". Once the instance is in the stand-by state it can be removed or detached. See Detach EC2 Instances from Your Auto Scaling Group.

To terminate an instance it should be put into stand-by state; see Instance Stand-by. Once the instance is in stand-by, you can proceed to terminate.

To avoid an accidental removal of any particular instance from the Auto Scale group, it can be made as Scale-In protected. If an instance is Scale-In protected, it won’t be terminated due to a Scale-In event.

Please refer to the following link to put an instance into Scale-In protected state.

https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-instance-termination.html

Important

It is recommended to make the minimum number of instances which are healthy (the target IP should be healthy, not just the EC2 instance) as Scale-In protected.

Any changes in configuration won’t be automatically reflected on already running instances. Changes will be reflected on upcoming devices only. Any such changes should be manually pushed to already existing devices.

If you are facing issues while manually updating the configuration on existing instances, we recommend removing these instances from the Scaling Group and replacing them with new instances.

Change the ASAv Admin Password

A change to the ASAv password requires the user to change it on each device manually for running instances. For new ASAv devices to be onboarded, the ASAv password will be taken from the Lambda environment variables. See Using AWS Lambda Environment Variables.

You can change many things in AWS post deployment, such as the Auto Scale Group, Launch Configuration, CloudWatch events, Scaling Policies etc. You can import your resources into a CloudFormation stack or create a new stack from your existing resources.

See Bringing Existing Resources Into CloudFormation Management for more information about how to manage changes performed on AWS resources.

In order to export CloudWatch logs please refer to Export Log Data to Amazon S3 Using the AWS CLI.