Introduction
This document describes the issue on sessmgr going to WARN state due to huge number of HTTP flows. This issue is reported on Cisco Aggregrated Service Routers (ASR) 5x00.
Problem
Sessmgr status is WARN and high memory utilization.
******** show task resources *******
Thursday July 24 17:44:58 IST 2014
task cputime memory files sessions
cpu facility inst used allc used alloc used allc used allc S status
----------------------- --------- ------------- --------- ------------- ------
4/0 sessmgr 3 26% 100% 1.86G 1.86G 34 500 1766 28160 I warn
These Error logs are generated in the process.There is no subscriber impact due to this error log. As per design once the call is rejected from sessmgr which is in WARN state, system tries at different sessmgrs and call goes through.
[sessmgr 10018 error] [4/0/6812 <sessmgr:3> sessmgr_func.c:44683] [software internal system syslog] Sessmgr-3 full (35200 effective number of calls, 1777 calllines in use, 51146 free flows, 31221 free aaa_sessions, 1777 used-mem-credits, 1777 used-sess-credits, 1948360 mem-usage, 1945600 mem-limit, 0 ecs-queue-usage, 70400 ecs-queue-limit, 16850 ecs-num-flows, 400000 ecs-max-flows, 2334720 ecs-mem-limit[ecs-flow/mem-values:valid], 0x86 limit-flags) - call rejected
Troubleshoot
Capture show support details output and check for the command outputs to troubleshoot further.
The memory problem is related with the amount of flows that the sessmgr handles. The correlation can be seen between sessmgr having high memory consumption and high amount of flows.
******** debug acsmgr show memory usage *******
Thursday July 24 17:50:06 IST 2014
------------------------------------------------------------------------------
! ! Caches Count !
Instance Memory ! Flows ! Callline Data-Session TCP OOO !
! Current Max ! Total Free Total Free Total Free!
--------------------------------------------------------------------------------
1 865.68M 43365 64360 5500 1178 56140 12775 1102 1064
2 852.05M 43879 64767 5500 1178 60150 16271 1102 1067
3 1902.68M 17252 276519 4400 2631 44110 26858 551 541
For affected sessmgrs (and for one unaffected), collect these command outputs, where x is the Sessmgr instance.
show messenger proclet facility sessmgr instance <x> heap
show messenger proclet facility sessmgr instance <x> system heap
task core facility sessmgr instance <x>
show active-charging flows instance <x>
show profile facility sessmgr active depth 8 head 201
show task resources faciltity sessmgr instance <x> max
Check if unoptimized rules and group of ruledefs consume lot of memory.
debug acsmgr show rule-optimization-information
debug acsmgr show grp-of-rdef-optimization-information
The highest memory consumption is due to these functions based on the command outputs.
acs_http_pkt_inspection()
acsmgr_alloc_buffer()
snx_add_dbufs()
sn_aaa_alloc_session_block()
sgx_imsa_bind_user()
You can also check Max No of Simultaneous HTTP Flows attained by Call lines
******** debug acsmgr show flow-stats max-simultaneous-flows http *******
Thursday July 24 17:50:04 IST 2014
Histogram of Max No of Simultaneous HTTP Flows attained by Calllines
No Of Flows No Of Calllines
1 to 10 964712518
11 to 20 384105002
21 to 40 232987189
41 to 100 148938918
101 to 200 115919586
201 to 500 86729303
501 to 1000 69975385
1001 to 2000 59635906
2001 to 5000 50743511
5001 to 10000 44566999
> 10000 1044671491
******** debug acsmgr show flow-stats cumulative http *******
Thursday July 24 17:50:03 IST 2014
Histogram of Total Cumulative HTTP Flows by Calllines
No Of Flows No Of Calllines
1 to 10 964712485
11 to 20 384104980
21 to 40 232987175
41 to 100 148938911
101 to 200 115919583
201 to 500 86729297
501 to 1000 69975377
1001 to 2000 59635907
2001 to 5000 50743509
5001 to 10000 44567004
> 10000 1044671452
You can conclude that there are huge number of HTTP sessions being allocated and this could be due to the heavy HTTP traffic. Also there are almost 1044671491 Calllines, which have greater than 10000 HTTP flows at a time. This leads to high memory usage.
Solution
You have the CLI to limit the number of flows per subscriber
flow limit-across-applications
Cisco would recommend to configure flow limit-across-applications to 5000 as recommended under all affected Rule-bases where huge number of HTTP Traffic can be seen.
This is the procedure to configure the command
In local context under Global configuration.
# active-charging service ECS
(config-acs)# rulebase GOLIVE
(config-rule-base)# flow limit-across-applications 5000
More information about this command.
flow limit-across-applications
This command allows you to limit the total number of simultaneous flows per Subscriber/APN sent to a rulebase regardless of the flow type, or limit flows based on the protocol type under the Session Control feature.
Product:
ACS
Privilege:
Security Administrator, Administrator
Mode:
Exec > ACS Configuration> Rulebase Configuration
active-charging service service_name > rulebase rulebase_name
Entering the above command sequence results in the following prompt:
[local]host_name(config-rule-base)#
Syntax
flow limit-across-applications { limit | non-tcp limit | tcp limit }no flow limit-across-applications [ non-tcp | tcp ] no
If previously configured, deletes the flow limit-across-applications configuration from the current rulebase.
flow limit-across-applications limit
Specifies the maximum number of flows across all applications for the rulebase.
limit must be an integer from 1 through 4000000000.
Default: No limits
non-tcp limit
Specifies the maximum limit of non-TCP type flows.
limit must be an integer from 1 through 4000000000.
Default: No limits
tcp limit
Specifies the maximum limit of TCP flows.
limit must be an integer from 1 through 4000000000.
Default: No limits
Usage:
Use this command to limit the total number of flows allowed for a rulebase regardless of flow type, or limit flows based on the protocol—non-TCP (connection-less) or TCP (connection-oriented).
If a subscriber attempts to exceed these limits system discards the packets of new flow. This limit processing of this command has following aspects for UDP, TCP, ICMP and some of the exempted flows:
- UDP/ICMP: System waits for the flow timeout before updating the counter and removing it from the count of number of flows.
- TCP: After a TCP flow ends, system waits for a short period of time to accommodate the retransmission of any missed packet from one end. TCP flows those are ended, but are still in wait period for timeout are exempted for this limit processing.
- Exempted flows: System exempts all the other flows specified with the flow limit-for-flow-type command in the ACS Charging Action Configuration Mode set to no.
Example:
This command defines the maximum number of 200000 flows for the rulebase:
flow limit-across-applications 200000
Feedback