S5700因收到大量STP TC報(bào)文導(dǎo)致CPU沖高
2014/11/24 15:14:06點(diǎn)擊:
問(wèn)題描述
某用戶反饋其企業(yè)網(wǎng)絡(luò)中,其中一臺(tái)S5700交換機(jī),CPU異常,經(jīng)常達(dá)到90%以上。
組網(wǎng)拓?fù)?故障設(shè)備為ZZXS57SW5F01-A4):
組網(wǎng)拓?fù)?故障設(shè)備為ZZXS57SW5F01-A4):
告警信息
查詢?cè)O(shè)備CPU信息,設(shè)備最近曾出現(xiàn)CPU沖高記錄
=====================================================
===============display cpu-usage===============
=====================================================
CPU Usage Stat. Cycle: 60 (Second)
CPU Usage : 18% Max: 97%
CPU Usage Stat. Time : 2014-10-07 11:19:29
CPU utilization for five seconds: 18%: one minute: 18%: five minutes: 18%
Max CPU Usage Stat. Time : 2014-09-11 16:37:54.
查詢?cè)O(shè)備日志有大量TC報(bào)文日志產(chǎn)生:
Oct 7 2014 11:06:20-05:13 ZZXS57SW5F01-A4 %%01INFO/4/SUPPRESS_LOG(l)[15]:Last message repeated 1 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
Oct 7 2014 11:05:19-05:13 ZZXS57SW5F01-A4 %%01INFO/4/SUPPRESS_LOG(l)[16]:Last message repeated 3 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
Oct 7 2014 11:04:12-05:13 ZZXS57SW5F01-A4 %%01INFO/4/SUPPRESS_LOG(l)[17]:Last message repeated 3 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
=====================================================
===============display cpu-usage===============
=====================================================
CPU Usage Stat. Cycle: 60 (Second)
CPU Usage : 18% Max: 97%
CPU Usage Stat. Time : 2014-10-07 11:19:29
CPU utilization for five seconds: 18%: one minute: 18%: five minutes: 18%
Max CPU Usage Stat. Time : 2014-09-11 16:37:54.
查詢?cè)O(shè)備日志有大量TC報(bào)文日志產(chǎn)生:
Oct 7 2014 11:06:20-05:13 ZZXS57SW5F01-A4 %%01INFO/4/SUPPRESS_LOG(l)[15]:Last message repeated 1 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
Oct 7 2014 11:05:19-05:13 ZZXS57SW5F01-A4 %%01INFO/4/SUPPRESS_LOG(l)[16]:Last message repeated 3 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
Oct 7 2014 11:04:12-05:13 ZZXS57SW5F01-A4 %%01INFO/4/SUPPRESS_LOG(l)[17]:Last message repeated 3 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
處理過(guò)程
1、因未在故障時(shí)采集信息,無(wú)法知道是具體那些進(jìn)程沖高,懷疑為大量TC報(bào)文導(dǎo)致設(shè)備FTS類(lèi)進(jìn)程沖高,設(shè)備一直產(chǎn)生TC報(bào)文日志,首先確定此TC報(bào)文是本設(shè)備產(chǎn)生的,還是從其它設(shè)備收到的。使用display stp tc-bpdu statistics命令查詢此TC報(bào)文是否是在本設(shè)備端產(chǎn)生,還是其它設(shè)備收到的。經(jīng)查詢?cè)撛O(shè)備與核心B互連端口TC報(bào)文入方向一直增漲,且同時(shí)轉(zhuǎn)發(fā)至其它接入層交換機(jī)。
[ZZXS57SW5F01-A4]display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/51 29272/63 0/0
0 GigabitEthernet0/0/52 3/18363 0/0
2、可以判定此TC報(bào)文不是本設(shè)備產(chǎn)生的,使用相同命令逐層排查T(mén)C報(bào)文入方向設(shè)備。確認(rèn)此TC報(bào)文是在網(wǎng)絡(luò)中的那一臺(tái)設(shè)備上產(chǎn)生的。
核心B:
<ZZXS57SW0B-C2>display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/1 12495/13 0/0
0 GigabitEthernet0/0/2 135/8349 0/0
0 GigabitEthernet0/0/3 13430/19 0/0
0 GigabitEthernet0/0/4 13784/14 0/0
0 GigabitEthernet0/0/5 14200/17 0/0
0 GigabitEthernet0/0/6 14687/10 0/0
0 GigabitEthernet0/0/14 14164/16 0/0
0 GigabitEthernet0/0/15 14164/16 0/0
0 GigabitEthernet0/0/16 14625/12 0/0
0 Eth-Trunk1 11012/4 0/0
核心A:
<ZZXS57SW0A-C2>display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/1 16754/7 0/0
0 GigabitEthernet0/0/2 17112/1 0/0
0 GigabitEthernet0/0/3 17462/11 0/0
0 GigabitEthernet0/0/4 17793/4 0/0
0 GigabitEthernet0/0/5 18118/5 0/0
0 GigabitEthernet0/0/6 18415/3 0/0
0 GigabitEthernet0/0/14 17791/3 0/0
0 GigabitEthernet0/0/15 18113/6 0/0
0 GigabitEthernet0/0/16 18435/4 0/0
0 Eth-Trunk1 4/11010 0/0
接入ZZXS57SW5F02-A4
<ZZXS57SW5F02-A4>display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/51 63/29347 0/0
0 GigabitEthernet0/0/52 15/14712 0/0
3、當(dāng)查詢到其中一臺(tái)接入設(shè)備時(shí),發(fā)現(xiàn)其TC報(bào)文只有在出方向上不斷有增漲計(jì)數(shù),有此可見(jiàn)該TC報(bào)文為此設(shè)備產(chǎn)生,可使用display stp topology-change 查詢?cè)揟C報(bào)文的信息?梢圆樵兊皆撛O(shè)備GigabitEthernet0/0/51端口不斷由阻塞變?yōu)榉砰_(kāi)后,由于狀態(tài)變?yōu)镈ETECTED而觸發(fā)拓?fù)渥兓?br /> [ZZXS57SW5F02-A4]display stp topology-change
CIST topology change information
Number of topology changes :8561
Time since last topology change :0 days 0h:0m:19s
Topology change initiator(detected) :GigabitEthernet0/0/51
Topology change last received from :3400-a3e0-379c
Number of generated topologychange traps : 2615
Number of suppressed topologychange traps: 71
4、使用命令display interface brief查詢?cè)摻尤朐O(shè)備端口信息,發(fā)現(xiàn)該設(shè)備GigabitEthernet0/0/51端口入方向有大量錯(cuò)包,說(shuō)明此接口入方向光纖線纜有問(wèn)題,排查線纜故障后問(wèn)題解決。
GigabitEthernet0/0/51 up up 0.01% 0.02% 38068638 0
根因
[ZZXS57SW5F01-A4]display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/51 29272/63 0/0
0 GigabitEthernet0/0/52 3/18363 0/0
2、可以判定此TC報(bào)文不是本設(shè)備產(chǎn)生的,使用相同命令逐層排查T(mén)C報(bào)文入方向設(shè)備。確認(rèn)此TC報(bào)文是在網(wǎng)絡(luò)中的那一臺(tái)設(shè)備上產(chǎn)生的。
核心B:
<ZZXS57SW0B-C2>display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/1 12495/13 0/0
0 GigabitEthernet0/0/2 135/8349 0/0
0 GigabitEthernet0/0/3 13430/19 0/0
0 GigabitEthernet0/0/4 13784/14 0/0
0 GigabitEthernet0/0/5 14200/17 0/0
0 GigabitEthernet0/0/6 14687/10 0/0
0 GigabitEthernet0/0/14 14164/16 0/0
0 GigabitEthernet0/0/15 14164/16 0/0
0 GigabitEthernet0/0/16 14625/12 0/0
0 Eth-Trunk1 11012/4 0/0
核心A:
<ZZXS57SW0A-C2>display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/1 16754/7 0/0
0 GigabitEthernet0/0/2 17112/1 0/0
0 GigabitEthernet0/0/3 17462/11 0/0
0 GigabitEthernet0/0/4 17793/4 0/0
0 GigabitEthernet0/0/5 18118/5 0/0
0 GigabitEthernet0/0/6 18415/3 0/0
0 GigabitEthernet0/0/14 17791/3 0/0
0 GigabitEthernet0/0/15 18113/6 0/0
0 GigabitEthernet0/0/16 18435/4 0/0
0 Eth-Trunk1 4/11010 0/0
接入ZZXS57SW5F02-A4
<ZZXS57SW5F02-A4>display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/51 63/29347 0/0
0 GigabitEthernet0/0/52 15/14712 0/0
3、當(dāng)查詢到其中一臺(tái)接入設(shè)備時(shí),發(fā)現(xiàn)其TC報(bào)文只有在出方向上不斷有增漲計(jì)數(shù),有此可見(jiàn)該TC報(bào)文為此設(shè)備產(chǎn)生,可使用display stp topology-change 查詢?cè)揟C報(bào)文的信息?梢圆樵兊皆撛O(shè)備GigabitEthernet0/0/51端口不斷由阻塞變?yōu)榉砰_(kāi)后,由于狀態(tài)變?yōu)镈ETECTED而觸發(fā)拓?fù)渥兓?br /> [ZZXS57SW5F02-A4]display stp topology-change
CIST topology change information
Number of topology changes :8561
Time since last topology change :0 days 0h:0m:19s
Topology change initiator(detected) :GigabitEthernet0/0/51
Topology change last received from :3400-a3e0-379c
Number of generated topologychange traps : 2615
Number of suppressed topologychange traps: 71
4、使用命令display interface brief查詢?cè)摻尤朐O(shè)備端口信息,發(fā)現(xiàn)該設(shè)備GigabitEthernet0/0/51端口入方向有大量錯(cuò)包,說(shuō)明此接口入方向光纖線纜有問(wèn)題,排查線纜故障后問(wèn)題解決。
GigabitEthernet0/0/51 up up 0.01% 0.02% 38068638 0
根因
STP組網(wǎng)中產(chǎn)與STP計(jì)算的設(shè)備互連端口因鏈路質(zhì)量不好,導(dǎo)致設(shè)備STP頻繁收斂,產(chǎn)生大量TC報(bào)文,導(dǎo)致收到此TC報(bào)文的設(shè)備部分CPU進(jìn)程沖商,影響業(yè)務(wù)正常運(yùn)行。
建議與總結(jié)
全局配置stp tc-protection
配置后可以保證設(shè)備頻繁收到TC報(bào)文時(shí),每2秒周期內(nèi)最多只處理1次表項(xiàng)刷新。從而減少M(fèi)AC、ARP表項(xiàng)頻繁刷新對(duì)設(shè)備的造成的負(fù)擔(dān)。
配置后可以保證設(shè)備頻繁收到TC報(bào)文時(shí),每2秒周期內(nèi)最多只處理1次表項(xiàng)刷新。從而減少M(fèi)AC、ARP表項(xiàng)頻繁刷新對(duì)設(shè)備的造成的負(fù)擔(dān)。
- 上一篇:S5700交換機(jī)配置SNMP團(tuán)體名字符串長(zhǎng)限制? 2014/11/24
- 下一篇:AR2200雙鏈路出口配置VRRP與NQA聯(lián)動(dòng)切換不生效 2014/11/24