網頁

2023/7/17

Golang goroutine leak by Mutex.Lock sync.runtime_SemacquireMutex block

今天發生系統卡住效能問題,發現原因是前陣子實作的WebSocket連線造成大量的goroutine未被釋放(leak)。


檢查pprof的goroutine頁面查看,可以看到有近三千條的goroutine卡在sync.runtime_SemacquireMutex

由於每個使用者的多條WebSocket連線都是用同一個map[string]map[string]*websocket.Conn來儲存並在存取時用Mutex.Lock取得鎖,在使用者斷線時邏輯沒有適當關閉連線造成持續等待並造成鎖未釋放,當新的連線請求近來存取map[string]map[string]*websocket.Conn時就被阻塞。

goroutine profile: total 3018
2983 @ 0x43ac76 0x44bc1e 0x44bbf5 0x468085 0x475c65 0x477256 0x477235 0xaf4c25 0xb191a2 0xb19188 0xb16a67 0xb16a4f 0xb17226 0xb17212 0xbf0b52 0xbf0b36 0xae1a31 0xae1699 0xae11dd 0x6e9c4c 0x6e4e67 0x46bf61
#	0x468084	sync.runtime_SemacquireMutex+0x24                                           runtime/sema.go:77
#	0x475c64	sync.(*Mutex).lockSlow+0x164                                                sync/mutex.go:171
#	0x477255	sync.(*Mutex).Lock+0x35                                                     sync/mutex.go:90
#	0x477234	sync.(*RWMutex).Lock+0x14                                                   sync/rwmutex.go:147
#	0xaf4c24	app/msgnc/internal/application/service.(*WsService).Login+0x64              app/msgnc/internal/application/service/ws_impl.go:28
#	0xb191a1	app/msgnc/internal/application/usecase.(*NotifUseCase).LoginWebSocket+0x281 app/msgnc/internal/application/usecase/notif.go:39
#	0xb19187	app/msgnc/internal/handler/ws.(*NotifHandler).Exec+0x267                    app/msgnc/internal/handler/ws/notif.go:40
#	0xb16a66	github.com/gin-gonic/gin.(*Context).Next+0x3c6                              github.com/gin-gonic/gin@v1.8.1/context.go:173
#	0xb16a4e	app/pkg/util.RecordHttpMiddleware+0x3ae                                     app/pkg/util/middleware.go:58
#	0xb17225	github.com/gin-gonic/gin.(*Context).Next+0x45                               github.com/gin-gonic/gin@v1.8.1/context.go:173
#	0xb17211	app/pkg/util.HandleErrorResponseMiddleware+0x31                             app/pkg/util/middleware.go:91
#	0xbf0b51	github.com/gin-gonic/gin.(*Context).Next+0x471                              github.com/gin-gonic/gin@v1.8.1/context.go:173
#	0xbf0b35	app/msgnc/router.corsMiddleware.func1+0x455                                 app/msgnc/router/router.go:85
#	0xae1a30	github.com/gin-gonic/gin.(*Context).Next+0x670                              github.com/gin-gonic/gin@v1.8.1/context.go:173
#	0xae1698	github.com/gin-gonic/gin.(*Engine).handleHTTPRequest+0x2d8                  github.com/gin-gonic/gin@v1.8.1/gin.go:616
#	0xae11dc	github.com/gin-gonic/gin.(*Engine).ServeHTTP+0x1dc                          github.com/gin-gonic/gin@v1.8.1/gin.go:572
#	0x6e9c4b	net/http.serverHandler.ServeHTTP+0x30b                                      net/http/server.go:2947
#	0x6e4e66	net/http.(*conn).serve+0x606                                                net/http/server.go:1991

15 @ 0x43ac76 0x44bc1e 0x44bbf5 0x468085 0xaf644e 0xaf642f 0xaf4e1a 0xb191a2 0xb19188 0xb16a67 0xb16a4f 0xb17226 0xb17212 0xbf0b52 0xbf0b36 0xae1a31 0xae1699 0xae11dd 0x6e9c4c 0x6e4e67 0x46bf61
#	0x468084	sync.runtime_SemacquireMutex+0x24                                           runtime/sema.go:77
#	0xaf644d	sync.(*RWMutex).RLock+0x6d                                                  sync/rwmutex.go:71
#	0xaf642e	app/msgnc/internal/application/service.(*WsService).CloseConn+0x4e          app/msgnc/internal/application/service/ws_impl.go:158
#	0xaf4e19	app/msgnc/internal/application/service.(*WsService).Login+0x259             app/msgnc/internal/application/service/ws_impl.go:39
#	0xb191a1	app/msgnc/internal/application/usecase.(*NotifUseCase).LoginWebSocket+0x281 app/msgnc/internal/application/usecase/notif.go:39
#	0xb19187	app/msgnc/internal/handler/ws.(*NotifHandler).Exec+0x267                    app/msgnc/internal/handler/ws/notif.go:40
#	0xb16a66	github.com/gin-gonic/gin.(*Context).Next+0x3c6                              github.com/gin-gonic/gin@v1.8.1/context.go:173
#	0xb16a4e	app/pkg/util.RecordHttpMiddleware+0x3ae                                     app/pkg/util/middleware.go:58
#	0xb17225	github.com/gin-gonic/gin.(*Context).Next+0x45                               github.com/gin-gonic/gin@v1.8.1/context.go:173
#	0xb17211	app/pkg/util.HandleErrorResponseMiddleware+0x31                             app/pkg/util/middleware.go:91
#	0xbf0b51	github.com/gin-gonic/gin.(*Context).Next+0x471                              github.com/gin-gonic/gin@v1.8.1/context.go:173
#	0xbf0b35	app/msgnc/router.corsMiddleware.func1+0x455                                 app/msgnc/router/router.go:85
#	0xae1a30	github.com/gin-gonic/gin.(*Context).Next+0x670                              github.com/gin-gonic/gin@v1.8.1/context.go:173
#	0xae1698	github.com/gin-gonic/gin.(*Engine).handleHTTPRequest+0x2d8                  github.com/gin-gonic/gin@v1.8.1/gin.go:616
#	0xae11dc	github.com/gin-gonic/gin.(*Engine).ServeHTTP+0x1dc                          github.com/gin-gonic/gin@v1.8.1/gin.go:572
#	0x6e9c4b	net/http.serverHandler.ServeHTTP+0x30b                                      net/http/server.go:2947
#	0x6e4e66	net/http.(*conn).serve+0x606	

沒有留言:

張貼留言