今天發生系統卡住效能問題，發現原因是前陣子實作的WebSocket連線造成大量的goroutine未被釋放（leak）。
檢查pprof的goroutine頁面查看，可以看到有近三千條的goroutine卡在sync.runtime_SemacquireMutex。
由於每個使用者的多條WebSocket連線都是用同一個map[string]map[string]*websocket.Conn來儲存並在存取時用Mutex.Lock取得鎖，在使用者斷線時邏輯沒有適當關閉連線造成持續等待並造成鎖未釋放，當新的連線請求近來存取map[string]map[string]*websocket.Conn時就被阻塞。
goroutine profile: total 3018
2983 @ 0x43ac76 0x44bc1e 0x44bbf5 0x468085 0x475c65 0x477256 0x477235 0xaf4c25 0xb191a2 0xb19188 0xb16a67 0xb16a4f 0xb17226 0xb17212 0xbf0b52 0xbf0b36 0xae1a31 0xae1699 0xae11dd 0x6e9c4c 0x6e4e67 0x46bf61
# 0x468084 sync.runtime_SemacquireMutex+0x24 runtime/sema.go:77
# 0x475c64 sync.(*Mutex).lockSlow+0x164 sync/mutex.go:171
# 0x477255 sync.(*Mutex).Lock+0x35 sync/mutex.go:90
# 0x477234 sync.(*RWMutex).Lock+0x14 sync/rwmutex.go:147
# 0xaf4c24 app/msgnc/internal/application/service.(*WsService).Login+0x64 app/msgnc/internal/application/service/ws_impl.go:28
# 0xb191a1 app/msgnc/internal/application/usecase.(*NotifUseCase).LoginWebSocket+0x281 app/msgnc/internal/application/usecase/notif.go:39
# 0xb19187 app/msgnc/internal/handler/ws.(*NotifHandler).Exec+0x267 app/msgnc/internal/handler/ws/notif.go:40
# 0xb16a66 github.com/gin-gonic/gin.(*Context).Next+0x3c6 github.com/gin-gonic/gin@v1.8.1/context.go:173
# 0xb16a4e app/pkg/util.RecordHttpMiddleware+0x3ae app/pkg/util/middleware.go:58
# 0xb17225 github.com/gin-gonic/gin.(*Context).Next+0x45 github.com/gin-gonic/gin@v1.8.1/context.go:173
# 0xb17211 app/pkg/util.HandleErrorResponseMiddleware+0x31 app/pkg/util/middleware.go:91
# 0xbf0b51 github.com/gin-gonic/gin.(*Context).Next+0x471 github.com/gin-gonic/gin@v1.8.1/context.go:173
# 0xbf0b35 app/msgnc/router.corsMiddleware.func1+0x455 app/msgnc/router/router.go:85
# 0xae1a30 github.com/gin-gonic/gin.(*Context).Next+0x670 github.com/gin-gonic/gin@v1.8.1/context.go:173
# 0xae1698 github.com/gin-gonic/gin.(*Engine).handleHTTPRequest+0x2d8 github.com/gin-gonic/gin@v1.8.1/gin.go:616
# 0xae11dc github.com/gin-gonic/gin.(*Engine).ServeHTTP+0x1dc github.com/gin-gonic/gin@v1.8.1/gin.go:572
# 0x6e9c4b net/http.serverHandler.ServeHTTP+0x30b net/http/server.go:2947
# 0x6e4e66 net/http.(*conn).serve+0x606 net/http/server.go:1991
15 @ 0x43ac76 0x44bc1e 0x44bbf5 0x468085 0xaf644e 0xaf642f 0xaf4e1a 0xb191a2 0xb19188 0xb16a67 0xb16a4f 0xb17226 0xb17212 0xbf0b52 0xbf0b36 0xae1a31 0xae1699 0xae11dd 0x6e9c4c 0x6e4e67 0x46bf61
# 0x468084 sync.runtime_SemacquireMutex+0x24 runtime/sema.go:77
# 0xaf644d sync.(*RWMutex).RLock+0x6d sync/rwmutex.go:71
# 0xaf642e app/msgnc/internal/application/service.(*WsService).CloseConn+0x4e app/msgnc/internal/application/service/ws_impl.go:158
# 0xaf4e19 app/msgnc/internal/application/service.(*WsService).Login+0x259 app/msgnc/internal/application/service/ws_impl.go:39
# 0xb191a1 app/msgnc/internal/application/usecase.(*NotifUseCase).LoginWebSocket+0x281 app/msgnc/internal/application/usecase/notif.go:39
# 0xb19187 app/msgnc/internal/handler/ws.(*NotifHandler).Exec+0x267 app/msgnc/internal/handler/ws/notif.go:40
# 0xb16a66 github.com/gin-gonic/gin.(*Context).Next+0x3c6 github.com/gin-gonic/gin@v1.8.1/context.go:173
# 0xb16a4e app/pkg/util.RecordHttpMiddleware+0x3ae app/pkg/util/middleware.go:58
# 0xb17225 github.com/gin-gonic/gin.(*Context).Next+0x45 github.com/gin-gonic/gin@v1.8.1/context.go:173
# 0xb17211 app/pkg/util.HandleErrorResponseMiddleware+0x31 app/pkg/util/middleware.go:91
# 0xbf0b51 github.com/gin-gonic/gin.(*Context).Next+0x471 github.com/gin-gonic/gin@v1.8.1/context.go:173
# 0xbf0b35 app/msgnc/router.corsMiddleware.func1+0x455 app/msgnc/router/router.go:85
# 0xae1a30 github.com/gin-gonic/gin.(*Context).Next+0x670 github.com/gin-gonic/gin@v1.8.1/context.go:173
# 0xae1698 github.com/gin-gonic/gin.(*Engine).handleHTTPRequest+0x2d8 github.com/gin-gonic/gin@v1.8.1/gin.go:616
# 0xae11dc github.com/gin-gonic/gin.(*Engine).ServeHTTP+0x1dc github.com/gin-gonic/gin@v1.8.1/gin.go:572
# 0x6e9c4b net/http.serverHandler.ServeHTTP+0x30b net/http/server.go:2947
# 0x6e4e66 net/http.(*conn).serve+0x606
沒有留言:
張貼留言