如何解决Windows .NET进程在GC中挂起
我的.NET x86进程挂死了。这是用于处理背景作业的Windows服务。它通过linq2sql，RabbitMQ，Kafka，Redis和其他一些技术来使用SQL Server。
由于某种原因，此过程会不时停止处理任何作业，而解决该问题的唯一方法是重新启动它。这是非常罕见的事情，但很烦人。
我有一个处于这种奇怪状态的内存转储和一个PerfView记录。
在内存转储中，我看到了〜600个托管和非托管线程。我发现进程中的许多线程被锁定在一些库（在linq2sql和NLog内部）和我们自己的代码中的某些Monitor上。 锁链在执行linq2sql查询的许多线程上结束，该线程具有如下堆栈跟踪：https://prnt.sc/v60k71
似乎这些线程正在尝试获取此锁：https://prnt.sc/v60m2b
有一个线程可以完成此任务：https://prnt.sc/v5bz4z
此线程也处于某种锁定或等待状态。在下一个屏幕截图中，我们可以看到它正在尝试分配一些内存，并且正在等待GC结束：https://prnt.sc/v60os0
此线程的主要问题是它处于一天之类的状态！之所以这么说是因为我发现该线程堆栈中的某些对象中包含DateTime。现在这些对象创建的瞬间，而这就像创建此内存转储的前一天一样。可以说，所有其他线程也被锁定了一天。也许它们会在很短的时间内解锁，但这似乎不太可能。
我试图调查GC的工作。我们使用服务器GC，该计算机具有24个核心，因此在此过程中我们具有24个GC线程。其中的23个具有如下的stacktrace：https://prnt.sc/v58910
这里是该线程的PerfView跟踪，因此我们可以看到它在Sleep中等待了大约60％的时间，并且似乎在其他有用的工作上花费了另外40％的时间：https://prnt.sc/v5c82p
正如我所见，在gc_heap::mark_steal的{{3}}中有sleep(1)，所以我很确定是这样。
其中一个GC线程具有以下堆栈跟踪：https://github.com/dotnet/runtime/blob/master/src/coreclr/src/gc/gc.cpp
这是该线程的PerfView：https://prnt.sc/v5wwxi
因此，看来GC正在运行，但是效果很差？为什么？如何解决它或如何不让进程进入这种状态？
.NET Framework版本：
0:336> lmv m mscor*
Browse full module list
start end module name
72020000 7342e000 mscorlib_ni (no symbols)
Loaded symbol image file: mscorlib.ni.dll
Image path: C:\Windows\assembly\NativeImages_v4.0.30319_32\mscorlib\c7625296f6cee933f010d4fca91738db\mscorlib.ni.dll
Image name: mscorlib.ni.dll
Browse all global symbols functions data
Has CLR image header,track-debug-data flag not set
Timestamp: Thu Sep 12 22:32:19 2019 (5D7A9D43)
CheckSum: 0141EB31
ImageSize: 0140E000
File version: 4.8.4042.0
Product version: 4.0.30319.0
File flags: 8 (Mask 3F) Private
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
Information from resource tables:
CompanyName: Microsoft Corporation
ProductName: Microsoft® .NET Framework
InternalName: mscorlib.dll
OriginalFilename: mscorlib.dll
ProductVersion: 4.8.4042.0
FileVersion: 4.8.4042.0 built by: NET48REL1LAST_C
PrivateBuild: DDBLD481
FileDescription: Microsoft Common Language Runtime Class Library
LegalCopyright: © Microsoft Corporation. All rights reserved.
Comments: Flavor=Retail
这是GC堆：
Number of GC Heaps: 24
------------------------------
Heap 0 (007028b0)
generation 0 starts at 0xdb3ffa3c
generation 1 starts at 0xdb3ffa30
generation 2 starts at 0x01441000
ephemeral segment allocation context: none
segment begin allocated size
01440000 01441000 0243fc2c 0xffec2c(16772140)
77470000 77471000 7846ff88 0xffef88(16773000)
88810000 88811000 8980ffac 0xffefac(16773036)
8c810000 8c811000 8d80ffe8 0xffefe8(16773096)
bc010000 bc011000 bd00ff9c 0xffef9c(16773020)
f4010000 f4011000 f500ffc4 0xffefc4(16773060)
db010000 db011000 db3ffa48 0x3eea48(4123208)
Large object heap starts at 0x19441000
segment begin allocated size
19440000 19441000 19493a58 0x52a58(338520)
ce810000 ce811000 d6811010 0x8000010(134217744)
Heap Size: Size: 0xe43af58 (239316824) bytes.
------------------------------
Heap 1 (00708410)
generation 0 starts at 0xfb16f168
generation 1 starts at 0xfb16f15c
generation 2 starts at 0x02441000
ephemeral segment allocation context: none
segment begin allocated size
02440000 02441000 0343ff9c 0xffef9c(16773020)
f3010000 f3011000 f400ffac 0xffefac(16773036)
91010000 91011000 9200ffdc 0xffefdc(16773084)
ad010000 ad011000 ae00ff74 0xffef74(16772980)
c6010000 c6011000 c700ff94 0xffef94(16773012)
f7010000 f7011000 f800ffc4 0xffefc4(16773060)
fb010000 fb011000 fb16f174 0x15e174(1433972)
Large object heap starts at 0x19c41000
segment begin allocated size
19c40000 19c41000 19c41010 0x10(16)
Heap Size: Size: 0x6157f74 (102072180) bytes.
------------------------------
Heap 2 (0070df70)
generation 0 starts at 0xeb00ffcc
generation 1 starts at 0xeb00ffc0
generation 2 starts at 0x03441000
ephemeral segment allocation context: none
segment begin allocated size
03440000 03441000 044354a0 0xff44a0(16729248)
63cb0000 63cb1000 64cb0000 0xfff000(16773120)
81010000 81011000 8200ffec 0xffefec(16773100)
8a810000 8a811000 8b80fcc0 0xffecc0(16772288)
b6010000 b6011000 b700fd34 0xffed34(16772404)
9f010000 9f011000 a000ffec 0xffefec(16773100)
ea010000 ea011000 eb00ffd8 0xffefd8(16773080)
Large object heap starts at 0x1a441000
segment begin allocated size
1a440000 1a441000 1a876ed0 0x435ed0(4415184)
Heap Size: Size: 0x7423d14 (121781524) bytes.
------------------------------
Heap 3 (007142d8)
generation 0 starts at 0xd84d4740
generation 1 starts at 0xd84d4734
generation 2 starts at 0x04441000
ephemeral segment allocation context: none
segment begin allocated size
04440000 04441000 0543ffa8 0xffefa8(16773032)
66cb0000 66cb1000 67caffec 0xffefec(16773100)
83810000 83811000 8480ffdc 0xffefdc(16773084)
9b010000 9b011000 9c00ffc0 0xffefc0(16773056)
b2010000 b2011000 b300ffdc 0xffefdc(16773084)
c9010000 c9011000 ca00ff68 0xffef68(16772968)
f9010000 f9011000 fa00ffe4 0xffefe4(16773092)
d8010000 d8011000 d84d474c 0x4c374c(4994892)
Large object heap starts at 0x1ac41000
segment begin allocated size
1ac40000 1ac41000 1ac65030 0x24030(147504)
Heap Size: Size: 0x74e05d4 (122553812) bytes.
------------------------------
Heap 4 (00719e38)
generation 0 starts at 0xeeb07994
generation 1 starts at 0xeeb07988
generation 2 starts at 0x05441000
ephemeral segment allocation context: none
segment begin allocated size
05440000 05441000 0643fc98 0xffec98(16772248)
5d0c0000 5d0c1000 5e0bff20 0xffef20(16772896)
85810000 85811000 8680f3b4 0xffe3b4(16769972)
8b810000 8b811000 8c810000 0xfff000(16773120)
ba010000 ba011000 bb00fff0 0xffeff0(16773104)
a3010000 a3011000 a400fee4 0xffeee4(16772836)
ee010000 ee011000 eeb079a0 0xaf69a0(11495840)
Large object heap starts at 0x1b441000
segment begin allocated size
1b440000 1b441000 1b441010 0x10(16)
Heap Size: Size: 0x6aef7f0 (112130032) bytes.
------------------------------
Heap 5 (007239a0)
generation 0 starts at 0xe0befdc4
generation 1 starts at 0xe0befdb8
generation 2 starts at 0x06441000
ephemeral segment allocation context: none
segment begin allocated size
06440000 06441000 0743fe4c 0xffee4c(16772684)
64cb0000 64cb1000 65caffdc 0xffefdc(16773084)
86810000 86811000 8780ff88 0xffef88(16773000)
a5010000 a5011000 a600fcf0 0xffecf0(16772336)
bf010000 bf011000 c000ff00 0xffef00(16772864)
e0010000 e0011000 e0c04940 0xbf3940(12532032)
Large object heap starts at 0x1bc41000
segment begin allocated size
1bc40000 1bc41000 1bc65030 0x24030(147504)
cc010000 cc011000 ce011018 0x2000018(33554456)
Heap Size: Size: 0x7c12328 (130097960) bytes.
------------------------------
Heap 6 (0072a508)
generation 0 starts at 0xda8386f4
generation 1 starts at 0xda8386e8
generation 2 starts at 0x07441000
ephemeral segment allocation context: none
segment begin allocated size
07440000 07441000 0843f4d8 0xffe4d8(16770264)
f5010000 f5011000 f600ffcc 0xffefcc(16773068)
9a010000 9a011000 9b00ff34 0xffef34(16772916)
b9010000 b9011000 ba00fe20 0xffee20(16772640)
da010000 da011000 da838700 0x827700(8550144)
Large object heap starts at 0x1c441000
segment begin allocated size
1c440000 1c441000 1c48ff10 0x4ef10(323344)
Heap Size: Size: 0x4871808 (75962376) bytes.
------------------------------
Heap 7 (00730068)
generation 0 starts at 0xe6c4b564
generation 1 starts at 0xe6c4b558
generation 2 starts at 0x08441000
ephemeral segment allocation context: none
segment begin allocated size
08440000 08441000 09438b64 0xff7b64(16743268)
ef010000 ef011000 f000fff4 0xffeff4(16773108)
82010000 82011000 8300ffd8 0xffefd8(16773080)
af010000 af011000 b000fff4 0xffeff4(16773108)
c7010000 c7011000 c800ff08 0xffef08(16772872)
e6010000 e6011000 e6c572a8 0xc462a8(12870312)
Large object heap starts at 0x1cc41000
segment begin allocated size
1cc40000 1cc41000 1cc41010 0x10(16)
Heap Size: Size: 0x5c39ce4 (96705764) bytes.
------------------------------
Heap 8 (00735bc8)
generation 0 starts at 0xf22a4fb0
generation 1 starts at 0xf22a4fa4
generation 2 starts at 0x09441000
ephemeral segment allocation context: none
segment begin allocated size
09440000 09441000 0a43c36c 0xffb36c(16757612)
7b470000 7b471000 7c470000 0xfff000(16773120)
8f010000 8f011000 9000f0fc 0xffe0fc(16769276)
a4010000 a4011000 a500fff4 0xffeff4(16773108)
bb010000 bb011000 bc00e704 0xffd704(16766724)
d7010000 d7011000 d800dcd0 0xffccd0(16764112)
f2010000 f2011000 f22a4fbc 0x293fbc(2703292)
Large object heap starts at 0x1d441000
segment begin allocated size
1d440000 1d441000 1d441010 0x10(16)
Heap Size: Size: 0x62857fc (103307260) bytes.
------------------------------
Heap 9 (0073b728)
generation 0 starts at 0xe201100c
generation 1 starts at 0xe2011000
generation 2 starts at 0x0a441000
ephemeral segment allocation context: none
segment begin allocated size
0a440000 0a441000 0b432154 0xff1154(16716116)
67cb0000 67cb1000 68c86368 0xfd5368(16601960)
92010000 92011000 9300fbd8 0xffebd8(16772056)
a9010000 a9011000 aa00ffa8 0xffefa8(16773032)
be010000 be011000 bf00fc18 0xffec18(16772120)
de010000 de011000 deddf1a4 0xdce1a4(14475684)
e2010000 e2011000 e2011018 0x18(24)
Large object heap starts at 0x1dc41000
segment begin allocated size
1dc40000 1dc41000 1dc41010 0x10(16)
Heap Size: Size: 0x5d90e20 (98111008) bytes.
------------------------------
Heap 10 (00741288)
generation 0 starts at 0xe59f9eb4
generation 1 starts at 0xe59f9ea8
generation 2 starts at 0x0b441000
ephemeral segment allocation context: none
segment begin allocated size
0b440000 0b441000 0c43ff28 0xffef28(16772904)
7d470000 7d471000 7e46ffd0 0xffefd0(16773072)
93010000 93011000 9400ff48 0xffef48(16772936)
aa010000 aa011000 ab00fa8c 0xffea8c(16771724)
c3010000 c3011000 c400fcb4 0xffecb4(16772276)
e5010000 e5011000 e59f9ec0 0x9e8ec0(10391232)
Large object heap starts at 0x1e441000
segment begin allocated size
1e440000 1e441000 1e465030 0x24030(147504)
Heap Size: Size: 0x5a07470 (94401648) bytes.
------------------------------
Heap 11 (007475f0)
generation 0 starts at 0xdc01100c
generation 1 starts at 0xdc011000
generation 2 starts at 0x0c441000
ephemeral segment allocation context: none
segment begin allocated size
0c440000 0c441000 0d437d90 0xff6d90(16739728)
78470000 78471000 7946ffe0 0xffefe0(16773088)
89810000 89811000 8a80ff74 0xffef74(16772980)
a8010000 a8011000 a900ff38 0xffef38(16772920)
c4010000 c4011000 c500d770 0xffc770(16762736)
dd010000 dd011000 dde6d38c 0xe5c38c(15057804)
dc010000 dc011000 dc03289c 0x2189c(137372)
Large object heap starts at 0x1ec41000
segment begin allocated size
1ec40000 1ec41000 1ec41010 0x10(16)
Heap Size: Size: 0x5e6dfc4 (99016644) bytes.
------------------------------
Heap 12 (0074d150)
generation 0 starts at 0xf05c5e00
generation 1 starts at 0xf05c5df4
generation 2 starts at 0x0d441000
ephemeral segment allocation context: none
segment begin allocated size
0d440000 0d441000 0e434334 0xff3334(16724788)
65cb0000 65cb1000 66cad148 0xffc148(16761160)
96010000 96011000 9700ff14 0xffef14(16772884)
b4010000 b4011000 b500afcc 0xff9fcc(16752588)
a1010000 a1011000 a200df68 0xffcf68(16764776)
f0010000 f0011000 f05c5e0c 0x5b4e0c(5983756)
Large object heap starts at 0x1f441000
segment begin allocated size
1f440000 1f441000 1f441010 0x10(16)
Heap Size: Size: 0x559a0e0 (89759968) bytes.
------------------------------
Heap 13 (00754aa8)
generation 0 starts at 0xfc01100c
generation 1 starts at 0xfc011000
generation 2 starts at 0x0e441000
ephemeral segment allocation context: none
segment begin allocated size
0e440000 0e441000 0f4369f0 0xff59f0(16734704)
fe010000 fe011000 ff00ffec 0xffefec(16773100)
8e010000 8e011000 8f00e6d4 0xffd6d4(16766676)
ac010000 ac011000 ad00fe00 0xffee00(16772608)
c1010000 c1011000 c200fc6c 0xffec6c(16772204)
e3010000 e3011000 e3c67088 0xc56088(12935304)
fc010000 fc011000 fc011018 0x18(24)
Large object heap starts at 0x1fc41000
segment begin allocated size
1fc40000 1fc41000 1fc41010 0x10(16)
Heap Size: Size: 0x5c45bcc (96754636) bytes.
------------------------------
Heap 14 (0075e800)
generation 0 starts at 0xe4c1ec74
generation 1 starts at 0xe4c1ec68
generation 2 starts at 0x0f441000
ephemeral segment allocation context: none
segment begin allocated size
0f440000 0f441000 10437ef8 0xff6ef8(16740088)
5c0c0000 5c0c1000 5d0bffe8 0xffefe8(16773096)
84810000 84811000 8580ff9c 0xffef9c(16773020)
9c010000 9c011000 9d00ffec 0xffefec(16773100)
b8010000 b8011000 b900ee4c 0xffde4c(16768588)
9e010000 9e011000 9f00ff54 0xffef54(16772948)
e4010000 e4011000 e4c1ec80 0xc0dc80(12639360)
Large object heap starts at 0x20441000
segment begin allocated size
20440000 20441000 20441010 0x10(16)
Heap Size: Size: 0x6bfe898 (113240216) bytes.
------------------------------
Heap 15 (00764150)
generation 0 starts at 0xfd01100c
generation 1 starts at 0xfd011000
generation 2 starts at 0x10441000
ephemeral segment allocation context: none
segment begin allocated size
10440000 10441000 1143fe74 0xffee74(16772724)
69cb0000 69cb1000 6acafe88 0xffee88(16772744)
94010000 94011000 9500fc30 0xffec30(16772144)
b0010000 b0011000 b100fc4c 0xffec4c(16772172)
c5010000 c5011000 c600f4c4 0xffe4c4(16770244)
e1010000 e1011000 e1ef4660 0xee3660(15611488)
fd010000 fd011000 fd011018 0x18(24)
Large object heap starts at 0x20c41000
segment begin allocated size
20c40000 20c41000 20c41010 0x10(16)
Heap Size: Size: 0x5edd0c4 (99471556) bytes.
------------------------------
Heap 16 (0076f1d8)
generation 0 starts at 0xf117d5d8
generation 1 starts at 0xf117d5cc
generation 2 starts at 0x11441000
ephemeral segment allocation context: none
segment begin allocated size
11440000 11441000 1243fac8 0xffeac8(16771784)
68cb0000 68cb1000 69caffb0 0xffefb0(16773040)
98010000 98011000 9900fac0 0xffeac0(16771776)
b7010000 b7011000 b800ffbc 0xffefbc(16773052)
a2010000 a2011000 a300fa74 0xffea74(16771700)
f1010000 f1011000 f117d5e4 0x16c5e4(1492452)
Large object heap starts at 0x21441000
segment begin allocated size
21440000 21441000 21479050 0x38050(229456)
Heap Size: Size: 0x519e59c (85583260) bytes.
------------------------------
Heap 17 (0077aa98)
generation 0 starts at 0xfa01100c
generation 1 starts at 0xfa011000
generation 2 starts at 0x12441000
ephemeral segment allocation context: none
segment begin allocated size
12440000 12441000 13438aac 0xff7aac(16743084)
58e00000 58e01000 59dffff0 0xffeff0(16773104)
87810000 87811000 8880fe90 0xffee90(16772752)
ab010000 ab011000 ac00ffd8 0xffefd8(16773080)
c0010000 c0011000 c100e214 0xffd214(16765460)
f6010000 f6011000 f6f98b74 0xf87b74(16284532)
fa010000 fa011000 fa011018 0x18(24)
Large object heap starts at 0x21c41000
segment begin allocated size
21c40000 21c41000 21c65030 0x24030(147504)
Heap Size: Size: 0x5f9d6d4 (100259540) bytes.
------------------------------
Heap 18 (0077dea8)
generation 0 starts at 0xe8848d64
generation 1 starts at 0xe8848d58
generation 2 starts at 0x13441000
ephemeral segment allocation context: none
segment begin allocated size
13440000 13441000 144198cc 0xfd88cc(16615628)
7e470000 7e471000 7f46f07c 0xffe07c(16769148)
80010000 80011000 80fb0a68 0xf9fa68(16382568)
9d010000 9d011000 9e00fbac 0xffebac(16772012)
b5010000 b5011000 b600ffe8 0xffefe8(16773096)
a0010000 a0011000 a100d47c 0xffc47c(16761980)
e8010000 e8011000 e885928c 0x84828c(8684172)
Large object heap starts at 0x22441000
segment begin allocated size
22440000 22441000 22441010 0x10(16)
Heap Size: Size: 0x67b865c (108758620) bytes.
------------------------------
Heap 19 (0077c4a0)
generation 0 starts at 0xdfb2d62c
generation 1 starts at 0xdfb2d620
generation 2 starts at 0x14441000
ephemeral segment allocation context: none
segment begin allocated size
14440000 14441000 154341c8 0xff31c8(16724424)
79470000 79471000 7a46ff3c 0xffef3c(16772924)
90010000 90011000 9100ffc4 0xffefc4(16773060)
a6010000 a6011000 a700ffec 0xffefec(16773100)
c2010000 c2011000 c300595c 0xff495c(16730460)
df010000 df011000 dfb33848 0xb22848(11675720)
Large object heap starts at 0x22c41000
segment begin allocated size
22c40000 22c41000 22c41010 0x10(16)
Heap Size: Size: 0x5b07268 (95449704) bytes.
------------------------------
Heap 20 (0077f8b0)
generation 0 starts at 0xeb85d3e0
generation 1 starts at 0xeb85d3d4
generation 2 starts at 0x15441000
ephemeral segment allocation context: none
segment begin allocated size
15440000 15441000 1643fdf8 0xffedf8(16772600)
7c470000 7c471000 7d46ffc4 0xffefc4(16773060)
95010000 95011000 9600fedc 0xffeedc(16772828)
b1010000 b1011000 b200ffec 0xffefec(16773100)
ca010000 ca011000 cb00ff24 0xffef24(16772900)
eb010000 eb011000 eb85d3ec 0x84c3ec(8700908)
Large object heap starts at 0x23441000
segment begin allocated size
23440000 23441000 23441010 0x10(16)
Heap Size: Size: 0x5846fa4 (92565412) bytes.
------------------------------
Heap 21 (00779090)
generation 0 starts at 0xed72ecf8
generation 1 starts at 0xed72ecec
generation 2 starts at 0x16441000
ephemeral segment allocation context: none
segment begin allocated size
16440000 16441000 17437418 0xff6418(16737304)
e9010000 e9011000 ea009284 0xff8284(16745092)
99010000 99011000 9a00fbf8 0xffebf8(16772088)
b3010000 b3011000 b400fb5c 0xffeb5c(16771932)
cb010000 cb011000 cc00fd68 0xffed68(16772456)
ed010000 ed011000 ed72ed04 0x71dd04(7462148)
Large object heap starts at 0x23c41000
segment begin allocated size
23c40000 23c41000 23c41010 0x10(16)
Heap Size: Size: 0x570886c (91261036) bytes.
------------------------------
Heap 22 (00774278)
generation 0 starts at 0xf801100c
generation 1 starts at 0xf8011000
generation 2 starts at 0x17441000
ephemeral segment allocation context: none
segment begin allocated size
17440000 17441000 1842e658 0xfed658(16701016)
4f720000 4f721000 5071d2f8 0xffc2f8(16761592)
ec010000 ec011000 ed00ff40 0xffef40(16772928)
a7010000 a7011000 a800ff4c 0xffef4c(16772940)
bd010000 bd011000 be00fe18 0xffee18(16772632)
d9010000 d9011000 d9da7da0 0xd96da0(14249376)
f8010000 f8011000 f8011018 0x18(24)
Large object heap starts at 0x24441000
segment begin allocated size
24440000 24441000 24441010 0x10(16)
Heap Size: Size: 0x5d7d3bc (98030524) bytes.
------------------------------
Heap 23 (007812b8)
generation 0 starts at 0xe7670e10
generation 1 starts at 0xe7663454
generation 2 starts at 0x18441000
ephemeral segment allocation context: none
segment begin allocated size
18440000 18441000 1943f7b0 0xffe7b0(16770992)
7a470000 7a471000 7b46de30 0xffce30(16764464)
97010000 97011000 9800fd74 0xffed74(16772468)
ae010000 ae011000 af00ff1c 0xffef1c(16772892)
c8010000 c8011000 c900ff28 0xffef28(16772904)
e7010000 e7011000 e7670e1c 0x65fe1c(6684188)
Large object heap starts at 0x24c41000
segment begin allocated size
24c40000 24c41000 24c41010 0x10(16)
Heap Size: Size: 0x5657fc4 (90537924) bytes.
------------------------------
GC Heap Size: Size: 0x986ab2d4 (2557129428) bytes.
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。