如何解决对于相同的pytorch型号,GTX 1080 ti,Tesla k80,Tesla v100上的不同内存分配
我尝试在pytorch中通过3种不同的GPU(GeForce GTX 1080 ti,tesla k80,tesla v100)加载distilbert模型。根据pytorch cuda profiler,所有这些GPU的内存消耗都是相同的(534MB)。但是“ nvidia-smi”对每个内存显示不同的内存消耗(GTX 1080 ti-1181MB,特斯拉k80-898MB,特斯拉v100-1714MB)。
我选择了v100,因为它有额外的内存,希望容纳更多的进程。因此,与k80相比,我无法在v100中容纳更多进程。
版本:Python 3.6.11,translators == 2.3.0, 火炬== 1.6.0
任何帮助将不胜感激。
以下是GPU中的内存消耗。
---------------- GTX 1080ti ---------------------
2020-10-19 02:11:04,147 - CE - INFO - torch.cuda.max_memory_allocated() : 514.33154296875
2020-10-19 02:11:04,147 - CE - INFO - torch.cuda.memory_allocated() : 514.33154296875
2020-10-19 02:11:04,147 - CE - INFO - torch.cuda.memory_reserved() : 534.0
2020-10-19 02:11:04,148 - CE - INFO - torch.cuda.max_memory_reserved() : 534.0
“ nvidia-smi”的输出:
2020-10-19 02:11:04,221 - CE - INFO - | ID | Name | Serial | UUID || GPU temp. | GPU util. | Memory util. || Memory total | Memory used | Memory free || display mode | display active |
2020-10-19 02:11:04,222 - CE - INFO - | 0 | GeForce GTX 1080 Ti | [Not Supported] | GPU-58d5d4d3-07a1-81b4-ba67-8d6b46e342fb || 50C | 15% | 11% || 11178MB | 1181MB | 9997MB || disabled | disabled |
----------------特斯拉k80 ---------------------
2020-10-19 12:15:37,030 - CE - INFO - torch.cuda.max_memory_allocated() : 514.33154296875
2020-10-19 12:15:37,031 - CE - INFO - torch.cuda.memory_allocated() : 514.33154296875
2020-10-19 12:15:37,031 - CE - INFO - torch.cuda.memory_reserved() : 534.0
2020-10-19 12:15:37,031 - CE - INFO - torch.cuda.max_memory_reserved() : 534.0
“ nvidia-smi”的输出:
2020-10-19 12:15:37,081 - CE - INFO - | ID | Name | Serial | UUID || GPU temp. | GPU util. | Memory util. || Memory total | Memory used | Memory free || display mode | display active |
2020-10-19 12:15:37,081 - CE - INFO - | 0 | Tesla K80 | 0324516191902 | GPU-1e7baee8-174b-2178-7115-cf4a063a8923 || 50C | 3% | 8% || 11441MB | 898MB | 10543MB || disabled | disabled |
---------------- Tesla v100 ---------------------
2020-10-20 08:18:42,952 - CE - INFO - torch.cuda.max_memory_allocated() : 514.33154296875
2020-10-20 08:18:42,952 - CE - INFO - torch.cuda.memory_allocated() : 514.33154296875
2020-10-20 08:18:42,953 - CE - INFO - torch.cuda.memory_reserved() : 534.0
2020-10-20 08:18:42,953 - CE - INFO - torch.cuda.max_memory_reserved() : 534.0
“ nvidia-smi”的输出:
2020-10-20 08:18:43,020 - CE - INFO - | ID | Name | Serial | UUID || GPU temp. | GPU util. | Memory util. || Memory total | Memory used | Memory free || display mode | display active |
2020-10-20 08:18:43,020 - CE - INFO - | 0 | Tesla V100-SXM2-16GB | 0323617004258 | GPU-849088a3-508a-1737-7611-75a087f18085 || 29C | 0% | 11% || 16160MB | 1714MB | 14446MB || Enabled | disabled |
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。