Python -记录内存使用情况

vu8f3i0k  于 2023-05-05  发布在  Python
关注(0)|答案(3)|浏览(236)

在python 3中有没有一种方法可以在程序运行时记录内存（ram）的使用情况？
一些背景信息。我使用slurm在一个hpc集群上运行模拟，在那里我必须在提交作业之前保留一些内存。我知道我的工作需要大量的内存，但我不确定需要多少。所以我想知道是否有一个简单的解决方案来记录内存随时间的变化。

python

3条答案

按热度按时间
7nbnzgx9

7nbnzgx91#

您可以使用memory_profiler包来实现这一点。只需将装饰器@profile添加到函数中，就会得到如下输出：

Line #    Mem usage  Increment   Line Contents
==============================================
 3                           @profile
 4      5.97 MB    0.00 MB   def my_func():
 5     13.61 MB    7.64 MB       a = [1] * (10 ** 6)
 6    166.20 MB  152.59 MB       b = [2] * (2 * 10 ** 7)
 7     13.61 MB -152.59 MB       del b
 8     13.61 MB    0.00 MB       return a

否则，最简单的方法是在之后使用sacct -l -j <JobId>命令（查找MaxRSS列）询问Slurm，以便您可以适应进一步的作业。
此外，您可以在运行程序时使用top命令来了解其内存消耗情况。查找RES列。

赞(0）分享  回复(0）举报  2023-05-05
xggvc2p6

xggvc2p62#

可以使用subprocess模块。下面是bash命令free的示例输出

$ free -m
             total       used       free     shared    buffers     cached
Mem:          7979       7678        300          0        109       4628
-/+ buffers/cache:       2941       5038
Swap:         2046        360       1686

Python程序-

import subprocess
result = subprocess.check_output(['bash','-c', 'free -m'])
free_memory = result.split('\n')[1].split()[3]
# print free_memory
# 300

如果您想检查某个进程的内存使用情况或定期记录它，那么您可以根据您的用例使用pmap或其他实用程序，然后解析输出。

赞(0）分享  回复(0）举报  2023-05-05
rsaldnfx

rsaldnfx3#

bash脚本跟踪python（或任何其他）进程的资源

这里有一些示例脚本，可能会很方便。这样做的好处是，你不需要修改你的Python代码。缺点：你需要一个bash。当然，你也可以从一个文件运行一个python脚本。在bash脚本中包含python代码只是为了演示目的。

脚本

#!/bash/bin
set -e

LOG_OUTPUT="output.log"
LOG_RESOURCES="resources.log"

PYTHON_CODE="
import numpy as np
from datetime import datetime
import time

arr = np.zeros(1000)

print('start', datetime.now(), flush=True)
for ii in range(5000000):  # work
    arr += 1
    
print('break start', datetime.now(), flush=True)
time.sleep(3)  # break
print('break end', datetime.now(), flush=True)

for ii in range(5000000):  # work
    arr += 1
print('end', datetime.now(), flush=True)
"

rm -f ${LOG_OUTPUT}
rm -f ${LOG_RESOURCES}

log_process () {
  while ps -p $1 > /dev/null
  do
    top -p $1 -b -d 1 -n 1 >> $LOG_RESOURCES
    printf "\n\n" >> $LOG_RESOURCES
    sleep 1.
  done
}

python -c "${PYTHON_CODE}" >> ${LOG_OUTPUT} 2>&1 &  # run python in background
PID_PYTHON=$!  # python process id
echo "PID_PYTHON=${PID_PYTHON}"
tail -f ${LOG_OUTPUT} &  # show output on terminal
log_process $PID_PYTHON  # log needed resources to file

终端输出

$ bash monitor_python_script.sh 
PID_PYTHON=6807
start 2023-05-04 10:08:56.510581
end 2023-05-04 10:09:01.770897
break start 2023-05-04 10:09:04.280212
break end 2023-05-04 10:09:07.283365
end 2023-05-04 10:09:14.875994

日志文件输出：output.log

start 2023-05-04 10:08:56.510581
break start 2023-05-04 10:09:04.280212
break end 2023-05-04 10:09:07.283365
end 2023-05-04 10:09:14.875994

日志文件资源：resources.log

top - 10:08:56 up  1:47,  0 users,  load average: 0.08, 0.10, 0.09
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s): 16.1 us, 10.4 sy,  0.0 ni, 73.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29084.9 free,   3086.8 used,   2072.7 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30686.5 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:00.38 python

top - 10:08:57 up  1:47,  0 users,  load average: 0.08, 0.10, 0.09
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s): 14.6 us,  0.5 sy,  0.0 ni, 84.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29083.8 free,   3087.9 used,   2072.7 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30685.5 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:01.53 python

top - 10:08:58 up  1:47,  0 users,  load average: 0.23, 0.13, 0.10
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s): 15.2 us,  0.0 sy,  0.0 ni, 84.4 id,  0.5 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29083.9 free,   3087.8 used,   2072.8 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30685.6 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:02.69 python

top - 10:09:00 up  1:47,  0 users,  load average: 0.23, 0.13, 0.10
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s): 14.3 us,  0.0 sy,  0.0 ni, 85.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29083.1 free,   3088.5 used,   2072.8 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30684.9 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:03.85 python

top - 10:09:01 up  1:47,  0 users,  load average: 0.23, 0.13, 0.10
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s): 14.3 us,  0.0 sy,  0.0 ni, 85.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29083.3 free,   3088.3 used,   2072.8 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30685.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:05.01 python

top - 10:09:02 up  1:47,  0 users,  load average: 0.23, 0.13, 0.10
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.1 us,  0.5 sy,  0.0 ni, 92.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.5 st
MiB Mem :  34244.4 total,  29096.4 free,   3075.2 used,   2072.8 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30698.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:06.17 python

top - 10:09:03 up  1:47,  0 users,  load average: 0.29, 0.14, 0.11
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.1 us,  0.0 sy,  0.0 ni, 92.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29096.4 free,   3075.2 used,   2072.8 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30698.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:07.32 python

top - 10:09:04 up  1:47,  0 users,  load average: 0.29, 0.14, 0.11
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29096.9 free,   3074.7 used,   2072.8 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30698.6 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 S   0.0   0.1   0:08.05 python

top - 10:09:05 up  1:47,  0 users,  load average: 0.29, 0.14, 0.11
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.5 us,  0.0 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29096.6 free,   3074.9 used,   2073.0 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30698.3 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 S   0.0   0.1   0:08.05 python

top - 10:09:07 up  1:47,  0 users,  load average: 0.29, 0.14, 0.11
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.4 us,  0.5 sy,  0.0 ni, 98.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29087.0 free,   3084.6 used,   2072.8 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30688.8 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 S   0.0   0.1   0:08.05 python

top - 10:09:08 up  1:47,  0 users,  load average: 0.29, 0.14, 0.11
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.6 us,  0.0 sy,  0.0 ni, 92.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29086.5 free,   3085.1 used,   2072.9 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30688.4 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:08.95 python

top - 10:09:09 up  1:47,  0 users,  load average: 0.35, 0.16, 0.11
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8.1 us,  0.0 sy,  0.0 ni, 91.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29086.7 free,   3084.9 used,   2072.9 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30688.6 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:10.11 python

top - 10:09:10 up  1:47,  0 users,  load average: 0.35, 0.16, 0.11
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8.0 us,  0.0 sy,  0.0 ni, 92.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29087.0 free,   3084.5 used,   2072.9 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30688.9 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:11.27 python

top - 10:09:11 up  1:47,  0 users,  load average: 0.35, 0.16, 0.11
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8.1 us,  0.0 sy,  0.0 ni, 91.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29086.5 free,   3085.1 used,   2072.9 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30688.4 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:12.43 python

top - 10:09:12 up  1:47,  0 users,  load average: 0.35, 0.16, 0.11
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.1 us,  0.0 sy,  0.0 ni, 92.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29086.3 free,   3085.3 used,   2072.9 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30688.2 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:13.59 python

top - 10:09:13 up  1:47,  0 users,  load average: 0.40, 0.17, 0.12
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8.1 us,  0.0 sy,  0.0 ni, 91.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29086.2 free,   3085.3 used,   2072.9 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30688.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:14.74 python
赞(0）分享  回复(0）举报  2023-05-05
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com