在我的硕士论文中,我想进一步开发波罗的海地区的PyPSA-Eur能源系统模型。在我开始开发之前,我必须成功地运行模型。我正在运行PyPSA-通过从Github克隆代码并使用Visual Studio Code在Ubuntu WSL环境中运行代码来解决我之前遇到的权限错误。我没有对现有的代码,我根据environment.yaml文件更新了所有的包。Github上的PyPSA-Eur仓库:
https://github.com/PyPSA/pypsa-eur
我经常遇到的错误是在运行Snakemake --cores all in the terminal时,在规则build_renewable_profiles中出现错误。错误消息表明,在Snakemake管道中运行build_renewable_profiles规则的进程由于高内存使用率而被杀死。这可能是由于工作进程超出了其内存限制。
我尝试通过在Snakefile中添加脚本来增加Dask Worker的内存限制。此外,我尝试通过在WSL环境的用户目录中添加.wslconfig文件来增加WSL环境的内存限制,指定8 GB的内存。然而,这两种方法都没有解决错误。我的系统总内存为16 GB。
我恳请大家帮助我解决这类错误,如图所示。解决这个错误使我能够成功地运行PyPSA模型,并继续为我的硕士论文开发PyPSA-eur能源系统。
非常感谢,Stijn
运行snakemake后的完整终端消息的第一个图像:
x1c 0d1x的数据
运行snakemake后的完整终端消息的第二张图片,包括错误消息:
的
运行snakemake --cores all
后包含错误消息的终端部分:
INFO:__main__:Calculate landuse availabilities...
INFO:__main__:Completed availability calculation (95.49s)
INFO:atlite.convert:Convert and aggregate 'pv'.
2023-12-27 14:44:34,179 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 1.38 GiB -- Worker memory limit: 1.91 GiB
2023-12-27 14:44:36,119 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 1.36 GiB -- Worker memory limit: 1.91 GiB
2023-12-27 14:44:37,196 - distributed.worker.memory - WARNING - Worker is at 81% memory usage. Pausing worker. Process memory: 1.55 GiB -- Worker memory limit: 1.91 GiB
[Wed Dec 27 14:44:38 2023]
Error in rule build_renewable_profiles:
jobid: 10
input: resources/networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
output: resources/profile_solar.nc
log: logs/build_renewable_profile_solar.log (check log file(s) for error details)
conda-env: /home/cfl/pypsa-balticsea/.snakemake/conda/a193a967e4b3183c6023115ed840b879_
RuleException:
CalledProcessError in file /home/cfl/pypsa-balticsea/rules/build_electricity.smk, line 309:
Command 'set -euo pipefail; /home/cfl/miniconda3/envs/pypsa-eur/bin/python3.11 /home/cfl/pypsa-balticsea/.snakemake/scripts/tmpwdpwcq1a.build_renewable_profiles.py' died with <Signals.SIGKILL: 9>.
File "/home/cfl/pypsa-balticsea/rules/build_electricity.smk", line 309, in __rule_build_renewable_profiles
File "/home/cfl/miniconda3/envs/pypsa-eur/lib/python3.11/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
/home/cfl/miniconda3/envs/pypsa-eur/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 24 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Complete log: .snakemake/log/2023-12-27T144217.073372.snakemake.log
字符串
2条答案
按热度按时间cczfrluj1#
规则
build_renewable_profiles
可能是相当资源密集型的,并且可能并行运行多次(对于太阳能,陆上风能,海上风能等)。你可以再试一次:
字符串
这将一次只执行一个规则,最大限度地减少内存消耗。
b09cbbtk2#
Fabian的方法将使用大量内存来限制规则,但也会消除任何并行性。您有一些更好的选择:
字符串
使用
snakemake --resources lots_of_memory=1
调用。这将只允许规则的一个示例运行,但其他作业仍然可以同时运行。如果其他作业使用大量内存,这仍然可能导致崩溃。mem_mb
的使用,并在系统snakemake --resources mem_mb=16000
的约束下执行snakemake。Snakemake将负责计费,但 * 不会强制执行 * 限制。如果您说一个规则使用10 GB,但实际使用12 GB,则不会被snakemake杀死。这种方法更耗时,可能仍然存在问题,但如果需要的话,它将允许您轻松地过渡到集群/云。