读取自动缩放的 Ray 工作节点的日志

如何解决读取自动缩放的 Ray 工作节点的日志

我们在 kubernetes 上运行具有自动缩放功能的 ray 任务。有时，一名工人死亡，我们会得到以下信息：

WARNING worker.py:1114 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem,check the logs for the dead worker. Task ID: 874253cc82fde8d4ffffffffffffffffffffffff02000000 Worker ID: 98dcd78c9bebbc2176a14fc230b40afdd574dea608dd573aebeae00e Node ID: d4a7b594aec2970b02dde480f2eb2c6070b7886b333d8f1cb1087758 Worker IP address: 10.40.4.2 Worker port: 10010 Worker PID: 307

我们正在使用类似于 autoscaler/kubernetes/default.yaml 的 config.yaml。

我们面临的问题是工作线程运行的节点是自动缩放的，导致节点经常在我们有时间读取日志（在 /tmp/ray 中）之前被缩放。

有什么办法可以保留自动缩放（工作）节点的日志？

我尝试通过调用 ray.init(_temp_dir="/our_mounted_filestore/persistent_ray_logs") 将临时目录设置为指向工作人员之间共享的 filestore，但这似乎不起作用。

我也尝试在 worker_start_ray_commands 中添加 --temp-dir=<path>，但这只会出现以下错误：

PermissionError: [Errno 13] Permission denied: '/filestore/ray_temp'
mkdir(name,mode)
File "/home/ray/anaconda3/lib/python3.8/os.py",line 223,in makedirs
makedirs(head,exist_ok=exist_ok)
File "/home/ray/anaconda3/lib/python3.8/os.py",line 213,in makedirs
os.makedirs(directory_path,exist_ok=True)
File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/_private/utils.py",line 797,in try_to_create_directory
try_to_create_directory(self._temp_dir)
File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/node.py",line 278,in _init_temp
self._init_temp(redis_client)
File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/node.py",line 166,in init
node = ray.node.Node(
File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/scripts/scripts.py",line 580,in start
return __callback(*args,**kwargs)
File "/home/ray/anaconda3/lib/python3.8/site-packages/click/core.py",line 763,in invoke
return ctx.invoke(self.callback,**ctx.params)
File "/home/ray/anaconda3/lib/python3.8/site-packages/click/core.py",line 1404,in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/ray/anaconda3/lib/python3.8/site-packages/click/core.py",line 1668,in invoke
rv = self.invoke(ctx)
File "/home/ray/anaconda3/lib/python3.8/site-packages/click/core.py",line 1062,in main
return self.main(*args,line 1137,in call
return cli()
File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/scripts/scripts.py",line 1808,in main
sys.exit(main())
File "/home/ray/anaconda3/bin/ray",line 8,in
Traceback (most recent call last):