Ray 和 numba jit 不兼容

分类：编程问答

如何解决Ray 和 numba jit 不兼容

我想结合 Ray 和 numba 进行并行计算。但是，Ray 无法使用 numba 正常工作。

如果 ray 和 numba 内核在同一个 notebook 中，则它工作正常，即

import ray
from numba import jit

ray.init(ignore_reinit_error=True)

@ray.remote
def f1():

    @jit
    def compute(x,y):
        return x+y
    re = compute(1,2)
    return re
print(ray.get(f1.remote()))

但是，如果我将以下内容放入 f.py 文件中，则会失败：

# the following is in f.py
import ray
@ray.remote
def f1():
    
    from numba import jit,float64

    @jit
    def f(x,y):
        # A somewhat trivial example
        return x + y
    
    return f(1.,2.)

如果我运行，我会得到一个错误：

import ray
from numba import jit

ray.init(ignore_reinit_error=True)

from f import f1

print(ray.get(f1.remote()))

这是错误：

2021-03-16 16:42:07,067 INFO services.py:1172 -- View the Ray dashboard at http://127.0.0.1:8265
(pid=3673426) /data2/zhangjuenjie/zjj/CRBMG/f.py:9: NumbaWarning: 
(pid=3673426) Compilation is falling back to object mode WITH looplifting enabled because Function f failed at nopython mode lowering due to: can't compile f1.<locals>.f: import of module f failed
(pid=3673426)   @jit
---------------------------------------------------------------------------
RayTaskError(ModuleNotFoundError)         Traceback (most recent call last)
<ipython-input-1-78c9292bc84b> in <module>
      6 from f import f1
      7 
----> 8 print(ray.get(f1.remote()))

~/anaconda3/envs/crbmg/lib/python3.8/site-packages/ray/_private/client_mode_hook.py in wrapper(*args,**kwargs)
     45         if client_mode_enabled and _client_hook_enabled:
     46             return getattr(ray,func.__name__)(*args,**kwargs)
---> 47         return func(*args,**kwargs)
     48 
     49     return wrapper

~/anaconda3/envs/crbmg/lib/python3.8/site-packages/ray/worker.py in get(object_refs,timeout)
   1454                     worker.core_worker.dump_object_store_memory_usage()
   1455                 if isinstance(value,RayTaskError):
-> 1456                     raise value.as_instanceof_cause()
   1457                 else:
   1458                     raise value

RayTaskError(ModuleNotFoundError): ray::f1() (pid=3673426,ip=210.45.78.32)
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/weakref.py",line 131,in __getitem__
    o = self.data[key]()
KeyError: '_ZN08NumbaEnv1f2f112$3clocals$3e5f$241E8pyobject8pyobject'

During handling of the above exception,another exception occurred:

ray::f1() (pid=3673426,ip=210.45.78.32)
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/funcdesc.py",line 93,in lookup_module
    return sys.modules[self.modname]
KeyError: 'f'

During handling of the above exception,ip=210.45.78.32)
  File "python/ray/_raylet.pyx",line 473,in ray._raylet.execute_task
  File "python/ray/_raylet.pyx",line 476,line 480,in ray._raylet.execute_task
  File "/data2/zhangjuenjie/zjj/CRBMG/f.py",line 14,in f1
    return f(1.,2.)
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/dispatcher.py",line 433,in _compile_for_args
    raise e
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/dispatcher.py",line 366,in _compile_for_args
    return self.compile(tuple(argtypes))
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/compiler_lock.py",line 32,in _acquire_compile_lock
    return func(*args,**kwargs)
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/dispatcher.py",line 857,in compile
    cres = self._compiler.compile(args,return_type)
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/dispatcher.py",line 77,in compile
    status,retval = self._compile_cached(args,line 91,in _compile_cached
    retval = self._compile_core(args,line 104,in _compile_core
    cres = compiler.compile_extra(self.targetdescr.typing_context,File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/compiler.py",line 602,in compile_extra
    return pipeline.compile_extra(func)
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/compiler.py",line 352,in compile_extra
    return self._compile_bytecode()
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/compiler.py",line 414,in _compile_bytecode
    return self._compile_core()
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/compiler.py",line 394,in _compile_core
    raise e
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/compiler.py",line 385,in _compile_core
    pm.run(self.state)
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/compiler_machinery.py",line 339,in run
    raise patched_exception
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/compiler_machinery.py",line 330,in run
    self._runPass(idx,pass_inst,state)
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/compiler_lock.py",**kwargs)
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/compiler_machinery.py",line 289,in _runPass
    mutated |= check(pss.run_pass,internal_state)
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/compiler_machinery.py",line 262,in check
    mangled = func(compiler_state)
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/object_mode_passes.py",line 120,in run_pass
    lowered = backend_object_mode()
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/object_mode_passes.py",line 115,in backend_object_mode
    return self._py_lowering_stage(state.targetctx,File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/object_mode_passes.py",in _py_lowering_stage
    lower = pylowering.PyLower(targetctx,library,fndesc,interp)
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/lowering.py",line 37,in __init__
    self.env = Environment.from_fndesc(self.fndesc)
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/environment.py",line 23,in from_fndesc
    inst = cls(fndesc.lookup_globals())
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/funcdesc.py",line 81,in lookup_globals
    return self.global_dict or self.lookup_module().__dict__
  File "/data2/zhangjuenjie/anaconda3/envs/crbmg/lib/python3.8/site-packages/numba/core/funcdesc.py",line 95,in lookup_module
    raise ModuleNotFoundError(
ModuleNotFoundError: can't compile f1.<locals>.f: import of module f failed

这很奇怪，因为什么都没发生。

解决方法

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。

相关推荐

导入项目后报错问题
依赖报错 idea导入项目后依赖报错，解决方案：https://blog.csdn.net/weixin_42420249/article/details/81191861 依赖版本报错：更换其他版本 无法下载依赖可参考：https://blog.csdn.net/weixin_42628809/a
idea不能识别yaml文件
使用mybatis plus常见错误
错误1：代码生成器依赖和mybatis依赖冲突 启动项目时报错如下 2021-12-03 13:33:33.927 ERROR 7228 [ main] o.s.b.d.LoggingFailureAnalysisReporter : *************************** APPL
gradle常见问题与错误
错误1：gradle项目控制台输出为乱码 # 解决方案：https://blog.csdn.net/weixin_43501566/article/details/112482302 # 在gradle-wrapper.properties 添加以下内容 org.gradle.jvmargs=-Df
Mybatis Plus传入参数0不起作用
错误还原：在查询的过程中，传入的workType为0时，该条件不起作用 &lt;select id=&quot;xxx&quot;&gt; SELECT di.id, di.name, di.work_type, di.updated... &lt;where&gt; &lt;if test=&qu
linux中make编译源码包失败
报错如下，gcc版本太低 ^ server.c:5346:31: 错误：‘struct redisServer’没有名为‘server_cpulist’的成员 redisSetCpuAffinity(server.server_cpulist); ^ server.c: 在函数‘hasActiveC
微服务启动错误：Command line is too long. Shorten command line for
解决方案1 1、改项目中.idea/workspace.xml配置文件，增加dynamic.classpath参数 2、搜索PropertiesComponent，添加如下 &lt;property name=&quot;dynamic.classpath&quot; value=&quot;tru
vue常见错误
删除根组件app.vue中的默认代码后报错：Module Error (from ./node_modules/eslint-loader/index.js): 解决方案：关闭ESlint代码检测，在项目根目录创建vue.config.js，在文件中添加 module.exports = { lin
spark常见错误
查看spark默认的python版本 [root@master day27]# pyspark /home/software/spark-2.3.4-bin-hadoop2.7/conf/spark-env.sh: line 2: /usr/local/hadoop/bin/hadoop: No s
matplotlib报错：AttributeError: module 'backend_interagg' has no attribute 'FigureCanvas'. Did you mean: 'FigureCanvasAgg'?
使用本地python环境可以成功执行 import pandas as pd import matplotlib.pyplot as plt # 设置字体 plt.rcParams[&#39;font.sans-serif&#39;] = [&#39;SimHei&#39;] # 能正确显示负号 p
gitlab登录失败，报错：This challenge page was accidentally cached by an intermediary and is no longer available.
设置时间 控制面板
后端开发常见错误
错误1：Request method ‘DELETE‘ not supported 错误还原：controller层有一个接口，访问该接口时报错：Request method ‘DELETE‘ not supported 错误原因：没有接收到前端传入的参数，修改为如下 参考 错误2：cannot r
docker常见错误
错误1：启动docker镜像时报错：Error response from daemon: driver failed programming external connectivity on endpoint quirky_allen 解决方法：重启docker -&gt; systemctl r
idea常见错误
错误1：private field ‘xxx‘ is never assigned 按Altʾnter快捷键，选择第2项 参考：https://blog.csdn.net/shi_hong_fei_hei/article/details/88814070 错误2：启动时报错，不能找到主启动类 #
pip安装依赖失败
报错如下，通过源不能下载，最后警告pip需升级版本 Requirement already satisfied: pip in c:\users\ychen\appdata\local\programs\python\python310\lib\site-packages (22.0.4) Coll
maven常见错误
错误1：maven打包报错 错误还原：使用maven打包项目时报错如下 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:3.2.0:resources (default-resources)
cloud项目中常见错误
错误1：服务调用时报错 服务消费者模块assess通过openFeign调用服务提供者模块hires 如下为服务提供者模块hires的控制层接口 @RestController @RequestMapping(&quot;/hires&quot;) public class FeignControl
springboot常见错误
错误1：运行项目后报如下错误 解决方案 报错2：Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project sb 解决方案：在pom.
springmvc拦截器中使用redisTemplate报空指针异常
参考 错误原因 过滤器或拦截器在生效时，redisTemplate还没有注入 解决方案：在注入容器时就生效 @Component //项目运行时就注入Spring容器 public class RedisBean { @Resource private RedisTemplate&lt;String
vite报错：npm ERR! command C:WINDOWSsystem32cmd.exe /d /s /c C:UsersychenAppDataLocalTempnpx-dde1c848.cmd
使用vite构建项目报错 C:\Users\ychen\work&gt;npm init @vitejs/app @vitejs/create-app is deprecated, use npm init vite instead C:\Users\ychen\AppData\Local\npm-
pythonJavaScriptjavaHTMLPHPreactjsC#AndroidCSSNode.jssqlrpython-3.xMysqLjQueryc++pandasFlutterangularIOSdjangolinuxswifttypescript路由器JSON路由器设置无线路由器h3c华三华三路由器设置华三路由器电脑软件教程arraysdocker软件图文教程Cvue.jslaravelspring-boot