現在,我麵臨一個問題,因為“databricks-connect”我們的集群上運行時更新到10.4。從那以後,我不能加載jar spark-avro了。通過運行下麵的代碼
從pyspark。sql進口SparkSession火花= SparkSession.builder.config (“spark.jars。包”、“org.apache.spark: spark-avro_2.12:3.3.0”) .getOrCreate ()
我得到以下錯誤:
的jar包存儲在:C:\Users\lazlo\.ivy2 \ jars org.apache.spark # spark-avro_2.12作為依賴項添加:::解決依賴關係:org.apache.spark #火花-提交-家長- dc011dfd - 9 - d25 - 4 - d6f - 9 - d0e - 354626 - e7c1f8; 1.0相依:(默認)發現org.apache.spark # spark-avro_2.12; 3.3.0中部發現org.tukaani # xz; 1.8中部發現org.spark-project.spark #未使用;中部1.0.0::決議報告::解決156 ms:工件dl 4:女士:模塊使用:org.apache.spark # spark-avro_2.12;從中央3.3.0(默認)org.spark-project.spark #未使用;從中央1.0.0(默認)org.tukaani # xz; 1.8從中央(默認)- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | |模塊| | |工件數量配置| | |搜索| dwnlded |驅逐| | |數量dwnlded | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -默認| | 3 | 0 | 0 | 0 | | 3 | 0 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -::檢索::表示。火花#火花-提交-家長- dc011dfd - 9 - d25 - 4 - d6f - 9 - d0e - 354626 - e7c1f8相依:0(默認)工件複製,3已經檢索(0 5 kb / ms) 22/08/16 13:15:57警告殼:沒有發現winutils。exe: {}
…
回溯(最近調用最後):文件“C: / Aifora /倉庫/ test_poetry / tmp_jars。py”, 4號線,在火花= SparkSession.builder.config (“spark.jars。包”、“org.apache.spark: spark-avro_2.12:3.3.0”) .getOrCreate()文件“C: \ lazlo \ AppData \ \用戶本地緩存\ pypoetry \ \ virtualenv \ test-poetry-vvodToDL-py3.8 \ lib \網站\ pyspark \ sql \會話。py”, 229行,getOrCreate sc = SparkContext.getOrCreate”(sparkConf)文件C: \ lazlo \ AppData \ \用戶本地緩存\ pypoetry \ \ virtualenv \ test-poetry-vvodToDL-py3.8 \ lib \網站\ pyspark \上下文。py”, 400行,在getOrCreate SparkContext(參看= conf或SparkConf())文件“C: \ lazlo \ AppData \ \用戶本地緩存\ pypoetry \ \ virtualenv \ test-poetry-vvodToDL-py3.8 \ lib \網站\ pyspark \上下文。py”, 147行,在__init__自我。_do_init(主瀏覽器名稱、sparkHome pyFiles,環境,batchSize,序列化器,文件C: \“用戶\ lazlo \ AppData \本地緩存\ pypoetry \ \ virtualenv \ test-poetry-vvodToDL-py3.8 \ lib \網站\ pyspark \上下文。py”, 210行,在_do_init自我。_jsc = jsc或self._initialize_context (self._conf._jconf)文件C: \“用戶\ lazlo \ AppData \本地緩存\ pypoetry \ \ virtualenv \ test-poetry-vvodToDL-py3.8 \ lib \網站\ pyspark \上下文。py”, 337行,在_initialize_context返回self._jvm.JavaSparkContext”(jconf)文件C: \ lazlo \ AppData \ \用戶本地緩存\ pypoetry \ \ virtualenv \ test-poetry-vvodToDL-py3.8 \ lib \網站\ py4j \ java_gateway。py”, 1568行,在__call__ return_value = get_return_value(文件C: \用戶\ lazlo \ AppData \本地緩存\ pypoetry \ \ virtualenv \ test-poetry-vvodToDL-py3.8 \ lib \網站\ py4j \協議。py”, 326行,get_return_value提高Py4JJavaError (py4j.protocol。Py4JJavaError:調用None.org.apache.spark.api.java.JavaSparkContext時發生一個錯誤。
如果重要的是:我使用windows機器(windows 11)通過詩歌和管理包。在這裏我pyproject.toml
(工具。詩歌)的名字= " test_poetry " version = " 1.37.5”描述=”“作者= [“lazloo xp”] [[tool.poetry。源]]name = " * * * _nexus " url = " https://nexus.infrastructure.xxxx.net/repository/pypi-all/simple/ "默認= true tool.poetry.dependencies python =”= = 3.8。* ^ 10.4“databricks-connect =
