我一直在試圖連接到紅移Databrick的文檔和驗證,我用我的集群上運行時版本11.3,我有讀/寫權限tempdir桶。但是,我無法加載數據從紅移到火花在筆記本數據幀。
我的代碼:
rsSQL = f“select *}{模式。{tbl_name},{場}({data_date})“rsRDD =(火花。讀.format(“紅移”).option(“查詢”,rsSQL) .option (“tempdir”、“s3: / / MYBUCKET”) .option (“url”, f " jdbc:紅移:/ / {rs('主機']}”).option(“用戶”,rs(“用戶”)).option(“密碼”,rs [' pwd ']) .option (“forward_spark_s3_credentials”,真的).load ())
錯誤了:
Py4JJavaError:調用o659.load時發生一個錯誤。:. lang。在com.databricks.spark.redshift.RedshiftRelation NullPointerException。anonfun模式1美元美元(RedshiftRelation.scala: 80) scala.Option.getOrElse (Option.scala: 189)美元com.databricks.spark.redshift.RedshiftRelation.schema lzycompute (RedshiftRelation.scala: 73) com.databricks.spark.redshift.RedshiftRelation.schema (RedshiftRelation.scala: 72) org.apache.spark.sql.execution.datasources.DataSource.resolveRelation (DataSource.scala: 498) org.apache.spark.sql.DataFrameReader.loadV1Source (DataFrameReader.scala: 375) org.apache.spark.sql.DataFrameReader anonfun負載2美元美元。(DataFrameReader.scala: 331) scala.Option.getOrElse (Option.scala: 189) org.apache.spark.sql.DataFrameReader.load (DataFrameReader.scala: 331) org.apache.spark.sql.DataFrameReader.load sun.reflect.NativeMethodAccessorImpl (DataFrameReader.scala: 223)。invoke0(本機方法)sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java: 62) sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java: 43) java.lang.reflect.Method.invoke (Method.java: 498) py4j.reflection.MethodInvoker.invoke (MethodInvoker.java: 244) py4j.reflection.ReflectionEngine.invoke (ReflectionEngine.java: 380) py4j.Gateway.invoke (Gateway.java: 306) py4j.commands.AbstractCommand.invokeMethod (AbstractCommand.java: 132) py4j.commands.CallCommand.execute (CallCommand.java: 79) py4j.ClientServerConnection.waitForCommands (ClientServerConnection.java: 195) py4j.ClientServerConnection.run (ClientServerConnection.java: 115) java.lang.Thread.run (Thread.java: 750)
根據文檔,運行時11.2及以上,磚運行時包含了紅移的JDBC驅動程序,可以使用的紅移字格式選項,但這並不似乎是為我工作,有什麼我失蹤嗎?
