我用databricks-connect訪問遠程集群。一切正常,我可以設置斷點並詢問結果,同樣在努力執行以下代碼:
val testDF = spark.createDataFrame (spark.sparkContext
.emptyRDD(行),StructType (
數組(
StringType StructField (“prop1”),
StringType StructField (“prop2”),
StringType StructField (“prop3”),
)))
logLn(“測試DF數:”+ testDF.count ())
我從Intellij運行。任何幫助或指針會感謝。
更新:
我第一集群目標是11.3。*我有錯誤。我試著測試針對7.3和它像預期的那樣工作。然後我10.4測試,得到了上述錯誤。
全文的錯誤:
. io .StreamCorruptedException:無效的類型代碼:00
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1700)
java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java: 2431)
java.io.ObjectInputStream.readSerialData (ObjectInputStream.java: 2355)
java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java: 2213)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1669)
java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java: 2431)
java.io.ObjectInputStream.readSerialData (ObjectInputStream.java: 2355)
java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java: 2213)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1669)
java.io.ObjectInputStream.readObject (ObjectInputStream.java: 503)
java.io.ObjectInputStream.readObject (ObjectInputStream.java: 461)
scala.collection.immutable.List SerializationProxy.readObject美元(List.scala: 527)
在sun.reflect.NativeMethodAccessorImpl。invoke0(本地方法)
sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java: 62)
sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java: 43)
java.lang.reflect.Method.invoke (Method.java: 498)
java.io.ObjectStreamClass.invokeReadObject (ObjectStreamClass.java: 1184)
java.io.ObjectInputStream.readSerialData (ObjectInputStream.java: 2322)
java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java: 2213)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1669)
java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java: 2431)
java.io.ObjectInputStream.readSerialData (ObjectInputStream.java: 2355)
java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java: 2213)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1669)
java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java: 2431)
java.io.ObjectInputStream.readSerialData (ObjectInputStream.java: 2355)
java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java: 2213)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1669)
java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java: 2431)
java.io.ObjectInputStream.readSerialData (ObjectInputStream.java: 2355)
java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java: 2213)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1669)
java.io.ObjectInputStream.readArray (ObjectInputStream.java: 2119)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1657)
java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java: 2431)
java.io.ObjectInputStream.readSerialData (ObjectInputStream.java: 2355)
java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java: 2213)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1669)
java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java: 2431)
java.io.ObjectInputStream.readSerialData (ObjectInputStream.java: 2355)
java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java: 2213)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1669)
java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java: 2431)
java.io.ObjectInputStream.readSerialData (ObjectInputStream.java: 2355)
java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java: 2213)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1669)
java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java: 2431)
java.io.ObjectInputStream.readSerialData (ObjectInputStream.java: 2355)
java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java: 2213)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1669)
java.io.ObjectInputStream.readObject (ObjectInputStream.java: 503)
java.io.ObjectInputStream.readObject (ObjectInputStream.java: 461)
scala.collection.immutable.List SerializationProxy.readObject美元(List.scala: 527)
在sun.reflect.NativeMethodAccessorImpl。invoke0(本地方法)
sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java: 62)
sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java: 43)
java.lang.reflect.Method.invoke (Method.java: 498)
java.io.ObjectStreamClass.invokeReadObject (ObjectStreamClass.java: 1184)
java.io.ObjectInputStream.readSerialData (ObjectInputStream.java: 2322)
java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java: 2213)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1669)
java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java: 2431)
java.io.ObjectInputStream.readSerialData (ObjectInputStream.java: 2355)
java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java: 2213)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1669)
java.io.ObjectInputStream.readArray (ObjectInputStream.java: 2119)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1657)
java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java: 2431)
java.io.ObjectInputStream.readSerialData (ObjectInputStream.java: 2355)
java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java: 2213)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java: 1669)
java.io.ObjectInputStream.readObject (ObjectInputStream.java: 503)
java.io.ObjectInputStream.readObject (ObjectInputStream.java: 461)
在org.apache.spark.sql.util.ProtoSerializer。anonfun deserializeObject美元1美元(ProtoSerializer.scala: 7248)
…