我有一組火花Dataframes轉換成臨時視圖火花運行SQL。然後,我刪除我的邏輯/使用完成後。刪除步驟拋出一個奇怪的錯誤,我不知道如何解決。尋找一些建議在修複它。注意,集群是統一作為共享集群。
df.createOrReplaceTempView (prefix_updates) target.toDF () .createOrReplaceTempView (prefix_main) sql_begin =‘選擇’+ prefix_updates +”。*從“+ prefix_updates +”內連接“+ prefix_main +”在“merged_inserts =火花。sql (sql_begin + merge_match_ins + prefix_updates”,“+ +”。ReplicationUTCDateTime > .ReplicationUTCDateTime“+ prefix_main +) merged_inserts.write.format .mode(“δ”)(“追加”).saveAsTable(目錄+ '。' +模式+ '。' + table_name) spark.catalog.dropTempView (prefix_updates) spark.catalog.dropTempView (prefix_main)
spark.catalog之前的代碼。dropTempView工作很好,沒有錯誤。但當添加視圖中刪除語句我得到這個錯誤:
Py4JError:調用o1620.dropTempView時發生一個錯誤。跟蹤:
py4j.security。公共布爾org.apache.spark.sql.internal.CatalogImpl.dropTempView Py4JSecurityException:方法(以)不是白名單org.apache.spark.sql.internal.CatalogImpl類類
py4j.security.WhitelistingPy4JSecurityManager.checkCall (WhitelistingPy4JSecurityManager.java: 473)
py4j.Gateway.invoke (Gateway.java: 305)
py4j.commands.AbstractCommand.invokeMethod (AbstractCommand.java: 132)
py4j.commands.CallCommand.execute (CallCommand.java: 79)
py4j.ClientServerConnection.waitForCommands (ClientServerConnection.java: 195)
py4j.ClientServerConnection.run (ClientServerConnection.java: 115)
java.lang.Thread.run (Thread.java: 750)
