我的代碼:
從great_expectations。數據源導入SparkDFDatasource
從pyspark。sql進口SparkSession
session_name = ' mk_spark_session '
火花= SparkSession.builder.appName (session_name) .getOrCreate ()
數據源= SparkDFDatasource(火花)
查詢= " SELECT * FROM test_large_table限製10 "
df = spark.sql(查詢)
打印(df)
我在Azure磚運行以上代碼。
火花版本是3.4.0
pyspark是3.4.0
py4j是0.10.9.7
我得到以下錯誤
py4j.security。公共org.apache.spark.SparkConf Py4JSecurityException:構造函數(布爾)不是白名單。
Py4JError:調用None.org.apache.spark.SparkConf時發生一個錯誤。跟蹤:
py4j.security。公共org.apache.spark.SparkConf Py4JSecurityException:構造函數(布爾)不是白名單。
py4j.security.WhitelistingPy4JSecurityManager.checkConstructor (WhitelistingPy4JSecurityManager.java: 451)
py4j.Gateway.invoke (Gateway.java: 256)
py4j.commands.ConstructorCommand.invokeConstructor (ConstructorCommand.java: 80)
py4j.commands.ConstructorCommand.execute (ConstructorCommand.java: 69)
py4j.ClientServerConnection.waitForCommands (ClientServerConnection.java: 195)
py4j.ClientServerConnection.run (ClientServerConnection.java: 115)
java.lang.Thread.run (Thread.java: 750)
