問題
你會斷斷續續NullPointerException保存數據時出錯。
Py4JJavaError:調用o2892.save時發生錯誤。: java.lang.NullPointerException at org.apache.spark.sql. execute .adaptive. optimizeskewedjoin .$anonfun$getMapSizesForReduceId$1(OptimizeSkewedJoin.scala:167) at org.apache.spark.sql. execute .adaptive. optimizeskewedjoin .$anonfun$getMapSizesForReduceId$1(OptimizeSkewedJoin.scala:167) at scala.collection. traversabllike .$anonfun$map$1(traversabljoin .scala:238) at scala.collection.IndexedSeqOptimized.foreach(indexedseqoptimize .scala:36) ....在sun.reflect.DelegatingMethodAccessorImpl.invoke sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java: 62) (DelegatingMethodAccessorImpl.java: 43) java.lang.reflect.Method.invoke (Method.java: 498) py4j.reflection.MethodInvoker.invoke (MethodInvoker.java: 244) py4j.reflection.ReflectionEngine.invoke (ReflectionEngine.java: 380) py4j.Gateway.invoke (Gateway.java: 295) py4j.commands.AbstractCommand.invokeMethod (AbstractCommand.java: 132)py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:251) at java.lang.Thread.run(Thread.java:748)
導致
如果自適應查詢執行(AQE) (AWS|Azure)已啟用，您正在連接數據。如果啟用了AQE，傾斜連接也會啟用。
如果任何shuffle數據由於集群縮放事件而失敗，它將生成一個NullPointerException錯誤。