問題

你想讀數據拚花或δ格式,你得到一個損壞的鑲花頁麵錯誤。

. lang。RuntimeException:損壞的拚花在com.databricks.sql.io.parquet.NativeColumnReader頁麵。readBatchNative(本機方法)com.databricks.sql.io.parquet.NativeColumnReader.readBatch (NativeColumnReader.java: 477) com.databricks.sql.io.parquet.DatabricksVectorizedParquetRecordReader.nextBatch (DatabricksVectorizedParquetRecordReader.java: 346) org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue (VectorizedParquetRecordReader.java: 159) org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext (RecordReaderIterator.scala: 40) org.apache.spark.sql.execution.datasources.FileScanRDD立刻1美元立刻2.美元美元getnext (FileScanRDD.scala: 236) org.apache.spark.util.NextIterator.hasNext (NextIterator.scala: 73) org.apache.spark.sql.execution.datasources.FileScanRDD不久美元1.美元hasnext (FileScanRDD.scala: 204) org.apache.spark.sql.catalyst.expressions.GeneratedClass GeneratedIteratorForCodegenStage1美元。scan_nextBatch_0(未知源)在org.apache.spark.sql.catalyst.expressions.GeneratedClass GeneratedIteratorForCodegenStage1美元。agg_doAggregateWithoutKey_0(未知源)在org.apache.spark.sql.catalyst.expressions.GeneratedClass GeneratedIteratorForCodegenStage1美元。processNext(未知源)org.apache.spark.sql.execution.BufferedRowIterator.hasNext (BufferedRowIterator.java: 43) org.apache.spark.sql.execution.WholeStageCodegenExec anonfun 13美元立刻1.美元美元hasnext (WholeStageCodegenExec.scala: 640) scala.collection.Iterator不久美元11.美元hasnext (Iterator.scala: 409) org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write (BypassMergeSortShuffleWriter.java: 125) org.apache.spark.scheduler.ShuffleMapTask.runTask (ShuffleMapTask.scala: 99) org.apache.spark.scheduler.ShuffleMapTask.runTask (ShuffleMapTask.scala: 55) org.apache.spark.scheduler.Task.doRunTask (Task.scala: 139) org.apache.spark.scheduler.Task.run (Task.scala: 112) org.apache.spark.executor.Executor TaskRunner anonfun 13.美元美元申請(Executor.scala: 497)美元org.apache.spark.util.Utils .tryWithSafeFinally (Utils.scala: 1526)美元org.apache.spark.executor.Executor TaskRunner.run (Executor.scala: 503) java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java: 1149)美元java.util.concurrent.ThreadPoolExecutor Worker.run (ThreadPoolExecutor.java: 624) java.lang.Thread.run (Thread.java: 748)

導致

這個錯誤可以發生在鋪快讀者,讀者,矢量化讀者被啟用。

解決方案

如果你遇到損壞的鑲花頁麵錯誤,您應該禁用快速讀者,本地讀者,讀者和矢量化在集群或筆記本,然後讀操作再試。

如果錯誤仍然存在,即使這些選項禁用,打開一個與磚的支持。

禁用快速讀者

集spark.databricks.io.parquet.fastreader.enabled來假在集群的火花配置(AWS|Azure|GCP)禁用快速鋪讀者在集群級別。

你也可以禁用快速讀者在筆記本級別通過運行:

% scala spark.conf.set (“spark.databricks.io.parquet.fastreader.enabled”,“假”)

禁用本地讀者

集spark.databricks.io.parquet.nativeReader.enabled來假在集群的火花配置禁用本地集群級別鑲花的讀者。

你也可以禁用本地讀者在筆記本級別通過運行:

% scala spark.conf.set (“spark.databricks.io.parquet.nativeReader.enabled”,“假”)

禁用矢量化讀者

集spark.sql.parquet.enableVectorizedReader來假在集群的火花配置禁用矢量化鋪讀者在集群級別。

你也可以禁用筆記本級別的矢量化讀者通過運行:

% scala spark.conf.set (“spark.sql.parquet.enableVectorizedReader”,“假”)