Bigquery -磚的集成問題。

[ekar-databricks]

我試圖讓Bigquery數據磚使用筆記本。後在此基礎上的步驟https://docs.www.eheci.com/external-data/bigquery.html。我相信這一步我犯了一些錯誤,以下錯誤。我試著給. json文件輸入,但還是同樣的錯誤。

麵對這個錯誤:

“Py4JJavaError:調用o451.load時發生一個錯誤。

com.google.cloud.spark.bigquery.repackaged.com.google.inject.ProvisionException:無法提供,請參閱以下錯誤:

1)在自定義服務提供方程序錯誤,. lang。IllegalArgumentException:一個項目ID需要這個服務但不能決定從建設者或環境。請設定一個項目使用builder ID。

com.google.cloud.spark.bigquery.SparkBigQueryConnectorModule.provideSparkBigQueryConfig (SparkBigQueryConnectorModule.java: 65)

而定位com.google.cloud.spark.bigquery.SparkBigQueryConfig

[匿名]

@Madhan Potluri:

你看到的錯誤消息表明,有一個問題與項目ID加載數據時從BigQuery磚。為了解決這個問題,您可以在代碼中顯式地設置項目ID。

這裏有一個例子代碼片段展示了如何設置項目ID加載數據時從BigQuery磚:

從穀歌進口操作係統。oauth2進口service_account #將路徑設置為你的服務帳戶JSON密鑰文件service_account_json = ' /路徑/ / service_account。json #設置項目ID project_id = ' your-project-id ' #加載服務帳戶關鍵憑證= service_account.Credentials.from_service_account_file (service_account_json) #設置為穀歌的雲項目ID係統環境變量。環境[' GOOGLE_CLOUD_PROJECT '] = project_id #加載數據從BigQuery磚df = spark.read.format (BigQuery) \ .option(“表”、“your_dataset.your_table”) \ .option(憑證,憑證)\ .load ()

一定要用你的穀歌的實際項目ID替換your-project-id雲項目,和

your_dataset。與實際BigQuery your_table表你想加載數據。同時,確保服務帳戶JSON密鑰文件路徑是正確的。

我希望這可以幫助!讓我知道如果你有任何進一步的問題。

[ekar-databricks]

我試著上麵的方法,但仍然得到以下錯誤。

我們需要編碼的關鍵。json文件嗎?如果是的,請建議明確的步驟。

194 def德科(*:任何* *千瓦:有)- >任何:

195年試題:

- - > 196返回f(*, * *千瓦)

197年除了Py4JJavaError e:

198 = convert_exception轉換(e.java_exception)

/磚/火花/ python / lib / py4j-0.10.9.5-src.zip / py4j /協議。py get_return_value(答案,gateway_client target_id,名稱)

324值= OUTPUT_CONVERTER[型](回答[2],gateway_client)

325如果[1]= = REFERENCE_TYPE回答:

- - > 326提高Py4JJavaError (

327”時發生一個錯誤調用{0}{1}{2}。\ n”。

328格式(target_id”。“名字),值)

Py4JJavaError:調用o1217.load時發生一個錯誤。

com.google.cloud.spark.bigquery.repackaged.com.google.inject.ProvisionException:無法提供,請參閱以下錯誤:

1)在自定義服務提供方程序錯誤,. lang。IllegalArgumentException: com.google.cloud.spark.bigquery.repackaged.com.google.common.io.BaseEncoding DecodingException美元:無法識別的字符:<

com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.connector.common.BigQueryClientModule.provideBigQueryCredentialsSupplier (BigQueryClientModule.java: 46)

而定位com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.connector.common.BigQueryCredentialsSupplier

的第三個參數com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.connector.common.BigQueryClientModule.provideBigQueryClient (BigQueryClientModule.java: 63)

com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.connector.common.BigQueryClientModule.provideBigQueryClient (BigQueryClientModule.java: 63)

而定位com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.connector.common.BigQueryClient

1錯誤

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.InternalProvisionException.toProvisionException (InternalProvisionException.java: 226)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.InjectorImpl得到1.美元(InjectorImpl.java: 1097)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.InjectorImpl.getInstance (InjectorImpl.java: 1131)

com.google.cloud.spark.bigquery.BigQueryRelationProvider.createRelationInternal (BigQueryRelationProvider.scala: 80)

在com.google.cloud.spark.bigquery.BigQueryRelationProvider.createRelation (BigQueryRelationProvider.scala: 48)

org.apache.spark.sql.execution.datasources.DataSource.resolveRelation (DataSource.scala: 387)

org.apache.spark.sql.DataFrameReader.loadV1Source (DataFrameReader.scala: 375)

在org.apache.spark.sql.DataFrameReader。anonfun負載2美元美元(DataFrameReader.scala: 331)

scala.Option.getOrElse (Option.scala: 189)

org.apache.spark.sql.DataFrameReader.load (DataFrameReader.scala: 331)

org.apache.spark.sql.DataFrameReader.load (DataFrameReader.scala: 223)

在sun.reflect.GeneratedMethodAccessor361。調用(未知源)

sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java: 43)

java.lang.reflect.Method.invoke (Method.java: 498)

py4j.reflection.MethodInvoker.invoke (MethodInvoker.java: 244)

py4j.reflection.ReflectionEngine.invoke (ReflectionEngine.java: 380)

py4j.Gateway.invoke (Gateway.java: 306)

py4j.commands.AbstractCommand.invokeMethod (AbstractCommand.java: 132)

py4j.commands.CallCommand.execute (CallCommand.java: 79)

py4j.ClientServerConnection.waitForCommands (ClientServerConnection.java: 195)

py4j.ClientServerConnection.run (ClientServerConnection.java: 115)

java.lang.Thread.run (Thread.java: 750)

引起的:. lang。IllegalArgumentException: com.google.cloud.spark.bigquery.repackaged.com.google.common.io.BaseEncoding DecodingException美元:無法識別的字符:<

com.google.cloud.spark.bigquery.repackaged.com.google.common.io.BaseEncoding.decode (BaseEncoding.java: 218)

com.google.cloud.spark.bigquery.repackaged.com.google.api.client.util.Base64.decodeBase64 (Base64.java: 106)

com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.connector.common.BigQueryCredentialsSupplier.createCredentialsFromKey (BigQueryCredentialsSupplier.java: 71)

在com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.connector.common.BigQueryCredentialsSupplier。< init > (BigQueryCredentialsSupplier.java: 45)

在com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.connector.common.BigQueryClientModule.provideBigQueryCredentialsSupplier (BigQueryClientModule.java: 53)

在com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.connector.common.BigQueryClientModule FastClassByGuice美元b1b60333.invoke美元(<生成>)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.ProviderMethod FastClassProviderMethod.doProvision美元(ProviderMethod.java: 264)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.ProviderMethod.doProvision (ProviderMethod.java: 173)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.InternalProviderInstanceBindingImpl CyclicFactory.provision美元(InternalProviderInstanceBindingImpl.java: 185)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.InternalProviderInstanceBindingImpl CyclicFactory.get美元(InternalProviderInstanceBindingImpl.java: 162)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.ProviderToInternalFactoryAdapter.get (ProviderToInternalFactoryAdapter.java: 40)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.SingletonScope得到1.美元(SingletonScope.java: 168)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.InternalFactoryToProviderAdapter.get (InternalFactoryToProviderAdapter.java: 39)

在com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.SingleParameterInjector.inject (SingleParameterInjector.java: 42)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.SingleParameterInjector.getAll (SingleParameterInjector.java: 65)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.ProviderMethod.doProvision (ProviderMethod.java: 173)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.InternalProviderInstanceBindingImpl CyclicFactory.provision美元(InternalProviderInstanceBindingImpl.java: 185)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.InternalProviderInstanceBindingImpl CyclicFactory.get美元(InternalProviderInstanceBindingImpl.java: 162)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.ProviderToInternalFactoryAdapter.get (ProviderToInternalFactoryAdapter.java: 40)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.SingletonScope得到1.美元(SingletonScope.java: 168)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.InternalFactoryToProviderAdapter.get (InternalFactoryToProviderAdapter.java: 39)

com.google.cloud.spark.bigquery.repackaged.com.google.inject.internal.InjectorImpl得到1.美元(InjectorImpl.java: 1094)

…20多

引起的:com.google.cloud.spark.bigquery.repackaged.com.google.common.io.BaseEncoding DecodingException美元:無法識別的字符:<

com.google.cloud.spark.bigquery.repackaged.com.google.common.io.BaseEncoding Alphabet.decode美元(BaseEncoding.java: 492)

com.google.cloud.spark.bigquery.repackaged.com.google.common.io.BaseEncoding Base64Encoding.decodeTo美元(BaseEncoding.java: 972)

com.google.cloud.spark.bigquery.repackaged.com.google.common.io.BaseEncoding.decodeChecked (BaseEncoding.java: 233)

com.google.cloud.spark.bigquery.repackaged.com.google.common.io.BaseEncoding.decode (BaseEncoding.java: 216)