我在一個集群上運行jupyter筆記本配置:
12.2 LTS(包括Apache火花3.3.2,Scala 2.12)
工作類型:i3。超大30.5 gb內存,4核
最小2和最大8個工人
鼠標光標= conn.cursor ()。執行(““SELECT * FROM (SELECT *,隨機()作為樣本從資源類型=“類型1”和created_at > = 2022-01-01就是和created_at < = 2022-06-30就是)在桑普示例< . 05;——返回5%的行”“”)#熊貓DataFrame df = DataFrame (cursor.fetchall()) # # #得到列名field_names =[我[0]我在cursor.description] df。列= field_names
致命錯誤:Python的內核是反應遲鈍。- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Python程序退出,退出碼137 (SIGKILL:死亡)。這可能是由一個伯父錯誤引起的。檢查你的命令的內存使用。過程的最後10 KB的stderr和stdout下麵可以找到。看到司機日誌完全日誌。- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -最後的消息stderr: ks / python / lib / python3.9 /網站/ IPython /核心/ ultratb。py”, 1112行,在structured_traceback FormattedTB返回。structured_traceback(文件“/磚/ python / lib / python3.9 /網站/ IPython /核心/ ultratb。py”, 1006行,在structured_traceback文件“/磚/ python / lib / python3.9 /網站/ stack_data /核心。py”, 649行,included_pieces pos = scope_pieces.index (self.executing_piece)文件“/磚/ python / lib / python3.9 /網站/ stack_data /跑龍套。py”, 145行,cached_property_wrapper價值= obj.__dict__ [self.func。__name__] = self.func(obj) File "/databricks/python/lib/python3.9/site-packages/executing/executing.py", line 164, in only raise NotOneValueFound('Expected one value, found 0') executing.executing.NotOneValueFound: Expected one value, found 0 Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/redshift_connector/core.py", line 1631, in execute ps = cache["ps"][key] KeyError: ('SELECT * \nFROM \n SELECT * \n FROM tbl\n WHERE event_type = %s\n AND created_at >= %s\n AND created_at <= %s \n \n LIMIT 5', ((, 0, ), (, 0, ), (, 0, )))
數據是相當大的,大概300 - 400 m行。我需要修改什麼配置?或優化查詢?並行處理等。
