磚與Conda 6.0運行時(不支持)
請注意
這個版本是不再可用。如果你想使用Conda來管理Python庫和環境,使用支持版本機器學習的磚運行時。
磚運行時6.0 Conda(β)允許您利用Conda管理Python庫和環境。這個運行時提供了兩根Conda環境選擇在創建集群:
磚的標準環境包括許多流行的Python包的更新版本。這個環境是作為現有筆記本電腦上運行的替代數據磚運行時。這是默認磚Conda-based運行時環境。
磚最小環境包含所需的最低數量的包PySpark和磚Python筆記本的功能。這個環境是理想的,如果你想自定義運行時各種Python包。
包括支持數據磚(遺留)庫工具(dbutils.library)。
請注意
Scala、Java和R在磚6.0運行時庫Conda磚運行時的6.0是相同的。詳情,請參閱磚運行時的6.0(不支持)發行說明。信息如何使用磚和Conda運行時,看到的磚與Conda運行時。
新功能
看到磚6.0運行時新功能。
改進
看到磚6.0運行時改進。
已知的問題
默認情況下,每個Python筆記本運行在自己的孤立Conda環境。這個孤立的環境是克隆從根Conda環境。因為克隆是一項昂貴的操作,在某些情況下,您可能會遇到以下問題:
如果集群實例類型沒有本地存儲,創建集群可能會失敗的錯誤:
無法開始的火花。這可能發生在安裝兼容庫或初始化腳本時失敗了。databricks_error_message:火花未能開始:超時後……秒
同時附加許多Python筆記本一個集群(例如,由安排工作或筆記本工作流)可能會導致一些筆記本無法連接。
以上如果您遇到任何問題,您不需要在隔離環境中運行Python筆記本(也就是說,您的集群不是共享),您可以禁用創建一個孤立的Python環境每個Python筆記本通過設置
spark.databricks.libraryIsolation.enabled來假在火花配置。設置這個標誌也禁用dbutils.library。如果你升級安裝Conda, Conda的新版本可能不包括修複Conda發行9104 (Conda列表失敗如果“記錄”文件有重複的條目)。如果你升級Conda看看失敗將Python筆記本或使用
conda列表的錯誤TypeError:“<”不支持之間的實例的“NoneType”和str的在司機日誌或筆記本,要麼使用一個版本的Conda解決或避免升級Conda安裝在這個版本。
庫
以下是出口environment.yml文件默認根與Conda磚6.0運行時的環境。
磚的標準
的名字:databricks-standard渠道:- - - - - -違約依賴關係:- - - - - -_libgcc_mutex = 0.1 =主要- - - - - -asn1crypto = 0.24.0 = py37_0- - - - - -backcall = 0.1.0 = py37_0- - - - - -布拉斯特區= 1.0 = openblas- - - - - -寶途= 2.49.0 = py37_0- - - - - -boto3 = 1.9.162 = py_0- - - - - -botocore = 1.12.163 = py_0- - - - - -ca證書= 2019.1.23 = 0- - - - - -certifi = 2019.3.9 = py37_0- - - - - -cffi = 1.12.2 = py37h2e261b9_1- - - - - -chardet = 3.0.4 = py37_1003- - - - - -密碼= 2.6.1 = py37h1ba5d50_0- - - - - -cython = 0.29.6 = py37he6710b0_0- - - - - -decorator = 4.4.0 = py37_1- - - - - -docutils = 0.14 = py37_0- - - - - -idna = 2.8 = py37_0- - - - - -ipython = 7.4.0 = py37h39e3cac_0- - - - - -ipython_genutils = 0.2.0 = py37_0- - - - - -絕地= 0.13.3 = py37_0- - - - - -jmespath = 0.9.4 = py_0- - - - - -krb5 = 1.16.1 = h173b8e3_7- - - - - -libedit = 3.1.20181209 = hc058e9b_0- - - - - -libffi = 3.2.1 = hd88cf55_4- - - - - -libgcc-ng = 8.2.0 = hdf63c60_1- - - - - -libgfortran-ng = 7.3.0 = hdf63c60_0- - - - - -libopenblas = 0.3.6 = h5a2b251_1- - - - - -libpq = 11.2 = h20c2e04_0- - - - - -libstdcxx-ng = 8.2.0 = hdf63c60_1- - - - - -ncurses = 6.1 = he6710b0_1- - - - - -nomkl = 3.0 = 0- - - - - -numpy = 1.16.2 = py37h99e49ec_0- - - - - -numpy-base = 1.16.2 = py37h2f8d375_0- - - - - -openssl = 1.1.1b = h7b6447c_1- - - - - -熊貓= 0.24.2 = py37he6710b0_0- - - - - -parso = 0.3.4 = py37_0- - - - - -容易受騙的人= 0.5.1 = py37_0- - - - - -pexpect = 4.6.0 = py37_0- - - - - -pickleshare = 0.7.5 = py37_0- - - - - -皮普= 19.0.3 = py37_0- - - - - -prompt_toolkit = 2.0.9 = py37_0- - - - - -psycopg2 = 2.7.6.1 = py37h1ba5d50_0- - - - - -ptyprocess = 0.6.0 = py37_0- - - - - -pycparser = 2.19 = py37_0- - - - - -pygments = 2.3.1 = py37_0- - - - - -pyopenssl = 19.0.0 = py37_0- - - - - -pysocks = 1.6.8 = py37_0- - - - - -python = 3.7.3 = h0371630_0- - - - - -python-dateutil = 2.8.0 = py37_0- - - - - -pytz = 2018.9 = py37_0- - - - - -readline = 7.0 = h7b6447c_5- - - - - -= 2.21.0 = py37_0請求- - - - - -s3transfer = 0.2.1 = py37_0- - - - - -scikit-learn = 0.20.3 = py37h22eb022_0- - - - - -scipy = 1.2.1 = py37he2b7bc3_0- - - - - -setuptools = 40.8.0 = py37_0- - - - - -6 = 1.12.0 = py37_0- - - - - -sqlite = 3.27.2 = h7b6447c_0- - - - - -statsmodels = 0.9.0 = py37h035aef0_0- - - - - -tk = 8.6.8 = hbc83047_0- - - - - -traitlets = 4.3.2 = py37_0- - - - - -urllib3 = 1.24.1 = py37_0- - - - - -wcwidth = 0.1.7 = py37_0- - - - - -輪= 0.33.1 = py37_0- - - - - -xz = 5.2.4 = h14c3975_4- - - - - -zlib = 1.2.11 = h7b6447c_3- - - - - -皮普:- - - - - -周期計= = 0.10.0- - - - - -kiwisolver = = 1.1.0- - - - - -matplotlib = = 3.0.3- - - - - -pyarrow = = 0.13.0- - - - - -pyparsing = = 2.4.2- - - - - -seaborn = = 0.9.0前綴:/磚/ conda / env / databricks-standard
磚最小
的名字:databricks-minimal渠道:- - - - - -違約依賴關係:- - - - - -_libgcc_mutex = 0.1 =主要- - - - - -backcall = 0.1.0 = py37_0- - - - - -布拉斯特區= 1.0 = openblas- - - - - -ca證書= 2019.1.23 = 0- - - - - -certifi = 2019.3.9 = py37_0- - - - - -decorator = 4.4.0 = py37_1- - - - - -ipython = 7.4.0 = py37h39e3cac_0- - - - - -ipython_genutils = 0.2.0 = py37_0- - - - - -絕地= 0.13.3 = py37_0- - - - - -libedit = 3.1.20181209 = hc058e9b_0- - - - - -libffi = 3.2.1 = hd88cf55_4- - - - - -libgcc-ng = 8.2.0 = hdf63c60_1- - - - - -libgfortran-ng = 7.3.0 = hdf63c60_0- - - - - -libopenblas = 0.3.6 = h5a2b251_1- - - - - -libstdcxx-ng = 8.2.0 = hdf63c60_1- - - - - -ncurses = 6.1 = he6710b0_1- - - - - -nomkl = 3.0 = 0- - - - - -numpy = 1.16.2 = py37h99e49ec_0- - - - - -numpy-base = 1.16.2 = py37h2f8d375_0- - - - - -openssl = 1.1.1b = h7b6447c_1- - - - - -熊貓= 0.24.2 = py37he6710b0_0- - - - - -parso = 0.3.4 = py37_0- - - - - -pexpect = 4.6.0 = py37_0- - - - - -pickleshare = 0.7.5 = py37_0- - - - - -皮普= 19.0.3 = py37_0- - - - - -prompt_toolkit = 2.0.9 = py37_0- - - - - -ptyprocess = 0.6.0 = py37_0- - - - - -pygments = 2.3.1 = py37_0- - - - - -python = 3.7.3 = h0371630_0- - - - - -python-dateutil = 2.8.0 = py37_0- - - - - -pytz = 2018.9 = py37_0- - - - - -readline = 7.0 = h7b6447c_5- - - - - -setuptools = 40.8.0 = py37_0- - - - - -6 = 1.12.0 = py37_0- - - - - -sqlite = 3.27.2 = h7b6447c_0- - - - - -tk = 8.6.8 = hbc83047_0- - - - - -traitlets = 4.3.2 = py37_0- - - - - -wcwidth = 0.1.7 = py37_0- - - - - -輪= 0.33.1 = py37_0- - - - - -xz = 5.2.4 = h14c3975_4- - - - - -zlib = 1.2.11 = h7b6447c_3- - - - - -皮普:- - - - - -pyarrow = = 0.13.0前綴:/磚/ conda / env / databricks-minimal