大家好,
我試圖加載SparkNLP (l墨水如果需要更詳細的模型)Mlflow注冊表。
為此,我跟隨一個教程和實施以下代碼:
進口mlflow。pyfunc類LangDetectionModel (mlflow.pyfunc.PythonModel): def __init__(自我):從sparknlp super () . __init__ ()。從sparknlp pretrained PretrainedPipeline進口。pretrained進口PipelineModel # sparknlp模型嵌入到自我。_model = PipelineModel.load (“/ mnt / sparknlp_models / detect_language_375 /”) def預測(自我,eval_data_lang_detect): #變換函數申請朗detetction list_columns = eval_data_lang_detect。列model_output = self._model.transform (eval_data_lang_detect)。選擇(list_columns + [F.col (language.result) .getItem (0))) .withColumnRenamed (“language.result [0] ', ' sparknlp_column ')返回model_output model_path =“my-langdetect-model reg_model_name”=“NlpieLangDetection”sparknlp_model = LangDetectionModel ()
#日誌MLflow實體和保存模型mlflow.set_tracking_uri (“sqlite: / / / mlruns.db”) #保存conda環境這個模型。conda_env ={“渠道”:“違約”、“conda-forge”,“依賴性”:[“python = {}”.format (PYTHON_VERSION),“皮普”),“皮普”:[‘mlflow’,‘cloudpickle = = {}’.format (cloudpickle.__version__), ' NlpieLangDetection = = 0.0.1 '],“名字”:“mlflow-env”}
#保存模型mlflow.set_experiment(' /用戶/(電子郵件保護)/ Language_Detection_Translation mlflow LangDetectionTest”)。start_run (run_name =“Nlpie語言檢測”)運行:model_path = f mlflow“{model_path} - {run.info.run_uuid}”。log_param mlflow.pyfunc(“算法”、“SparNLPLangDetection”)。save_model(路徑= model_path python_model = sparknlp_model conda_env = conda_env)
我得到一個錯誤在最後一段代碼,我試圖Mlflow注冊表保存模型。
下麵的錯誤讓我得到:
_thread TypeError:不能泡菜”。RLock的對象--------------------------------------------------------------------------- TypeError Traceback (most recent call last) in 4 model_path = f"{model_path}-{run.info.run_uuid}" 5 mlflow.log_param("algorithm", "SparNLPLangDetection") ----> 6 mlflow.pyfunc.save_model(path=model_path, python_model=sparknlp_model, conda_env=conda_env) /databricks/python/lib/python3.8/site-packages/mlflow/pyfunc/__init__.py in save_model(path, loader_module, data_path, code_path, conda_env, mlflow_model, python_model, artifacts, signature, input_example, pip_requirements, extra_pip_requirements, **kwargs) 1467 ) 1468 elif second_argument_set_specified: -> 1469 return mlflow.pyfunc.model._save_model_with_class_artifacts_params( 1470 path=path, 1471 python_model=python_model, /databricks/python/lib/python3.8/site-packages/mlflow/pyfunc/model.py in _save_model_with_class_artifacts_params(path, python_model, artifacts, conda_env, code_paths, mlflow_model, pip_requirements, extra_pip_requirements) 162 saved_python_model_subpath = "python_model.pkl" 163 with open(os.path.join(path, saved_python_model_subpath), "wb") as out: --> 164 cloudpickle.dump(python_model, out) 165 custom_model_config_kwargs[CONFIG_KEY_PYTHON_MODEL] = saved_python_model_subpath 166 else: /databricks/python/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py in dump(obj, file, protocol, buffer_callback) 53 compatibility with older versions of Python. 54 """ ---> 55 CloudPickler( 56 file, protocol=protocol, buffer_callback=buffer_callback 57 ).dump(obj) /databricks/python/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py in dump(self, obj) 631 def dump(self, obj): 632 try: --> 633 return Pickler.dump(self, obj) 634 except RuntimeError as e: 635 if "recursion" in e.args[0]: TypeError: cannot pickle '_thread.RLock' object
請讓我知道如果你需要任何進一步的細節。
提前感謝您的支持。
