我想保存模型分布式訓練後通過以下代碼
從spark_tensorflow_distributor進口進口sys MirroredStrategyRunner mlflow進口。mlflow keras mlflow.keras.autolog ()。log_param (“learning_rate”, 0.001)從sklearn進口tensorflow tf導入時間。從sklearn model_selection train_test_split進口。數據導入load_breast_canc #添加,因為databrick不允許canc ....def火車():戰略= tf.distribute.experimental.MultiWorkerMirroredStrategy () # tf.distribute.experimental.CollectiveCommunication。NCCL模型=沒有strategy.scope (): data = load_breast_canc() #添加,因為databrick不允許canc ....X_train、X_test y_train y_test = train_test_split(數據。數據,數據。目標,test_size N = 0.3), D = X_train。從sklearn形狀#數量的觀察和變量。預處理進口StandardScaler標量= StandardScaler () X_train = scaler.fit_transform (X_train) X_test = scaler.transform = tf.keras.models (X_test)模型。順序([tf.keras.layers.Input(形狀= (D)), tf.keras.layers。密度(1激活=乙狀結腸)#使用乙狀結腸函數為每個時代])model.compile(優化器=“亞當”,#使用自適應動量損失= binary_crossentropy,指標=[“準確性”])#火車模型r =模型。適合(X_train y_train validation_data = (X_test y_test))打印(“火車得分:”模型。評估(X_train y_train) #和準確性mlflow.keras評估收益損失。log_model(模型、“mymodel”) MirroredStrategyRunner (num_slots = 4, use_custom_strategy = True) .run(火車)
@https: / /github.com/tensorflow/ecosystem/blob/master/spark/spark-tensorflow-distributor/spark_tensorflow_distributor/mirrored_strategy_runner.py
我有幾個問題
- 設置num_slots = 4將導致mlflow日誌4模型,每個模型並不擅長預測的數據集,但是。我希望首席節點日誌一個模型至少有80%的準確率,是否有辦法拯救隻有一個模型或合並模型嗎?
- 沒有mlflow如何保存模型。日誌,保存通過dbutil我會競態條件,但目前還不清楚從節點的主要節點的火花分銷商
- 是每個節點獲得的所有數據,而不是部分數據?
