텐서보드를 사용하고 있다면(tensorboard_callback = keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)), histogram_freq=1 인자를 아래와 같이 바꿔주세요.
histogram_freq=0이렇게 해도 killed 된다면 그냥 tensorboard 안사용하는 것도 방법이더라구요..
참조한 글 :
(In case anyone else runs into this issue). I ran into the same problem - Keras would train for one epoch and then the console just says Killed with no other error messages, and the weird thing was that it only happened when I train on a larger dataset (1,000 samples was fine, 100,000 samples threw the error). Turns out the problem was with the histogram_freq argument in the TensorBoard callback. I set histogram_freq=0 and everything worked fine. I guess calculating the histograms of all activations in a large network takes up a lot of memory.
출처 : https://github.com/keras-team/keras/issues/3476
Process "Killed" suddenly · Issue #3476 · keras-team/keras
Hi, Most of the times, my training stops abruptly with just a "killed" msg at the end of logs. Epoch 1/1 296960/296960 [==============================] - 641s - loss: 2.3065 - val_loss: 3...
github.com