Reconnecting remote Jupyter Notebook and get current cell output

And if you use a .py file instead of a .ipynb file (jupyter notebook), and inside this .py file you print the results to test the operation of your code.

To convert from .ipynb to .py file you can use this command:

'jupyter nbconvert --to script example.ipynb'

Now, you can work with a python script instead a jupyter notebook file, this will make things easier.

In your script write prints() in the stages you think necessary in order that you can see it in Tmux terminal. So you can kill your training whenever you want (ctr+c) or not, Tmux can save the session if you want, just tape 'ctr-b + d' to detach from de session

This is a still OPEN issue in Jupiter Notebook Official website. See https://github.com/jupyterlab/jupyterlab/issues/2833 "Reconnect to running session: keeping output"

I'm curently facing the same problem and I found this discussion. Mentioned Papermill works quite well. Just use something like:

nohup papermill --request-save-on-cell-execute --no-progress-bar input.ipynb output.ipynb &

input.ipnb notebook with your sourcecode.

output.ipnb processed notebook where you can see the output.

--request-save-on-cell-execute prints cell output into the output.ipnb notebook after the cell is completed.

--no-progress-bar disables showing progress bar which is quite useless if you do all the work in one cell.

nohup is there so papermill keeps running after you logout from server and $ to perform it in backgroud.

All Papermill options can be found there.