Migrating Your Workflow to a Remote Server

We will discuss to popular ways to use Python and Conda on a server:

  1. VSCode Remote-SSH

  2. Jupyter Notebooks with SSH Tunneling

  3. Python scripts run in a detached terminal with screen or tmux

VSCode and Remote-SSH

The VSCode Extension Remote-SSH allows you to SSH into a server through the code-editor. You will then be able to navigate your files on the server, load conda environments, and save files with the exact same interface you use on your laptop.

  1. Install the Remote-SSH extension in VSCode

  2. Open the control panel (cmd + shift + p)

  3. Type Remote-SSH and select Remote-SSH: Connect to Host...

  4. Select + Add New SSH Host

  5. Enter the credentials for your server

  6. Enter your password

This will log you into your server and will open a new VSCode window (which will be in the filesystem on the server). The first time you do this, you may have to wait a few minutes after logging in while VSCode installs the necessary tools on the server side. After you have done this the first time, you will be able to select the host name from the list of known hosts (instead of adding a new one) and input your password. You will no longer need to type out ssh username@ip_address.

Keep in mind that your conda environment will have to exist on the server to run your code. So you will need to make one there!

Jupyter Notebooks with SSH Tunneling

  1. Log into your server (ssh username@your-server-address)

  2. Activate a conda environment on the server that has Jupyter lab installed.

    1. When you first get access to the server you will want to download and install conda with miniforge and create your environment.

  3. Run jupyter-lab --no-browser in the folder you want to work from

  4. Jupyter lab will start, and the output in the terminal will include a line that looks like Or copy and paste one of these URLs: http://localhost:8888/lab?token=some-long-token

    1. The port number (in this case 8888) is the port on the server that the jupyter lab information is piped to

    2. We need to link a port on our local computer to the port on the server and then we can open the webpage like we are used to

  5. Open a new terminal on your local computer and run ssh -L 8888:localhost:8888 username@your-server-address

    1. This links the local port 8888 (the first number) to port 8888 on the server (the second number).

    2. These numbers don’t have to be the same, but the second one has to match what Jupyter lab told us on the server.

    3. If port 8888 on our laptop is taken, we can try another number. ssh -L 8890:localhost:8888 username@your-server-address.

    4. Enter your password to complete the SSH command. Now the tunnel is configured! Leave this terminal open as long as you want to keep tunneling Jupyter lab from the server to your local computer.

  6. Copy and paste that URL from the Jupyter lab output on the server into your local web browser (http://localhost:8888/lab?token=some-long-token)

Running Python Scripts in a Detached Terminal

The tmux command allows you to open terminals in which you can start a program that won’t stop even when the terminal window is closed. This is useful for running on a server because a normal terminal will kill whatever process is running if you close the window, close your computer, or lose your internet connection. That is not a practical way to work if we need to run programs that take a long time!

The screen command is an alternative to tmux. You can use help, man, or ChatGPT to figure out how exactly to use screen or tmux for your project.

This workflow would look something like:

  1. Develop your Python script on a small set of data. This could be done in a Jupyter notebook or python script on the server or on your laptop.

  2. Organize the analysis you need to do into a script or set of scripts.

  3. When logged into the server, open a terminal and type the appropriate tmux or screen command. This will open a “detached” terminal, the details of which will be based the command you entered.

  4. Activate your conda environment in the detached terminal.

  5. Run your Python script from the command line. When you exit the terminal, it will kill your SSH session but it will not kill the program.

    1. Alternatively you could have a bash script that runs a whole list of Python programs.

Once you have executed your Python or bash script, you can close the terminal and your programs will continue to run on the server.

Using Git to Move and Track your Work

Git repositories provide an easy way to move work between locations (and to make sure you don’t lose anything because you are switching between platforms). To use Git on a server:

  1. Log into the server

  2. Generate a new SSH key pair (the key pair should end up in your home directory on the server at ~/.ssh/)

  3. Add the public key to your GitHub profile through Settings -> SSH and GPG Keys -> Add SSH key

Once you have done the above, you can use your Git account to track the work you do on the server. Your research project repo can be downloaded on both the server and on your local computer. When you make changes in one place, you can add, commit, and push them. When you return to the other location you can do a git pull to see the updated work.

An example workflow:

  1. Starting a project from scratch: Create a repo on your local computer or in your GitHub account. Clone that repo to the server. Do your work in the repo. Before you log out, add and commit the changes. Return to your local computer and git pull to see the updated project without being logged into the computer. This is a great way to keep figures up to date. You will generate new figures, commit and push both the figures and the code that generated them, and then pull them to your local computer. This negates the need to use scp or rsync to continually move around copies of your figures and prevents confusion about which code generated which version of the figures.

  2. Tracking an existing project: Log onto your server. Navigate into a directory where you have code for your project. Use git init to make the project into a repository (make sure to have a .gitignore file!). Push the repo to your GitHub. Proceed with your work on the server. Add and commit (and push) any changes before you log out. Return to your local computer, use the link on GitHub to clone your repository. Now you can see the work you were doing on the server!