Run GUI Apps over SSH – NVIDIA Jetson

Most people know that you can run command line programs over SSH. But a lot of people don’t realize that you can run graphics programs over SSH through a feature called X11 Forwarding. Looky here:

X Window System

The X Window System version 11 (commonly referred to as X11) is a windowing system commonly used in Linux. It’s fundamental to the graphical user interface (GUI) in most Linux distributions, including the NVIDIA Jetsons.

X11 follows a client-server model. The X server communicates with the hardware, such as the monitor, keyboard, and mouse. The X clients are the various applications that use the server to display GUI elements. This architecture allows support for what is termed network transparency. The client and the server can be on separate machines. In other words, an application may run on one machine, and display on a different one. Of course, the X11 client and server may be on the same machine so that the GUI shows up on an attached monitor.

Window Management

X11 itself doesn’t dictate how windows look or behave. Instead, there are Window Managers to handle the task. Desktop environments have a default window manager. For example, a Jetson with a GNOME desktop environment uses the “Mutter” window manager. The window manager controls the placement, appearance and decorations (like border and title bars) of windows.

Extensions for X11 like OpenGL for providing 3D graphics capabilities and others implement modern features like compositing add flexibility.

X11 Protocol

X11 uses a network protocol to communicate between the server and clients. This protocol allows X clients to send requests to the X server (like opening a window or drawing graphics), and receive responses and events (like keyboard or mouse input) from the server.

Since applications don’t write directly to hardware (as in more modern operating systems such as Windows or Macintosh OS), the stream can be redirected over the network.

Note that Wayland, which some think of as the successor to X11 on Linux, does not support network transparency. Wayland is simpler and more modern. It also ditches the network bit. The work around is much like that of other operating systems, a remote desktop or similar.

X11 Forwarding

X11 Forwarding is the feature which allows a user to run a graphical application on a remote server and have the application’s display (its window) appear on their local machine. You can think of it as a simple forwarding of the network stream. The application on the remote machine is sending display messages to your local machine.

The terminology is a little tricky. It’s the opposite of what you would normally expect. The “server” is the system that displays the application (i.e., the user’s local machine). The “client” is the system where the application is running (i.e., the remote machine).

It’s pretty straightforward how it works:

• The user connects from their local machine (X Server) to the remote machine (X Client) via a secure shell (SSH) connection.
• When setting up the connection, the user enables X11 forwarding. That tells the SSH server on the remote to forward X11 protocol data over the SSH connection.
• When the user runs a graphical application on the remote machine, the application sends its X11 data to the SSH server.
• The SSH server sends this data through the SSH encrypted SSH tunnel to the user’s local machine.
• The SSH client on the user’s local machine receives the data and passes it to the local X11 server. The X11 server then displays the applications window, just as if it were running locally.

The terminology clouds the explanation a bit. Conceptually, when the remote application makes a drawing call, SSH sends it to the local machine to draw it. The rest is just plumbing.

Mostly the Same

The local graphics are mostly the same as they would display on the remote with a monitor attached. However, remember that the remote and the local machines may have different window managers. The windows may look a bit different.

Naturally, you expect lag in the process. It depends on how fast the network connection is, and the amount of data that you’re piping over the network. You don’t expect raw, uncompressed video to be displayed as fast as if it was running completely on one machine. Same with a wired versus wireless connection.

That’s why most video gets highly compressed before you send it over a network connection. It’s so ubiquitous that you don’t think about it most of the time. Until you have to.

Why X11 Forwarding ?

Most operating systems (you can do this with Linux too) use VNC or remote desktop or some other similar solution to viewing remote machines. This is fairly heavyweight; you have to keep track of the entire graphic state of the desktop. This has gotten better over the years with use of compression, faster machines and so on. But it’s still has a lot of overhead.

On the other hand, X11 forwarding brings you only the window in which you are interested. It’s very useful when you are doing remote development. You can use tools like Visual Studio Code on a local host to program a Jetson remotely, yet still run the application you are developing on the local machine too.

Also, it can be a little meta having a remote desktop along with your local desktop on the screen at once.

Running SSH with X11 Forwarding

Oh, and the bit about running SSH. There are two flags of interest. -X (X11 with Security Restrictions) and -Y (Trusted X11 Forwarding). -X enables X11 forwarding with restrictions. It activates a secure form of X11 forwarding to prevent remote clients from performing, uh, “unauthorized” activities. -X is “untrusted”.

-Y enables “trusted” X11 forwarding. This removes some of the security restrictions. Probably a bad idea if you’re not on a trusted network or out in the big bad Internet.

A typical command would look like:

$ ssh <option> <user name>@<ip_address>

for example:

$ ssh -X jim@192.168.1.12

That’s reads as open a trusted X11 connection under the user name jim at the IP address 192.168.1.12.

Other Operating Systems

Both Windows and Macintosh have 3rd party programs of X11 server implementations. On Macintosh, many people user XQuartz. On a Windows box, many people use Putty as their SSH client and Xming for their X11 server.

Certainly worth checking out if you are curious or want to work from a different machine.