NOTE: It is now recommended to flash the Jetson using the NVIDIA SDK Manager for this functionality. This post was written before that the Native Boot feature was available in the SDK Manager GUI. With the advent of JetPack 4.6 (L4T 32.6.1) it is now possible to boot the Jetson AGX Xavier and Jetson Xavier NX from external storage. Looky here:
Background
In the JetPack 4.6 release Cboot now has NVMe driver support for the Jetson Xavier NX and Jetson AGX Xavier series. This enablesnables loading kernel, kernel-dtb and initrd from the root file system on NVMe.
There is also a new flashing tool which can flash internal or external media connected to the Jetson Xaviers. The new tool uses initial RAM disk for flashing and is up to 1.5x faster when flashing compared to the previous method!
If you have read the highlights from the press release, you already know all of that. Let’s talk about the new flashing tool in a little more detail.
For reference, the JetsonHacks scripts are in the JetsonHacks repository on Github: bootFromExternalStorage
You’ll need a NVMe drive in your Jetson to follow this tutorial. Only Gen3 drives have been tested, there appear to be issues with Gen4. These drives are available in different sizes of course, but 500GB seems like a good price/capacity. Here’s some suggestions (Amazon affiliate links):
This process does not require that the NVIDIA SDK Manager GUI application be present. These are all command line scripts that can be run in a Terminal. For many people, this better fits their workflow.
Flashing with initrd
The new flashing method uses initrd (initial RAM Disk) to flash both internal media and external media connected to a Jetson device. This method uses initrd and USB device mode. Initrd flash supports several workflows such as:
- Flashing internal storage devices
- Flashing external storage devices such as NVMe SSD and USB drives
- Enabling A/B rootfs on external storage devices
- Enabling disk encryptions on external storage devices
- Flashing individual partitions
- Flashing fused Jetson devices
- Flashing a Massflash blob to normal and fused Jetson devices
- Generating separate images for external and internal storage devices, then flashing the combined images
Note: Jetson AGX Xavier series devices use boot firmware that is stored only on internal eMMC memory. Consequently this type of device cannot boot from USB or NVMe SSD until its internal eMMC has been flashed.
As usual, tools this flexible have a little bit of a learning curve. There are a group of scripts in the Linux_for_Tegra/tools/kernel_flash directory which handle the above mentioned tasks. It is worth reading the README_initrd_flash.txt file to get a feel for what it takes accomplish the task you are trying to accomplish.
Preparing the Host
You will need to install dependencies on the x86 host to support flashing the Jetson. NVIDIA officially supports Ubuntu 16.04 and 18.04 for the flashing scripts. Then you will minimally need three archives:
- Jetson Xavier Board Support Package (BSP)
- Sample Root File System (rootfs)
- Secure Boot
Here are Direct downloadable links to JetPack 4.6 and L4T debian packages for Jetson
There are helper scripts in the JetsonHacks bootFromExternalStorage repository. The time for the entire process of downloading and flashing the Jetson will depend on the speed of your Internet connection and host computer. Here it was ~ 1 hour. To download the helper scripts and run them:
$ git clone https://github.com/jetsonhacks/bootFromExternalStorage.git
$ cd bootFromExternalStorage
$ ./install_dependencies.sh
$ ./get_jetson_files.sh
There are a couple of tricks here and there in the scripts to put everything in the right place.
Flashing the Jetson
Once the archives are expanded and put in the correct place, put the Jetson into Force Recovery mode. This is different on each Xavier. The video shows a Xaiver NX. You can use the command lsusb to determine if the host can ‘see’ the Xavier. There should be an entry with the NVidia Corp tag when the Xavier is connected to the host in Force Recovery mode. Default is to flash to the NVMe SSD. You will find this works best if you actually have a NVMe SSD installed (64GB minimum; more is better) before attempting to flash.
You can use the convenience script in the repository to flash the Jetson:
$ flash_jetson_external_storage.sh
It takes ~ 20 minutes to install everything in this manner. After flashing, the Jetson will be in ‘oem-config’ mode, ready to be setup. At this point, we’re done on the host side.
Note on Flashing
The script flash_jetson_external_storage.sh wraps a NVIDIA script which calls the initrd flashing script. After everything is setup, this boils down to:
$ ./nvsdkmanager_flash.sh –storage nvme0n1p1
For a Jetson Xavier NX, this flashes the internal QSPI-NOR memory and puts the rootfs and other partitions on the NVMe SSD. The Jetson AGX Xavier does not have QSPI memory, so the equivalent on eMMC is flashed instead.
Setting up the Jetson
You can choose to setup the Jetson in headless mode or desktop mode. In the video, we chose desktop mode. In either case, the next step is to install the NVIDIA JetPack packages on the Jetson. Switch from the host to the Jetson. Finish the Jetson configuration, then:
$ git clone https://github.com/jetsonhacks/bootFromExternalStorage.git
$ cd bootFromExternalStorage
./install_jetson_default_packages.sh
The default packages are from the meta-package nvidia-jetpack and includes all of our Jetson friends like:
- CUDA
- cuDNN
- TensorRT
- VisionWorks
- VPI
- OpenCV
plus a couple of other of packages to replicate the same experience as the default Xavier NX SD card.
Conclusion
With the convenience scripts, setting up your Jetson Xavier to boot from external storage should be straightforward. This is the NVIDIA ‘approved’ way, so Over The Air (OTA) updates should work going forward too.
Note that NVIDIA is working on the SDK Manager GUI application to replicate this type of functionality. That’s certainly worth checking out!
Notes
# this command create username ubuntu and password ubuntu
$ sudo ./tools/l4t_create_default_user.sh -u ubuntu -p ubuntu </blockquote>
Then reflash the Jetson.
Initial Release
- JetPack 4.6, L4T 32.6.1
- Tested on Jetson Xavier NX, NVMe SSD
- Host: Ubuntu 16.04 (should work with Ubuntu 18.04 also)
- Experimental Ubuntu 20.04 support added
40 Responses
Great news! Just a quick question: does it require a clean NVME? What if I already have data on my NVME drive? Would it be wiped out when I run the flashing script?
The procedure will reformat the NVMe, which will wipe any data you have on it. Thanks for reading!
Thanks!! Was finally able to get this to work from the latest version of Proxmox VE (Debian Buster). Sadly my only option for an X86 Linux distro at the moment.
BUT, it needed:
sudo (yep, it’s Proxmox…apt install sudo)
uuidgen (apt install uuid-runtime)
strings (apt install binutils)
Python 2 (!!! 2002 called and wants its snake back….apt install python)
parted (apt install parted)
Python and parted were late (i.e., painful) failures. parted was especially tricky since it failed silently (i.e., it said SUCCESS!! but failed to boot for pretty obvious reasons).
But it’s running happily now. Thanks for all you do for the Jetson community. So I guess the real question is, between USB booting on the Nano and SSD booting on the Xavier, what the heck am I supposed to do with all of my Micro SD cards now?!? You stinker…
Wow, that sounds like a lot of work! I am glad you were able to flash your Jetson.
You can do like I do. Keep your Micro SD cards next to your SD cards and small, unusable thumb drives. Those are right next to the R/W DVDs. Those are right next to the R/W CD ROMS. So that I can easily find them, I put those next to the disk cartridges and 3.5″ floppies. Of course, it makes sense to store those next the the 5 1/4″ floppies, and the 8″ floppies. On the shelf below, keep the paper tape, punch cards and program listings because you will find that you won’t access those as much anymore.
Thanks for reading!
Wonderful tutorial. I used Ubuntu 20.04 on a parallels VM installed on a Mac to implement this. Two issues got me. I had to install Qemu and Python 2.7 as additional dependencies. I also had to manually set the read / write status of the folder to get the script to work.
Glad to have a method for direct to NVMe SSD.
Cheers
Thank you for the kind words, and thank you for your feedback.
As you know, NVIDIA does not officially support Ubuntu 20.04 for these scripts, which leads to the issues you encountered.
In your ‘install_dependencies.sh’ file, do you have the line:
I added this which I had hoped would solve the qemu issue.
I’ve been working on a Python workaround. The file /usr/bin/python is a symbolic link to the Python environment. The NVIDIA scripts use this. For 20.04, they decided to leave the link empty, as the directive is to explicitly call python3.
If you use Python3, I have been told you get warnings, but it still works.
The idea is to set the symbolic link to Python3 if it is not set, flash, and then put it back to its original state.
It worked for me in Ubuntu 20.03 LTS Physical Machine Mode
One of the note is it require 8G of extra space on Host machine to download the needed files
Excellent to hear! Thanks for sharing, and thanks for reading.
Nice job.
I really don’t understand in which part OS is installed.
get_jetson_files.sh downloads the BSP and rootfs in a compressed form. Then it converts them into a disk image. The disk image represents a base Ubuntu 18.04 OS.
flash_jetson_external_storage.sh – Copies the disk image to the Jetson, and updates the flash memory to handle booting the Jetson. After this step, the Jetson boots into Ubuntu.
install_jetson_default_packages.sh – Installs the JetPack libraries on to the Jetson. These libraries are Jetson specific, and provide CUDA and machine learning and vision processing.
Hope this helps.
Is there any possibility to install cloud-init at some point of the folder preparing and use it as can be done with user-data file as rpi?
https://cloudinit.readthedocs.io/en/latest/topics/examples.html
Or, is there anything similar to this for jetsons which I don’t know?
Is there something similar?
There appears to be support for Ubuntu. Did it not work when you tried?
I really don’t know in which place of your workflow I should try to use it. Normal use is: by setting up in /boot folder user-data file like example url I described in my first comment right here.
The scripts here are an executable version of NVIDIA documentation about using initrd. The scripts install the L4T version of Ubuntu operating system on the Jetson. The scripts allow the system to run the rootfs from an external drive.
This is lower level operating system material. I suggest that you do not use them since they do not meet your workflow criteria.
So, Is there no way, for example, to execute things at first boot?
That’s what oem-config does. You can configure the system however you want.
The oem-config does it trough a config file? In addition, cloud-init config are quite standard, even is able to deploy cheff or puppet config files.
On a headed target device (one equipped with a display), oem-config runs as a GUI application. On a headless target device (one without a display), it runs as a character interface application.
You should ask your question on the official NVIDIA Jetson developer forums, where a large group of developers and NVIDIA engineers share their experience. Also, consider using Yocto if you need to a more defined BSP. See: https://github.com/OE4T/meta-tegra
https://forums.developer.nvidia.com/t/l4t-cloud-init/188742/11
I will take a look to Yocto, but sounds like I am going to have specific issues, above all for using last cuda updates.
hi, thx for this video. I ran the installation and succeeded.
But.. I can’t install nano, htop, etc.
Which repositories i need to add?
You will need to look up which repositories those tools are in. Typically you would:
$ sudo apt update
before trying to add any additional tools.
I fixed it. Forgot the run ./install.. on the jetson xavier after flashing.
Installation completed successfully, allhough still running ubuntu 18.04.
What steps are needed to upgrade to 20.04?
I’m glad you were able work. Please ask your question on how to upgrade to 20.04 on the official NVIDIA Jetson Xavier forum.
Hi,I followed the video and succed,but when I move the SSD to another jetson NX board it cannot startup, only black screen flash slowly. is this normal? When i move hard disk from one PC to another same type PC, it always can startup.thanks
Should this work with a TX2 SOM? Does not seem to:
[ 0.1461 ] Boot Rom communication
[ 0.1499 ] tegrarcm_v2 –chip 0x19 0 –rcm rcm_list_signed.xml –skipuid
[ 0.1522 ] RCM version 0X16
[ 0.1544 ] Boot Rom communication failed
[ 5.1913 ]
Error: Return value 3
Command tegrarcm_v2 –chip 0x19 0 –rcm rcm_list_signed.xml –skipuid
Reading board information failed.
Error: Failed to generate images for external device
Cleaning up…
So.. I did all the steps and nothing changed on my jeston..
After “successful” flashing when I boot jetson it just starts like nothing changed
I have all my data and setting still the same, SSD not formatted
I even tried “sudo ./tools/l4t_create_default_user.sh -u ubuntu -p ubuntu” command
I have ubuntu 16.04 x86 on host, jetson xavier nx (mic-710 aix) and Kingston SA400M8/240 SSD
My flashing logs can be found there: https://disk.yandex.ru/d/tf_B5NhWkbLXHQ
Please tell me what I’m missing there
Thx for your work
when i’m doing the fist jetson flashing:
$ flash_jetson_external_storage.sh
I’m getting a timeout on step 3: start the flashing process
any idea how to succeed?
Hello Jim, thanks for the video. After flashing it gives a success message but afterwards I cant do anything on the jetson. If i immediately connect the monitor, keyboard and mouse all i see is a cursor and nothing happens. If i try to reboot the jetson, nothing happens or seems to happen. even the nvidia logo does not come. What could be the problem. I should note that I had formatted the ssd as ext4 also I am using the 64bit ubutnu 18.04. Could that be the cause?
Hard to tell what the issue might be. If you are using the latest JetPack, you should use the SDK manager to set this up instead of these scripts. The scripts were written before that functionality was available.
thanks. thats the approach I took indeed. Worked like a charm. Glad to know its a legit method, cause I had just sort of happened to try it because I saw the option in the sdkmanager.
Hello, thanks for the tutorial, everything seems to work great (with Ubuntu 20.04 as a host machine by the way). The thing is, I want to enable A/B rootfs but I can’t figure out how. I have managed to do it for an sd card, just by running the “flash.sh” script with the ROOTFS_AB=1 option. Do you happen to know the way for that? Thanks!
I haven’t worked with that feature at all. It is worth asking in the official NVIDIA Jetson forums the procedure. It may be possible to accomplish using the SDK Manager.
Hello, I’m trying to do the flash using Virtual Box Ubuntu 18. “lsusb” command shows that device is connected:
Bus 001 Device 008: ID 0955:7e19 NVidia Corp.
But during flashing I got an error:
Reading ECID … *** Error: ECID read failed
Is any issues with Windows-based Virtual Box?
VMs are not supported by the SDK manager. It has to do with USB pass through.
Sluge,
Did you switch to flashing with an Ubuntu 18 physical machine? Like you, I’m using a VirtualBox Ubuntu 18.04 VM to do the flashing. I encountered a different issue, “CPU bootloader not running on device”. I received the same error regardless flashing with Nvidia SDK Manager or running flash.sh on the command line. Just want to see if it’s worthy trying to get flashing to work on VM or simply give up on VM and use a physical machine.
Hello kangalow,
Thank you for everything you do for our community.
A sort of broad question but I was wondering if you could help me understand. When I download Jetpack 4.6.2 from SDK Manager for the Jetson it is 30.2 Gb yet when flashed to the Jetson, it fits neatly within its 16Gb EMMC. Why is this the case? Thanks
-Will
You are welcome! SDK Manager includes a lot of different parts. You can think of SDK Manager “assembling” or compiling an image to flash onto the Jetson. There are many bits and pieces that it puts together. Some of these parts can be compressed, other parts are simply not needed. Also, a typical desktop may have different drive sector sizes, so that the download may take up extra physical space in comparison.
Or it’s some dark magic. One of the two. Thanks for reading!
Hi, can I use this method with a USB drive?
You should be able to. You’ll need to change to sda1. USB is a little trickier. Remember, USB doesn’t guarantee the enumeration id of a mass storage device. If you have two USB drives attached to the Jetson, you aren’t guaranteed that the one that you place the rootfs on is going to be sda1. For flashing, see: https://github.com/jetsonhacks/bootFromExternalStorage#flash_jetson_external_storagesh
Thanks for reading.