With the introduction of JetPack 4.5, it is now possible to boot your Jetson Nano from a USB drive! Looky here:
Background
The NVIDIA Jetson Nano Developer Kits (A02, B01, and 2GB) boot and run from a micro-SD card. It is now possible to set the Nano up to boot and run from a USB drive.
There are a couple of reasons to boot from USB. First, it is more reliable over the long term than using the micro SD card. SD cards aren’t really designed to handle the files of an operating system, and tend to be less reliable than USB drives in general.
The second reason is that USB is much faster than the SD card speed, usually between 4 to 10 times depending on the application. This makes things much snappier!
USB drives are available in various flavors. You’ll see them as SSD, which stands for Solid State Drive, HDD, which stands for Hard Disk Drive, and flash drive which are sometimes referred to as thumb drives or USB sticks.
For this application, HDD is the most reliable long term, SSD is an excellent alternative and is faster. HDD tends to be slower than SSD and tends to use more power, but offers more storage for the same price. Some HDD drives can be powered externally. Flash drives aren’t as suitable for this application for much the same reason as SD cards, but some people still use them.
People also like the extra storage that many USB drives afford. This makes it easy to store lots of programs and large amounts of data, such as video files and large data models.
Here are a few drives we’ve had success with (affiliate links):
Samsung T7 500 GB USB SSD: https://amzn.to/3vdp62b
Samsung T5 500 GB USB SSD: https://amzn.to/2PtE7bK
Western Digital 2TB External Hard Drive: https://amzn.to/3t7A7jH
These are all available in a variety of sizes, some careful hunting will find a drive which meets your needs.
Disk Formatting and Partitions
Here are some terms that you will see when we talk about drives on Linux. Partitions, GPT and Ext4. Sometimes the terms are jumbled together, making them hard to understand.
Physical devices consist of a collection of sectors. These are the physical bits were data are stored. Software creates something known as a Partition, which maps and groups sectors together. In that way, each region of the device can be managed independently.
Partition descriptions are stored in a Partition Table. A physical drive can have one or more partitions.
Because a Partition is treated as a separate logical volume, drives with multiple partitions appear as though they are multiple physical drives.
Partitions are handy for storing things like file systems, swap disks and so on.
You have heard the term ‘format the drive’ before. When we talk about formatting a drive for Linux, this is a two step process. First we format the drive to define the partition layout, and then we format partitions for a specific use.
Linux uses GPT for the layout of a partition table on a physical storage device using globally unique identifiers, or GUIDs. It is the Linux preferred partitioning system for drives larger than 2TB. This results in a table at a known location which tells the system where partitions are physically located on the drive. In other words, a map!
Once the drive is partitioned, we are ready to format a partition for the Linux file system. Different partitions can have different format types.
This is how you might setup a dual boot Windows and Linux machine on a PC, for example. One partition for the Windows filesystem, and another for the Linux filesystem.
For our Jetson, we want to format a partition as Ext4. Ext4 is a format for a journaling file system for Linux. We will put the rootfs of the Jetson on this partition. Rootfs is simply the root file system, we usually think about this as all of the files on the system.
QSPI-NOR?
In previous versions of JetPack, the system boots from information stored on specific partitions on the SD card. For JetPack 4.5+, this changes.
On the Jetson module, there is flash memory. Typically you will see this referred to as QSPI-NOR. NOR is a type of flash memory. QSPI stands for Quad Serial Protocol Interface, a highway for transferring data to and from the Jetson Tegra chip to the NOR flash memory.
When JetPack goes through its setup process, it transfers information to boot the system from the SD card to the QSPI-NOR. Consequently, the Jetson can use this information to boot the system without having to have a SD card present.
There are a couple of more things to know. If there is no SD card, then the Jetson will look to the USB drive for a rootfs. There are other devices from which the Jetson will be able to boot if they are setup correctly, such as a NVMe drive. In our case, we will just concern ourselves with the USB drive.
There is a special file in the /boot directory which tells Linux where the root file system is located. We will need to modify this file, named extlinux.conf to point to the new location of the root.
You can follow the instructions from NVIDIA in case you want to create your bootable USB drive on your host PC. Here we are going to create the drive from the Jetson itself.
Install
You will need to do an initial setup of the Jetson with JetPack 4.5+ in order to load updated firmware into the Jetson Module QSPI-NOR flash memory. Follow the ‘Getting Started’ instructions on the JetPack site: https://developer.nvidia.com/embedded/jetpack
The JetPack archives are located here: https://developer.nvidia.com/embedded/jetpack-archive
During the initial setup of L4T 32.5+, the firmware for the Jetson Nano developer kits relocates the boot firmware from the micro SD card to the Jetson module integrated QSPI-NOR flash memory. This also changes the layout of the SD card. This layout is now analagous to the BIOS in a PC.
There is a repository on the JetsonHacksNano account on Github, bootFromUSB which contains convenience scripts to help with this task.
Clone the repository, and switch to that repositories’ directory.
$ git clone https://github.com/jetsonhacksnano/bootFromUSB
$ cd bootFromUSB
Step 1
Prepare a USB drive (preferably USB 3.0+, SSD, HDD) by formatting the disk with the GPT partitioning scheme. This will erase the data that is on the drive, be warned. Then create a partition, and set the format type to Ext4. In the video, we use the ‘Disks’ application. See the video for a walk through. It is easier if you only plug in one USB drive during this procedure. When finished, the disk should show as /dev/sda1 or similar. Note: Make sure that the partition is Ext4, as NTSF will appear to copy correctly but cause issues later on. Typically it is easiest to set the volume label for later use during this process.
Step 2
Copy the application area of the micro SD card to the USB drive. The script copyRootToUSB.sh copies the contents of the entire system micro SD card to the USB drive. Naturally, the USB drive storage should be larger than the micro SD card. Note: Make sure that the USB drive is mounted before running the script. In order to copyRootToUSB:
usage: ./copyRootToUSB.sh [OPTIONS]
-d | --directory Directory path to parent of kernel
-v | --volume_label Label of Volume to lookup
-p | --path Device Path to USB drive (e.g. /dev/sda1)
-h | --help This message
In the video, we:
$ ./copyRootToUSB.sh -p /dev/sda1
Step 3
Modify the /boot/extlinux/extlinux.conf file on the USB drive. An entry should be added to point to the new rootfs (typically this is /dev/sda1). There is a sample configuration file: sample-extlinux.conf in the repository.
Modify the /boot/extlinux/extlinux.conf file located on the USB drive. This is in a system protected area, so you will need privileges to change the file, ie ‘sudo gedit’. Make a copy of the ‘PRIMARY’ entry and rename it sdcard.
In the PRIMARY entry change the location of the root to point to the USB drive, ie change ‘root=/dev/mmcblk0p1’ which is the address of the SD card. Provided in this repository is a sample configuration file: sample-extlinux.conf as an example.
While using root=/dev/sda1 in the extlinux.conf works, it can be a good idea to use the PARTUUID of the disk to identify the disk location. Because USB devices are not guaranteed to enumerate in the same order every time, it is possible that that /dev/sda1 points to a different device. This may happen if an extra flash drive is plugged into the Jetson along with the USB boot drive, for example.
The UUID of the disk in the GPT partition table is called the PARTUUID. This is a low level descriptor. Note that there is another identifier, referred to as UUID, which is given by the Linux file system. Use the PARTUUID for this application, as UUID has been reported to cause issues at the present time in this use case.
There is a convenience file: partUUID.sh which will determine the PARTUUID of a given device. This is useful in determining the PARTUUID of the USB drive. Note: If the PARTUUID returned is not similar in length to the sample-extlinux.conf example (32a76e0a-9aa7-4744-9954-dfe6f353c6a7), then it is likely that the device is not formatted correctly.
$ ./partUUID.sh
While this defaults to sda1 (/dev/sda1), you can also determine other drive PARTUUIDs. The /dev/ is assumed, use the -d flag. For example:
$ ./partUUID.sh -d sdb1
After saving the extlinux.conf file, you are ready to test things out. Shutdown the Jetson, and remove the SD card. Boot the Jetson, and the Jetson will then boot from the USB drive. There will be a slight pause as it looks for which drive to boot from. Make sure that your USB drive is plugged in when you attempt to boot the machine, as this is not hot swappable.
If you encounter issues, you can always boot from the SD card again.
Then you are all set! Enjoy.
Notes
Once you flash your Jetson with JetPack 4.5+, the QSPI-NOR changes and may not properly boot a disk that was created using JetPack 4.4 and lower. In order to restore the QSPI-NOR you will have to run the SDK Manager from a pre-JetPack 4.5 release on a host PC. The SDK Manager will refresh the QSPI-NOR for the earlier version, and should be able to work with the earlier card.
There is an issue with JetPack 4.5 for which you will need a workaround. This was fixed in JetPack 4.5.1. The workaround is in the releases of the bootFromUSB repository, but you should use JetPack 4.5.1 if possible.
Release Notes
March, 2021
- JetPack 4.5.1
- L4T 32.5.1
- Samsung T5 250GB
- Tested on Jetson Nano
50 Responses
Thank you Jim, and Happy St Patrick’s Day!
Ive been waiting for this!
Boot from USB worked a treat only problem is that the jetson cant see the SD card like at the end of the video. Can you advise me on what I could do to fix this?
It doesn’t even show up in the disks application
Good! My Nano boot again from SSD. Thanks!
Good to hear. Thanks for reading!
My question is whether I can use different image in the external USB drive. I dont want the same images in the sdcard and the USB drive.
I want to boot a different image like this one :
https://forums.developer.nvidia.com/t/xubuntu-core-xfce4-18-04-2-custom-image-for-the-jetson-nano/76481
I it possible I use Balena Etcher to flash the external USB drive and use sdcard extlinux.conf to boot into the USB ?
Is it possibe
I don’t know anything about that image. Good luck on your project!
Hi Jim. I followed your tutorial to boot from my SSD and it worked quite well. Just one comment and a question for you:
1) Comment: I had to power my SSD externally because the time between power being delivered to USB was sometimes too long, so that the Jetson either reverted back to SD Card or got stuck in the boot sequence. It may therefore be beneficial to mention this in your tutorial.
2) Question:After moving my root to SSD, jetson-io doesn’t work anymore. I get the error “findfs partlabel=app could not resolve”. I understand this is because my SSD partition is not called APP. If you have an idea how to easily fix this, would be great.
Thank you for the great work!
Cheers
Florian
I am glad you found the article useful.
You can set the name of the partition using the Disks application. Select the partition that you would like to name (partition 1 in our case). Select the gear icon, and select ‘Edit Filesystem…’ from the menu. You will then be prompted to change the name of the partition, change it to APP. Select the ‘Change’ button to save the change.
As an alternative, you can use the CLI parted command. For example, open a Terminal. If your partition is /dev/sda1 as shown in the video, then:
where 1 represents the partition number. The blkid command should list /dev/sda1 with the name APP.
Thanks for reading!
Worked like a charm! After that only had to clean up dtb files and jetson-io was back on track!
Cheers
Florian
I am glad you were able to get it to work.
Nvidia docs seem to indicate I can do this with my TX1. Since I can’t remove the eMMC do I just delete the /boot/extlinux/extlinux.conf file from the eMMC device to get it to continue searching for a boot device and finding the USB next in sequence?
I believe that there is a way to set the boot order that was posted in the official NVIDIA Jetson forums. Thanks for reading!
Does this work for Xavier NX kits also? I am no longer up to date with all the details
This method does not work with the Xavier NX.
Many thanks for this, Jim!
I’ve run into an issue. It seems that my nano won’t boot directly off of usb without keeping the sdcard into the nano and modifying its extlinux.conf to point to the usb first. I think this is because the nano won’t power on the drive before extlinux.conf is loaded and therefore it can’t find the boot drive and just gets stuck on the boot screen. This might be solved if using an external powered hub for the drive but I do not have one of those to test.
Are you aware of any solutions to this? Can we force the nano to power on usb devices early?
You are welcome, even though I’m not sure how useful it is to you if it doesn’t work. Which Nano are you using, and which USB drive? I’ve done a little investigating on a Nano 2GB, which works in the opposite way. If it is plugged directly into the Nano, it works. If it is through a powered USB hub it does not.
The issue appears to be very early on in the boot process, where the drive is not recognized when the USB subsystem starts and enumerates the devices. This is long before Linux itself is loaded. Examining the Serial Console output could help determine if this is the cause in your case. I am not sure of a good solution as it is mired down in the boot loaders. They may be some script work around for U-Boot/CBoot to reset the USB subsystem after it enumerates the first time, but it requires more knowledge than I possess about this part.
I am having the issue where I cannot run jetson-io.py when booting from the USB drive. I did name my partition to APP per the conversation above. I just get a screen that flashes the outline and then immediately exits. Any further info on how you got jetson-io.py to work on the USB drive? Thanks.
Can Jetson Xavier NX boot from NVMe without eMMC/SD card?
On March 10, 2021 you posted “Jetson Nano – Boot from USB” utilizing JetPack 4.5+, which supersedes a previous post using an older versions of JetPack that relied on the system boot from information stored on specific partitions on the SD card. You stated that “when JetPack 4.5+ goes through its setup process, it transfers information to boot the system from the SD card to the QSPI-NOR. Consequently, the Jetson can use this information to boot the system without having to have a SD card present.
My question is, have you explored Jetson Xavier NX boot from NVMe using the same methodology, without relying on the eMMC/SD card?
Thank you in advance
Best regards,
André
I’ve run into a problem with a current USB conversion. when running my first sudo apt-get upgrade I see many errors. Is there something unique in my installation? text follows:
Script started on 2021-07-01 14:09:58-0600
wsprdaemon@wspr8:~$ sudo apt-get upgrade
[sudo] password for wsprdaemon:
Reading package lists… Done
Building dependency tree
Reading state information… Done
Calculating upgrade… Done
The following packages were automatically installed and are no longer required:
apt-clone archdetect-deb bogl-bterm busybox-static cryptsetup-bin
dpkg-repack gir1.2-timezonemap-1.0 gir1.2-xkl-1.0 grub-common
kde-window-manager kinit kio kpackagetool5 kwayland-data kwin-common
kwin-data kwin-x11 libdebian-installer4 libkdecorations2-5v5
libkdecorations2private5v5 libkf5activities5 libkf5attica5
libkf5completion-data libkf5completion5 libkf5declarative-data
libkf5declarative5 libkf5doctools5 libkf5globalaccel-data libkf5globalaccel5
libkf5globalaccelprivate5 libkf5idletime5 libkf5jobwidgets-data
libkf5jobwidgets5 libkf5kcmutils-data libkf5kcmutils5 libkf5kiocore5
libkf5kiontlm5 libkf5kiowidgets5 libkf5newstuff-data libkf5newstuff5
libkf5newstuffcore5 libkf5package-data libkf5package5 libkf5plasma5
libkf5quickaddons5 libkf5solid5 libkf5solid5-data libkf5sonnet5-data
libkf5sonnetcore5 libkf5sonnetui5 libkf5textwidgets-data libkf5textwidgets5
libkf5waylandclient5 libkf5waylandserver5 libkf5xmlgui-bin libkf5xmlgui-data
libkf5xmlgui5 libkscreenlocker5 libkwin4-effect-builtins1 libkwineffects11
libkwinglutils11 libkwinxrenderutils11 libqgsttools-p1 libqt5designer5
libqt5help5 libqt5multimedia5 libqt5multimedia5-plugins
libqt5multimediaquick-p5 libqt5multimediawidgets5 libqt5opengl5
libqt5quickwidgets5 libqt5sql5 libqt5test5 libxcb-composite0 libxcb-cursor0
libxcb-damage0 os-prober python3-dbus.mainloop.pyqt5 python3-icu python3-pam
python3-pyqt5 python3-pyqt5.qtsvg python3-pyqt5.qtwebkit python3-sip
qml-module-org-kde-kquickcontrolsaddons qml-module-qtmultimedia
qml-module-qtquick2 rdate tasksel tasksel-data
Use ‘sudo apt autoremove’ to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
3 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up nvidia-l4t-bootloader (32.5.1-20210614115125) …
3448-300-0000–1–jetson-nano-devkit-mmcblk0p1
Starting bootloader post-install procedure.
ERROR. Procedure for bootloader update FAILED.
Cannot install package. Exiting…
dpkg: error processing package nvidia-l4t-bootloader (–configure):
installed nvidia-l4t-bootloader package post-installation script subprocess returned error exit status 1
Setting up nvidia-l4t-xusb-firmware (32.5.1-20210614115125) …
3448-300-0000–1–jetson-nano-devkit-mmcblk0p1
Starting xusb firmware post-install procedure.
ERROR. Procedure for xusb firmware update FAILED.
Cannot install package. Exiting…
dpkg: error processing package nvidia-l4t-xusb-firmware (–configure):
installed nvidia-l4t-xusb-firmware package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of nvidia-l4t-initrd:
nvidia-l4t-initrd depends on nvidia-l4t-xusb-firmware (= 32.5.1-20210614115125); however:
Package nvidia-l4t-xusb-firmware is not configured yet.
dpkg: error processing package nvidia-l4t-initrd (–configure):
dependency problems – leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
Errors were encountered while processing:
nvidia-l4t-bootloader
nvidia-l4t-xusb-firmware
nvidia-l4t-initrd
E: Sub-process /usr/bin/dpkg returned an error code (1)
wsprdaemon@wspr8:~$ exit
exit
Script done on 2021-07-01 14:10:54-0600
Hello,
very good tutorial. In former times I could run from USB following “rootOnUSB”.
I now tried boot from usb, but unfortunately I get errors when running the copy script with:
./copyRootToUSB.sh -p /dev/sda1
The script stopped with errors:
”
rsync is already the newest version (3.1.2-2.1ubuntu1.1).
0 upgraded, 0 newly installed, 0 to remove and 261 not upgraded.
8,901,257,952 93% 41.51MB/s 0:03:24 (xfr#33052, ir-chk=1015/40914)
rsync: write failed on “/media/tom/jetsonssd/usr/lib/libreoffice/share/gallery/$
rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.2]
”
Any ideas what I am doing wrong?
I am using Jetson Nano Version 1 with JetPack 4.5.1.
Thanks for any help,
regards
Thomas
Code 11 usually means disk full.
Thanks, for Sure the SSD has enogh space. Ist ist because the SD Card Runs Out of memory? It is a 32 GB Card with just the jetson Image in IT.
Any luck with a clean 4.6 install? I’ve tried a couple of times, however no luck yet.
I could install the Usb drive with 4.6 without problems.
With 4.5 I had problems.
I have noticed that the jetson cant find mu SSD unless I un-plug it and plug it back in while its looking for the boot target.
It sometimes wouldn’t auto mount either so may be thats related
I’m also curious if anyone has been successful using this procedure with a clean JP 4.6 install? I’ll start jumping down some rabbit holes if it is my specific hardware that is the root of the problem. But if it’s a 4.6 issue, well, that could be a big rabbit hole. Might consider taking a left at Albuquerque.
More than likely a 4.6 issue. I won’t be able to look at in for a while.
Well, I think the problem may be one of (im)patience.
I flashed a 4.5.1 image, followed the instructions, and had no issue. So then I did an apt upgrade and things went sour. It seems like there’re still some hard-coded mmcblk0p1 references (perhaps /etc/nv_boot_control.conf??) that the upgrade scripts are trying to reference. At least the error messages still reference this partition. So the upgrade failed, as did all subsequent dpkg repair attempts. Kinda hosed.
So I decided to try to push through with another clean 4.6 install. Seemed to fail as before. But unlike before, I decided to just let it sit while I wailed in anguish. And it decided to work. Apparently that first boot will sit (seemingly) stalled on a […falling back to…] something or another message. But after a minute or two this finally clears and the GUI pops up. Perhaps it’s just updating the QSPI-NOR or running some diagnostics. But subsequent reboots are snappy. So…yeah! All seems to be working.
Just don’t try to upgrade from 4.5.1. And be a little patient on that first reboot to SSD.
Okay, me AGAIN. Oddly, this worked fine on one Nano and not on the other. Or I should say it works without the SD card on one, but requires the SD card to be inserted on the other. Boots to the USB SSD, though, as long as my SD’s extlinux.conf points to the SSD.
I think the one that works without the SD inserted is the older revision (A02?) and the one that doesn’t is the new one (B01?).
Does this match others’ experience with this? I don’t mind leaving an SD in one of my Nano’s (unlike Jim, I don’t have a suitable shrine of obsolete removeable media on which to display it). But I’m a little OCD with this kind of thing, and this doesn’t seem like something that SHOULD be required.
i tried the script with jetpack 4.6 and jetson nano A02.
the script successfully completes the copy, but there is no extlinux.conf file in the boot directory.
I noticed instead that in grub.conf, the UUID of the PC HOST are copied, rather than the UUID of the sda1 USB partition.
I tried to replace them with the UUIDs of the sda1 partition, but it still does not boot from a USB stick.
it could be that with 4.6 the boot system has changed
The script should be run on the Jetson itself, not a host PC.
Hello !!!
Work fine with a Jetson 2Go and a Samsung MZNLN512HMJP SSD drive 🙂
Thank you very much !!!
You are welcome. Thanks for sharing your results, and thanks for reading!
Thank you for this, I was able to get my Nano 4GB mostly working with this. It works for a hard boot from power, but if the system soft reboots from a reboot command it fails to boot until power cycled. I get repeated mount: /mnt/: can’t find PARTUUID=….. messages then just end at a bash prompt eventually. I downloaded 4.6 and everything works, just not if it ever has to soft reboot. Do you have any suggestions where to look? I am using the PARTUUID from partUUID.sh.
Thanks
Hi there!
What an excellent video! I was wondering if you know if this is possible with the EMMC module? Or if it only works with the SD Card developer kit version.
I haven’t tried it with the eMMC module, so I don’t know. Thanks for reading!
Thank you very much for this, I followed the video but it was impossible to boot using UUID, so I used the path /dev/dba1 … and it works fine so far.
You are welcome, and thanks for reading!
Thanks Ser, I did everything according to the manual using an NVMe drive and it works!
I used Gparted for partitioning.
I am glad you were able to get it to work! Thanks for reading!
I tried the Jetpack 4.61 which is the latest, I followed faithfully the instructions but encountered multiple errors during boot up like there are so many missing files. Has anyone tried Jetpack 4.61? If so have you encountered any issues? My board is B01.
What type of USB drive are you using? Are you using a powered USB hub?
I am using Jetpack 4.61 4GB jetson nano, the screen just hung up at the Nvidia green logo.
I tried both sda2 (my sda1 is a swap partition) and partUUID with same problem. The initial sreen always show a missing SD sign at upper RHS. Not sure if in 4.6 the way the flash memory firmware is updated differently?
I dont really want to modify the boot extlinux.conf in SD card and keep it to boot from USB SSD as it looks a bit silly but probably would work. (BTW i am using a power hub for the SATA SSD)
It’s difficult to say what the issue might be from the information provided. Typically you need to change the extlinux.conf file to boot from the correct device. The serial debug console can shed further light on the issue.
Great tutorial
After my first attempt the Nano hung at the second nvidia splash screen.
Plugged into the serial debug port and could see that it wasn’t finding the USB storage. This was because the disk wasn’t powering up in time so had to alter the boot delay in UBOOT to twenty seconds and all is well.
I’m glad you were able to get it to work. Waiting for the hardware to initialize is a little bit tricky when trying to boot the system. Thanks for reading!
Fails on with JetPak 4.6 requires initrd flash.
See NVIDIA forum for rationale.
Request that the website include a disclaimer and or is pulled.
https://forums.developer.nvidia.com/t/boot-from-external-drive/182883
It’s not clear what procedure you followed and failed. Is this a NVIDIA development kit, or a third party with a eMMC instead of SD card. Did you flash the QSPI, and what USB drive did you use?
Hi,
I have recently carried out this procedure (on a jetson nano 4GB), and it worked like a charm. So thank you very much for that. However, I now have an image on the SSD which I have been working on and now would like to back that up, before I make any further major changes.
Please note, I only have the one SSD drive (the one with the image and in use), but would like to create and hold a backup (for restore) on a laptop, thus if possible I would like it to also be a reduced size image i.e. not the actual size of the SSD, but only the necessary image.
Even at this late date, could you please point me in the right direction as how i can do this.
Thank you for any response you can give me
Can anyone please post an example of how extlinux.conf should look like? I fail to understand what we’re supposed to do at step 3. Cheers
As stated in the article, there is a sample extlinux.conf in the repository called sample-extlinux.conf
Good luck on your project.