NOTE: THIS ARTICLE HAS BEEN UPDATED WITH AN EASIER METHOD. PLEASE SEE: Jetson Nano – Run From USB Drive.
Using a USB Drive as the root file system on a Jetson Nano Developer Kit provides many advantages, including faster disk access and more storage. Looky here:
Background
For external storage, the Jetson Nano uses a Micro SD Card. The SD card holds the operating system, applications and any data in use. In the overall scheme of things, this device provides relatively slow access. Also, even though Micro SD cards are much better now than they have been in the past, the cards have a reputation of low reliability in long term or heavy use.
Most desktop/laptop computers use a different type of external storage, such as a Hard Disk Drive (HDD) or Solid Sstate Disk (SSD). Typically these are built into the computer, though you can add an external one also.
In fact, we can add one of those types of drives to the Jetson Nano through the USB 3.0 port! We will cover setting up our USB drive so that it can act as the “root file system” for the Nano. Conceptually, this means that the Nano is running off of the USB drive.
Typically most larger computers boot directly to the disk drive. However, this is not possible using the current configuration boot loader on the Jetson Nano. The boot loader understands USB 2.0, the Nano speaks USB 3.0.
Remember that the term ‘boot’ is shorthand for the slang term ‘bootstrapping’, that is, pulling ones self up by their own boot straps. The Jetson uses a two step boot process. The basic idea is that the boot loader loads a memory image with minimal support for key attached peripherals, and then loads over itself with the Linux kernel specified. It’s really clever and quite tricky.
The solution here is to tell the boot loader to use the USB drive as the new root file system. But in order to be able to do that, we have to embed the USB 3 driver directly into the Linux kernel itself (the kernel file is called ‘Image’). Therefore we end up building a new kernel to include the USB 3 driver, and replacing the old one
Installation
Note: Before we get started, a major shout out goes out to Syonyk’s Project Blog for deciphering how to place the Tegra USB firmware in to the kernel. A very nice write up is available there. Thank you!
USB Drives
In the video, we took a look at the following drives:
- Samsung T5 500 GB USB SSD: https://amzn.to/2PtE7bK
- Samsung 860 Pro 256GB SATA III: https://amzn.to/2UEWQCb
- SATA SSD to USB cable: https://amzn.to/2UEB1md
- Western Digital 2TB External Hard Drive: https://amzn.to/2GH50WY
Install
You should do this on a freshly flashed Micro SD card. On the JetsonHacksNano account on Github, there is a repository named rootOnUSB. Clone the repository, and then switch to the repositories directory:
$ git clone https://github.com/JetsonHacksNano/rootOnUSB
$ cd rootOnUSB
You are now ready to build the kernel:
$ ./buildKernel.sh
Note: Depending on which version of L4T you are running, you may have to “git checkout” the appropriate release to match the release you are running.
It takes ~ 45 minutes to download and build the kernel. You should reboot the machine to make sure the Image build is correct. You are now ready to copy the root file system from the Micro SD card to the USB drive.
First, prepare a USB drive by formatting it Ext4 with at least one partition. In the video, we assign a volume label. If you need instructions for formatting, check out the video where we do so using the ‘Disks’ application.
Make sure that the USB drive is mounted (you can open a file browser on it to make sure it is mounted). To copy the root:
$ ./copyRootToUSB.sh -v <Volume Label>
OR
$ ./copyRootToUSB.sh -d <Directory Path>
An example of the first and the latter:
$ ./copyRootToUSB.sh -v NanoSSD500
OR
$ ./copyRootToUSB.sh -d /media/jetsonhacks/NanoSSD500
It will take several minutes to copy over all the files. Note: Removable media like USB drives are typically in the /media/<user name> directory.
The final step is to modify /boot/extlinux/extlinux.conf to tell the Nano to switch over to the USB drive. Make a backup of the file, and then duplicate the primary entry. Rename the label of the original entry, and then set the primary entry to point to the USB device. Typically the USB drive will be /dev/sda1 (1 is the partition). The important line needs to be:
APPEND ${cbootargs} rootfstype=ext4 root=/dev/sda1 rw rootwait
There is a file named sample-extlinux.conf which is a sample of the a modified extlinux.conf file might look like. Notice that the label on the second entry is emmc, which represents the Micro SD device address. If you use the serial console for debugging, you will see two entries that you can boot, “primary” and “emmc”. The first is the USB drive, the second the Micro SD card. This can come in handy if the machine decides not to boot after you make your changes.
Benchmarks
If you use the Micro SD card, you typically get average read rates ~ 87 MB/s and access times of ~ 0.52 msec. With a SSD, you get average read rates of ~367 MB/s and access times of 0.29 msec. About a 4X speedup! A HDD drive performs about the same as the Micro SSD, but tends to be much more reliable over time.
Notes
- Even though the Jetson Nano has 4 USB 3.0 connectors, they all go through one USB hub. Therefore, you end up sharing the USB bandwidth with all the other USB peripherals. Bandwidth can fill up quick!
- If you use swap memory on a SSD best practice is to use a swap file, not a swap partition. The advantage over a swap partition is that the swap file will move around the disk as it is written and deleted. That way the wear leveling algorithms can help manage the loads. A partition is always in the same place.
- Mostly recently tested on Jetson Nano, L4T 32.2 [JetPack 4.2.1]
- In the video, tested on Jetson Nano, L4T 32.0.1 [JetPack 4.2]
73 Responses
Well done!! I was able to follow you and now have a fast, powerful little
machine WOOT, WOOT!!
I am glad you got it to work. Thanks for reading!
Very interesting, thank you for this article!
Is it possible to do the same with high speed usb flash drive instead of SSD? Would there be any speed advantages over SD card?
It is faster. It is not recommended to use a flash drive for a long period of time, as they do not have the needed wear leveling algorithms aboard. They exhibit much the same issues as SD cards under extended use. Thanks for reading!
Hi. Thank you for sharing this tutorial first. I just followed the instruction and there is an issue comes up. When I tried to build the Kernel. it stop at “Firmware blobs root directory (extra_firmware_dir) [firmware] (NEW)”. Should I type some dir in there? Thank you!
You are welcome. You can hit return, it defaults to the directory named ‘firmware’. Thanks for reading!
Thank you so much! It works!
Hi. is there a minimum requirement for space of the SD card which is used for building kernel? Is 16G SD card ok for this?
I have not tried it on a 16GB, I would use a minimum of 32 personally. Thanks for reading!
Hi, the 16GB is too small, so either use a 32 or mount /usr/src on the external drive while building.
You may want to emphasize more on your statement, “You should do this on a freshly flashed Micro SD card.” After waiting for nearly 2 hours for the compile to finish, I realized that it was waiting for input. The previous kernel was modified and it was waiting for my decision to input or continue. It displayed the change that was made and stop the process. No input question or indicator. Not even a blinking cursor. I hit the enter key and it continued.
Thanks for the article! Its still chugging along as I write this.
Will it work on jetson xavier?
These scripts do not work on the Jetson Xavier. The kernel image on the Xavier is signed, and must be flashed from a PC. Thanks for reading.
Once a new kernel is built and booting to SSD is working and one decides to apply other changes to the kernel, what is the process the repeat these steps? The new kernel will be built on the SSD and does one just need to copy the newly created image to the SD card?
That’s correct. There are a couple of caveats:
The new Image needs to be placed onto the SD card /boot directory and
any modules that you modified/created are on the USB drive. Backup as needed.
Thanks for reading!
We are now at Jetpack 4.2.2 and I made the SSD a while back with Jetpack 4.2.
I would like to figure out how I can update the SSD to 4.2.2 without needing to reinstall and recompile all the programs I have on the SSD. After updating the SD card to 4.2.2 my SSD will not boot anymore and if I copy the SD card (with 4.2.2) to the SSD I believe it will loose the installed programs or am I mistaken? I have a backup of SD card but I dont have a backup of the SSD to test this.
Is it correct to say that the USB boot modification does not allow for future firmware upgrades?
Previous versions of JetPack are not compatible with 4.2.2. You can use your host computer to recover data from the SSD. There is an update to this article which will save you about 45 minutes when you set up a SSD drive: Jetson Nano – Run From USB Drive
Your comment in the notes of “A partition is always in the same place.” is not entirely correct. SSD devices (not USB thumb drives or flash cards) do their own internal wear leveling using over-provisioning. While a partition is logically in the same space as you write to the device the physical location moves around based on the device’s wear leveling algorithm. If you are going to use a SSD I would recommend purchasing an enterprise SSD rather than a consumer SSD as an enterprise SSD contains more over-provisioning, faster speeds, and a longer lifetime.
Your blog is is amazing!
Thank you so much for all this.
Thank you for the kind words, and thank you for reading!
Hi, i tried running the above but kept hitting –
rsync: write failed on “/media/user1/myvolumename/usr/bin/lshw”: Read-only file system (30)
I even tried using sudo but the same error. Any advice?
I do not know what that error means, so I do not have any advice to give.
Lookup dmesg any UAS error, my usb3.1 m2 ssd had these troubles too: after copying gigs of data (like the root fs) the ssd issued an UAS error, and was remounted as read-only.
A usb_storage_qirck=xxxx.yyyy.u in the extlinux boot, and/or in /etc/modprobe.d/ … blacklist
had solved my problem (or the linux usb3.0 troubles)…
@Teik Hooi Beh and others having that problem:
“Read-only file system” nearly always is an indicator of the target file system having an error. When the error is sensed, the file system in question is re-mounted read-only. As long as the file system is not root, you can unmount it, and check/repair it.
Assume we are talking about /dev/sda1:
umount /dev/sda1
fsck /dev/sda1
If you have already pivoted to the drive, you can’t simply umount it. The easiest fix is a re-boot. During the re-boot, the system notices an error, and automatically does an fsck. As a last resort, you can plug the drive into another linux system, and do an fsck from there. While doing that, I noticed that the ext4 file system placed onto the ssd by the Jetson was incompatible with a kernel 5.1 on another machine. I reformatted the SSD to the ext4 of the kernel 5.1 machine, and never had a problem since.
@Marcos: “A usb_storage_qirck=xxxx.yyyy.u in the extlinux boot, and/or in /etc/modprobe.d/ … blacklist
had solved my problem” ???? – Could you please be a bit more specific?
If the culprit are UAS errors in your log, it could be a known problem with usb3.0 on linux, with usb3.x adapter that have problem with UAS, but work flawlessly on usb2.
The solution is to use: lsusb to find vendor ID and productID, both are 4 digit codes (XXXX:YYYY), these codes are needed to tell the kernel module loader that a specific kernel module for this device is blacklisted.
In our case we want that the problematic USB adapter does NOT load the UAS kernel module.
By adding the “usb-storage.quirks=XXXX:YYYY:u” to the /boot/exxlinux/extlinux.conf, the kernel will not load it at the startup.
For temporary usb drives that will be automounted:
Edit or create the file /etc/modprobe.d/uas.conf that contain “options usb-storage quirks=XXXX:YYYY:u” , where XXXX:YYYY is your specific USB adapter vendorID, productID.
This was super helpful! Worked first time with no hassles. Thanks so much!
You are welcome, and thanks for reading!
on the final reboot i get the error
tegra-xusb 700900000.xusb: cannot find firmware…retry after 1 second
tegra-xusb 700900000.xusb: Direct firmware load for tegra21x_xusb_firmware failed with error -2
tegra-xusb 700900000.xusb: Falling back to user helper
those three lines repeat over time
This could be a result of several things. The most likely causes tend to be an incorrect extlinux.conf file, followed by not having the USB drive connected.
was able to resolve the problem by starting fresh with a brand new, larger micro sd card. I was using 32GB but the drive was old and very slow and may have not been a good card. After attempting again I noticed a warning about being out of memory and I think I just missed it the first time. The new sd card resolved the problem and was much faster as well. Thanks for these great tutorials!
Thank you for your great instructional and easy to follow videos and all the work you put into them. Single question, after following your instructions and changing the root to the SSD I now want to incorporate swapfile, since the default is /mnt do I need to change the usage instructions to another directory? Thank you in advance
I’m glad you find the articles useful. I don’t know how to answer your question. The rootfs is copied over to the USB drive. Everything pretty much stays the same. I don’t know what usage instructions are. Thanks for reading!
Nice recipe, thank you! I have expanded on it a bit to improve on the kerbnel, mainloy to make it work with the CFS firewall.
Configure configureKernel.sh in scripts as follows:
#!/bin/bash
# Copyright (c) 2016-19 Jetsonhacks
# MIT License
# Configure kernel to include tegra usb firmware
# SOURCE_TARGET and KERNEL_RELEASE should be set in caller
cd “$SOURCE_TARGET”kernel/kernel-$KERNEL_RELEASE
cp /lib/firmware/tegra21x_xusb_firmware ./firmware/
# CONFIG_EXTRA_FIRMWARE=”tegra21x_xusb_firmware”
FIRMWARE_NAME=”tegra21x_xusb_firmware”
bash scripts/config –file .config \
–set-val CONFIG_NETFILTER_XT_MATCH_MULTIPORT m \
–set-val CONFIG_NETFILTER_XT_MATCH_RECENT m \
–set-val CONFIG_NETFILTER_XT_MATCH_OWNER m \
–set-val CONFIG_NF_NAT_REDIRECT m \
–set-val CONFIG_IP_NF_TARGET_REDIRECT m \
–set-val CONFIG_NETFILTER_XT_TARGET_REDIRECT m \
–set-val CONFIG_CGROUP_HUGETLB y \
–set-val CONFIG_CFQ_GROUP_IOSCHED y \
–set-val CONFIG_INET_ESP m \
–set-val CONFIG_IP_VS_PROTO_TCP y \
–set-val CONFIG_IP_VS_PROTO_UDP y \
–set-val CONFIG_NET_L3_MASTER_DEV y \
–set-val CONFIG_DM_THIN_PROVISIONING m \
–set-val CONFIG_DM_BUFIO m \
–set-val CONFIG_DM_BIO_PRISON m \
–set-val CONFIG_DM_PERSISTENT_DATA m \
–set-val CONFIG_IPVLAN m \
–set-str CONFIG_EXTRA_FIRMWARE “$FIRMWARE_NAME” \
–set-str CONFIG_EXTRA_FIRMWARE_DIR “firmware”
Also, add ./makeMudules.sh to the end of buildKernel.sh
# Make the kernel
echo “Making kernel”
cd $BUILD_REPOSITORY
./makeKernel.sh
./makeModules.sh
if [ $? -eq 0 ]; then
# Put a copy of the current Image here
cp /boot/Image $INSTALL_DIR/Image.orig
./copyImage.sh
echo “New Image created and placed in /boot”
fi
That step would be very important for any other changes that affect modules
Thank you, this sounds very useful! Thanks for reading!
Also, the first thing I would do is connect a decent 5V/5A power supply via the barrel jack. The Jetson already taxes even the best 2.5A supply connected via micro USB. Connecting an SSD makes the situation worse. If you can’t find a suitable 5A PSU, use an old/retired PC power supply. The soldering is left as an exercise to the student.
Be aware that this hack turns the connection from high-speed USB3 to very low-speed USB2, simply to get around a mis-behaving USB3 adatpter. It would be a last-resort option. Getting a better USB-to-SATA adapter probably would be preferred first.
See https://unix.stackexchange.com/questions/239782/connection-problem-with-usb3-external-storage-on-linux-uas-driver-problem
Thank you for this great tutorial.
Unfortunately I got the error when executing ./copyRootToUSB.sh that the I hav a read onlz file sysytem.
E.g. mkdir “/media/tom/jetsonSsd/var/metrics” failed: Read-only file system (30)
So can you help me to tell what I am doing wrong?
Thank you.
Tom
Unless you have set a file system to read only, or mounted it as read-only, “read-only- file system” means that the machine has detected an error of the file system, and has set the respective partition to read-only to protect it from developing more errors.
If the ssd has the error, you need to to umount the ssd and fsck the file system on the ssd.
Thank you for your help.
I tried as follows:
sudo ./copyRootToUSB.sh -v jetsonSsd
Destination Target:
Device Path: /dev/sda1
Destination Target: /mnt/jetsonSsd
Target: /mnt/jetsonSsd
E: Konnte Sperre /var/lib/dpkg/lock-frontend nicht bekommen – open (11: Resource temporarily unavailable)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
9,165,199,647 98% 24.71MB/s 0:05:53 (xfr#84184, ir-chk=1110/129090)
rsync: write failed on “/mnt/jetsonSsd/usr/src/kernel/kernel-4.9/vmlinux.o”: Read-only file system (30)
rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.2]
Afterwards I made an fschk and got an OK status.
When I use NTFS the file system, copy works but execution not…
NTFS is a Windows format. For Ubuntu Linux, you should have the disk formatted as Ext4.
Yes, I just wanted to say that the SSD is working. With NTFS the script execution works, with ext4 file system not. So I am wondering why.
You need to create a partition of the ssd, and format it to ext4. It won’t work with ntfs.
Let’s say the ssd is /dev/sda. Create a /dev/sda1 partition, and format it to ext4. Mount /dev/sda1, and try creating a few files on it. If you succeed, you are good to go. If you don’t succeed, you need to rectify this situation before proceeding.
Recompiling the kernel and pivoting the root is not for the faint of heart, and it is not an exercise I recommend when learning.
Thanks.
Now I created the ext4 partition again, and now the script copied all. And afterwards bootup worked :-).
I do not know why it did not work for me at the beginning. I tried it before I created the NTFS around 4 times with ext4. Ok, but now it worked.
Thanks for your help.
BR
Gratuliere
I goofed this up by botching the modification to /boot/extlinux/extlinux.conf I’d formatted the drive with 2 partitions and so I needed sda2 rather than sda1. What bugs me is that if I plug a USB SD adapter into a USB slot, This can become Sdb2:
Disk /dev/sda: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: 88841F98-3EC8-4741-95F4-B36210B18889
Device Start End Sectors Size Type
/dev/sda1 40 409639 409600 200M EFI System
/dev/sda2 411648 976773119 976361472 465.6G Linux filesystem
vs:
Disk /dev/sdb: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: 88841F98-3EC8-4741-95F4-B36210B18889
Device Start End Sectors Size Type
/dev/sdb1 40 409639 409600 200M EFI System
/dev/sdb2 411648 976773119 976361472 465.6G Linux filesystem
Is there a way to express it by label rather than something that is dependent on what is plugged in??
I believe that you can specify the UUID, you can look it up as I recall it’s something like root=UUID={device UUID}
I’ve successfully done this once before, but NVIDIA released a new Image (https://developer.nvidia.com/jetson-nano-sd-card-image-r322) and now I get a “Jetson Board Mismatch” error when I run the buildKernel script. Will this repo be updated for the 32.2 target, or should I try to get a hold of 32.1? Or can I just modify the script to look for 32.2 instead?
Thanks!
I am currently working on it. There are several repositories which are tied to the same code, so it will take about a week before it is available.
Oh, awesome! I appreciate all the hard work!!
ty keep up the great work
You are welcome, and thanks for reading!
These instructions do not work with the release of jetpack 4.2. I upgraded to the new jetpack and when i tried to rootOn USB, I got this error
jetson1@jetson1-desktop:~/rootOnUSB$ ./buildKernel.sh
Jetson Model: NVIDIA Jetson Nano Developer Kit
Jetson L4T:
==== Jetson Board Mismatch! =============
Currently this script works for the jetson-nano.
This processor appears to be a NVIDIA Jetson Nano Developer Kit, which does not have a corresponding script
Exiting
jetson1@jetson1-desktop:~/rootOnUSB$
Anybody have a solution?
I forgot to mention that I am working with the “Jetson nano developers kit” and this rootOnUSB worked on it just fine until i upgraded to the new jet pack 4.2 that was released on July 19 2019. And it should be mentioned that this does not work with the new jetpack 4.2 next to the video title.
The current release on Github works with JetPack 4.2. The new release you mention is JetPack 4.2.1, which is not compatible with the scripts currently.
Hi, the L4T R32.2 has released now, but the script can not work on this release, will it be any update of this script to support the R32.2?
Thank you for your amazing post. I’ve had a problem when i plug my external HDD to my jetson. After i plugged in my external hdd, jetson crash and dead. Before that, i use power source with 5V and 2A. Is that bcs my power source?
Thank you for the kind words. The Jetson by itself uses almost 2A. If you add more peripherals, there is not enough power. In earlier articles, we covered adding a 4A, 5V power supply. Thanks for reading!
does this support nvme?
I have not tried it.
Easiest way to do :
1) burn the jetson image on your ssd usb drive
2) change the /boot/extlinux/exlinux.conf to add rootfstype=ext4 root=/dev/sda1 rw rootwait
3) do a reboot
The jetson will use the usb drive instead of the sdcard to boot up.
Enjoy!
I don’t quite understand. Do you still need to have the SD card present to boot the Jetson?
Just FYI… you do NOT have to rebuild the kernel to use a USB drive as the root partition. All you have to do is rebuild the initrd which you can do on the device in a few minutes.
Details here:
https://devtalk.nvidia.com/default/topic/1062777/jetson-nano/jetson-nano-r32-2-1-partuuid-not-working-while-booting-from-usb/post/5382600/#5382600
Thank you George, this looks very promising!
You’re welcome. Unfortunately, yes you do still need the sdcard until tboot, cboot and u-boot all can initialize the usb 3 controller and detect mass storage devices connected to it.
hi , my hard disk is not visible when i connect with my jetson nano
I don’t know what that means. How are you trying to look at it?
Hi,
I’d like to know where should I modify the /boot/extlinux/extlinux.conf file ? On the micro SD card or on the SSD Drive ? Also should I keep the micro SD card in the socket while booting with the SSD drive, please ?
Thanks and best regards,
Pascal
Modify the SD card version, and keep the SD card installed. The Nano boots from the SD card. There is now an easier method to set this up that does not require rebuilding the kernel. Please see: https://wp.me/p7ZgI9-317
Thanks Jim!
Love your videos! I hope in the future you can work on a Boot from Network video, and if possible expand your expertise from the Jetson Nano to the new nVidia Shield (TX1+)
hello, quick question.
is all of this done directly from the jetson nano?
Yes.
Thanks for this. But this page needs to be updated 🙂 no need to compile kernel anymore
I do not know what you mean. The first line of the article is:
NOTE: THIS ARTICLE HAS BEEN UPDATED WITH AN EASIER METHOD. PLEASE SEE: Jetson Nano – Run From USB Drive.