It is a simple task to a sea of gigabyte goodness to the NVIDIA Jetson Xavier NX by adding a NVMe SSD. This is the best performance upgrade you can make for your Jetson. Looky here:
Background
Even though they share the same form factor, the Jetson Xavier NX Developer Kit has a nice addition in comparison to the Jetson Nano. A M.2 Key M slot. With the M.2 Key M slot, we can easily add a Solid State Drive (SSD). The M.2 Key M slot uses the Non-Volatile Memory Express (NVMe) protocol that runs over PCIe.
There are two types of SSDs that use Key M. When you install a SSD card in the slot, the card needs to be the NVMe PCIe type.
The SSD is typically 5-20x faster than the flash memory in a SD card, so anytime you do disk access things speed up considerably.
This process works on the Jetson AGX Xavier too. See Notes below.
Note: Before upgrading using Over The Air (OTA) updates, read the notes below in Upgrading.
Materials and Tools
In the video, we install a Western Digital 500GB NVMe SSD. There are several different sizes and brands of these types of devices, we also get good results with the Samsung variety. We also use our trusty iFixit Pro Tech Toolkit which contains a variety of useful tools for just this purpose.
Hardware Installation
The M.2 Key M connector is on the underside of the carrier board. A retaining screw is at the end of the M.2 slot. Remove the retaining screw. From the factory, the screw may have some blue thread locker on it, which may require a little elbow grease to start the removal process.
Next install the the SSD card into the Key M connector, and then use the same retaining screw to hold it in place.
Disk Configuration
After the hardware install, it is time to configure the disk under Ubuntu. Hook the Xavier back up to a keyboard, mouse and monitor. In the video above, we use mostly GUI tools, please refer to the video for a walk through.
The basic steps are to format the disk, and then create partition(s). In the video, we just allocate most of the disk space to one big ol’ partition. However you may want to be a little fancier. Being Linux, there are tools for doing this through the command line, as all the pros will tell you.
At this point, you can now begin using the SSD. It will show up on your desktop as an unmounted drive.
Running from SSD
We go a little further here. After the Jetson boots the image from the SD card, we switch the rootfs to point to the SSD. In effect, the system will now run from the SSD, the SD card is only there to boot the system.
On the JetsonHacks account on Github, there is a repository rootOnNVMe. Clone the repository:
$ git clone https://github.com/jetsonhacks/rootOnNVMe
and switch over to that repository’s directory:
$ cd rootOnNVMe
Next, copy the rootfs of the eMMC/SD card to the SSD
$ ./copy-rootfs-ssd.sh
Finally, we will add a service which will run a script when the system starts up. The script will “pivot the root” to the SSD so that the system will run from the SSD.
$ ./setup-service.sh
There’s a chapter in the video ‘Tech Talk Time’ that walks through how the script works. After setting up the service, reboot for the changes to take effect.
The performance of your system will feel much snappier after this simple modification.
Boot Notes
These script changes the rootfs to the SSD after the kernel image is loaded from the eMMC/SD card. For the Xavier NX, you will still need to have the SD card installed for booting. As of this writing, the default configuration of the Jetson NX does not allow direct booting from the NVMe.
Upgrading
Once this service is installed, the rootfs will be on the SSD. If you upgrade to a newer version of L4T using OTA updates (using the NVIDIA .deb repository), you will need to also apply those changes to the SD card that you are booting from.
Typically this involves copying the /boot* directory and /lib/modules/<kernel name>/ from the SSD to the SD card. If they are different, then modules load will be ‘tainted’, that is, the modules version will not match the kernel version.
Special Shout Out!
This is taken from the NVIDIA Jetson AGX Xavier forum https://forums.developer.nvidia.com/t/how-to-boot-from-nvme-ssd/65147/22, written by user crazy_yorik (https://forums.developer.nvidia.com/u/crazy_yorick). Thank you crazy_yorik!
Notes
- Initial Release, May 2020
- JetPack 4.4 DP
- Tested on Jetson Xavier NX, Jetson AGX Xavier
Installation of the SSD is different on the AGX Xavier. See: Install NVMe SSD on NVIDIA Jetson Developer Kit. After formatting the drive, software installation is the same as shown in the article.
46 Responses
Hi, I immediately went out and bought a SSD for my NX and I am very happy with the snappy results. I worked from the video that I found to be fantastic with humor too.
My only concern is that if and when I flash a new micro SD card, will I be able to handle it? Right now we are on Jetpack 4.4, and there will certainly be a 4.5 in the near future. I have read crazy_yorik’s forum comments, but that is beyond me, I was good with your video and hope that you will lead us through any updates without starting all over again.
It depends on how you decide to upgrade your Jetson. If you do an over the air update, then you will need to copy the new kernel image and /boot directory over to the SD card. Typically you would also copy over the modules too. Thanks for reading!
I have a question. I have been reading your articles on changing the swap file on the nano and was trying to use resizeSwapMemory article to reduce the size of my swap file. I also used this article to run the NX from an SSD which is working great.
However, is this creating any issues with the swap file? I notice that I have a 8GB swfile on the micro SD card and a 4GB swapfile in the SSD. Activity Monitor sees nearly 12 GB of swap (8GB swap file and 4GB of zram). I would like to be using a swap file on the faster ssd vs. the sd.
Hi! I bought a WD Black 500GB Nvme SSD but it’s not showing up on the Xavier NX. Could my NVME slot or SSD be faulty? Any possible solutions or ways of investigating?
I do not know what you mean by “not showing up”.
Since we are dealing with the same hardware, it could be in the process and hopefully not the hardware. Did you follow the video step-by-step? I do not remember all the details, but the video went through the formatting of the disk and then I followed the process step by step. Watch the video again and see if you missed something.
Thanks for the excellent guidance – very much appreciated!
So if a new version of jetpack comes out and we have already setup booting to SSD, how do you recommend updating the whole system?
Thanks!
This process no longer works. I ran the scripts, but it failed to identify the new NVME disk, so I changed it from /dev/nvmeonn1p1 to /dev/nvme0n1 and mounted it to /mnt seems to work fine and browsing the folders shows a copy of the FS.
I then run the startup scripts, but nothing is changed, the system boots from and loads from the SDHC and /mnt isn’t even mounting.. where else can I look?
I do not understand your question. What does “failed to identify the new NVME disk” mean? How did you partition the disk?
The script assumes /dev/nvmeon1p1 and the Samsung Evo 970 Plus seemed to install at /dev/nvme0n1. So I mounted /dev/nvme0n1 to /mnt and re-ran the scripts. When the device reboots, it boots from the SD card and loads from the SD card. /mnt shows up, but it isn’t mounted until I access it. Are there items in the second script that hard code /dev/nvmeon1p1? It seems that everything points to /mnt, but it doesn’t work. How can I force it, or do this manually without the script?
I got it working. I just changed all the instances of /nvme0n1p1 to /nvme0n1. It’s in a few locations including a couple files in the data folder.
I am glad you got it to work. However, the scripts assume that you partition your disk as shown in the video (that is what the p1 indicates).
Oh right. I think by the time I turn 50 I will start paying attention to details. Thanks. Got it working correctly with the newest jet pack. Your contributions are incredible and valuable and much appreciated. Don’t let internet jerks like me discourage you.
Thanks for the note. I don’t think this is trolling, I know you were probably a little frustrated. Unfortunately there are some things you need to learn the hard, expensive way if you know what I mean.
If you want to see an inappropriate response, look down a couple of comments and look at Alain’s comment. Note that the main issue encountered was something wrong with the NVIDIA update process, unrelated to the anything in this article. Fortunately when I was a young lad I was taught never to take criticism from someone from whom I would not take advice. Turns out that’s all you need to know when dealing with people on the inter-webs.
As with most computer stuff, missing one of the steps can ruin everything afterwards. BTW, I think once you’re 50 you pay even less attention to details unless that’s your personality type to begin with. Thanks for reading!
Thank you for this article, could you please give us more details on the UPGRADING part ? I already bricked a jetson Xavier nx (“cpu bootloader is not running on device”) and I suspect this was caused by the Nvidia OTA bootloader running on a device that booted from the SSD. I updated everything on the micro sd card, then ran your script to boot from SSD, but today I am seeing on the update list an OTA bootloader update waiting, very afraid to run it …
Well, with new jetpack 4.4 update, the Xavier NX does not boot anymore.
I think you proposed this method to quickly. With major update, everything goes down.
Many thanks for this poor method. Now, i have to restart from zero. Great. Really great.
Too soon? This was published in May. You don’t state how you upgraded, or what steps you took for taking into account that the rootfs is on the NVMe drive.
I do not know why you believe you need to restart from zero, but good luck on your project.
That isn’t the fault of this process to move root to .m2 its the upgrade its self that was broke. Blame NVIDIA, not these guys. ( and that upgrade bug is supposed to be fixed by now )
Can the Jetson Nano be upgraded to a SSD?
Thanks,
Brent
The Jetson Nano does not have a slot for a NVMe SSD like the Xavier NX. Typically you would attach a SSD through a USB port. Thanks for reading!
Is seems this procedure works, but gets broken if the user takes a JetPack updates which come out periodically. Is there ways to either avoid this breaking and still run the OS off NVMe. Or is there a way to quickly fix the issue? And if so, what are the steps?
I’m working on the articles for the upgrade steps. However, upgrading is a two step process. First, upgrade the system (running from the SSD). Before rebooting, copy over the /boot directory to the SD card. Something like:
$ sudo mkdir /mnt/sd/
$ sudo mount /dev/mmcblk0p1 /mnt/sd/
$ sudo rsync -a /boot/ /mnt/sd/boot/
I don’t think so, but you may have to copy over the modules
$ sudo rsync -a /lib/modules/4.9.140-tegra/ /mnt/sd/lib/modules/4.9.140-tegra/
I have not tested this enough to make sure it is bullet proof, but others have had good success with it.
Thanks. This worked for me.
I just copied boot and modules over to the SD *before* a reboot after upgrades. Worked just fine. After, might not be a good idea. May not boot, depending on how much of a change.
Love It
I tried your commands after the OTA upgrade and then rebooted with success. I was a bit like a dog watching TV, but so far it works. Thanks. I look forward to your new article on the subject just to see if we can make any tweaks to the system. Any chance of booting from the SSD in the future>
Hi, there was another OTA upgrade for Ubuntu and NVIDIA upgrades. This time I just let the upgrade take place and rebooted with no issues showing up yet. Should we still be using the following commands? I have not done this yet.
$ sudo mkdir /mnt/sd/
$ sudo mount /dev/mmcblk0p1 /mnt/sd/
$ sudo rsync -a /boot/ /mnt/sd/boot/
Is it possible to put the small SD boot part on the on board eMMC and then remove the SD card?
I don’t know what this means. Are you talking about a Jetson Xavier NX module? The regular dev kit does not have on board eMMC.
Perhaps he has a production module? But i thought the production ones didnt have SD. ( and too lazy to look )
For object detection and crowd counting, how large of a SSD is required? 1TB? 2TB?
I am not sure what the minimal requirements are, you can check on the NVIDIA Jetson forums. However, most of the demos that I have seen will run comfortably on a 128-256GB SSD.
From when this article was written a number of things have changed for the NX. We are now on JP 4.4 and we are getting OTA updates all the time. I am finally at a point after this last OTA Ubuntu 20.04 update that my NX is no longer working. I have backed up the SSD and am thinking about just using the SSD for backup data and not running off of the SSD as intended in this article. Do you have a better idea? Please advise.
That sounds like a reasonable path to take.
I have been doing a sync to the boot SD after every system update as habit ( its scripted anyway, so one command to run for me ), and as of last week i have not had any issues. As mine came with the kit and i have not bought any production modules I dont have one with emmc, only SD + m.2 at this point.
Hello I have a question about running a file on boot up.
I have a terminal command that I could like to run at boot up.
Now that I have done the upgrade to boot to the NVMe PCIe how do I set this file up to run at boot up?
I can confirm that this script works on my TX2 with a Samsung NVME SSD 🙂
Thanks!
Charles
I am glad you were able to get it to work! Thank for reading!
Hi,
worked really smoothly, thanks for your time to write the scripts!
I’ve not noticed any immediate speed improvements, is there a way to verify that I’m actually running from SSD (and not SD anymore)?
Thank you very much!
-Phil
“is there a way to verify that I’m actually running from SSD (and not SD anymore)?”
Did you find any solution for that?
NEED HELP!!!
I did all steps in the tutorial on the xavier nx and after reboot it seemed to work. The SSD was detected and the SD seemed to be just a normal drive that was plugged in. However, I accidentally (I was to stupid to read that the SD need to be plugged in all the time) removed the SD and reboot. Only black screen, nothing happend, no boot, nothing. I inserted the SD card again, power on, nothing happend, only black screen. I even removed the SSD, flashed a new SD card and inserted it. Nothing is happening, only black screen. It seems that the Xavier NX is dead.
Any suggestions how to fix it??? Appreciate your help.
I’m facing the same problem, did you find the solution?
When I did this on jetson AGX Xavier, I got the following error message:
nvidia@xaiver:~/apps/rootOnNVMe$ ./copy-rootfs-ssd.sh
[sudo] password for nvidia:
sudo: rsync: command not found
Could you please let me know hot to fix it?
Well, it clearly says rsync is missing. If true, install it.
NEED HELP!!!
I did all steps in the tutorial on the xavier nx and after reboot it seemed to work. The SSD was detected and the SD seemed to be just a normal drive that was plugged in. However, I accidentally (I was to stupid to read that the SD need to be plugged in all the time) removed the SD and reboot. Only black screen, nothing happend, no boot, nothing. I inserted the SD card again, power on, nothing happend, only black screen. I even removed the SSD, flashed a new SD card and inserted it. Nothing is happening, only black screen. It seems that the Xavier NX is dead.
Any suggestions how to fix it??? Appreciate your help.
Probably need to follow these instructions to debug the issue.
https://jetsonhacks.com/2019/04/19/jetson-nano-serial-console/