In production, I faced issue once when servers had no inode space left but disk space was available. This post is about how to troubleshoot such issue.
First of all, we will check disk space usage with $df command
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 33030016 10407780 22622236 32% /
tmpfs 368748 0 368748 0% /lib/init/rw
varrun 368748 56 368692 1% /var/run
tmpfs 368748 0 368748 0% /dev/shm
So, lot of disk space available here.
Now, We will do $df –i to check the inode usage. If you have less 100 inode usage that means you should free your inode usage by deleting a huge amount of small unused files.
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda 2080768 2080768 0 100% /
tmpfs 92187 3 92184 1% /lib/init/rw
varrun 92187 38 92149 1% /var/run
varlock 92187 4 92183 1% /var/lock
tmpfs 92187 1 92186 1% /dev/shm
How will you find them?
for i in /*; do echo $i; find $i |wc -l; done
This will help you to find the number of files in each directory. If you find a huge amount of empty files in any directory, better to delete them.
sudo rm -rf /home/culprit_user/culprit_directory
Use this command to remove them.
Now, check inode usage again using $df -i