Excess temporary files can lead to inode shortage

[wchest]

We have just come across an issue on our Strider CI server where use of npm module caching leads to very large number of folders and files being created in /tmp. We use strict module caching on our repos and were alerted to this issue when our tests failed because of the following error:

Error: ENOSPC: no space left on device, open '/home/[user]/strider/builds/data/techchange-shawhowa-573f4bb9e8c8eb04091498f7/node_modules/lodash/_baseInverter.js

We checked and had plenty of disk space on our server, so a bit more inspection alerted us to the fact that all of the inodes on our server were being consumed by the npm temporary files.

I haven't dug enough into the code to figure out where these temporary files are being created and not sure if there's a programmatic solution to this, but at the very least it might be helpful to provide a warning to people using the caching that they may need to clear out their /tmp folder on a regular basis to prevent inode issues.

[crobinson42]

@wchest I ran into this problem today. My builds started failing due to No space left on device. ie:

wk/images: Cannot mkdir: No space left on device
tar: lib/node_modules/npm/node_modules/request/node_modules/hawk/images: Cannot mkdir: No space left on device

...a warning to people using the caching that they may need to clear out their /tmp folder on a regular basis to prevent inode issues.

What is taking up space in your /tmp dir?

I use strict caching for my node_modules also but have installed Strider in my home dir /home/admin/.strider. When running du -h (or using ncdu package) I see the folder /home/admin/.strider/cache is using Gigs of disk space. After looking into the directory a little further it appears the hashed node modules are not being deleted when a fresh hash is created and node_modules are installed again.

/home/admin/.strider/cache/crobinson42/strider-test-repo/node/modules
admin@jenkins:~/.strider/cache/crobinson42/strider-test-repo/node/modules$ ls -li
total 20
 459580 drwxrwxr-x 84 admin admin 4096 Jul 27 16:28 4609bfab919d621a758cb8b1d8634ee9
1043345 drwxrwxr-x 84 admin admin 4096 Jul 27 17:08 714060a7b2c32cbc505077c33e7aba12
 674544 drwxrwxr-x 84 admin admin 4096 Jul 28 23:11 a84199a611bf36dab5b3dbcceda18610
 632830 drwxrwxr-x 84 admin admin 4096 Jul 28 23:18 alpha-release
 680805 drwxrwxr-x 73 admin admin 4096 Jul 28 22:43 master

[knownasilya]

Strider-CD/strider-node#26

[wchest]

@crobinson42 I believe it was cached versions of node modules that were taking up space in /tmp for us. However, it wasn't the actual amount of storage space that was the issue, it was the excessive number of inodes being used due the large number of files created with each cache. Either of the strategies proposed by @knownasilya in #26 should solve the issue though. For now we just have a cronjob clearing out the cached node modules every week. Not a great solution in the long run, but working for now.

[crobinson42]

I'm now experiencing this error. Error: ENOSPC: no space left on device, open

After running df -i to see inode usage on the linux box, there are none free. This linux machine is dedicated to StriderCD...

/tmp$ df -i
Filesystem      Inodes   IUsed IFree IUse% Mounted on
/dev/vda1      1310720 1310720     0  100% /
none             62722       2 62720    1% /sys/fs/cgroup
udev             61642     402 61240    1% /dev
tmpfs            62722     317 62405    1% /run
none             62722       1 62721    1% /run/lock
none             62722       1 62721    1% /run/shm
none             62722       3 62719    1% /run/user