RocksDB loads a new librocksdbjni*.so file into the /tmp directory each time, causing a "No space left on device" error.

[liang636600]

Hi, @adamretter
I’m running a RaftJava project that uses RocksDB as its underlying storage. After running the project more than 1000 times, I encountered an error from RocksDB:

Caused by: java.lang.RuntimeException: Unable to load the RocksDB shared library
java.io.IOException: No space left on device
    at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:67)
    at org.rocksdb.RocksDB.<clinit>(RocksDB.java:35)

I then checked the /tmp directory, and part of its contents are shown in the attached image. I believe that RocksDB is loading a new librocksdbjni*.so file every time, which eventually leads to running out of space in the /tmp directory.

I think this is an issue with RocksDB, and a potential fix would be to check whether a librocksdbjni*.so file already exists in the /tmp directory before attempting to load it, and if it does, skip the loading process.

Steps to reproduce the behavior

The version of RocksDB I’m using is as follows:

<dependency>
    <groupId>org.rocksdb</groupId>
    <artifactId>rocksdbjni</artifactId>
    <version>5.1.4</version>
</dependency>

I'm looking forward to your response. Thank you!

[adamretter]

@liang636600 those files are created as temporary files, and as we call File#deleteOnExit() it should be deleted when the JVM stops, see: https://github.com/facebook/rocksdb/blob/main/java/src/main/java/org/rocksdb/NativeLibraryLoader.java#L133

[liang636600]

Hi, @adamretter. Thank you very much for your response and the valuable insights. However, I was wondering if there might be an opportunity for improvement of the createTemp function(

rocksdb/java/src/main/java/org/rocksdb/NativeLibraryLoader.java

Line 112 in 138c7b6

private File createTemp(final String tmpDir, final String libraryFileName) throws IOException {

).

The primary goal of this function is to create a temporary file to copy the library to. It might be beneficial to first check whether a related library file already exists in the /tmp/ directory. If such a file exists, reusing it could potentially avoid creating a new temporary file. This adjustment could help prevent issues like the one I encountered, where repeatedly creating new files caused the /tmp/ directory to run out of disk space.

Thank you for considering this suggestion, and I look forward to your thoughts!

[adamretter]

The primary goal of this function is to create a temporary file to copy the library to. It might be beneficial to first check whether a related library file already exists in the /tmp/ directory. If such a file exists, reusing it could potentially avoid creating a new temporary file.

This could lead to a race condition. As the user could have more than one JVM running RocksJava - it is impossible to know whether an existing file is in-use or will be deleted when a different JVM exits.

This adjustment could help prevent issues like the one I encountered, where repeatedly creating new files caused the /tmp/ directory to run out of disk space.

I am wondering how you were able to end up in this situation considering that the JVM is meant to delete the file when it stops. Any ideas?

[liang636600]

My project is a pseudo-distributed system, and every time I run it, I start 30 nodes, each of which uses RocksDB for storage. As a result, each run creates a librocksdbjni*.so file in the /tmp directory for each node, totaling 30 such files. The system is shut down by killing the processes using kill -9 with the process IDs. I believe the main issue is that the JVM does not exit properly, which causes the librocksdbjni*.so files in my /tmp directory to accumulate with each run. Thank you for your response, you’re very kind!

[lizhimins]

When running unit tests for my project, I also found many so file in /tmp.

/tmp/librocksdbjni13154662332689126021.so
/tmp/librocksdbjni16490315955904600167.so
/tmp/librocksdbjni4939072163525448187.so
/tmp/librocksdbjni47661560017069608.so
/tmp/librocksdbjni10726819897969016968.so
/tmp/librocksdbjni15249121562563185552.so
/tmp/librocksdbjni1559163457289158633.so
/tmp/librocksdbjni9930408341636859259.so