0 votes
asked in MicroStream for Java by (240 points)

I was testing my app now that I have microstream connected to it and when running encountered the following error.

What I see happening is when I run a lot of process sequentially but very quickly the size if the directory continues to grow eventually to 34Gb, then got this exception.  I have 4 channels, and the 2nd channel directory is 34Gb and has up to 'channel_2_9555.dat'.

I have seen when I throw less processes it grew to 8G and when finished successfully it reported 25Mb space.  Is it buffering data 1st and it cannot cope fast enough to free it up.

Is there any way to recover the state of the directory/database in it's current state or is it unrecoverable.

Exception in thread "MicroStream-StorageChannel-2" one.microstream.storage.exceptions.StorageException: Cannot obtain lock for file /opt/foo/bar/data/channel_2/channel_2_9910.dat
	at one.microstream.storage.types.StorageLockedFile.openLockedFileChannel(StorageLockedFile.java:81)
	at one.microstream.storage.types.StorageInventoryFile.New(StorageInventoryFile.java:29)
	at one.microstream.storage.types.StorageNumberedFile$Default.inventorize(StorageNumberedFile.java:127)
	at one.microstream.storage.types.StorageFileManager$Default.createNewStorageFile(StorageFileManager.java:454)
	at one.microstream.storage.types.StorageFileManager$Default.createNextStorageFile(StorageFileManager.java:531)
	at one.microstream.storage.types.StorageFileManager$Default.checkForNewFile(StorageFileManager.java:525)
	at one.microstream.storage.types.StorageFileManager$Default.incrementalTransferEntities(StorageFileManager.java:1482)
	at one.microstream.storage.types.StorageFileManager$Default.incrementalDissolveStorageFile(StorageFileManager.java:1430)
	at one.microstream.storage.types.StorageFileManager$Default.internalCheckForCleanup(StorageFileManager.java:1378)
	at one.microstream.storage.types.StorageFileManager$Default.incrementalFileCleanupCheck(StorageFileManager.java:1286)
	at one.microstream.storage.types.StorageChannel$Default.houseKeepingCheckFileCleanup(StorageChannel.java:241)
	at one.microstream.storage.types.StorageChannel$Default.houseKeeping(StorageChannel.java:214)
	at one.microstream.storage.types.StorageChannel$Default.work(StorageChannel.java:299)
	at one.microstream.storage.types.StorageChannel$Default.run(StorageChannel.java:351)
	at java.base/java.lang.Thread.run(Thread.java:835)
Caused by: java.nio.file.FileSystemException: /opt/foo/bar/data/channel_2/channel_2_9910.dat: Too many open files
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
	at java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
	at java.base/java.nio.channels.FileChannel.open(FileChannel.java:345)
	at one.microstream.io.XIO.openFileChannelRW(XIO.java:552)
	at one.microstream.storage.types.StorageLockedFile.openLockedFileChannel(StorageLockedFile.java:56)

 

1 Answer

0 votes
answered by (9.7k points)

Hello,

To recover from that error, you need to start the storage again. It is currently not possible to recover from that at runtime.

As the exception stated there are to many open files. Do you know how many files are opened?

Under heavy load Microstream may open quite a lot of file (more then 1000) at the same time.

 

A possible solution would be to increase the max file size for storage files. The default is 8MB. See https://manual.docs.microstream.one/data-store/configuration/housekeeping#file-sizes-and-payload. Some hundreds MB should be no problem.

Another Option could be to check if your operation systems limits are quite low (for Linux see the “ulimit” command)

 

If you can provide use a running code snippet, we can also have a look on that, if you like.

 

Best regards

commented by (240 points)
There were 3915 files open.

The ulimit on the box is 'unlimited' and this is a pretty serious machine.

To explain a little of what I am doing is I am doing the equivalent of a 'seed' of the DB.  So I am running through as quickly as possible and pushing the data into the DB.

I am using the JCache implementation you have so using the 'get', 'putIfAbsent'.  So what is confusing to me is why so much file space being consumed when actually a lot of the data is already in the cache.  As the seed runs it is checking the cache to see if already present, otherwise call the putIfAbsent and from the final 25Mb you can tell it's not that much data that needs to be written.  I am assuming you are using the off heap space to do something.

Question1: If I set the default file size to 100Mb from the 8Mb while doing this, for example.  Once the seed process is complete if I stop the application, modify the size back to 8Mb will it re-size the existing files based on the new setting?  As I said after I ran the 1st phase of the seed I saw there was 8G of file space being consumed, but when it was finished only 25Mb was used.

Question2: If I adjust the gap % so it is lower than 75%, let's say to 50%, will this cause more files to be open or less?  Additionally after the seed and I stop start the app changing the % will the files be re-based.  So if it was 50% and changed it to 75%, House Keeping would clean the files up?
commented by (240 points)

FYI, ur documentation needs updating

The example using 'setFileEvaluator' should be 'setDataFileEvaluator'

EmbeddedStorageManager storage = EmbeddedStorage.Foundation(
	Storage.ConfigurationBuilder()
		.setFileEvaluator(Storage.DataFileEvaluator(1024*1024, 1024*1024*8, 0.75))
		.createConfiguration())
	.start();

Should actually be...

EmbeddedStorageManager storage = EmbeddedStorage.Foundation(
	Storage.ConfigurationBuilder()
		.setDataFileEvaluator(Storage.DataFileEvaluator(1024*1024, 1024*1024*8, 0.75))
		.createConfiguration())
	.start();

 

Notes: Every question must be a separate forum post. Headline: Formulate your question shortly and precisely. Thank you!
Powered by Question2Answer
...