0 votes
asked in MicroStream for Java by (240 points)
This is occurring when the application starts up.

1) What would cause this issue?

2) Is there anyway to recover the DB from this?

 

Caused by: one.microstream.storage.exceptions.StorageException: Problem in channel 0

    at one.microstream.storage.types.StorageChannelTask$Abstract.checkForProblems(StorageChannelTask.java:65)

    at one.microstream.storage.types.StorageChannelTask$Abstract.waitOnCompletion(StorageChannelTask.java:127)

    at one.microstream.storage.types.StorageSystem$Default.startThreads(StorageSystem.java:285)

    at one.microstream.storage.types.StorageSystem$Default.internalStartUp(StorageSystem.java:454)

    at one.microstream.storage.types.StorageSystem$Default.start(StorageSystem.java:524)

    at one.microstream.storage.types.StorageSystem$Default.start(StorageSystem.java:1)

    at one.microstream.storage.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:208)

    at one.microstream.storage.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:1)

    at net.atpco.metadata.microstream.configuration.MicrostreamCacheConfiguration.microStreamStorageManager(MicrostreamCacheConfiguration.java:77)

    at net.atpco.metadata.microstream.configuration.MicrostreamCacheConfiguration$$EnhancerBySpringCGLIB$$c0bfca08.CGLIB$microStreamStorageManager$2(<generated>)

    at net.atpco.metadata.microstream.configuration.MicrostreamCacheConfiguration$$EnhancerBySpringCGLIB$$c0bfca08$$FastClassBySpringCGLIB$$35e2c403.invoke(<generated>)

    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)

    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)

    at net.atpco.metadata.microstream.configuration.MicrostreamCacheConfiguration$$EnhancerBySpringCGLIB$$c0bfca08.microStreamStorageManager(<generated>)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.base/java.lang.reflect.Method.invoke(Method.java:566)

    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)

    ... 105 common frames omitted

Caused by: one.microstream.storage.exceptions.StorageException: Inconsistent store timestamps between channels

    at one.microstream.storage.types.StorageChannelTaskInitialize$Default.fallbackDetermineConsistentStoreTimestamp(StorageChannelTaskInitialize.java:151)

    at one.microstream.storage.types.StorageChannelTaskInitialize$Default.determineConsistentStoreTimestamp(StorageChannelTaskInitialize.java:136)

    at one.microstream.storage.types.StorageChannelTaskInitialize$Default.getConsistentStoreTimestamp(StorageChannelTaskInitialize.java:58)

    at one.microstream.storage.types.StorageChannelTaskInitialize$Default.succeed(StorageChannelTaskInitialize.java:188)

    at one.microstream.storage.types.StorageChannelTaskInitialize$Default.succeed(StorageChannelTaskInitialize.java:1)

    at one.microstream.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.synchronizedComplete(StorageChannelSynchronizingTask.java:61)

    at one.microstream.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.complete(StorageChannelSynchronizingTask.java:109)

    at one.microstream.storage.types.StorageChannelTask$Abstract.processBy(StorageChannelTask.java:217)

    at one.microstream.storage.types.StorageChannel$Default.work(StorageChannel.java:281)

    at one.microstream.storage.types.StorageChannel$Default.run(StorageChannel.java:351)

    at java.base/java.lang.Thread.run(Thread.java:834)

1 Answer

0 votes
answered by (9.7k points)
Hello,

Unfortunately, I can’t say very much about that exception

Did you encounter any errors before you restarted the storage?
I fear that I can’t help you very much without a reproduceable code sample.

I will forward your questions to the core developers maybe they have some ideas on that.
 

Best regards.
commented by (240 points)
Harald,

1) Pretty difficult to reproduce as I'm not sure what is making this happen and at this time little insights into when it occurs.  This is like a batch/seed process and it can take hours before the problem manifests itself

2) The data file max is configured to 200Mb and the entityCacheThreshhold is at 4Gb.

3) When this occurred the file storage has consumed 368Gb across all channels and there are 929 files open in channel_0 and 183Gb file storage.  Channel_2 also had 942 files with 185Gb file storage

I can only assume it has too many files open and encounters a problem.  This occurred once before and I up'd the file.max size, and I can do this again to see if it resolved the problem.  My main concern is the amount of file storage consumed when in reality it should not be anywhere that size.

I am not sure I am throwing too much data at microstream and it cannot keep up and whether I should slow down the process as I am assume it does housekeeping and clean-up, but if I constantly throw data at it, can it maintain the housekeeping side?
commented by (9.7k points)

Hello,

The housekeeping has a limited time slice. Under heavy load it may be happen that the storage grows faster then it can be cleaned up. You may have a look at
https://manual.docs.microstream.one/data-store/configuration/housekeeping
and
https://manual.docs.microstream.one/data-store/housekeeping
for more details and how to configure it.

 

Regarding the large storage size:

Do you store all those data in one large Collection?
If so, maybe you can split it in smaller ones or even in some kind of a tree.
This way you can gain more control of the amount of written data.
 

Storing collections again and again can create quite a lot of overhead because every time a collection is stored the references to all contained elements need to be written too (the referenced objects are not written again).

 

Best regards

commented by (100 points)

Right now I'm performing some tests with Microstream to check whether it is suitable for our needs.
I'm using Version 4 (base-04.00.00-MS-GA.jar ...)

I was trying a small example on how to change existing DB objects, when I came across the same error.
What I did (roughly):
- Created a small DB with 40 objects (in 2 classes) stored in 8 channels.
- added a variable to one of the two classes
- filled the new variable in some of the objects
- when I try to read the DB again, the error happens.
- each time I restart the application, the error mentions a different channel. ("Problem in channel 0")

I also can't reproduce the error. When I do the same all over again it works.
 

Situation now is, that I have a DB which can't be started.
When I try to start it, the error appears.
I tried to call "storageManager.issueFullFileCheck()" but that fails, because that can only be done on a started DB.

Is there a strategy to solve this problem? Right now it seems like a worst case situation, where I can't access the data any more.

 

C:\...\ibmjdk-8-20160719_312156-LEAP-1\bin\java.exe ... (started from IntelliJ 2019.3.3 Ultimate)
Exception in thread "main" one.microstream.storage.exceptions.StorageException: Problem in channel 0
	at one.microstream.storage.types.StorageChannelTask$Abstract.checkForProblems(StorageChannelTask.java:65)
	at one.microstream.storage.types.StorageChannelTask$Abstract.waitOnCompletion(StorageChannelTask.java:127)
	at one.microstream.storage.types.StorageSystem$Default.startThreads(StorageSystem.java:295)
	at one.microstream.storage.types.StorageSystem$Default.internalStartUp(StorageSystem.java:473)
	at one.microstream.storage.types.StorageSystem$Default.start(StorageSystem.java:550)
	at one.microstream.storage.types.StorageSystem$Default.start(StorageSystem.java:1)
	at one.microstream.storage.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:208)
	at one.microstream.storage.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:1)
	at strukturaenderung.DbStrukturaendernAddColumn.erzeugeStorage(DbStrukturaendernAddColumn.java:125)
	at strukturaenderung.DbStrukturaendernAddColumn.go(DbStrukturaendernAddColumn.java:29)
	at strukturaenderung.DbStrukturaendernAddColumn.main(DbStrukturaendernAddColumn.java:23)
Caused by: one.microstream.storage.exceptions.StorageException: Inconsistent store timestamps between channels
	at one.microstream.storage.types.StorageChannelTaskInitialize$Default.fallbackDetermineConsistentStoreTimestamp(StorageChannelTaskInitialize.java:151)
	at one.microstream.storage.types.StorageChannelTaskInitialize$Default.determineConsistentStoreTimestamp(StorageChannelTaskInitialize.java:136)
	at one.microstream.storage.types.StorageChannelTaskInitialize$Default.getConsistentStoreTimestamp(StorageChannelTaskInitialize.java:58)
	at one.microstream.storage.types.StorageChannelTaskInitialize$Default.succeed(StorageChannelTaskInitialize.java:188)
	at one.microstream.storage.types.StorageChannelTaskInitialize$Default.succeed(StorageChannelTaskInitialize.java:1)
	at one.microstream.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.synchronizedComplete(StorageChannelSynchronizingTask.java:61)
	at one.microstream.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.complete(StorageChannelSynchronizingTask.java:109)
	at one.microstream.storage.types.StorageChannelTask$Abstract.processBy(StorageChannelTask.java:217)
	at one.microstream.storage.types.StorageChannel$Default.work(StorageChannel.java:388)
	at one.microstream.storage.types.StorageChannel$Default.run(StorageChannel.java:475)
	at java.lang.Thread.run(Thread.java:785)

Process finished with exit code 1

 

commented by (9.7k points)
Unfortunatelly we still cannot reproduce this error.

Is there any chance that you could provide us your code and the corrupted Storage?
commented by (9.7k points)
An additional Question:

Did you change the channel count?
E.g. created the initial storage with default settings and then tried to load it again with a differently configured channel count?
commented by (100 points)
No, the number of channels were not changed. It was the whole time set to 8 channels.
commented by (100 points)
If that helps, I can provide a the corrupt DB as a zip file, for analysis. Via PM or some other direct way.

Some source code which I used to create the DB can be provided as well. But I can also not reproduce the error right now with the source code.
commented by (9.7k points)
Hello,

Many thanks for your offer to supply the affected storage and the related code to us.

As the Forum does not support exchanging private messages or uploads, we could do that via Github if you like.

Otherwise I could contact you using the email address you used to register at the forum if you agree.

 

Best regards
commented by (100 points)
Yes, please use my email so I can send it to you directly.
Notes: Every question must be a separate forum post. Headline: Formulate your question shortly and precisely. Thank you!
Powered by Question2Answer
...