Durability option: writing to stage storage - NoSQL Database

Hi all,
Please, can someone explain how does that durability option work?
Where and how can we set that option?
What are the aim and the benefit of that option?
Is that option available in the current version of Oracle NoSQL ?

user962305 wrote:
Hi all,
Please, can someone explain how does that durability option work?
Where and how can we set that option?
What are the aim and the benefit of that option?
Is that option available in the current version of Oracle NoSQL ?Please see http://docs.oracle.com/cd/NOSQL/html/javadoc/oracle/kv/KVStore.html for information on the API including when/where you can use the Durability option. If you have more questions after reading through that, I'll be happy to answer them.
In general, Durability is meant to allow the program to specify, on a per-operation basis, the level of durability for writes and updates.
Yes, it is available in all versions of Oracle NoSQL.
Charles Lamb
Edited by: Charles Lamb on Jan 6, 2012 9:02 AM 

Excuse-me Charles, I was talking of the two dimensions of durability I read here
http://www.nocoug.org/download/2011-11/Marie-Anne_Neimat_NoSQL_Database.pdf page 21
It is written there that we can choose if we want to commit to RAM or commit to Disk.
I can not see any information on that flexibility any where? How and when can we choose that? 

Both dimensions to durability are specified using the API Charlie pointed to. The Durability parameter describes both. Also see:

There are three elements of Durability: master sync policy, replica sync polcy, and replica ack policy. The first two specify the level of durability for a transaction's data on the master and replica, resp. The last part lets the user specify the number of replicas which must ack a transaction before it is ack'd to the user.
Sync policy (the first two elements of durability that I mentioned above) can be either no_sync (just write to the JVM memory), write_no_sync (just write to the OS but do not force to disk), or sync (force to disk). I believe this is what you were referring to. The javadoc that I (and Mark) referenced above has more details.
Charles Lamb 

I am not sure I have understood sync policy.
Documentation says:
Do not write or synchronously flush the log on transaction commit.
Write and synchronously flush the log on transaction commit.
Write but do not synchronously flush the log on transaction commit.
What do you mean by “flush the log”?
Where do you need to write apart form log file?
Are there a log file for current transactions and another file for committed transactions?
With write_no_sync, when do you decide to flush the log? When it is full?
With sync, if there is a crash why should we loose data if there are on disk even if there was no flush on the log?

There is only one log. NoSQL DB uses BDB Java Edition, which is an append-only storage system.
By "write" we mean issue a file system write. Such writes are not guaranteed to be durable, because the file system and hardware may buffer the data. An application (JVM) crash after a write will not cause data loss, but a system/OS crash may cause data loss.
By "synchronously flush" we mean issue a file system fsync. This pushes the data all the way to the storage device, so data loss will not occur even if there is a system/OS crash.

So in plain english you are saying:

Reading it again my previous post is wrong. It should be:
Probably like user962305, the "write" wording is confusing me. I expect "write" as the ultimate operation, somehow more glorified than "sync". 

Your previous post was correct so I'm going to repeat it:
NO_SYNC = no write, no fsync
WRITE_NO_SYNC = write, no fsync
SYNC = write and fsync
I think these names are historical and I can see how they are confusing.

greybird wrote:
Your previous post was correct so I'm going to repeat it:
NO_SYNC = no write, no fsync
WRITE_NO_SYNC = write, no fsync
SYNC = write and fsync
I think these names are historical and I can see how they are confusing.To be even more specific, write means that the Java program (BDB JE) issues a RandomAccessFile.write() call which issues a write(2) call to the underlying file system. Hence, the data goes from the JVM to the OS. This protects against JVM failure, but not system failure.
Charles Lamb


JCAPS 513 Semaphores

Hiya all,
Can someone give me hints on how to use semaphores within JCAPS? I never used semaphores in java before, but I know what they are from other languages.
I have a jcd which writes entries to a XML file and I have other jcd's that will read the same XML file, therfore I know I need to have a semaphore to make sure the XML file will stay sound and complete :)
There are numerous solutions to your problem, depending on how you want to implement it. If you use an eInsight BPEL process to synchronize the writing to and reading from the XML file, you can just make sure only one instance of the process can run simultaneously, thus making sure the read and write operations are separated.
If you want to use semaphores in Java, its a little more complicated. But generally, you can protect both a method, and a code segment in Java using the synchronized keyword like:
public synchronized void doIt() {
public void doIt() {
//do something
synchornized(this) {
//Synchronized code
read more here:
Edited by: maparicio on Feb 12, 2008 4:52 AM 
Your answer gave me the correct push, especially the link :)
- Remold 
Writing to a file in order to immediately (or concurrently) read from it in a single solution does not appear to be a correct design for an integration solution. To pass data, whether xml or not, between JCDs one could use a JMS queue. This would be much more efficient and would not require synchronisation.
The original post describes a solution (write to file foloowed by a read from the same file).
What is the problem that needs to be solved?
I am creating a JavaCAPS application (or better multiple applications) where all traffice is routed via queues. One of these jcd's is a sort of routing application which directs all incomming messages to different queues based on incomming message type. Other jcd's after this routing application are being used to change the message (from SAP IDOC to EDIFACT) and add information to the new message based on the initial sender. Also we need to do different things with during the changing of the message (like what to do when one of the incomming messages fails, should all messages be marked failed or just the failing message).
What we are trying to setup (and I am half way with this right now) is a single XML-file where the routing and decision information is stored. This XML-file can be dynamically changed when a specific message is recieved.
I have thought of using queues to ask and deliver the information, but I got stuck with the need to dynamically change the needed information.
Our customer only bought eGate, so no eInsight functionallities.
- Remold 
Hoi Remold,
Do you have any volumetrics ? The solution you're building has an implicit assumption you're running everything from within a single AppServer.
An often used way is to implement a distributed caching mechanism which allows for injection to broadcast changes, e.g. with JMS Request Reply using Durable Subscribers to Topics.
JavaSpaces are another intriguing option.
For the rest... looks a bit like a scary solution given the description of what is needed such as grouped failure mechanisms and such..
Groetjes and take care

jms appropriate for this? + related questions

so we currently have 2 separate systems.
On the external system usesr can enter info into a db.
On the internal site, a process runs every 15 minutes to pull all new entries on the external site db into the internal site db.
I am about to suggest that jms be used to replace that process, because
1. it seems appropriate
2. the current process is buggy.
the client is extremely worried about security. Are there any known jms vulnerabilities that can be exploited if a firewall port is left open for JMS? (of course, a port is currently left open for the existing process, but I have to ask...)
from my understanding of jms, if an insert fails on the internal side, we can shunt the messages off to an error queue, and use jmx to try and reinsert. Is that correct?
Is there any way to stop an MDB from reading more messages off the Q if there is an error?
Message was edited by:
Are there any known jms vulnerabilities that can be exploited if a firewall port is left open for JMSYou'll have to check with the JMS vendor you decide upon. Industrial strength implementations like MQ allow you to encrypt traffic between the two systems using digital certificates, which makes things about as secure as you're likely to get.
if an insert fails on the internal side, we can shunt the messages off to an error queue, and use jmx to try and reinsert. Is that correct?JMS is just a specification. You have to decide how you want to handle errors then write your code accordingly. Some general principles that may help...
Treat business errors and system errors differently. A business is one where the data is invalid, e.g. field too long for a database column. It doesn't matter how many times you attempt to reinsert the message it will always fail. The only way to handle this error is to fix the message or fix the data in the database.
Technical errors are things like network outages which can be recovered from. In this case it may be a good idea to keep retrying the message, but also raise some kind of alert.
Error queues can be a great idea in some circumstances, but you need to consider whether message order is important. What happens if update 1 gets moved to an error queue and update 2 is inserted before update 1 was applied. Is your data still valid?
You also need to think about transactionality. Unless the entire transaction (from remote db to local db) occurs in a distributed transaction you will always run the risk of lost or duplicate messages. Will duplicate creates be rejected? will duplicate (and out of sequence) updates cause problems? Can you afford to lose the occaisional message?
These problems aren't unique to messaging and if they haven't been thought about could explain why your current system is buggy.
Is there any way to stop an MDB from reading more messages off the Q if there is an error? Don't know about this one (maybe with JMX). You can certainly do the equivalent if you write your one JMS Listeners (just stop the connection), or if you use a middleware product such as IBM Message Broker (WBIMB).
Last thing to think about is that while JMS is a reasonable solution for this type of problem there must be data synchronisation tools out there to do this type of thing for you. 
Thanx for your response. Your advice about handling errors in business logic one way and technical errors differently is good.
Message order is important, to a degree. We will be passing two types of objects over the queue, call them A and B. Relationally, every instance of B depends on an instance of A, so every B for an A needs that A inserted before it can be inserted. So some Bs can be inserted if there is a problem with a particular A, if those Bs don't depend on the problem A.
I realize we can use 2 Qs, one for As and one for Bs, but I think one Q that handles both is better, becuase order is handled for us. The external app cant have Bs created without an A already existing, so when a new A gets created, it is automatically on the Q before any of its Bs.
Errors get interesting, though; if an A fails to insert, all its Bs will also fail to insert. I was going to suggest an error Q that all fails get shunted to an error Q. This should happen in order, so clearing the error Q (plan on using JMX) should allow all fails to get inserted in the correct order. Keeping in mind your advice, the error Q is only to be used for technical problems (network down, app server crash, whateveR). Since the 2 DBs are copies of each other, anything that gets inserted in the external should not fail to insert to the internal, except because of technical issue.
Which raises an interesting point. How to enqueue on the external side? I was orignally planning on doing it thusly -- we divide all our services into operations, so you can imagine, on the external side, having an insertAOperation and then an enqueueAOperation. Using spring, transactionalize each service method call, so that following happens:
If there is an error inserting, the enqueue operation never happens.
If there is an error enqueueing, roll back the transaction so the insert never happened.
Does that sound reasonable? The user should see a nice error message.
The other option would be to create a scheduled bean that calls a service method that reads all new messages since its last run, then enqueues them. I don't really like this option -- figuring this out is a pain, but it wont stop a user from inserting if there is a Q problem....

Concurrency between jcds in JCAPS

It will be helpful if some one can clear my following doubt. We have a scenario where four or five jcds attempt to
read a common file at same time. As all jcds tries to read the common file at same time is there any possibility
for having concurrency problem between jcds.
We are developing a project in JCAPS 512. Our project design is as follows
1) A jcd will get triggered when it receive message to queue from source system. After receiving the message
this particular jcd will send an individual jms msg to approximately 12 queues where each queue is listened
by one individual jcd.
2) The JMS msg content will have the file location and file name to read.
3) As soon as jcds receive JMS message from respective queue it will start reading the file from location.
4) Now in this case all the jcds will attempt to read the same file at same time.
Please let us know whether above scneario will open up concurrency issue in JCAPS. If there can be a
concurrency issue how to avoid the issue.
That's an interesting design. If all the JCDs that do the 'getting' (are they using FTP?) and don't have a post-transfer of rename or delete, the file will stay there.
How will you clean up the messages? How would you know when all of the JCDs have picked up the message?
It seems that an easier way to do this will be to have one JCD pick up the file, and put it into a topic that all the other JCDs subscribe to. This way cleanup is easy (just set post-transfer to delete) and FTP traffic is greatly reduced (1 jcd getting the file instead of multiple JCDs).
If you're worried about picking up the files too slowly, you can still use the same concept as before, where you are sending a message containing where the file is, and what file to pick up. The messsages go to a queue, where one 'get' JCD is connected to it. If the messages start to pile up on the queue, the application server will load another instance of your get component. Just be careful about FTP connection pools (you could run out if the application server spawns too many instances).
Hope any of that helps! :)
hi *,
i would fully agree to adam if the payload does not exeed a certain threshhold. (in our case would be ~ 30 MB)
if it does your way would be ok too because it allows you SIMPLE access to the file without having to worry about e.g. streaming.
if it goes beyond your heap limitations be careful that the jcd does not load all payload at a time into heap 12*30MEGS = at least 360 megs......
but the questions about cleaning etc remain...
regards chris 
Hi All,
Thanks a lot for all the reply. Basically we don't use FTP for reading the file, files are in local system where JCAPS
logicalhsot runs. As per our plan every month once we are planning to clean up the file by unix script.
As per our design we are planning to read file by invoking a java class which uses InputStreamReader. We are
not going to use BatchLocalFile. We don't have any specific reason for doing this other than adheringThere is no
specific reason in doing this.
Our main concern is whether there will be any concurrency issue when several jcds attempt to read the file at
same time[Basically the jcd which populates the message to respective destination queues will populate one by
one to all queues. First to QueueA, then QueueB and then QueueC etc.]. After receiving messages from queue
each jcd will try to read the same file which will be available in common location.
Whether above design will open up any concurrency issue. Please advise.
As long as you are going to do only a read operation on the file, i dont think you need to worry about concurrency issue. Are you going to perform write operations also?

Consumer/Multi Master writing

We would like to increase the speed of out architecure and have 2 consumers and 2 multimaster. Some of our aps write and some just read. My question is will the consumers still be getting written to? Doesn't it write timestamps and login counts etc? Hoe much copacity does this take up? Isn't that defeating the purpose of having just a readable comsumer? 
A replication consumer is a server that cannot be directly written to by a client, only by a replication supplier.
It does not store timestamps and login counts to disk, it only keeps them in memory. So if the server goes down, that information is lost. It doesn't take up much capacity.
What should a read-only consumer do in this case?

Event based listener

Hi Experts,
I have a requirement to develop a listener for application B, which have to receive xml input from another application (application A).
As soon as the xml file is received, the data has to be updated in the database of application B.
This has to be near real time.
Any ideas on how to achieve this.
Obviously you will have to build something observable into B and have it observe itself, but the question is about as vague as it could possibly be. For example:
As soon as the xml file is receivedReceived how? 
Say, it is placed in a shared folder for both applications. 
So it's clear. Read about Observer pattern as EJP suggested and implement it, that's all. 
Hi Opal,
Thanks for your reply. I will go through that.
I need to suggest how to receive the msg from the application. I thought of asking for a xml to be placed in some shared folder.
Is there any better approach than this way.
vicky wrote:
Hi Opal,
Thanks for your reply. I will go through that.
I need to suggest how to receive the msg from the application. I thought of asking for a xml to be placed in some shared folder.
Is there any better approach than this way.there are many better approaches than this way. but, you continue to be very vague about your requirements/restrictions, so it's hard to give more detailed suggestions. at the very highest level, there are a couple of classes of communications channels:
- communicate asynchronously
- read/write a shared resource (e.g. your current suggestion, one app writes to disk, the other reads)
- use some sort of queueing package
- communicate synchronously
- use some form of network call (e.g. sockets or some higher level abstraction)
shared disk is probably the slowest and most error prone communication mechanism. 
Yeah true, I agree, because of the reason I am not from development team.
I am tweaking an oracle product to somehow, get the data from the connected application.
But, with the discussions I had so far with the application administrator, I need to receive the messages asynchronously, and using the messages have to tweak the oracle default functionalities.
I am successful in tweaking out the default functionalites of oracle. But, stuck up at this point of receiving messages.
Both applications are present on different servers, using diff databases.
So, in this scenario, which method to pickup.
Also, let me know, what are the other things to be taken care of from the custom application side, so that the requirements will be more clear and I can pick up the right way of receiving messages.
Edited by: vicky on Feb 10, 2011 6:47 PM