MAP/REDUCE and Oracle NoSQL - NoSQL Database

Hi all,
I would like to know if there are some examples on how to run run map/reduce with Oracle NoSQL.
Is there any source code any where? Can you send me one example?
Where can we download all the necessary tools?
In Oracle Big Data Appliance is map/reduce used with Oracle NoSQL or with Hadoop?
Thanks 

user962305 wrote:
I would like to know if there are some examples on how to run run map/reduce with Oracle NoSQL.
Is there any source code any where? Can you send me one example?Take a look in the oracle.kv.hadoop.KVInputFormat javadoc. It discusses how to use Oracle NoSQL Database with Hadoop as well as referring to an example which is included in the distribution.
Where can we download all the necessary tools?
In Oracle Big Data Appliance is map/reduce used with Oracle NoSQL or with Hadoop?It would be used with both, whether or not you were on the BDA. You use the KVInputFormat to read data from Oracle NoSQL Database into Hadoop during map/reduce processing.
I hope this is useful.
Charles Lamb 

Hi Charles,
Can you please, explain where and what to download and install for this case?
Should we also install hadoop on the same replication nodes as Oracle NoSQL?
Is it possible to have an example with pre-loaded keys on Oracle NoSQL to perform the test?
Is there a version of Oracle NoSQL which comes with some key/value pairs?
I understand the following. Data in Oracle NoSQL will be loaded in hadoop first and then map/reduce is performed in haddop. Is it right?
I would like to know: Does that mean Oracle NoSQL can not run parallel operations? What is the aim in loading data to hadoop first if Oracle is able do perform parallel operations? Loading data from Oracle NoSQL to hadoop may take enormous time I suppose.
Thanks 

user962305 wrote:
Can you please, explain where and what to download and install for this case?Download [Oracle NoSQL Database from OTN|http://www.oracle.com/technetwork/database/nosqldb/downloads/index.html] .
Should we also install hadoop on the same replication nodes as Oracle NoSQL?It depends on your access patterns. In general, probably not, but there may be cases where you achieve better performance with Hadoop and the Rep Nodes co-located.
Is it possible to have an example with pre-loaded keys on Oracle NoSQL to perform the test?
Is there a version of Oracle NoSQL which comes with some key/value pairs?Look at the quickstart guide that comes with the above Oracle NoSQL Database package. There is a small HelloWorld example which you can use as the basis for creating a data set.
I understand the following. Data in Oracle NoSQL will be loaded in hadoop first and then map/reduce is performed in haddop. Is it right?Hadoop is a framework, which among other things happens to run Map/Reduce jobs. Your Map/Reduce job would use the KVInputFormat to read data from Oracle NoSQL Database and process it however it sees fit. It might write the output of the M/R to (say) HDFS. Or it might write it to (say) Oracle RDBMS. Or it might write it back to (say) Oracle NoSQL Database.
>
I would like to know: Does that mean Oracle NoSQL can not run parallel operations? What is the aim in loading data to hadoop first if Oracle is able do perform parallel operations? Loading data from Oracle NoSQL to hadoop may take enormous time I suppose.I am not sure I understand your question. Hadoop, by its nature will break a job into many subtasks. Those subtasks run in parallel, generally across many Hadoop nodes. Those subtasks may access Oracle NoSQL Database data. Hence, Oracle NoSQL Database is able to perform operations in parallel either on the same or different Rep Nodes.
Charles Lamb 

Thanks Charles.
For hadoop, where and which version should we used for Oracle NoSQL ? 

I think 0.2.20 is the current, no?
Charles Lamb

Related

Goldengate Java Adapter 12c

We currently use Goldengate for replication from source database to target database. Current Environment SourceOracle Database 12cOracle Goldengate 12cTargetOracle Database 12cOracle Goldengate 12c For some new set of tables we need to do the replication to a NoSQL database. I saw multiple blogs about this, but none of them was complete. Requesting if anyone who has done this, to share the step by step details on this. Basically I want to understand the part from source to java adapter with details. Any help would be appreciated. Thanks in advance.
Replication to nosql is the same as replication to any big data systems, so you need to install OGG for Java adapter, see Oracle GoldenGate Downloads I see that you describe your source and target. Note that you need a separate extract for J adapter from either database. Ok as with all Java adapter OGG expects a connection that will except XML , that is DMLs as encapsulate in XML code. You can configure OGG to write to a flat file to start and look at it. You can actually take this flat file and feed into your xml connection. So the next thing you need to configure outside of OGG is your xml connection. That said this is all but one way. You did not say what nosql (there are 100s). Are you talking about Oracle NoSql? If so there are other ways to get your data, you can do Oracle logs to Oracle nosql using one of the big data systems like Flume. Anyway as a start I will install the Java Adapter and write this out to a flat file. You can see what you are getting and then grab someone (or maybe it is you) who knows about your nosql database and ask a simple question, how do I get these xmls (from the flat file) into nosql. CheersKee Gan
Thanks Gan for the reply. We will need to use Oracle Goldengate itself for the replication and the target system is Oracle NoSQL. I am familiar with the part from Java to NoSQL, what I am looking for is step-by-step instructions till the Java code. What are the different configuration required, my source and target are in same system, so should I configure an extract, pump alone? or another extract is required? Looking for all such details or a pointer to a documentation is also fine.
The administration guide will provide the steps you need. All you need is the Java adapter extract. Start with writing to a flat file first as you will then not have to deal with comms issue. When this works then configure the extract to hook it up to nosql. See this document, it is a little old but totally applicable. http://www.ateam-oracle.com/oracle-goldengate-transactional-data-delivery-to-weblogic-java-messaging-service/  CheersKee Gan
Thanks Gan!

Oracle Golden Gate Synchronization

Want to use oracle golden gate for unidirectinoal synchronization between source database (11g running on AIX) and Target Database (12c running on Linux). Is this possible? Please advise on how it can be achieved.
Any advise and suggestions would be greatly appreciated!
Yes this is absolutely possible. It is also easy, just install the OGG version for the platform and DB type/version and capture from one end. Install OGG for 11g running on AIX which will create OGG trails which you can shipped using OGG extract pump. Then install OGG for Database (12c running on Linux) and replicat the shipped trails for your AIX onto your 12c DB. OGG is a decoupled system, ie you capture and create the OGG trails that you can use on almost all OGG supported platforms and DB. The same trails you created for AIX can be used on a Windows SQL 2016 for example. Go to Oracle GoldenGate Downloads and search for version you need. Obviously you need some education on OGG which you can google as there are many free guides and videos. This one by Oracle is a good starthttp://www.ateam-oracle.com/oracle-goldengate-tutorial-for-oracle-to-oracle-version-11-2/ There are some later ones like this http://www.oracle.com/technetwork/database/multitenant/learn-more/ogg-multitenant-2408073.pdf but I think it might be a little overwhelming if you have zero OGG. CheersKee Gan
Hi , Yes this is possible. Oracle GoldenGate is very easy to install. For each OS platform, you will be having different OGG version to download and install. But, before going in directly in to the installation and configuration, you need to check the pre-requisites and system requirements.  Please refer the below link, https://docs.oracle.com/goldengate/1212/gg-winux/GIORA.pdf  You need to check what all the objects you have and data types.  https://docs.oracle.com/goldengate/1212/gg-winux/GIORA/system_requirements.htm#GIORA110  https://docs.oracle.com/goldengate/c1221/gg-winux/GIORA/GUID-774E1D82-DCD4-4600-AD70-E613734550E0.htm#GIORA146  What type of Extract and Replicat you are going to use?  https://docs.oracle.com/goldengate/c1230/gg-winux/GGODB/choosing-capture-and-apply-modes.htm  Whether all the tables which you are replicating has the primary key or unique index? This is one of the major requirement/pre-requisite for OGG. http://www.ateam-oracle.com/oracle-goldengate-tables-without-keys/  As I mentioned OGG is easy to install, but to configure it according to you requirement needs some work to be done. You need to know more than the basic knowledge of OGG. Hope the above documents helps!!. Regards,Veera
Thanks for your quick response. So synchrinozation of cross oracle versions on cross OS platform is possible. Is this a perfect tool for real-time synchronization? Also is it possible to monitor and resolve conflicts during synchronization using this tool?
Hi , Yes, this is a perfect replication tool for the real-time synchronization. This is a Heterogeneous replication tool and hence independent of database and OS platform versions. Resolving the conflicts mainly used in Bi-Directional replication. In a uni-directional replication, if you perform the Initial Load perfectly, then there should be no data integrity issues.  We have data comparison tool call Oracle GoldenGate Veridata which is used to compare the data between the source and target. Generate reports and you can also synchronize the data using the same tool. Please do through the below link, https://www.oracle.com/middleware/data-integration/goldengate/veridata/index.html  Regards,Veera
Thank you Veera.
Nothing is perfect from Oracle.... but yep it works. Th rest is as explained by Veera. CheersKee Gan 
Thanks Kee Gan & Veera, Is there any diagramatic view available, which could explain how Golden Gate Synchronization (real-time) works?
There are many videos and documentation out there, but if you want something from me I have a video athttps://www.youtube.com/watch?v=w_tBLSCKrL0&t=138s The aim of my video is to make sure there is no marketing/sales and also concepts that are generally not explained by Oracle. It will also have a step by step explanation with animation on a simple straight uni-directional setup.CheersKee Gan
Hi , You can also refer my blog below where in I write all the articles related to Oracle GoldenGate from scratch (for example What is OGG?). www.oracle-scn.com Regards,Veera
Hi,  Can I use veridata to perform data comparison of unidirectional sync (with GG)? My target database will be accessesd directly by multiple set of users. However I also want whatever delta is being updated in source database by different set of users should get sync with target database at the system. Can we use veridata only for comparing the delta that is coming from source is in sync with target, while the target is also getting updated directly by few set of users?
You can use Veridata to compare any sort of replication, uni or bidirectional. If your target is also being updated then you can still compare (Veridata or otherwise is irrelevant) but you just have to understand if there are differences then it might be caused by target updates. You can also compare specific columns and don't have to compare everything. CheersKee Gan
Hi , Veridata is nothing but Verification of Data. Yes, you can compare the data between the source and target tables. If you see any data discrepancy, you can use the REPAIR option where your data will be synchronized automatically by the Veridata. Oracle GoldenGate Veridata is a high-speed data-comparison and repair solution that identifies, reports on, and fixes data discrepancies between heterogeneous databases without interrupting ongoing business processes http://www.oracle.com/us/products/middleware/059493.pdf   https://docs.oracle.com/goldengate/v1221/gg-veridata/GVDAD/about_ogg_veridata.htm#GVDAD114  Regards,Veera

Processing oracle relational data with hadoop

hello every body:
i would like if it possible to process oracle relational data with hadoop in order to get better performance
native parallel processing in oracle VS hadoop processing? who is the better?
how can we do it?
thank you at advance 
Hello,
This is the Oracle NoSQL Database forum. You asked about Oracle Database so you may want to try that forum instead of this one. I suggest that you ask about "In-Database Map/Reduce".
Charles

What are the differences between Oracle and other NoSQL database

Hi all,
I would like to know what the differences between Oracle and other NoSQL database are.
When and why should we use Oracle?
Is Oracle NoSQL database link with Big Data Appliance?
Can we use map-reduce on a single personal computer? How should we install Oracle NoSQL database to use map reduce on a single personal computer?
Do we also have eventual consistency with Oracle NoSQL database? Can we lose data if master node fails?
Are transactions ACID with Oracle NoSQL database? How can we prove it?
Thanks. 
893771 wrote:
Hi all,
I would like to know what the differences between Oracle and other NoSQL database are.
When and why should we use Oracle?I suggest that you start here:
http://www.oracle.com/technetwork/database/nosqldb/overview/index.html
Is Oracle NoSQL database link with Big Data Appliance?Yes, Oracle NoSQL Database will be a component of the Big Data Appliance.
Can we use map-reduce on a single personal computer? How should we install Oracle NoSQL database to use map reduce on a single personal computer?Yes, I believe you can run M/R on a single computer. Consult the various pieces of documentation available on the web. You may run Oracle NoSQL Database on the same computer that you are running M/R on, but it is likely that they will compete for CPU and IO resources and therefore performance may suffer.
Do we also have eventual consistency with Oracle NoSQL database? Yes.
Can we lose data if master node fails?If you run Oracle NoSQL Database with the default (recommended) durability settings, then if the master fails, a new one will be elected and data is not lost.
Are transactions ACID with Oracle NoSQL database? How can we prove it?Yes, each operation is executed in an ACID transaction. The API has the concept of "multi" operations which allow the caller to perform multiple operations on sets of records with the same major key, but different minor keys. Those operations are also performed within a transaction.
Charles Lamb

GoldenGate and JavaDB (Derby db)

Hello,
I would like to know if GoldenGate supports replication from Oracle dbs to Javadbs (old named was Derby). In particular is the version of Javadb 10 10.6.2.1 provided with Glassfish 3.1.1 supported and with which version of Golden gate?
Extra question: is it possible to set the replication one way instead of both way? I would need data to get replicated only one way from central Oracle database to a multiitude of javaDB databases.
Any documentation on this?
Other alternative than Godengate?
Thanks 
user553981 wrote:
Hello, I would like to know if GoldenGate supports replication from Oracle dbs to Javadbs (old named was Derby). Java DB is not supported as a source nor a target for GoldenGate. It's possible that ODI (Oracle Data Integrator) might be able to do what you're looking for.
Extra question: is it possible to set the replication one way instead of both way? I would need data
to get replicated only one way from central Oracle database to a multiitude of javaDB databases.GoldenGate by default is one-way replication; two-way replication is just a matter of setting up the same configuration in the other direction. You can set up data-capture from Oracle DB, and possibly (not trivially) use the Java API to take these change-events and apply them via your own JDBC implementation to Java DB. (Again, that's a bit of a project. This is not out-of-the-box functionality.)
Any documentation on this?The OGG Java API (aka the Java user-exit) is =>
* http://docs.oracle.com/cd/E18101_01/index.htm
* http://www.oracle.com/technetwork/middleware/goldengate/documentation/index.html
More likely you'll want to look at ODI =>
* http://www.oracle.com/technetwork/middleware/data-integrator/overview/index.html
* Data Integrator
I believe you can also use GG via ODI to capture changes from Oracle, and then use ODI to deliver the change events via JDBC to Java DB... but I'm not positive about that. (You might want to check in the ODI forum.)
Cheers,
-Mike

Categories

Resources