Focal Point Banner
Community Center Education Summit Technical Support User Groups
Let's Get Social!

Facebook Twitter LinkedIn YouTube
Focal Point    Focal Point Forums  Hop To Forum Categories  iWay Software Product Forum on Focal Point    [SOLVED]Method to move data on daily basis using Data Migrator
Go
New
Search
Notify
Tools
Reply
  
[SOLVED]Method to move data on daily basis using Data Migrator
 Login/Join
 
Member
posted
We need to setup an iWay Data Migrator process to move data from several tables in Oracle DB to DataWarehouse on daily basis.

If a source table contains a Datetime column, we can base on the date time to identify and easily move these records.
However, when a source table doesn't contain any date time column, what would be a proper way to select the newly inserted/updated records?

One way to solve this is to build triggers on the source tables to keep track of what being insert/update. Then iWay will base on this info to move data. This will work but i don't think this is a good approach.

Could you recommend any alternatives? Thanks a lot.

This message has been edited. Last edited by: Tamra,


iWay Data Migrator v7704M
Windows, All Outputs
 
Posts: 23 | Registered: January 29, 2014Reply With QuoteReport This Post
Platinum Member
posted Hide Post
You'll have to double check the manual but I belive that Oracle supports Change Data Capture.

Then you can set up DM to only get the changed data.

If that doesn't work then one hack is to create hashes of the concatenated fields you are interested in and compare that against your warehouse - not exactly efficient but much better than comparing field to field.

When I've used this I've added the hash as a separate field in the warehouse to speed up comparison.

Cheers

Stu


WebFOCUS 8.0.09
 
Posts: 233 | Location: Melbourne, Australia | Registered: February 07, 2007Reply With QuoteReport This Post
Guru
posted Hide Post
Hi David,

My suggestion to handle your ETL to tables without Datetime column is:

- Access your DW table, create a HOLD with IDs or max PK ( If its is auto increment );
- Access your Oracle DB filtering IDs NE (HOLD) or PK GT (HOLD)

I hope it helps you. Good lucky!


WebFOCUS 8.1.05 / APP Studio
 
Posts: 272 | Location: Brazil | Registered: October 31, 2006Reply With QuoteReport This Post
Guru
posted Hide Post
There is an optional add-on to DataMigrator that we call Change Data Capture that reads from the ORACLE Archive Logs to identify rows that have changed (insert, update or delete). It's documented in the Data Migrator User's Guide, Chapter 12, "Change Data Capture."

Note that for an ORACLE source it's only required that full logging is enabled. It does not require what ORACLE calls Change Data Capture (which is being deprecated) or Golden Gate.

While CDC is available in Release 7704 it's been improved since that release was first posted. I'd recommend using the current production Release 7706.

This message has been edited. Last edited by: Clif,
 
Posts: 373 | Location: New York City | Registered: May 03, 2007Reply With QuoteReport This Post
Member
posted Hide Post
Thanks a lot for all of your suggestion. I really appreciate it.

I'm taking a look at using the Change Data Capture Method in iWay 7704 for now.

We are also in a process of upgrading to iWay 7706 this year.


iWay Data Migrator v7704M
Windows, All Outputs
 
Posts: 23 | Registered: January 29, 2014Reply With QuoteReport This Post
  Powered by Social Strata  
 

Focal Point    Focal Point Forums  Hop To Forum Categories  iWay Software Product Forum on Focal Point    [SOLVED]Method to move data on daily basis using Data Migrator

Copyright © 1996-2018 Information Builders, leaders in enterprise business intelligence.