Focal Point Banner


As of December 1, 2020, Focal Point is retired and repurposed as a reference repository. We value the wealth of knowledge that's been shared here over the years. You'll continue to have access to this treasure trove of knowledge, for search purposes only. Moving forward, myibi is our community platform to learn, share, and collaborate. We have the same Focal Point forum categories in myibi, so you can continue to have all new conversations there. If you need access to myibi, contact us at myibi@ibi.com and provide your corporate email address, company, and name.


Connect to myibi
Focal Point    Focal Point Forums  Hop To Forum Categories  iWay Software Product Forum on Focal Point    [SOLVED]Method to move data on daily basis using Data Migrator

Read-Only Read-Only Topic
Go
Search
Notify
Tools
[SOLVED]Method to move data on daily basis using Data Migrator
 Login/Join
 
Member
posted
We need to setup an iWay Data Migrator process to move data from several tables in Oracle DB to DataWarehouse on daily basis.

If a source table contains a Datetime column, we can base on the date time to identify and easily move these records.
However, when a source table doesn't contain any date time column, what would be a proper way to select the newly inserted/updated records?

One way to solve this is to build triggers on the source tables to keep track of what being insert/update. Then iWay will base on this info to move data. This will work but i don't think this is a good approach.

Could you recommend any alternatives? Thanks a lot.

This message has been edited. Last edited by: Tamra,


iWay Data Migrator v7704M
Windows, All Outputs
 
Posts: 23 | Registered: January 29, 2014Report This Post
Guru
posted Hide Post
You'll have to double check the manual but I belive that Oracle supports Change Data Capture.

Then you can set up DM to only get the changed data.

If that doesn't work then one hack is to create hashes of the concatenated fields you are interested in and compare that against your warehouse - not exactly efficient but much better than comparing field to field.

When I've used this I've added the hash as a separate field in the warehouse to speed up comparison.

Cheers

Stu


WebFOCUS 8.2.03 (8.2.06 in testing)
 
Posts: 253 | Location: Melbourne, Australia | Registered: February 07, 2007Report This Post
Guru
posted Hide Post
Hi David,

My suggestion to handle your ETL to tables without Datetime column is:

- Access your DW table, create a HOLD with IDs or max PK ( If its is auto increment );
- Access your Oracle DB filtering IDs NE (HOLD) or PK GT (HOLD)

I hope it helps you. Good lucky!


WebFOCUS 8.1.05 / APP Studio
 
Posts: 272 | Location: Brazil | Registered: October 31, 2006Report This Post
Guru
posted Hide Post
There is an optional add-on to DataMigrator that we call Change Data Capture that reads from the ORACLE Archive Logs to identify rows that have changed (insert, update or delete). It's documented in the Data Migrator User's Guide, Chapter 12, "Change Data Capture."

Note that for an ORACLE source it's only required that full logging is enabled. It does not require what ORACLE calls Change Data Capture (which is being deprecated) or Golden Gate.

While CDC is available in Release 7704 it's been improved since that release was first posted. I'd recommend using the current production Release 7706.

This message has been edited. Last edited by: Clif,


N/A
 
Posts: 397 | Location: New York City | Registered: May 03, 2007Report This Post
Member
posted Hide Post
Thanks a lot for all of your suggestion. I really appreciate it.

I'm taking a look at using the Change Data Capture Method in iWay 7704 for now.

We are also in a process of upgrading to iWay 7706 this year.


iWay Data Migrator v7704M
Windows, All Outputs
 
Posts: 23 | Registered: January 29, 2014Report This Post
  Powered by Social Strata  

Read-Only Read-Only Topic

Focal Point    Focal Point Forums  Hop To Forum Categories  iWay Software Product Forum on Focal Point    [SOLVED]Method to move data on daily basis using Data Migrator

Copyright © 1996-2020 Information Builders