Focal Point Banner


As of December 1, 2020, Focal Point is retired and repurposed as a reference repository. We value the wealth of knowledge that's been shared here over the years. You'll continue to have access to this treasure trove of knowledge, for search purposes only.

Join the TIBCO Community
TIBCO Community is a collaborative space for users to share knowledge and support one another in making the best use of TIBCO products and services. There are several TIBCO WebFOCUS resources in the community.

  • From the Home page, select Predict: WebFOCUS to view articles, questions, and trending articles.
  • Select Products from the top navigation bar, scroll, and then select the TIBCO WebFOCUS product page to view product overview, articles, and discussions.
  • Request access to the private WebFOCUS User Group (login required) to network with fellow members.

Former myibi community members should have received an email on 8/3/22 to activate their user accounts to join the community. Check your Spam folder for the email. Please get in touch with us at community@tibco.com for further assistance. Reference the community FAQ to learn more about the community.


Focal Point    Focal Point Forums  Hop To Forum Categories  iWay Software Product Forum on Focal Point    [SOLVED]Method to move data on daily basis using Data Migrator

Read-Only Read-Only Topic
Go
Search
Notify
Tools
[SOLVED]Method to move data on daily basis using Data Migrator
 Login/Join
 
David Nguyen
Member
posted
We need to setup an iWay Data Migrator process to move data from several tables in Oracle DB to DataWarehouse on daily basis.

If a source table contains a Datetime column, we can base on the date time to identify and easily move these records.
However, when a source table doesn't contain any date time column, what would be a proper way to select the newly inserted/updated records?

One way to solve this is to build triggers on the source tables to keep track of what being insert/update. Then iWay will base on this info to move data. This will work but i don't think this is a good approach.

Could you recommend any alternatives? Thanks a lot.

This message has been edited. Last edited by: Tamra,


iWay Data Migrator v7704M
Windows, All Outputs
 
Posts: 23 | Registered: January 29, 2014Report This Post
StuBouyer
Guru
posted Hide Post
You'll have to double check the manual but I belive that Oracle supports Change Data Capture.

Then you can set up DM to only get the changed data.

If that doesn't work then one hack is to create hashes of the concatenated fields you are interested in and compare that against your warehouse - not exactly efficient but much better than comparing field to field.

When I've used this I've added the hash as a separate field in the warehouse to speed up comparison.

Cheers

Stu


WebFOCUS 8.2.03 (8.2.06 in testing)
 
Posts: 253 | Location: Melbourne, Australia | Registered: February 07, 2007Report This Post
Ricardo Augusto
Guru
posted Hide Post
Hi David,

My suggestion to handle your ETL to tables without Datetime column is:

- Access your DW table, create a HOLD with IDs or max PK ( If its is auto increment );
- Access your Oracle DB filtering IDs NE (HOLD) or PK GT (HOLD)

I hope it helps you. Good lucky!


WebFOCUS 8.1.05 / APP Studio
 
Posts: 272 | Location: Brazil | Registered: October 31, 2006Report This Post
Clif
Guru
posted Hide Post
There is an optional add-on to DataMigrator that we call Change Data Capture that reads from the ORACLE Archive Logs to identify rows that have changed (insert, update or delete). It's documented in the Data Migrator User's Guide, Chapter 12, "Change Data Capture."

Note that for an ORACLE source it's only required that full logging is enabled. It does not require what ORACLE calls Change Data Capture (which is being deprecated) or Golden Gate.

While CDC is available in Release 7704 it's been improved since that release was first posted. I'd recommend using the current production Release 7706.

This message has been edited. Last edited by: Clif,


N/A
 
Posts: 397 | Location: New York City | Registered: May 03, 2007Report This Post
David Nguyen
Member
posted Hide Post
Thanks a lot for all of your suggestion. I really appreciate it.

I'm taking a look at using the Change Data Capture Method in iWay 7704 for now.

We are also in a process of upgrading to iWay 7706 this year.


iWay Data Migrator v7704M
Windows, All Outputs
 
Posts: 23 | Registered: January 29, 2014Report This Post
  Powered by Social Strata  

Read-Only Read-Only Topic

Focal Point    Focal Point Forums  Hop To Forum Categories  iWay Software Product Forum on Focal Point    [SOLVED]Method to move data on daily basis using Data Migrator

Copyright © 1996-2020 Information Builders