Focal Point Banner


As of December 1, 2020, Focal Point is retired and repurposed as a reference repository. We value the wealth of knowledge that's been shared here over the years. You'll continue to have access to this treasure trove of knowledge, for search purposes only.

Join the TIBCO Community
TIBCO Community is a collaborative space for users to share knowledge and support one another in making the best use of TIBCO products and services. There are several TIBCO WebFOCUS resources in the community.

  • From the Home page, select Predict: WebFOCUS to view articles, questions, and trending articles.
  • Select Products from the top navigation bar, scroll, and then select the TIBCO WebFOCUS product page to view product overview, articles, and discussions.
  • Request access to the private WebFOCUS User Group (login required) to network with fellow members.

Former myibi community members should have received an email on 8/3/22 to activate their user accounts to join the community. Check your Spam folder for the email. Please get in touch with us at community@tibco.com for further assistance. Reference the community FAQ to learn more about the community.


Focal Point    Focal Point Forums  Hop To Forum Categories  iWay Software Product Forum on Focal Point    Data Migrator 7.7.03 slows d-o-o-o-o-w-w-w-w-n-n-n-n

Read-Only Read-Only Topic
Go
Search
Notify
Tools
Data Migrator 7.7.03 slows d-o-o-o-o-w-w-w-w-n-n-n-n
 Login/Join
 
John_Edwards
Virtuoso
posted
Data Migrator 7.7.03 seems to run slower and slower as a flow processes, starting out at 20,000 records per minutes even on tough transforms but dragging down to 50 or even less records per minute as the flow processes more and more records. At first I thought this was only for my XML transform, but now I'm seeing it with SQL*Server as well.

Question -- has anyone come up with ways to mitigate this? I'm at 2800 records and it has slowed to a crawl.

J.



 
Posts: 1012 | Location: At the Mast | Registered: May 17, 2007Report This Post
Clif
Guru
posted Hide Post
What load type and options are you using?

If you are using "Insert/Update" and an "If the record exists" value of "Reject" or "Update" then for every input record a SELECT is done against the target table for the key column(s) to see if there is a match. As the target table grows larger that takes longer and the load process slows.

Instead of "Reject" you could use "Include" which for a RDBMS target sends the record to the database which will reject it anyway with a unique constraint violation. In most cases this is faster.

If there won't be duplicates you can speed things up even more by using a load type of "Insert records from memory" which inserts a block of records at a time.

On the other hand if you need to use "Update" the best option for an RDBMS source would be license the Change Data Capture add-on for DataMigrator to process just the changes.


N/A
 
Posts: 397 | Location: New York City | Registered: May 03, 2007Report This Post
John_Edwards
Virtuoso
posted Hide Post
Your reply will go into the Keys-to-the-Kingdom doc I keep for just this kind of detailed intelligence.

I will look into the Change Data Capture add-on today. I have two separate clients that process insert/updates in bulk on a nightly basis.

Thanks for the quick reply.

J.



 
Posts: 1012 | Location: At the Mast | Registered: May 17, 2007Report This Post
John_Edwards
Virtuoso
posted Hide Post
Do you have any options for speeding up XML writes? The options you discuss aren't available for XML outputs, just Insert/Update.

J.



 
Posts: 1012 | Location: At the Mast | Registered: May 17, 2007Report This Post
Clif
Guru
posted Hide Post
Are you still using 7.7.03? Performance is improved somewhat in the current production Release 7.7.05. Especially if you use a Direct Load flow.


N/A
 
Posts: 397 | Location: New York City | Registered: May 03, 2007Report This Post
John_Edwards
Virtuoso
posted Hide Post
I am, and a very troubled 7.7.03 install means that we will not upgrade anytime soon. 7.7.03 does not provide a Direct Flow option for XML output, correct?

J.



 
Posts: 1012 | Location: At the Mast | Registered: May 17, 2007Report This Post
Clif
Guru
posted Hide Post
I just want to let you know that our developers have rewritten the XML output routines for 7.7.06 to dramatically enhance the throughput when writing very large XML documents. In one customer case a flow that previously took 30 minutes now completes in 30 seconds. Of course YMMV.

Because this is such a significant improvement and would benefit other customers I requested that this change be retrofitted into 7.7.05M. For John Edward however, I regret that it's not possible for earlier releases.

It's available in 7.7.05M gen 136 which is available for download for 64-bit Windows; other platforms can be requested from Hotline.


N/A
 
Posts: 397 | Location: New York City | Registered: May 03, 2007Report This Post
  Powered by Social Strata  

Read-Only Read-Only Topic

Focal Point    Focal Point Forums  Hop To Forum Categories  iWay Software Product Forum on Focal Point    Data Migrator 7.7.03 slows d-o-o-o-o-w-w-w-w-n-n-n-n

Copyright © 1996-2020 Information Builders