Focal Point Banner
Community Center Education Summit Technical Support User Groups
Let's Get Social!

Facebook Twitter LinkedIn YouTube
Focal Point    Focal Point Forums  Hop To Forum Categories  iWay Software Product Forum on Focal Point    Data Migrator 7.7.03 slows d-o-o-o-o-w-w-w-w-n-n-n-n
Go
New
Search
Notify
Tools
Reply
  
Data Migrator 7.7.03 slows d-o-o-o-o-w-w-w-w-n-n-n-n
 Login/Join
 
Master
posted
Data Migrator 7.7.03 seems to run slower and slower as a flow processes, starting out at 20,000 records per minutes even on tough transforms but dragging down to 50 or even less records per minute as the flow processes more and more records. At first I thought this was only for my XML transform, but now I'm seeing it with SQL*Server as well.

Question -- has anyone come up with ways to mitigate this? I'm at 2800 records and it has slowed to a crawl.

J.



 
Posts: 931 | Location: At the Mast | Registered: May 17, 2007Reply With QuoteReport This Post
Guru
posted Hide Post
What load type and options are you using?

If you are using "Insert/Update" and an "If the record exists" value of "Reject" or "Update" then for every input record a SELECT is done against the target table for the key column(s) to see if there is a match. As the target table grows larger that takes longer and the load process slows.

Instead of "Reject" you could use "Include" which for a RDBMS target sends the record to the database which will reject it anyway with a unique constraint violation. In most cases this is faster.

If there won't be duplicates you can speed things up even more by using a load type of "Insert records from memory" which inserts a block of records at a time.

On the other hand if you need to use "Update" the best option for an RDBMS source would be license the Change Data Capture add-on for DataMigrator to process just the changes.
 
Posts: 384 | Location: New York City | Registered: May 03, 2007Reply With QuoteReport This Post
Master
posted Hide Post
Your reply will go into the Keys-to-the-Kingdom doc I keep for just this kind of detailed intelligence.

I will look into the Change Data Capture add-on today. I have two separate clients that process insert/updates in bulk on a nightly basis.

Thanks for the quick reply.

J.



 
Posts: 931 | Location: At the Mast | Registered: May 17, 2007Reply With QuoteReport This Post
Master
posted Hide Post
Do you have any options for speeding up XML writes? The options you discuss aren't available for XML outputs, just Insert/Update.

J.



 
Posts: 931 | Location: At the Mast | Registered: May 17, 2007Reply With QuoteReport This Post
Guru
posted Hide Post
Are you still using 7.7.03? Performance is improved somewhat in the current production Release 7.7.05. Especially if you use a Direct Load flow.
 
Posts: 384 | Location: New York City | Registered: May 03, 2007Reply With QuoteReport This Post
Master
posted Hide Post
I am, and a very troubled 7.7.03 install means that we will not upgrade anytime soon. 7.7.03 does not provide a Direct Flow option for XML output, correct?

J.



 
Posts: 931 | Location: At the Mast | Registered: May 17, 2007Reply With QuoteReport This Post
Guru
posted Hide Post
I just want to let you know that our developers have rewritten the XML output routines for 7.7.06 to dramatically enhance the throughput when writing very large XML documents. In one customer case a flow that previously took 30 minutes now completes in 30 seconds. Of course YMMV.

Because this is such a significant improvement and would benefit other customers I requested that this change be retrofitted into 7.7.05M. For John Edward however, I regret that it's not possible for earlier releases.

It's available in 7.7.05M gen 136 which is available for download for 64-bit Windows; other platforms can be requested from Hotline.
 
Posts: 384 | Location: New York City | Registered: May 03, 2007Reply With QuoteReport This Post
  Powered by Social Strata  
 

Focal Point    Focal Point Forums  Hop To Forum Categories  iWay Software Product Forum on Focal Point    Data Migrator 7.7.03 slows d-o-o-o-o-w-w-w-w-n-n-n-n

Copyright © 1996-2018 Information Builders, leaders in enterprise business intelligence.