[CLOSED] Any way to get a Read count in Data Migrator?
I have a very simple routine that reads records from DB2 with the following format in the SQL Statement box:
select * from alert
I select records from one database this way, write them all to another. When I run this routine the ETLLog shows the number of records written to my destination table, but it does not show the number of records read.
Yes, in theory every record should write and it should be a perfectly even match, or an error thrown if not. But I have been given a requirement to show both the number of records read and the number of records written for accounting purposes, and I'm trying to figure out a way to get the records-read count to appear.
I can put '-TYPE &RECORDS' after the select command, and it prints the number of records read. But, I can't close the box and save the routine. It throws an error.
Any suggestions? Looking at SQL options but don't see one that fits the bill.This message has been edited. Last edited by: FP Mod Chuck,
July 26, 2018, 03:27 PM
Clif
The count of rows processed (read) and accepted (written) both appear in the log regardless as to whether you generate the SQL or type it in provided that you use load type insert/update. .
These same numbers are available in the statistics report after the flow has completed.
The SQL object is just for SELECT statements.
If you want to add a -TYPE statement to a flow, create a stored procedure with it, and then on the process flow tab it after the (embedded) data flow and connect the objects.
N/A
July 27, 2018, 08:42 AM
John_Edwards
This is what I get when I query ETLLog after the run:
------ Start of Log Record for core_db2p_all_rows/load_alert ------
Request - load_alert (Owner: admin) submitted.
coredb_alert_core type MS SQL Server Existing target
Issuing PREPARE
ROWS AFFECTED BY PASSTHRU COMMAND : 32967/DELETE
Bulk Insert active. INSERTSIZE = 10000
Starting Load
0 WARNING.. ON MATCH INCLUDE INPUTS DUPLICATE SEGMENTS
REFERENCE...AT TRANS 10000
Commit forced at: 10000 for 10000 row(s)
REFERENCE...AT TRANS 20000
Commit forced at: 20000 for 10000 row(s)
REFERENCE...AT TRANS 30000
Commit forced at: 30000 for 10000 row(s)
Commit forced at: 32964 for 2964 row(s)
Ending Load
Return Code = 0
Request: load_alert - finished processing
CPU Time : 2485
Finished
Elapsed run time 0:00:08
------ End of Log Record for core_db2p_all_rows/load_alert ------
How do I get those additional messages to print to the log file please?
July 27, 2018, 09:00 AM
John_Edwards
Yeah, and my ibif_max_messages is set at 20 on my I-Way server too. Need to figure out where to change that, and all the help sends me to the client console, where it's set to 20,000. So if you can explain how I change that in the I-Way service I'd appreciate it as well.
July 27, 2018, 11:10 AM
Clif
As I said "provided that you use load type insert/update." Since that does row-at-a-time processing we can count the rows.
You are using load type "Insert records from memory" with a blocksize of 10,000.
As documented in the DMUG
Insert records from memory: * Loads much faster, since a block of rows are inserted at once... * Does not provide row counts (the number of records inserted or rejected) in the detail log or statistics.
N/A
July 27, 2018, 12:56 PM
John_Edwards
Thank you Clif.
If anyone can offer any recommendations on either of the two issues above I'm all ears, I can't move to insert/update due to the size of what I'm moving, and I'd very much like to capture all log rows.