Focal Point Banner


As of December 1, 2020, Focal Point is retired and repurposed as a reference repository. We value the wealth of knowledge that's been shared here over the years. You'll continue to have access to this treasure trove of knowledge, for search purposes only.

Join the TIBCO Community
TIBCO Community is a collaborative space for users to share knowledge and support one another in making the best use of TIBCO products and services. There are several TIBCO WebFOCUS resources in the community.

  • From the Home page, select Predict: WebFOCUS to view articles, questions, and trending articles.
  • Select Products from the top navigation bar, scroll, and then select the TIBCO WebFOCUS product page to view product overview, articles, and discussions.
  • Request access to the private WebFOCUS User Group (login required) to network with fellow members.

Former myibi community members should have received an email on 8/3/22 to activate their user accounts to join the community. Check your Spam folder for the email. Please get in touch with us at community@tibco.com for further assistance. Reference the community FAQ to learn more about the community.


Focal Point    Focal Point Forums  Hop To Forum Categories  iWay Software Product Forum on Focal Point    ETL Flow not reporting on truncated data

Read-Only Read-Only Topic
Go
Search
Notify
Tools
ETL Flow not reporting on truncated data
 Login/Join
 
AnnaR
Member
posted
Hello,

Here is the scenario; sometimes I receive source files with columns sizes larger than the original configuration setup in master data. During data load, some rows do not get loaded b/c there is truncation on data. There is no flag in Data Migrator to indicate that data was not loaded in its entirety due to truncation. Theonly way is to look through the log files. Data Migrator flow reads/processes only the
rows with column size fitting the master data specs and rpeorts the flow execution wuth return code 0 - but it is hardly a success. I tried using `Record Logging` feature that is available on the flow properties with attribute `Format Errors`; however that did nothing to capture truncated rows. Is there some method to flag the flow if ther eis truncation on the data? I know , I can setup master data with larger sizes, but I am looking for something that can be setup on a flow as a flag to indicate that some rows were not loaded due to truncation of data.

Any suggestions?


WebFOCUS 7.7, iWay Data Migrator, Windows
Excel, PDF, HTML, XML
 
Posts: 19 | Registered: March 19, 2012Report This Post
Kerry
Expert
posted Hide Post
Hi Anna,

From our iWay manager: if you have not done so, please open a case with Customer Support Services for assistance. You may call 1-800-736-6130, or access online at InfoResponse.

Cheers,

Kerry


Kerry Zhan
Focal Point Moderator
Information Builders, Inc.
 
Posts: 1948 | Location: New York | Registered: November 16, 2004Report This Post
dhagen
Virtuoso
posted Hide Post
We have the same situation. Add this stored proc before and after the data flow:

-* Verify the process.
-DEFAULTH &&U_FILE='NONE'
-DEFAULTH &&CO = 'MAN'
-IF &&U_FILE NE 'NONE' GOTO :VERIFY;
 
-* Allocate a file to store source format problems.
-SET &&U_FILE_NAME = LOCASE(&&CO.LENGTH, &&CO, 'A&&CO.LENGTH') || '_emgfile' || &YYMD || EDIT(HHMMSS('A8'),'_99_$99_$99');
-SET &&U_FILE_EXT = &&U_FILE_NAME || '.text';
-SET &&U_FILE = 'baseapp/' || &&U_FILE_EXT ;
FILEDEF EMGFILE DISK &&U_FILE
SET EMGSRV=FILE
-RUN
-EXIT
 
-* job has finished running ... test to see if emgfile contains values
-:VERIFY
APP QUERY baseapp HOLD
TABLE FILE FOCAPPQ PRINT SIZE
WHERE FILENAME EQ '&&U_FILE_EXT';
ON TABLE HOLD AS T_FSIZE
END
-RUN 
-READFILE T_FSIZE
-RUN
-SET &&KILL_RPC = IF &SIZE EQ 0 THEN 'N' ELSE 'Y';
-SET &MSG1 = IF &&KILL_RPC EQ 'N' THEN ' ' ELSE '-TYPE (FOC0209) Source format errors, check following file for details';
-SET &MSG2 = IF &&KILL_RPC EQ 'N' THEN ' ' ELSE '-TYPE (FOC0209) File: ' | &&U_FILE;
-SET &FDEL = IF &&KILL_RPC EQ 'Y' THEN ' ' ELSE 'APP DELETEF BASEAPP &&U_FILE_NAME TEXT';
&MSG1.EVAL
&MSG2.EVAL
&FDEL.EVAL
-EXIT


For every row that gets removed from the query because of truncation, a line is added to the EMGFILE. If the EMGFILE has any values, then the process will set the KILL_RPC flag.

Side note: You might want to break the above into 2 procedures just in case you run multiple data flows - as RPC - in a single process flow.


"There is no limit to what you can achieve ... if you don’t care who gets the credit." Roger Abbott
 
Posts: 1102 | Location: Toronto, Ontario | Registered: May 26, 2004Report This Post
  Powered by Social Strata  

Read-Only Read-Only Topic

Focal Point    Focal Point Forums  Hop To Forum Categories  iWay Software Product Forum on Focal Point    ETL Flow not reporting on truncated data

Copyright © 1996-2020 Information Builders