Focal Point Banner


As of December 1, 2020, Focal Point is retired and repurposed as a reference repository. We value the wealth of knowledge that's been shared here over the years. You'll continue to have access to this treasure trove of knowledge, for search purposes only.

Join the TIBCO Community
TIBCO Community is a collaborative space for users to share knowledge and support one another in making the best use of TIBCO products and services. There are several TIBCO WebFOCUS resources in the community.

  • From the Home page, select Predict: WebFOCUS to view articles, questions, and trending articles.
  • Select Products from the top navigation bar, scroll, and then select the TIBCO WebFOCUS product page to view product overview, articles, and discussions.
  • Request access to the private WebFOCUS User Group (login required) to network with fellow members.

Former myibi community members should have received an email on 8/3/22 to activate their user accounts to join the community. Check your Spam folder for the email. Please get in touch with us at community@tibco.com for further assistance. Reference the community FAQ to learn more about the community.


Focal Point    Focal Point Forums  Hop To Forum Categories  WebFOCUS/FOCUS Forum on Focal Point     [SOLVED] how to validate csv input file

Read-Only Read-Only Topic
Go
Search
Notify
Tools
[SOLVED] how to validate csv input file
 Login/Join
 
Platinum Member
posted
Hi,

I need to read, validate and process a csv (COM) source file.
I know how to make a synonym and with a Webfocus procedure or DataMigrator flow I can read and process it.

My question is about validating the input. We would like to validate if each record has the correct number of fields, fields are not too long, numeric fields are numeric and date fields have valid dates, et cetera. If the source is a database table you do not have to do such validations, but csv is just a flat text file, so can contain anything that does not match the definitions.

What are best practices (or suggestions) how to do such validations?
I could just run it and Webfocus will surely throw some error if the input is not correct, but preferably we want to give more specific feedback about what is not correct.

Martin.

This message has been edited. Last edited by: Martin vK,


WebFocus 8206M, iWay DataMigrator, Windows, DB2 Windows V10.5, MS SQL Server, Azure SQL, Hyperstage, ReportCaster
 
Posts: 168 | Registered: March 29, 2013Report This Post
Expert
posted Hide Post
write a process that does a -READ against that file, record by record.
Evaluate the contents of each of the variables that you read in, one at a time.
Then do a -WRITE to a new file of the cleaned record.
Eg, if a field is expected to be an integer, and it has rubbish in it:
-READ MYRECORD &field1.A3. &field2.A4. etc
and &field1 is meant to be an integer,
you can test &field1.TYPE to be 'N' or 'A'
It will take alot of coding to properly handle each one of your input fields.
but that's the way to do it. And it can be fun.
I had a case where i had to -READ my file and parse the results into two separate entities, b/c one field was a txt box with a ton of special characters in it. It had come from a comment box on an input form somewhere.




In Focus since 1979///7706m/5 ;wintel 2008/64;OAM security; Oracle db, ///MRE/BID
 
Posts: 3811 | Location: Manhattan | Registered: October 28, 2003Report This Post
Virtuoso
posted Hide Post
As Susannah has said, it is not straightforward.

However, interestingly, I see from your signature that you have Hyperstage. As an assumption, I take it that this is not a one off file.

If this assumption is correct then one approach I might take is to utilise the power of the Hyperstage DLP (Distributed Load Processor).

Create a hyperstage table to mimic the structure that the CSV file should be in, with all the correct field formats, using standard SQL create schema/table syntax.

Then create a DLP call to load the CSV file into this Hyperstage table, placing the log (-l) and reject (-r) files into an accessible folder and ensure that the load continues after errors (-c -1). These reject and log files will give a picture of what, if anything, was wrong with the incoming data, (field lengths mainly here, but also valid numeric values and dates), to the Hyperstage table.

Now you can TABLE the Hyperstage data and further validate any data items that have to be within certain criteria.


Alan.
WF 7.705/8.007
 
Posts: 1451 | Location: Portugal | Registered: February 07, 2007Report This Post
Platinum Member
posted Hide Post
Thanks Susannah and Alan for your suggestions. I will work with both to see what works best for us.

Alan, where can I find documentation on the Hyperstage DLP? Since we bought Hyperstage 2 years ago we have received little documentation.

Martin.


WebFocus 8206M, iWay DataMigrator, Windows, DB2 Windows V10.5, MS SQL Server, Azure SQL, Hyperstage, ReportCaster
 
Posts: 168 | Registered: March 29, 2013Report This Post
Guru
posted Hide Post
When you create a synonym for a delimited flat file it's analyzed to determine the least restrictive data type to describe each field. You can specify how many rows to scan so that it won't take forever for a large file, but that also can affect the results if later rows contain different types of values.

Of course this automatic process doesn't know the intent. A field that contains all digits in the rows that are examined will be described as numeric, but if a row contains a non-numeric character than it's described as Alpha.

But what's correct? You may want to treat any non-numeric value as an error. Or it may be a field where some values are correctly character values. For example a Zip code field with mostly USA Zip codes but also contains some Canada postal codes.

Once you have a synonym that describes you delimited flat file the way you want you can use DataMigrator to load the data. In the target transformations on the Validate tab you can reject records that don't meet your criteria. With logging enabled for invalid records they can be written to a separate file for later review and analysis.


N/A
 
Posts: 397 | Location: New York City | Registered: May 03, 2007Report This Post
  Powered by Social Strata  

Read-Only Read-Only Topic

Focal Point    Focal Point Forums  Hop To Forum Categories  WebFOCUS/FOCUS Forum on Focal Point     [SOLVED] how to validate csv input file

Copyright © 1996-2020 Information Builders