Focal Point
(SOLVED) - Limitations of GETTOK function (another 4000 character limit)

This topic can be found at:
http://forums.informationbuilders.com/eve/forums/a/tpc/f/7971057331/m/5027003196

May 13, 2019, 08:52 AM
Troy_proffitt
(SOLVED) - Limitations of GETTOK function (another 4000 character limit)
Last week, I was having an issue with STRREP and the 4000 character length limitaton. The CTRAN function is a great substitute and doesn't seem to have the same limitation. This week, I'm trying to replace a GETTOK function that has the same problem. The character string is over 4000 characters long and includes a long concatenated list of variables (delimited by an "^" character). I was using GETTOK within a loop to break it up into individual variables like this:



-REPEAT :VAR_LOOP WHILE &return_flg EQ 'Y';
-*****GRAB PARAMETER BASES ON &int value*****
-SET &parm = GETTOK(&parm_list, &parm_list.LENGTH,&int.EVAL,'^' ,&parm_list.LENGTH,'A&parm_list.LENGTH');
-SET &parm_length = ARGLEN(&parm_list.LENGTH,&parm,I4);
-*****IF TITLE LENGTH IS 0, LOOP KNOWS TO EXIT*****
-IF &parm_length EQ 0 THEN GOTO :SKIP_THIS;
-SET &trim_parm = TRIMV('T',&parm,&parm_length,' ',1,'A&parm_length.EVAL');
-SET &title_sublength = POSIT(&trim_parm, &parm_length, '=', 1,I4);
-*****GRAB TITLE FROM STRING*****
-SET &parm_title = SUBSTR(&trim_parm.LENGTH,&trim_parm,1,&title_sublength ,&title_sublength,'A&title_sublength.EVAL');
-*****GRAB VALUES FROM STRING*****
-SET &parm_values = SUBSTR(&parm_length, &trim_parm, (&title_sublength + 1) ,&parm_length, &parm_length,'A&parm_length.EVAL');
-*****CREATE VARIABLE BY CONCATENATING TITLE VARIABLE AND VALUE VARIABLE
-SET &parm_title.EVAL = '&parm_values.EVAL';
-:SKIP_THIS;
-SET &return_flg = IF &parm_length EQ 0 THEN 'N' ELSE 'Y';
-SET &int = &int + 1;
-:VAR_LOOP;

Any ideas on how to replace the GETTOK function with something that can handle over 4000 characters?

This message has been edited. Last edited by: Troy_proffitt,


WF version 8105
all output
May 13, 2019, 09:31 AM
Wep5622
What about TOKEN()?


WebFOCUS 8.1.03, Windows 7-64/2008-64, IBM DB2/400, Oracle 11g & RDB, MS SQL-Server 2005, SAP, PostgreSQL 11, Output: HTML, PDF, Excel 2010
: Member of User Group Benelux :
May 13, 2019, 09:43 AM
Troy_proffitt
Unfortunately, it looks like TOKEN isn't available in 8009 which some of our enviornments are on. It does seem to be in 8105 which is helpful but not a total solution. Is there another way?

This message has been edited. Last edited by: Troy_proffitt,


WF version 8105
all output
May 13, 2019, 11:09 AM
Wep5622
Nothing particularly convenient comes to mind. But...

You could split the string into chunks of a manageable size for GETTOK.

Make sure that if you find a substring at the end of a line, concatenate the next line to it and repeat the last GETTOK - or combine the last result with the first one on the next line.

Another option is to push the work to an RDBMS that you can connect to. Some have particularly convenient functions for stuff like this, such as being able to return a result-set of tokens as a virtual table. Of course this does mean an extra round-trip for your parameters.


WebFOCUS 8.1.03, Windows 7-64/2008-64, IBM DB2/400, Oracle 11g & RDB, MS SQL-Server 2005, SAP, PostgreSQL 11, Output: HTML, PDF, Excel 2010
: Member of User Group Benelux :
May 13, 2019, 11:10 AM
dhagen
Your kind of pushing a boulder up hill here. Maybe you could give an overview of what you are trying to do. Specifically why such long variables to start with, and how you plan on using the final variables your process.

One of the good users of this forum might have an alternate approach.


"There is no limit to what you can achieve ... if you don’t care who gets the credit." Roger Abbott
May 13, 2019, 11:20 AM
Troy_proffitt
My company created it's own OEM type reporting portal that fits seamlessly with the rest of our website. It has similar features to the WF portal in that the user can select fields / sorts / filters...etc. When the user hits the run button, we accumulate all choices that the user makes into a single string and pass that to the webfocus fex at runtime. In the past this wasn't an issue since the string length was under the 4000 character limit of the functions we were using, but as our clients get more complicated, that string has increased to a level where it's causing the functions (like GETTOK and STRREP) to fail. Unfortunately, some of our servers are still on 8009 and some of the newer functions aren't available like REPLACE() and SUBSTRING() and TOKEN().


WF version 8105
all output
May 14, 2019, 07:29 AM
dhagen
How's your Java?

Off the top of my head I would (in this order):
1. Ask your portal developers to chop into multiple parameters with a size limit. On the WF side, if the parameter exists, then parse it.
2. Build a WFRS Java plug-in that can take the large parameter and parse it into the desired output as the final .fex that contains -SET ...; statements.
3. Build a Java Filter that overrides the getParameters[] method of the http request and break the string down into the final parameters values.

2 and 3 will work with all versions if WF so long as it is compiled with the lowest version of Java that your servers run with.


"There is no limit to what you can achieve ... if you don’t care who gets the credit." Roger Abbott
May 15, 2019, 09:20 AM
Troy_proffitt
Ok, We decided to move all servers to version 8105 so we will have access to TOKEN() and SUBSTRING() which should be able to handle larger strings. Unfortunately during testing I'm having issues when running this code. It fails when it tries to use SUBSTRING to create the &parm_values variable. If you change the function to look for characters 12 to 200 it works fine, but anything over 200 and it causes an error:

 -SET &REPORTPARAMETERS = '^corp_user=true^client_type=client^client_list=4209200,4209300,4209301,4209302,4209303,4209304,4209305,4209306,4209307,4209308,4209309,4209310,4209311,4209312,4209313,4209314,4209315,4209316,4209317,4209318,4209400,4209401,4209402,4209403,4209404,4209405,4209406,4209407,4209408,4209409,4209410,4209411,4209412,4209413,4209414,4209415,4209416,4209417,4209418,4209419,4209420,4209421,4209422,4209423,4209424,4209425,4209500,4209501,4209502,4209503,4209504,4209505,4209506,4209507,4209508,4209509,4209510,4209511,4209512,4209513,4209514,4209515,4209516,4209517,4209518,4209519,4209520,4209521,4209522,4209523,4209524,4209525,4209600,4209601,4209602,4209603,4209604,4209700,4209800,4209900,4210000,4210001,4210002,4210003,4210004,4210005,4210006,4210007,4210008,4210009,4210010,4210011,4210012,4210013,4210014,4210015,4210016,4210017,4210018,4210019,4210020,4210021,4210022,4210100,4210101,4210102,4210103,4210104,4210105,4210106,4210107,4210108,4210109,4210110,4210111,4210112,4210113,4210114,4210115,4210116,4210200,4210201,4210202,4210203,4210204,4210205,4210206,4210207,4210208,4210209,4210210,4210211,4210212,4210213,4210214,4210215,4210216,4210217,4210218,4210219,4210220,4210300,4210301,4210302,4210303,4210304,4210305,4210306,4210307,4210308,4210309,4210311,4210312,4210313,4210314,4210315,4210316,4210317,4210318,4210319,4210400,4210401,4210402,4210403,4210404,4210405,4210406,4210407,4210408,4210409,4210410,4210411,4210412,4210413,4210414,4210415,4210416,4210417,4210418,4210419,4210420,4210421,4210422,4210500,4210501,4210502,4210503,4210504,4210505,4210506,4210507,4210508,4210509,4210510,4210511,4210512,4210513,4210514,4210515,4210516,4210517,4210518,4210519,4210520,4210521,4210522,4210523,4210524,4210525,4210526,4210527,4210600,4210601,4210700,4210701,4210702,4210703,4210704,4210705,4210706,4210707,4210708,4210709,4210710,4210711,4210712,4210713,4210714,4210715,4210716,4210717,4210718,4210719,4210720,4210721,4210722,4210723,4210724,4210800,4210801,4210802,4210803,4210804,4210805,4210806,4210807,4210808,4210809,4210810,4210811,4210812,4210813,4210814,4210815,4210816,4210817,4210818,4210819,4210820,4210821,4210822,4210900,4210902,4210903,4210904,4210905,4210907,4210908,4210909,4210910,4210911,4210912,4210913,4210914,4210915,4210916,4210917,4210918,4210919,4210920,4210921,4210922,4211000,4211001,4211002,4211003,4211004,4211005,4211006,4211007,4211008,4211009,4211010,4211011,4211012,4211013,4211014,4211015,4211016,4211017,4211018,4211019,4211020,4211021,4211022,4211100,4211101,4211102,4211103,4211104,4211105,4211106,4211107,4211108,4211109,4211110,4211111,4211112,4211113,4211114,4211115,4211116,4211117,4211118,4211119,4211120,4211121,4211122,4211123,4211124,4211125,4211126,4211127,4211128,4211129,4211130,4211200,4211201,4211202,4211203,4211204,4211205,4211206,4211207,4211208,4211209,4211210,4211211,4211212,4211213,4211214,4211215,4211216,4211217,4211218,4211219,4211220,4211221,4211222,4211223,4211224,4211225,4211226,4211227,4211300,4211400,4211500,4211600,4211700,4211800,4211900,4212000,4212100,4269400,4454200,4454300,4454301,4454302,4454303,4454304,4454305,4454306,4454307,4454308,4454309,4454310,4454311,4454312,4454313,4454314,4454315,4454316,4454317,4454318,4454319,4454320,4454400,4454401,4454402,4454403,4454404,4454405,4454406,4454407,4454408,4454409,4454410,4454411^field_list=4,114'||
- '^WFFMT=EXL2K^mac_os=N^EXL2K=EXL2K^select_font=8^alt_color=No^paper_size=Letter^orientation=Portrait^template_id=132^program_id=17^user_template_id=0^template_title=My Saved                                     Report^begdt=04/01/2019^enddt=04/25/2019^report_title=Detailed Employee Changes^sort1=113^sort_sh1=true^active_term=undefined^undefined=undefined^undefined=undefined^undefined=undefined^undefined=undefined^undefined=undefined^hours=undefined^pay=undefined^deductions=undefined^tax_wage=undefined^inv_charge=undefined^termyn=Y^summaryyn=Y^date_type1=0^termyn_Y=termyn_Y^summaryyn_Y=summaryyn_Y^date_type1_0=date_type1_0^class_changes=true^valid_report_run=True^browser_ver=Chrome';
-TYPE REPORTPARAMETERS: &REPORTPARAMETERS
-*****MUST CONVERT TO NON AMPR CHAR IF SCHEDULED RUN BEFORE TRIMV IF NECESSARY*****
-SET &parm_list = CTRAN(&REPORTPARAMETERS.LENGTH, &REPORTPARAMETERS.QUOTEDSTRING, 38, 94, 'A&REPORTPARAMETERS.LENGTH');
-SET &parm = TOKEN(&parm_list, '^', 4);
-TYPE parm: &parm
-SET &parm_length = ARGLEN(&parm_list.LENGTH,&parm,I4);
-TYPE parm_length: &parm_length
-SET &title_sublength = POSIT(&parm_list, &parm_length, '=', 1,I4);
-TYPE title_sublength: &title_sublength
-*****GRAB TITLE FROM STRING*****
-SET &parm_title = SUBSTRING(&parm.QUOTEDSTRING,1,&title_sublength);
-TYPE parm_title: &parm_title
-*****GRAB VALUES FROM STRING*****
-SET &parm_values = SUBSTRING(&parm.QUOTEDSTRING,&title_sublength +1,&parm_length);
-TYPE parm_values &parm_values
 


I get a message that webpage is not responding, and in IIS logs, we're getting:

quote:
A process serving application pool 'DefaultAppPool' terminated unexpectedly. The process id was '5592'. The process exit code was '0xff'.



WF version 8105
all output
May 15, 2019, 12:03 PM
FP Mod Chuck
Troy

You better open a case with techsupport for this error.


Thank you for using Focal Point!

Chuck Wolff - Focal Point Moderator
WebFOCUS 7x and 8x, Windows, Linux All output Formats
May 15, 2019, 02:51 PM
Troy_proffitt
I already did Smiler I'm just trying to use all my resources to find a solution to this.


WF version 8105
all output
May 15, 2019, 04:48 PM
dhagen
This intrigued me, to I took an hour out of the day and tried a plugin.

/*
 * Created on 2019-05-15
 *
 */
package com.ibi.plugin;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.PrintWriter;

import ibi.callpgm.callpgm;
import ibi.callpgm.ibiAnswerSet;
import ibi.callpgm.ibianswr;
import ibi.trace.IBILogFactory;
import ibi.trace.ILogger;

public class ParseParams implements callpgm {

	private static final ILogger log = IBILogFactory.getEdaprintLogger();
	ibianswr answr = null;
	int currentRow = 0;
	String fexName = null;
	String inParams = null;

	public ibianswr execute(String arg0, String arg1, String[] params) throws Exception {
		/* init class variables */
		fexName = params[0];
		inParams = params[1];
		log.debug("Make file : " + fexName);
		
		/* set up answer set */
		answr = new ibianswr(1);
		answr.setColName(1, "RETURNCODE");
		answr.setColType(1, ibiAnswerSet.IBI_ALPHA);
		answr.setColSize(1, 1);
		return answr;
	}

	public ibianswr execute(String arg0, String arg1, Object arg2, String[] arg3) throws Exception {
		return null;
	}

	public Integer fetch() throws Exception {
		/* Only fetch once.  It returns or user id or it doesn't */
		if (currentRow++ <= 0) {
			if (WriteSets())
				answr.setColValue(1, "1");
			else 
				answr.setColValue(1, "0");
			return callpgm.IBI_DATA;
		} else {
			return callpgm.IBI_EOD;
		}
	}

	private boolean WriteSets() {
		boolean rtn = false;

		try {
			FileOutputStream fout = new FileOutputStream(fexName);
			PrintWriter pout = new PrintWriter(fout);

			/* split the parameters */
			String[] params = inParams.split("\\^");
			for (String param : params) {
				log.debug("In param loop for param: " + param);
				if (!param.equals("")) {
					String setStatement = "-SET &" + param.split("=")[0] + "='" + param.split("=")[1] + "';";
					log.debug("Set: " + setStatement.substring(0, Math.min(78,setStatement.length()-1)));
					pout.println(setStatement);
				}
			}
			
			/* close the file */
			pout.flush();
			pout.close();
			fout.flush();
			fout.close();

			/* make file writable to avoid delete issues with Unix and Linux */
			File outFile = new File(fexName);
			if (outFile.exists()) {
				outFile.setWritable(true);
				outFile.setReadable(true);
			}
			
			/* all is good */
			rtn = true;
					
		} catch (IOException e) {
			// TODO Auto-generated catch block
			log.info("IO Exception: " + e.getMessage());
			e.printStackTrace();
		}  catch (Exception e) {
			// TODO Auto-generated catch block
			log.info("General Exception: " + e.getMessage());
			e.printStackTrace();
		}
		
		return rtn;
	}

}
  


Compile and export to a JAR and put that on the JSCOM classpath.

Fex to test:
-* param test
-SET &PARAMS='^corp_user=true^client_type=client^client_list=4209200,...,4454411^field_list=4,114^WFFMT=EXL2K^mac_os=N^EXL2K=EXL2K^select_font=8^alt_color=No^paper_size=Letter^orientation=Portrait^template_id=132^program_id=17^user_template_id=0^template_title=My Saved';

-* set fex name to hold the final -SET statements
-SET &TEST=TEMPPATH(100,'A100') || 'mysets.fex';
-SET &TEST=TRUNCATE(&TEST);

-* run the plugin
-DEFAULT &RETURNCODE='X'
CALLJAVA com.ibi.plugin.ParseParams,"&TEST","&PARAMS" ;
TABLE FILE SQLOUT
PRINT *
ON TABLE HOLD AS JRUN
END
-RUN
-READFILE JRUN
-TYPE JRUN RETURNED &RETURNCODE :: 1=GOOD,0=BAD

-INCLUDE MYSETS
-? &c
  


Output:
0 NUMBER OF RECORDS IN TABLE=        1  LINES=      1
JRUN RETURNED 1 :: 1=GOOD,0=BAD
CURRENTLY DEFINED & VARIABLES STARTING WITH 'c':
&client_list  = 4209200,4209300,4209301,4209302,4209303,4209304,4209305,42093
&client_type  = client
&corp_user    = true  


Contents of mysets.fex
-SET &corp_user='true';
-SET &client_type='client';
-SET &client_list='4209200,...,4454411';
-SET &field_list='4,114';
-SET &WFFMT='EXL2K';
-SET &mac_os='N';
-SET &EXL2K='EXL2K';
-SET &select_font='8';
-SET &alt_color='No';
-SET &paper_size='Letter';
-SET &orientation='Portrait';
-SET &template_id='132';
-SET &program_id='17';
-SET &user_template_id='0';
-SET &template_title='My Saved';  


You could harden the code and call it in your site profile only when the param list exists.


"There is no limit to what you can achieve ... if you don’t care who gets the credit." Roger Abbott
May 21, 2019, 11:26 AM
Troy_proffitt
Just wanted to give a heads up that we figured out that we needed to change a value in TOMCAT:

Changed the two occurrences of maxPostSize="0" to maxPostSize="-1" and restarted tomcat
After troubleshooting some more we noticed an error with the maxHttpHeaderSize and under to two occurences of maxPostSize we added maxHttpHeaderSize="16384" and that appeared to resolve the issue when running the procedure from DevStudio and App Studio.


WF version 8105
all output