I'm having a bit of a problem with a graph report on some of our measured data, where the data contains a ridiculous amount of (x,y) points; nearly 108,000 per legend item!

I'm trying to reduce the data, but I'm having a bit of a hard time deciding what the correct strategy is here.

Currently, the graph is created from data from this TABLE request:

TABLE FILE UNI00020H_PLOTDATA0
PRINT yCol               -* Y-value; D20.4

BY legend                -* Plot line legend; A40
BY seq                   -* Plot point sequence no.; I10
BY xCol                  -* X-value; D20.4

ON TABLE HOLD AS UNI00020H_PLOTDATA FORMAT FOCUS
END

This data is used to create an XY-plot per legend item.

The sequence is very important in these plots, as it determines the order in which the points need to be connected (therefore this can not be implemented using an IBI JSChart, as IBI insists on sorting values BY xCol, which is a different sort order). For example, some of these plots are measurements around a tyre cross-section, inside and outside.

What I'm pretty sure that we need to do is grouping similar (xCol, yCol) points under subsequent seq values into a single averaged (x,y) value at an averaged sequence number (or min, or max; as long as the sort order remains unchanged).

The first step should be determining the resolution, I think as below?:

COMPUTE RESOLUTION/D20.8 = .01 * SQRT(((MAX.xCol - MIN.xCol) * (MAX.yCol - MIN.yCol)) / (&GRAPH_WIDTH * &GRAPH_HEIGHT));

And then we need to use that resolution to compute the averages of: (subsequent!) seq, xCol & yCol values as long as they are less than RESOLUTION apart from each-other. That seems to require Pythagoras:

COMPUTE distance/D20.8 = SQRT((xCol - LAST xCol) * (xCol - LAST xCol) + (yCol - LAST yCol) * (yCol - LAST yCol));

I can't wrap my mind around how to implement the grouping based on this though... Ideas?

P.S. I tried using BY ... IN-RANGES-OF, but apparently that only accepts integer ranges, while I have to deal with fractions.

This message has been edited. Last edited by: Wep5622,

lol, wep. What the h3ll you're doing?

A few tips:
Could you combine a few BY-fields to reduce the data? ( or force the chart to sort as wished ).


I once did something like your resolution thing a whole different way.

I made a HOLD sorted as desired and calculated percentiles of the total rows.

e.g. the HOLD file had 1500 lines. Percentile was: PREC = int(100 * ( line_nr / total_lines )).

Then used SUM and BY PREC to reduce to a maximum of 100 lines of data.
( just like IN-RANGES-OF ), but here you have control over decimals if needed.

Hope this helps?

Things get complicated when you're creating reports for scientists

The problem with only plotting every n-th point (same as with looking at the point density along only the X-axis or the Y-axis respectively) is that you're going to leave out points where the point density is low, while you keep superfluous points where the point density is high.

The latter isn't a big issue, it just causes a bit of overhead, but the first reduces the accuracy of the plot!

With some more time to think this over, what I'm looking for is a method to divide the graph into equidistant sections (along each respective curve). Within each equidistant section, we can simply average the values and only plot the averaged point.

That would be ideal.

Well, at least now I have the right jargon to search the internets!

I would DEFINE a field for your xCol value so that you can get onto an IBI JSChart. I sometimes cheat with blank spaces, which generally get compressed or removed entirely at run time

DEFINE FILE ABC
  SORT_ORDER/A25 = IF FIELD EQ 'FIRST_FIELD'  THEN '    FIRST_FIELD'
              ELSE IF FIELD EQ 'SECOND_FIELD' THEN '  SECOND_FIELD'
              ELSE IF FIELD EQ 'THIRD_FIELD'  THEN ' THIRD_FIELD'
              ELSE IF FIELD EQ 'FOURTH_FIELD' THEN '   FOURTH_FIELD';
END

SORT_ORDER actually will put FOURTH_FIELD in the second position in the above example because it has three spaces in front of it.

Yours will be more complicated due to the number of sort fields you have, but I'd wager you can make a couple of passes at the data to get it to be in a position to do a simple graph. Concatenating your sort fields via defines in one pass or many can make things easier at graph time.

That's generally my strategy -- report hard, graph easy. Graphing has enough complexity in it to get the appearance right, it's worth pre-working the data before you go after it in the graph tool.

Don't know if this is a new idea to you, but hope it helps.

No, sorry, JSCHART is really unsuitable for my case. I don't believe that there is any way to make that work correctly. A sort-field is certainly not going to help, as we already have that in the form of the 'seq' column.

The problem with IBI charts is that they insist on sorting values along the X-axis, while our sort order is defined in an (ungraphed) seq-field that dictates the order of (X,Y) pairs.

Try plotting a circle using JSCHART, I don't think you'll succeed

For this reason, these graphs use Google visualizations, where we manually generate the Javascript code that sets the plot-points (with a rather more efficient syntax than IBI uses in their JSCHARTs, I might add...). That's a bit of an implementation detail that I didn't want to complicate this already complicated issue with.

I found a reasonable solution!

We ended up defining a grid of virtual squares along both axis, with the side of each square about a 10th of a pixel (to be on the safe side).

Putting points inside those squares was simply done by COMPUTE-ing a field where we rounded their coordinates down to a single grid position, after which we could average all values per grid square.

So, first we calculate the adjusted coordinates:

TABLE FILE UNI00020H_PLOTDATA0
PRINT yCol
   COMPUTE BOX_X/P20.1 = RESOLUTION * INT(xCol / RESOLUTION);
   COMPUTE BOX_Y/P20.1 = RESOLUTION * INT(yCol / RESOLUTION);

   COMPUTE BOX_SEQ/I10 = IF (legend NE LAST legend) THEN 1
		ELSE IF (BOX_X NE LAST BOX_X) OR (BOX_Y NE LAST BOX_Y)
			THEN LAST BOX_SEQ +1
			ELSE LAST BOX_SEQ;

BY legend
BY seq
BY xCol
ON TABLE HOLD
END

And then we average the points inside each box:

TABLE FILE HOLD
SUM
   AVE.seq
   AVE.xCol
   AVE.yCol

BY legend
BY BOX_SEQ
END

And then we plot the graph.