Something doesn't seem quite right here regarding MISSING data.

I have a simple fex where I want to compare two fields in a HOLD file generated by an ACROSS statement.

In the sample code below I have two sets of reports: REPORT 1A, REPORT 1B and REPORT 2A, REPORT 2B. (I've also thrown in REPORT 0 to display the data I'm working with.

REPORT 1A and REPORT 2A display the data without a WHERE statement.

REPORT 1B has a WHERE statement and it seems to work - it shows no results because no values are equal.

REPORT 2B has a WHERE statement which is not working - it shows results even though the values are different.

The only difference between the two sets of reports is the extra BY statement in the second set.

I have HOLDMISS set to ON because it "Allows you to store missing data in a HOLD file. When TABLE generates a default value for data not found, it generates missing values".

I would like my EQ test to differentiate between a zero and missing data. I'd like to understand why the following two rows appear in REPORT 2B:

SEATS BODYTYPE    SALENGLAND SALITALY 
----- ----------- ---------- --------
    2 CONVERTIBLE          0      ??? 
    2 HARDTOP              0      ??? 

The code:

SET BYDISPLAY=ON
SET PAGE=NOPAGE
SET NODATA='???'
SET HOLDFORMAT=ALPHA
SET HOLDLIST=PRINTONLY
SET ASNAMES=ON
SET HOLDMISS=ON
SET COMPMISS=ON

TABLE FILE CAR
SUM
SALES
BY COUNTRY
BY SEATS
BY BODYTYPE
WHERE COUNTRY IN ('ENGLAND', 'ITALY')
HEADING
"REPORT 0"
END

-*-----------------------

TABLE FILE CAR
SUM
SALES
BY SEATS
ACROSS COUNTRY
WHERE COUNTRY IN ('ENGLAND', 'ITALY')
ON TABLE HOLD AS H001
END

TABLE FILE H001
PRINT *
HEADING
"REPORT 1A"
END

TABLE FILE H001
PRINT *
WHERE SALENGLAND EQ SALITALY
HEADING
"REPORT 1B"
END

-*-----------------------

TABLE FILE CAR
SUM
SALES
BY SEATS
BY BODYTYPE
ACROSS COUNTRY
WHERE COUNTRY IN ('ENGLAND', 'ITALY')
ON TABLE HOLD AS H002
END

TABLE FILE H002
PRINT
*
HEADING
"REPORT 2A"
END

TABLE FILE H002
PRINT
*
WHERE SALENGLAND EQ SALITALY
HEADING
"REPORT 2B"
END

Hi Francis

Interesting behaviour. I can almost understand why this happens, not sure if it has the element of least surprise though.

SALES is an integer field. In the hold file, the character used to show missing is '.'. This can be controlled by HNODATA. A basic comparison between a zero and a '.' becomes true. Wondering if this is because '.' translates to 0, or to blank, which in a numeric field is 0.

Converting Sales to Alpha, this will work straight off.

Or, keeping with Integer, or any numeric, the only approach seems to be to go back to basics and use a DEFINEd FLAG, FLAG/A1 MISSING ON NEEDS ALL = IF SALENGLAND EQ SALITALY THEN 'Y' ELSE 'N';

Or, setting HNODATA to -1, would work, but then missing data shows as -1. Not perfect.

So if there was a WHERE_MISSING_ON_NEEDS_ALL … type option, this would not occur.

Think you're right, Alan.
Because if you add in the last request'WHERE SALITALY NE MISSING AND SALENGLAND NE MISSING;' you then get the correct result. This means that the missing test is done correctly, but the eq test isn't quite correct, imho.

It looks like MISSING integer values get silently cast to 0 in equality tests.
That makes me wonder, if both are MISSING, is the result of the comparison still a match? What is the result of MISSING EQ MISSING?

I don't think the result of a comparison with MISSING should ever be true, just as it is with NULLs in SQL (http://www-cs-students.stanford.edu/~wlam/compsci/sqlnulls).
What would be the correct result for N EQ MISSING?

Is that FALSE?
If I don't have a pot of flowers and I want to know whether they're red, well, if they were red I could see them. They're not green or blue or any other colour either.

Or MISSING?
I'd love to tell you whether those flowers are red or not, but I don't have any flowers to compare so I can't answer that question at this time.

Or even UNKNOWN?
I don't know whether those flowers might have been red had I had them.

quote:

I don't think the result of a comparison with MISSING should ever be true

Ah, Philosophy 101.

The result of the FLAG above, with NEEDS ALL, is MISSING. With NEEDS SOME, is TRUE. So the NEEDS SOME, ignores the MISSING value, relying on the values that do exist, ergo there is a value so TRUE, whereas with NEEDS ALL, then there is no evaluation if any item does not exist, therefore becomes MISSING, or does not exist.

I believe if you look at the source data some have sales of Zero and some have sales of missing so the Zeroes are overriding the missing.

TABLE FILE CAR
SUM
SALES
BY SEATS
ACROSS COUNTRY
WHERE COUNTRY IN ('ENGLAND', 'ITALY')
ON TABLE HOLD AS H001
END

TABLE FILE H001
PRINT *
END

TABLE FILE CAR
PRINT
SALES
BY SEATS
ACROSS COUNTRY
WHERE COUNTRY IN ('ENGLAND', 'ITALY')
END

I'm sure you much more experienced focal pointers already thought of this, but a simple fix to me would be to compare apples to apples and make the

SET NODATA=0 

then use

   WHERE SALENGLAND EQ SALITALY AND (SALENGLAND NE 0 OR SALITALY NE 0 ); 

in report 2B.

Thanks very much for this conversation!

FYI, the NODATA character is for display only and has no impact on the EQ test. I deliberately did not use zero so I could see what's happening. I also tried changing the format from integer to Dnn to mimic my real-world program and that made no difference.

I ended up with this:

-* Include values that are not equal
WHERE SC_AMT1 IS MISSING OR SC_SMT2 IS MISSING OR SC_AMT1 NE SC_AMT2;

In my data, I know I will never have missing values in both columns.

This message has been edited. Last edited by: Francis Mariani,

You could also use function ASIS, which seems to know the difference between zero and missing values:

WHERE ASIS(SALENGLAND) EQ ASIS(SALITALY);

As a matter of interest - if you simply use a WHERE clause like:
WHERE SALENGLAND EQ 0;
you will get both zeroes and missing values.
I recall bumping into this previously, and it is in fact documented that way in the manual.
Just BTW - I tried the WHERE clause with ASIS as per Dan's suggestion on WF 8.0.01, and it isn't very happy with me:
(FOC263) EXTERNAL FUNCTION OR LOAD MODULE NOT FOUND: ASIS

I've always known ASIS to be a Dialogue-Manager-only function...

Hi Francis,
Yes, that is what I thought too.
Explains why it does not work then.