R help - Dec 2003 - read.spss question warning compression bias

Hello again

I have a file from SPSS in .sav format.

when I run

library(foreign)
cvar<-as.data.frame(read.spss("c:\\NDRI\\cvar\\data\\cvar2rev3.sav"))

I get a warning

Warning message: 
c:\NDRI\cvar\data\cvar2rev3.sav: Compression bias (0) is not the usual
value of 100. 

The data appear to be OK, but I am concerned.  

 (I tried searching the archives and the documenation for data import
export, but saw nothing).


Thanks as always

Peter

Peter L. Flom, PhD
Assistant Director, Statistics and Data Analysis Core
Center for Drug Use and HIV Research
National Development and Research Institutes
71 W. 23rd St
www.peterflom.com
New York, NY 10010
(212) 845-4485 (voice)
(917) 438-0894 (fax)

On Thu, 2003-12-11 at 09:44, Peter Flom wrote:
> Hello again
> 
> I have a file from SPSS in .sav format.
> 
> when I run
> 
> library(foreign)
>
cvar<-as.data.frame(read.spss("c:\\NDRI\\cvar\\data\\cvar2rev3.sav"))
> 
> I get a warning
> 
> Warning message: 
> c:\NDRI\cvar\data\cvar2rev3.sav: Compression bias (0) is not the usual
> value of 100. 
> 
> The data appear to be OK, but I am concerned.  
> 
>  (I tried searching the archives and the documenation for data import
> export, but saw nothing).
> 
> 
> Thanks as always
> 
> Peter

The error message appears to be coming from sfm-read.c in the package.
The particular code is at line 682:

  ext->bias = hdr.bias;
  if (ext->bias != 100.0)
    warning("%s: Compression bias (%g) is not the usual "
	    "value of 100.", h->fn, ext->bias);

I have not used SPSS, but I presume that as with other applications, it
can save the data files in either compressed or non-compressed formats.
>From reading the code in the above source file, there appears to be a
SPSS header structure that contains information on the nature of the
file and its contents.

One of the entries in the structure indicates whether or not the data
file is compressed. That "flag" is then stored in:

  ext->compressed = hdr.compressed;

at line 669. That value is either 0 (FALSE) or 1 (TRUE).

That information comes into play later on at line 1393, which is the
read_compressed_data function.

It would seem that if the data file is not compressed, which I presume
is the case with your file, the check of the compression bias at line
682 is superfluous.

Thus, if my read is correct, the fix (if one is needed) would be to add
a check in advance of the bias check code:

  if (ext->compressed)
  {
    ext->bias = hdr.bias;
    if (ext->bias != 100.0)
      warning("%s: Compression bias (%g) is not the usual "
              "value of 100.", h->fn, ext->bias);
  }


An additional question might be, if the file is not compressed, what is
the default bias value set by SPSS? If it is 0, then the check is
meaningless. On the other hand, if the default value is 100, whether or
not the file is compressed, then the warning message would serve a
purpose in flagging the possibility of other issues. Reasonably, that
setting may be SPSS version specific.

Hopefully, I am close. If so, your data should be correct and the
warning is just that, a "warning" and not an "error".


HTH,

Marc Schwartz

>So it would appear that if the above is correct, there is no user adjustment
to the bias value.
>The only scenario that I can envision is if the user SAVE's the
".sav" file in an uncompressed
>format, where the bias value **might** be set to 0.
>Perhaps a r-help reader with access to current SPSS manuals can confirm the
above.

The windows version 11.5.0 appears the same (I assume the negative sign on -99
was somehow dropped)

COMPRESSED and UNCOMPRESSED Subcommands

COMPRESSED saves the file in compressed form. UNCOMPRESSED saves the file in
uncom-pressed form.
In a compressed file, small integers (from ?99 to 155) are stored in one
byteinstead of the
eight bytes used in an uncompressed file. 

The only specification is the keyword COMPRESSED or UNCOMPRESSED. There are
noadditional specifications.

Compressed data files occupy less disk space than do uncompressed data files.

Compressed data files take longer to read than do uncompressed data files.

The GET command, which reads SPSS-format data files, does not need to specify
whetherthe files it reads are compressed or uncompressed.

Only one of the subcommands COMPRESSED or UNCOMPRESSED can be specified perSAVE
command. COMPRESSED is usually the default, though UNCOMPRESSED may bethe
default on some systems.

Ciao, Tom

_________________________________________________
 
Tom Mulholland
Senior Policy Officer
WA Country Health Service
Tel: (08) 9222 4062
 
The contents of this e-mail transmission are confidential and may be protected
by professional privilege. The contents are intended only for the named
recipients of this e-mail. If you are not the intended recipient, you are hereby
notified that any use, reproduction, disclosure or distribution of the
information contained in this e-mail is prohibited. Please notify the sender
immediately.


-----Original Message-----
From: Marc Schwartz [mailto:MSchwartz at medanalytics.com] 
Sent: Friday, 12 December 2003 3:56 AM
To: Thomas Lumley
Cc: r-help at stat.math.ethz.ch
Subject: Re: [R] read.spss question warning compression bias


On Thu, 2003-12-11 at 12:32, Thomas Lumley wrote:
> On Thu, 11 Dec 2003, Marc Schwartz wrote:
> >
> > An additional question might be, if the file is not compressed, what 
> > is the default bias value set by SPSS? If it is 0, then the check is 
> > meaningless. On the other hand, if the default value is 100, whether 
> > or not the file is compressed, then the warning message would serve 
> > a purpose in flagging the possibility of other issues. Reasonably, 
> > that setting may be SPSS version specific.
> >
> 
> I think the issue is that the format is not documented, so the author 
> of the code (Ben Pfaff) didn't know what a change in the value would 
> imply. If the file is apparently read correctly it seems that it 
> doesn't imply anything.
> 
> 	-thomas


Thanks for the clarification Thomas.

I did some searching of the PSPP site and found the following:

http://www.gnu.org/software/pspp/manual/pspp_18.html#SEC170

The compression bias is defined as:

flt64 bias;
        Compression bias. Always set to 100. The significance of this
        value is that only numbers between (1 - bias) and (251 - bias)
        can be compressed.
        

So it would seem to potentially impact aspects of the file compression data
structure, when compression is used.

I am not sure if the "Always set to 100" is unique to PSPP in how Ben
elected to do things. Presumably if that is always the case, even with SPSS, one
might reasonably wonder: why have it, if it does not vary?

It leaves things unclear as to under what circumstances this value would change.

I did some Googling and found the following text snippet from a presumably dated
SPSS manual for the syntax of the SAVE command:


SAVE OUTFILE=file 

[/VERSION={3**}] {2 } 

[/UNSELECTED=[{RETAIN}] {DELETE} 

[/KEEP={ALL** }] [/DROP=varlist] {varlist} 

[/RENAME=(old varlist=new varlist)...] 

[/MAP] 

[/{COMPRESSED }] {UNCOMPRESSED} 

**Default if the subcommand is omitted.


COMPRESSED and UNCOMPRESSED Subcommands 

COMPRESSED saves the file in compressed form. UNCOMPRESSED saves the file in
uncompressed form. In a compressed file, small integers (from
99 to 155) are stored in one byte instead of the eight bytes used in an
uncompressed file.

The only specification is the keyword COMPRESSED or UNCOMPRESSED. There are no
additional specifications.

Compressed data files occupy less disk space than do uncompressed data files. 

Compressed data files take longer to read than do uncompressed data files. 

The GET command, which reads SPSS-format data files, does not need to specify
whether the files it reads are compressed or uncompressed.

Only one of the subcommands COMPRESSED or UNCOMPRESSED can be specified per SAVE
command. COMPRESSED is usually the default, though UNCOMPRESSED may be the
default on some systems.




So it would appear that if the above is correct, there is no user adjustment to
the bias value. The only scenario that I can envision is if the user SAVE's
the ".sav" file in an uncompressed format, where the bias value
**might** be set to 0.

Perhaps a r-help reader with access to current SPSS manuals can confirm the
above.

Until demonstrated otherwise, it seems reasonable to leave the warning message
in place as a warning (as opposed to an error), though it might be helpful to
folks to add a comment to the read.spss help file on this for clarification. The
text might read:

"NOTE: You may receive the following message:

 Warning message: 
 FileName: Compression bias (X) is not the usual value of 100.

Where 'FileName' will be the file that you are reading and 'X'
will be a numeric value, possibly 0. This *may* be the result of reading an
UNCOMPRESSED SPSS file. It is recommended that you verify the integrity of your
imported SPSS data after using read.spss() if you receive this warning."


The wording is subject to change and of course, the integrity check should be
done under any circumstances... :-)

HTH,

Marc Schwartz

______________________________________________
R-help at stat.math.ethz.ch mailing list
https://www.stat.math.ethz.ch/mailman/listinfo/r-help