no warning if I/O buffer too small in pipe

[teuben]

In this example:
fitsccd tmp1.fits - | ccdstat -
for small files this works, as long as the file fits in memory, but there is no warning if not. then ccdstat will just report a 0 data array. Not good.

This should not happen, as filestruct allocated all data in memory when coming from pipe.

[teuben]

  ccdgen cube1 flat 1 1500,1500,1500; ccdfits cube1 cube1.fits
  ### Fatal error [ccdfits]: [image.c:323]: cannot allocate 18446744066349813248 bytes

this also fails, but it seem fitsio cannot handle files > 4GB

[teuben]

Converted the filestruct code to using off_t and size_t an odd (compiler?) bug came up where the following seemingly legal code failed to stop the while loop:

        size_t   len;

	while (--len >= 0) {			/*   loop copying data      */
	    *dat++ = *src++;			/*     byte by byte         */
	}

whereas the following code ran fine:

	for (size_t i=0; i<len; i++)
	  *dat++ = *src++;

maybe I need another eye on this @jcldc ??? Happy newyear JC !

Code is still in the "issue17" branch, not merged to master yet.

[teuben]

All of the examples given above now seem to work, except scanfits on huge fits files fails. fitshead works fine though. This must be an unrelated and scanfits specific error due to some never converted 32/64 bit code.

[teuben]

scanfits and the fits library module also needed a patch for using size_t instead of int. All example now workingl.

[jcldc]

Hi Peter,
an unsigned int variable cannot have a negative value. It leads to an underflow, which corrupt the memory, and involve an infinite while-loop and then crash. This is why while (--len >= 0) statement does not work.
Here is a simple example which reproduce the bug.

#include <stdio.h>
#define MAXLEN 10
int main()
{
    size_t len;
    int bsrc[MAXLEN], bdat[MAXLEN],i;
    int * src, * dat;
    
    len = MAXLEN;
    for (i=0; i<len; i++) {
        bsrc[i] = i;
    }
    src = bsrc;
    dat = bdat;
    printf("First len = %ld\n\n",len);
#if 1    
    while (--len >= 0) {
        printf("-> len=%lu",len);
        *dat++ = *src++;
    }
#else
    for (size_t i=0; i<len; i++)
	    *dat++ = *src++;
#endif    
}

[jcldc]

Actually, if arithmetic underflow occurs, such as when decrementing past zero, all bits in size_t flip to one, representing the maximum unsigned value, not a negative number.

[jcldc]

Actually, if arithmetic underflow occurs, such as when decrementing past zero, all bits in size_t flip to one, representing the maximum unsigned value, not a negative number.

I have updated the example code above, to display the real unsigned int value of len :
printf("-> len=%lu",len);

[teuben]

ah, indeed, i need to change the %ld to %lu in the code! In Issue 17 was merged already, so I''ll do that in the master branch.

[teuben]

I benchmarked the while() and for() loop, using the intel timers(3NEMO) routines. For small values of the array (say < 1e4) for() is just a tad slower than while(), but for much larger values of the array, the while() loop can be 2-3 times faster.