GIDForums  

Go Back   GIDForums > Computer Programming Forums > C Programming Language
User Name
Password
Register FAQ Members List Calendar Search Today's Posts Mark Forums Read

 
 
Thread Tools Search this Thread Rate Thread
  #1  
Old 28-Sep-2009, 02:24
gabatwork gabatwork is offline
New Member
 
Join Date: Sep 2009
Posts: 3
gabatwork is on a distinguished road

Writing data efficiently


Dear all I am a new user with some problems in programming.
I wrote a simple programme whose goal is to perform a data analysis to find some outliers. Everything work good except when I come to writing the remaining data into a file: it takes really a lot of time (almost 7 minutes).

So I tried a smaller programme (which I will post here) just to understand the problem: why it is so slow in printing into a file?? What am I doing wrong??

CPP / C++ / C Code:
#include
#include
#include
#include
#include "stat.h"

int numlines,i,size;
char cestino2[200];
void *malloc(size_t size);

int main(int argc, char *argv[]) {
float *Up, *Delta;
double *Est, *Nord;
FILE *in, *out;
if((in = fopen("dati.dat","r")) == NULL)
printf ("The file does not exist");
out = fopen("output.dat","w");
while(feof(in) == 0){
fgets(cestino2, 100, in);
numlines++; }
rewind(in);
size = numlines-1;
Est = (double *) malloc(size*sizeof(double));
Nord = (double *) malloc(size*sizeof(double));
Up = (float *) malloc(size*sizeof(float));
Delta = (float *) malloc(size*sizeof(float));

for(i = 0; i < size; i++) {
fscanf(in,"%lf%lf%f%f", &Est[i], &Nord[i], &Up[i], &Delta[i]);
}

for(i = 0; i < size; i++) {
fprintf(out,"%.3lf\t%.3lf\t%5.2f\t%5.2f\n", Est[i], Nord[i], Up[i], Delta[i]);
}

fclose(in);
fclose(out);

return 0;
}

The program red a file of 1500000 points and took almost 7 minutes to write them down in a new file (of 50 Mb)

Any help is accepted
Thank you in advance

Gabatwork
Last edited by LuciWiz : 28-Sep-2009 at 06:18. Reason: Please insert your C++ code between [cpp] & [/cpp] tags
  #2  
Old 28-Sep-2009, 04:36
gabatwork gabatwork is offline
New Member
 
Join Date: Sep 2009
Posts: 3
gabatwork is on a distinguished road

Re: writing data efficiently


Dear all,

It's always me. I have done some trials, compiling the source file with Visual c++ (microsoft). And this time it works much more faster!!!
So the problem moves from "what is wrong in my code" to "what is wrong in my compilation procedure??"

For the compilation of the program I use gcc. I have done a makefile which is as follows:

all : stat.o leggo.o leggo

stat.o : stat.c
gcc -g -std=c99-Wall -c stat.c

leggo.o : leggo.c
gcc -g -std=c99 -Wall -c leggo.c

leggo : leggo.o stat.o
gcc -g -std=c99 -Wall -o leggo.exe stat.o leggo.o

Then I just type in the terminal "make all" and everything is done.

Thank you in advance.
regards

Gabatwork
  #3  
Old 28-Sep-2009, 11:02
davekw7x davekw7x is offline
Outstanding Member
 
Join Date: Feb 2004
Location: Left Coast, USA
Posts: 5,218
davekw7x is a splendid one to beholddavekw7x is a splendid one to beholddavekw7x is a splendid one to beholddavekw7x is a splendid one to beholddavekw7x is a splendid one to beholddavekw7x is a splendid one to beholddavekw7x is a splendid one to behold

Re: writing data efficiently


Quote:
Originally Posted by gabatwork
...I have done some trials...
I wrote a little test program to see how long it takes to write 1,500,000 lines with four floating point numbers on each line:

CPP / C++ / C Code:
/* gendata.c */
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
int main()
{
    int i;
    int numpoints = 1500000;
    time_t starttime, endtime;
    char *outname = "dati.dat";
    FILE *outfile = fopen(outname, "w");
    if (!outfile) {
        printf("Can't open file %s for writing.\n", outname);
        return EXIT_FAILURE;
    }

#ifdef __GNUC__
    printf("%s: GNU compiler %d.%d.%d\n", __FILE__, 
            __GNUC__, __GNUC_MINOR__, __GNUC_PATCHLEVEL__);
#endif

#ifdef __BORLANDC__
    printf("%s: Borland compiler %d.\n", __FILE__, __BORLANDC__);
#endif

#ifdef _MSC_VER
    printf("%s: Microsoft compiler %d.\n", __FILE__, _MSC_VER);
#endif


    starttime = time(0);

    for (i = 0; i < numpoints; i++) {
        fprintf(outfile, "%f %f %f %f\n", 
                (double)i, (double)(i+1), (float)(i+2), (float)(i+3));
    }

    endtime = time(0);
    printf("Elapsed time = %.0f seconds.\n", difftime(endtime, starttime));

    fclose(outfile);
    return 0;
}


Results on my old Windows XP workstation:

Microsoft compiler from Visual C++ 2008 Express Edition (Free download.)
Code:
gendata.c: Microsoft compiler 1500. Elapsed time = 12 seconds.

Borland bcc32.exe from Turbo C++ (Free download.)
Code:
gendata.c: Borland compiler 1410. Elapsed time = 8 seconds.

With the cygwin port of gcc (Open Source---free download.)
Code:
gendata.c: GNU compiler 4.3.2 Elapsed time = 243 seconds.

With the mingw port of gcc, used in dev-cpp (Open Source---free download.)
Code:
gendata.c: GNU compiler 3.4.2 Elapsed time = 17 seconds.
On my Linux machine (with a somewhat faster processor), using gcc 4.1.2 elapsed time was something like 9 seconds.

Conclusion: File output of the cygwin port of gcc is considerably slower than the "native" compilers from Microsoft and Borland that I have. The mingw port used in dev-cpp uses Microsoft I/O routines and has a run time that is at least the same order of magnitude as the "native" compilers.

Bottom line: If it's my decision, I just about never use my Windows machine for "useful" work, but if I just "had to," and if file I/O speed was important, I would (probably) try something other than the cygwin gcc.


Regards,

Dave
Last edited by davekw7x : 28-Sep-2009 at 12:04.
  #4  
Old 28-Sep-2009, 11:40
gabatwork gabatwork is offline
New Member
 
Join Date: Sep 2009
Posts: 3
gabatwork is on a distinguished road

Re: writing data efficiently


Dear Dave,

Your explanation was perfect!
Thank you very much for the reply!!!!
I was exactly using cygwin on a windows machine. That was the reason.
This forum is great :-)
Regards

Gabatwork
  #5  
Old 28-Sep-2009, 12:03
davekw7x davekw7x is offline
Outstanding Member
 
Join Date: Feb 2004
Location: Left Coast, USA
Posts: 5,218
davekw7x is a splendid one to beholddavekw7x is a splendid one to beholddavekw7x is a splendid one to beholddavekw7x is a splendid one to beholddavekw7x is a splendid one to beholddavekw7x is a splendid one to beholddavekw7x is a splendid one to behold

Re: writing data efficiently


Quote:
Originally Posted by gabatwork
...This forum is great...

By seeing other people's problems I get a chance look at things from other points of view, and sometimes I can even get (and give) an explanation for some of life's little mysteries.

That's why I keep coming back: I always learn something here (always).


Regards,

Dave
 
 

Recent GIDBlogToyota - 2009 May Promotion by Nihal

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Memory leak when nothing is happening... How can I even debug this ? Algar MS Visual C++ / MFC Forum 10 19-Nov-2007 08:17
[Include] Doubly-linked List dsmith C Programming Language 6 14-Apr-2006 14:12
Strange C++ code memory leakage problem gaoanyu C++ Forum 7 04-Nov-2005 09:09
[GIM] Data Module - Contact dsmith C Programming Language 2 27-Jan-2005 17:30
[CONTEST?]Data Structure Test dsmith C Programming Language 2 06-Jun-2004 16:13

Network Sites: GIDNetwork · GIDWebHosts · GIDSearch · Learning Journal by J de Silva, The

All times are GMT -6. The time now is 02:44.


vBulletin, Copyright © 2000 - 2009, Jelsoft Enterprises Ltd.