File Read Problem. Size limit?

jsmain · #1 10-Jul-2006, 13:41

I have a binary file approx 6 meg in size.
I'm trying to parse the records from this file, but after x number of records, it crashes.

CPP / C++ / C Code:

#include "stdafx.h"
#include <stdio.h>
#include <iostream.h>
#include <fstream.h>

#pragma pack(1);
struct 
{	int Size;
}File;
struct RECORD {
	int Valid:8;
	char Comment[600];
};
#pragma pack();

int main()
{
	FILE * pFile;
	pFile=fopen("testfile.bin","rb"); //6 meg bin file.
	fseek(pFile,16,SEEK_SET);
	fread(&File, sizeof(File),1,pFile);
	printf("\tSize: %i \t\t",File.Size);

	fseek(pFile,20,SEEK_SET);
RECORD Record[1712];
	fread(&Record, sizeof(Record),1,pFile);
	
printf("\n\nFile Position: %i \n",ftell(pFile));
	fclose(pFile);
	return 0;
}

As the code is listed it works, but if I add just a single record, it no longer prints data to the screen, and if I make it 1716, I get the following error....

---------------------------
RFile.exe - Application Error
---------------------------
The exception unknown software exception (0xc00000fd) occurred in the application at location 0x00401bc7.

Click on OK to terminate the program
Click on CANCEL to debug the program
---------------------------
OK Cancel
---------------------------

Can someone explain why I cannot access the entire file? Make Suggestions?
I thought it was memory, but it runs the same on a 256 meg machine as it does on a 512 meg machine.

Many thanks!

WaltP · #2 10-Jul-2006, 16:04

CPP / C++ / C Code:

#include "stdafx.h"
#include <stdio.h>
#include <iostream.h>
#include <fstream.h>

#pragma pack(1);
struct 
{	int Size;     /// why make this a structure?
}File;

struct RECORD {
	int Valid:8;
	char Comment[600];
};
#pragma pack();

int main()
{
	FILE * pFile;
	pFile=fopen("testfile.bin","rb"); //6 meg bin file.
	fseek(pFile,16,SEEK_SET);
	fread(&File, sizeof(File),1,pFile);
	printf("\tSize: %i \t\t",File.Size);

	fseek(pFile,20,SEEK_SET);
        RECORD Record[1712];    // you are writing C -- this needs to be at 
                                          //the top of your function
	fread(&Record, sizeof(Record),1,pFile);
	
printf("\n\nFile Position: %i \n",ftell(pFile));
	fclose(pFile);
	return 0;
}

I'm not sure. How many bytes is Record when you create it? I calculate over 1M for your posted definition.

jsmain · #3 10-Jul-2006, 17:16

Quote:

Originally Posted by WaltP

CPP / C++ / C Code:

#include "stdafx.h"
#include <stdio.h>
#include <iostream.h>
#include <fstream.h>

#pragma pack(1);
struct 
{	int Size;     /// why make this a structure?
}File;

struct RECORD {
	int Valid:8;
	char Comment[600];
};
#pragma pack();

int main()
{
	FILE * pFile;
	pFile=fopen("testfile.bin","rb"); //6 meg bin file.
	fseek(pFile,16,SEEK_SET);
	fread(&File, sizeof(File),1,pFile);
	printf("\tSize: %i \t\t",File.Size);

	fseek(pFile,20,SEEK_SET);
        RECORD Record[1712];    // you are writing C -- this needs to be at 
                                          //the top of your function
	fread(&Record, sizeof(Record),1,pFile);
	
printf("\n\nFile Position: %i \n",ftell(pFile));
	fclose(pFile);
	return 0;
}

I'm not sure. How many bytes is Record when you create it? I calculate over 1M for your posted definition.

There is actually more in the size struct than Size, but it probably isn't necessary. Just easier to read, and I use size a few times in the actual program. This was pared down for simplicity.

As stated in the remarked code, the file is over 6 meg.

I'm confused why the " RECORD Record[1712];" should be at the top of the script. Shouldn't it be initialized just prior to the file read concerning that data? The 1712 value will actually be a read value once I can determine why I can't read the entire file currently.

WaltP · #4 10-Jul-2006, 17:24

Quote:

Originally Posted by jsmain

There is actually more in the size struct than Size, but it probably isn't necessary. Just easier to read, and I use size a few times in the actual program. This was pared down for simplicity.

Oh. OK then... That must be the reason for the fseek() I suppose...

Quote:

Originally Posted by jsmain

As stated in the remarked code, the file is over 6 meg.

Yes, but there's a limit to the size of buffer you can declare. Can't you read a record and process it? Then read the next?

Quote:

Originally Posted by jsmain

I'm confused why the " RECORD Record[1712];" should be at the top of the script. Shouldn't it be initialized just prior to the file read concerning that data? The 1712 value will actually be a read value once I can determine why I can't read the entire file currently.

In C++ yes it can go where you have it. But in C it should be with the other variable definitions, just after main()

CPP / C++ / C Code:

int main()
{
    FILE * pFile;
    RECORD Record[1712]; 
...

davekw7x · #5 10-Jul-2006, 19:10

Quote:

Originally Posted by jsmain

I thought it was memory, but it runs the same on a 256 meg machine as it does on a 512 meg machine.

Regardless of the amount of physical memory on your machine, all compilers (linkers, actually) impose some limit on the stack size. The stack is used to hold argument values and return addresses for function calls and is used as memory for all "automatic" variables (local variables declared inside of functions, including main()). Default stack sizes for Borland and Microsoft compilers that I am familiar with are about one Megabyte. Note that with a struct size of over 600 bytes and a declaration of 1700 or so of these beasties, you are knocking at the one megabyte door. (As I seem to recall, GNU compilers/linkers have default stack sizes of 8 Megabytes, but it may vary.)

You could look into your compiler documentation to see whether there is a command-line switch to set a larger stack size, but be aware that there may be some limit on how large you can make it.

Note that stack overflow due to declaring more memory than is available on the program stack may result in silent failures (the program simply returns to the command line without any visible action by the program itself), or may result in various kinds of access violation errors (sometimes called seg faults). I have seen attempts by compiler writers to add optional switches to generate code that detects stack overflow, but I don't know of any good solution. And note that the C language itself is deliberately ignorant of all hardware and other physical attributes of your system, so there is no "standard" way of detecting or solving this problem.

Whenever I have a program that I know is going to need a large amount of memory, I generally use dynamic memory allocation for large arrays (malloc() or one of its cousins for C, and new for C++). By the way: Is this really a C++ program? You have a couple of headers that prevent its being compiled as C (in addition to the implicit typedef on your structs, which won't work with C), but you show absolutely no C++ constructs or functions. So: what's up with that?

If it's a C++ program, then the headers should be <iostream> and <fstream>, not <iostream.h> and <fstream.h>, but since you aren't using anything from these, I wonder why they are there.

Anyhow, back to your problem: If you use dynamic memory allocation, the space is obtained from the operating system, and programs on 32-bit systems typically can have up to a couple of Gigabytes of memory (yes, more memory than is actually physically present in your system) --- it's "virtual" memory, whose contents are swapped into and out of physical RAM onto and off of disk storage (called the "swap" file).

Anyhow, try malloc() or new to get your array of structs and let us know how it works out.

Regards,

Dave

P.S. You might mention what compiler you are using, since it might make a difference if someone else wants to try to duplicate your problem.

WaltP · #6 11-Jul-2006, 02:04

I didnt' mention malloc() because I think (may wrongly I admit, but) you should look into reading a record at a time rather than the entire file. In general, a requirement to read an entire 6M file into memory is probably a poor design. Just something to think about...

jsmain · #7 11-Jul-2006, 07:53

The compiler is the standard VC++6.
The file is a relational type database. I must read all the records to search the database properly. In a sense, it's much like a CATrax Music database.... You have an album, and you have tracks assigned to that album.

The header info is strictly info I have gotten online. Without explanation, I don't necessarily understand it, and the code was a few years old. Couldn't find anything newer. I deleted alot of code to make it as simple as it is, so I likely did not realize I left in the unnecessary header info.

OK, So I'm a newbie!

I asked for explanations didn't I...

I desperately want to understand what I'm doing.

Ok, so I'm knocking on the 1 meg door! I get that impression also, as it reads to approx 1033680 before failing me. In this particular case, I need to process half the file, (3 meg) for the album information, and the other half for track information and album assignments.

The problem arises when the database grows. This 6 meg file houses info for about 9000+ tracks. I know a couple that have a database that includes over 300,000 tracks. That will be substantially larger than my 6 meg file.

I know my example is very primitive, but it was necessary to keep it to the point. There is a maximum number of albums listed in the database that gets me to the start of the tracks section, that again has a max number of records for containment. There is a reference ID in each track that tells me what album it belongs to.

If there is a better way to process this, I'm all ears(or eyes)

The overall goal is a win32 application that displays the content of this database, while remaining completely searchable.

I am very aware of the Swap File, and Virtual memory.

davekw7x · #8 11-Jul-2006, 11:11

Quote:

Originally Posted by jsmain

The compiler is the standard VC++6.
The file is a relational type database. I must read all the records to search the database properly. In a sense, it's much like a CATrax Music database.... You have an album, and you have tracks assigned to that album.

I don't mean any disrespect, but I'm not sure that a database program capable of handling 300,000 anythings is a suitable project for someone who is not proficient in at least the rudiments of C or C++ programming. Decisions that affect program design (such as whether it is really necessary to have the entire data base resident in program memory) are usually more effective when made in some kind of context based on experience.

Quote:

Originally Posted by jsmain

The header info is strictly info I have gotten online. Without explanation, I don't necessarily understand it, and the code was a few years old. Couldn't find anything newer. I deleted alot of code to make it as simple as it is, so I likely did not realize I left in the unnecessary header info.

And, getting bits and pieces of programs that you don't understand isn't necessarily a good way to learn. For example, what do you think the #pragma pack(1); does? (By the way, it should probably be #pragma pack(1) without the semi-colon.) Well, the answer is: it depends. For GNU and Borland compilers that I use from time to time, it causes the size of a RECORD structure to be 601 (without the #pragma, the size is 603). For Visual C++ version 6.0 it does nothing that I can see (the size of a RECORD is 604 with or without that particular pragma). What is the purpose of the struct? Is its first member supposed to have size 1 or size 4? Or what? If it's supposed to have a size of 1, then just use a char or unsigned char. If it's supposed to have a size of 4, then what's the purpose of the pack() stuff?

See what I mean about getting other people's stuff for starters?

Quote:

Originally Posted by jsmain

I desperately want to understand what I'm doing.

I respect that more than you could possibly know. My advice would be to start with tutorials or books with simple examples (Not all tutorials or books' examples are really good code, by the way, so you still might have to ask about stuff that doesn't seem to work as it should.)

Quote:

Originally Posted by jsmain

If there is a better way to process this...

Anyhow, for your problem of memory size:

If you are going to have data bases on the order of a few tens of megabytes, then having the entire data base in memory at one time is certainly practical with dynamic memory allocation (using malloc() or new). For larger data bases, you probably should consider techniques where your program keeps parts of the data base in memory and coordinates its own swapping to get stuff in and out, rather than getting a huge block from the operating system (with malloc() or new) and, therefore, claiming a very large percentage of your system's resources for this one program.

This is a whole new layer of programming complexity.

For example sorting data bases that are too large to fit into memory all at once is a different breed of cat than the usual compiler library-supplied sort functions. This is nothing new in the field of computer science or computer programming (it's sometimes called "external sorting"), but it is not real easy to grasp for people with no experience.

On the other hand, if you aren't familiar with the differences between malloc() in C and malloc() and new in C++, then how could anyone help you figure out a "better way" to process this?

For Standard C++, something like the following would be a way to get a large block of memory from the system:

CPP / C++ / C Code:

    RECORD *Record;
    try {
        Record = new RECORD[10000000];
    }
    catch (bad_alloc) {
       cout << "Memory allocation failed for RECORD[10000000]" << endl;
        // do something here to handle the error: maybe abort the program
    }

In fact, this doesn't work with Visual C++ version 6.0 (my version), and something like the following would be required

CPP / C++ / C Code:

    Record = new RECORD[10000000];
    if (!Record) {
        printf("Record is NULL\n");
        return 0;
    }

Visual C++ version 6.0 was developed and released at about the same time as the C++ standard was being written and approved, so there are a number of non-standard behavior patterns that remain in that version and might (or might not) be fixed in later releases.

You would have to experiment to see which would be appropriate for your system.

Regards,

Dave

jsmain · #9 11-Jul-2006, 11:34

Dave, I appriciate your comments.
I have been hen peck learning this for several years. I know of no better way of learning this than hands on. Today I'll mess with it, and then I might set it down for a couple months. Not the best way to deal with learning as I forget everything I learned, and have to relearn it each time I pick it up, however, my schedule doesn't permit me to stay on task for any length of time.

I have several books covering the basics. None adaquately cover File IO. If you could point one out, I would appriciate that even more.

Without schooling, how does one learn without asking questions?
My appologies if I am wasting anyones time here, including your own.

Yes! I want to learn! If that means finding a better source for my learning, then so be it. Books don't necessarily transmit a full understanding of the topic at hand, simply because they are a one way media. Forums are 2 way. Asking until one understands. Ask the right person, and you'll find an understanding.

I cannot, and will not wait several more years. I do not need to know everything, just what I'm trying to code today.

jsmain · #10 11-Jul-2006, 12:08

As a side note, I have been able to parse the file nearly completely, and save a template in 010 edit. I understand the file itself. I simply need to create a file to access it as the original application does. Whether it be the entire file at once, or a record at once. I would prefer smaller bites, but can I do that with a relational DB?

Currently, this particular memory problem is my roadblock. I'll remove each road block as I come across them.

To respond to the pragma pack comment, I recieved that as a tip from someone on the code project forums.

This is really a chicken or egg delemma!
How do I get the chicken without the egg, or the egg without the chicken.

Experience comes from what?

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Rate This Thread
Rate This Thread:

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Airport Log program using 3D linked List : problem reading from file	batrsau	C Programming Language	11	29-Feb-2008 08:44
100MB free webspace, cgi, php and with NO file size limit!	Zero	Free Web Hosting	0	27-Sep-2005 13:38
Determing size of a binary file	Dream86	C++ Forum	7	01-Jun-2005 11:10
How to detect end of file with read() function call?	nkhambal	C Programming Language	6	12-Oct-2004 02:08
Yet another CD burner problem: Lite-On LSC-24082K	Erwin	Computer Hardware Forum	1	22-May-2004 12:28

File Read Problem. Size limit?

Re: File Read Problem. Size limit?

Re: File Read Problem. Size limit?

Re: File Read Problem. Size limit?

Re: File Read Problem. Size limit?

Re: File Read Problem. Size limit?

Re: File Read Problem. Size limit?

Re: File Read Problem. Size limit?

Re: File Read Problem. Size limit?

Re: File Read Problem. Size limit?