Questions on General Software Design.

[kren.Z]

I've got a few questions on general software design, bare in mind that I'm a horrible programmer.


1. Reading/Writing Data from an external file (with source). I'm new to understanding binary format, any recommendations for tutorials would be greatly appreciated.  Here is what I'm working with, http://icculus.org/homepages/phaethon/q3a/formats/md2-schoenblum.html. I understand that the C headers contain all declared variables, but my problem is how do i read and parse that data so it is accesible (some type of array)? Is this the general approach to reading any type of external file (.jpg, .gif, .blah, dat etc etc) since all files are essential binary?





2.Reading/Writing Data from an external file(without source code). Now what if i don't have the source like the .md2 file? how would i go about reading the headers and source?



sorry if this makes little sense, i will elaborate further if needed

[peewee_RotA]

You'll have to elaborate because you touched on 3 broad subjects. Header files, IO Streams, and reading and displaying formated and binary files.

[kren.Z]

Thanks peewee, you hit what I'm having trouble with right on the nose. My general questions are a bit broad.


Since essentially all the files on a computer system are binary, it should be possible to Read/Write any file using binary format (even if it doesn't release the source like the .md2), correct?

Now with md2 project. I've declared all my variables the same as those in the md2 header. and i've loaded the data. Now how do i parse each variable from the md2 data to my variables, since they are integers they should be a specific amount of bytes etc...

[kren.Z]

oh and could someone explain bit significance and byte ordering like little endian and so on...

[reaper]

little endian vs big endian, is the order bits are stored in memory.  for instance if you stored an ip address in memory, is it read from left to right, or right to left.  this is dependant on the processor architectures, how one maps the memory addresses via the processor registers.

say you wanna take a text file, and parse data to remove windows line feed characters from a file.  you could use perl, which would be one line in your program, or use a utilitiy written in C (dos2unix).  the latter being 100+ lines of code, the perl program being one line.  C is good to use for bit and byte manipulation, managing memory to access data via pointers to memory locations, instead of copying data in memory, when you pass information between functions.  this can be very important when you're engineering a device, or just want to make your program run efficiently.  so depending on what type of file you are parsing, you may wish to use a different tool.

it looks like the md2 file has a format, and they even give your byte offsets, so you can decode the file.  everything on a computer is binary information in a sense, but if it wasn't human readable somehow doing practically anything would be very difficult.  say you know you have X value, at the 4th byte in the file,  you could code something in C to look for that value, change it, etc.  maybe someone else will provide some more insight

[kren.Z]

Thank you so much reaper 

[kren.Z]

what do you mean by offsets?

[reaper]

offset is a coordinate from a reference (like the beginning of a file).  take a look here, it looks like there is some code toward the bottom to read md2 files.  it also contains the files structure, so you could write that code, without reverse engineering the format.  manipulating the md2 file is a good question for quake programmers

doh, here's the link
http://tfc.duke.free.fr/coding/md2-specs-en.html

[math]

Hi!

little endian vs big endian, is the order bits are stored in memory.

That's just one half of the truth. Little-Endian and Big-Endian
also refer to (and that is usually the case) the order BYTES are
stored in a 16-, 32- or 64-bit word/register/variable.

you could use perl, which would be one line in your program, or use a
utilitiy written in C (dos2unix).  the latter being 100+ lines of code,
the perl program being one line.

In C (2 lines):

Code: [Select]

#include <stdio.h>
int main(int c, char **a) { for (; (c = fgetc(stdin)) - EOF; c - 13 ? putchar(c) : 0); return (0); }

In i386 asm (AT&T-syntax) with BSD-syscalls (1 line ;-)):

Code: [Select]

.globl _start;_start:pushl $1;pushl $c;pushl $0;mov $3,%eax;call kernel;cmpl $0,%eax;jle end;cmp $13,c;je _start;pushl $1;pushl $c;pushl $1;movl $4,%eax;call kernel;jmp _start;end:pushl $0;movl $1,%eax;call kernel;kernel:int $0x80;ret;.data;.lcomm c,1



Best regards,
math

[reaper]

regarding the program in C you did in one line, it is not the same program which is 100 lines - it doesn't accomplish the same things.  also the line (2 lines) are longer, and less human readable.

the assembly is obviously inefficient, and not a good choice to code something like that, and that's what I was pointing out, the various options for humans.  also look at the lines you wrote, it's a lot easier in perl, and you're a lot less worried about moving bits around.  why should I be worried about the stack, when I don't care about the stack, I want to parse text?

anyways, good information.

 

[quadz]

regarding the program in C you did in one line, it is not the same program which is 100 lines - it doesn't accomplish the same things.  also the line (2 lines) are longer, and less human readable.

Yes, it's obfuscated C code.  But it does remove carriage return characters from the input...

the assembly is obviously inefficient

To clarify... it's inefficient in terms of programmer time... But it's likely very efficient in terms of the bytesize of the compiled binary.


Regards,

[QwazyWabbit]

oh and could someone explain bit significance and byte ordering like little endian and so on...

Endianness is a property of the CPU architecture. It can refer to bits or bytes, although it's rare to see it referred to bits these days. In the days of minicomputers like the PDP-8 bit 15 was the least-significant bit (the one on the right) and had the value 2^0 and bit 0 had the value 2^15 and was on the left. (Left and right were the on the register panel and the toggle switches below it.) The PDP-8 was actually a 12-bit machine (bits 0 thru 11) but had extensions on the memory address to add 3 more bits to the most significant end of the address. The PDP-8A control panel had switches on the front panel for toggling in the programs. This front panel was later emulated on the MITS Altair 8800 that had a Intel 8080 microprocessor in it. In the case of the microprocessors, bit 0 was on the right, value 2^0 and bit 8 or bit 15 were on the left for registers or addresses. The endianness of the bit values were the same but when referring to the bit number the endianness was reversed. I don't think the PDP-8 had any bit-address mode instructions but if you referred to bit 7 in PDP-8's it's value was 2^8 or 256 vs. 2^7 or 128 in 8080's or any microprocessors that came later.

Little endian bit index order and enumerating them from right to left makes more sense because the bit address (offset) corresponds to the power of 2 it represents. Offset 0 is 2^0, offset n is 2^n.

Offset refers to the position of a byte in a file or memory block. The first byte has offset zero, or is the zeroth byte in the block, then offset 1, 2, 3 , etc. until offset n-1, n for any n items in a list. An offset refers to the order of the items in a list, the list could contain bytes, words, or double-words or could even be structures of bytes. Programmers always call offset of the first item in a list as offset zero. Some older computer languages allow counting to begin at 1 instead of 0.

Endianness in byte order refers to the way the bytes are stored in memory or even transmitted serially. If ABCD represents 4 bytes, then on little-endian machines they are stored in memory as DCBA where:

Offset    Byte
   0           D
   1           C
   2           B
   3           A

On big-endian machines they are stored as:

Offset    Byte
   0           A
   1           B
   2           C
   3           D

This is great when sending bytes serially since you can walk right down the list and they will be transmitted exactly as they are stored in memory as bytes and they will "read" left to right in this fashion. This is why big-endian was chosen as network byte order in the early days of the Internet. (The first networked computers were big-endian.)

This doesn't apply to byte strings however. Strings are always stored in memory from lowest to highest address first character to last character regardless of the endianness of the computer. It's only when you start talking about multi-byte values that it gets interesting.

Take the 16-bit example. If AB and CD represent the bytes of two 16 bit integers, the little endian machine will store them as:

Offset    Byte
   0           B
   1           A
   2           D
   3           C

The big-endian will store them as:

Offset    Byte
   0           A
   1           B
   2           C
   3           D

Storing or transmitting them from a big endian machine to a little endian machine must take this into account and they must know the nature of the data they are trying to transmit.