Fix downloaded files that are the wrong mimetype

[Chargers]

Does anyone know how to fix files that are downloaded with a web browser and interpreted as the wrong mimetype?
I have some files that are saved as html files, but are really binary.  The data is there; it is formatted wrong.... like it is on the wrong boundary or something....

and no, I can't download them correctly, they are not online anymore.

[reaper]

mime-type doesn't matter.

mime-type is an HTTP-header content-type

some web servers might not serve files if there's no mime type for resource.extension, but if you got the file, you have the file, just open it in whatever program.  the mime-type is used for like: resource is text/html, display in browser, resource is video/mp4, open with browser helper object X.  you could always just open the video in a video player if the mime-type was wrong.

[Chargers]

I appreciate the input, but unfortunately, that is not true.
I actually happen to have an example file that has been correctly downloaded as well as an incorrectly downloaded version of the same file.  So it is easy to see the differences.

Code: [Select]

vi ascii display of binary data:

corrupted:
PK^C^D^T^H$qA)»½ý>^^^[^A6<95>^C VAP_SIN2.DM2Äýks^\G<96>® ®¸Ì$^E<96>*K U<91>ê^Vö^^`,°·<94>C<85>µ<9d>^?5¿q>Í×^P-©9
;ÂÞCíSdµP])^USb&<81>rü}×ëqË^DHU÷Ø<81><95>øÔò<88><8c><8b><87>_<96>/_¾üÿ<99>'Ib<96>^XþÍmb^_ó<97>÷þ^?iÛ4'óÔe<93>¼måÈ?<89>åoå^S¿^^Yܦe
Þ¤e^A6âûÀ<ð×À£Àuà £À«Àià uà <81>ÿÎlÈY¸G±Mç^SÈ^FÙi=ö<9f><83>éc<86>ãµx¢óË>qÞV¼ÑûE9
ϱÁs<84>ç}<8e>ëlÒ²Ò{U}9òªc­|¨oü}êkQé<95>çÛyåÏ7^W+ñ!<99>+ý^AÏ<9f>^W<87>bN<9e><8b>ó"Óu^R1KËK¤§iÙxzÌ<8f><93>
Ýï<8d>òE÷+<97>:Oés<9d>^WÞßù|t~Ì×úWñ]Âß×7 <9e>'^\Ï\^F­Ç^[q<9d>ÍÉ<95>ÎÛ<84>ß<81>W<89>^_ß<8a><96>è¾<99><98>èy^S½<97>ßÏ
çü^]ß<9b>×^A^[=<97>éú8<8f>ùE<96>=ÃŽ<8b>^CòÜó;9)¦ä<99>Ã’OÀðûÏÅÇäUrM6IÅû[ráï<9f>,^Cçá¹<96><96>âù²uíϹ®s<94>¿<84>rà Rò<8f>õ<81><98>+Ý<8f>¯<8a>4^C«"Éæ¤Qn<82>ÌëÃ
Î<91>O<81><97>Kg%¾xãü¶R:¹íø^]~ï2ÉëÝv

uncorrupted:
PK^C^D^T^@^@^@^H^@$qA)»½ý>^^^[^A^@6<95>^C^@^L^@^@^@VAP_SIN2.DM2Äýks^\G<96>®^K®¸^@ÃŒ$^E<96>*K^MU<91>ê^Vö^^`,°·<94>C<85>µ<9d>^?5¿q>Í×^P-©9^L;ÂÞCíSdµP])^USb&<81>^@rü}×ëqË^DHU   [etc.]


od -c | sdiff output of first 8 lines:

0000000   P   K 003 004 024  \b   $   q   A   ) 273 275 375   | 0000000   P   K 003 004 024  \0  \0  \0  \b  \0   $   q   A 
0000020 001   6 225 003       V   A   P   _   S   I   N   2   | 0000020 375   > 036 033 001  \0   6 225 003  \0  \f  \0  \0 
0000040   P 304 375   k   s 034   G 226 256     256 270 314   | 0000040   P   _   S   I   N   2   .   D   M   2 304 375   k 
0000060   *   K       U 221 352 026 366 036   `   , 260 267 2 | 0000060 226 256  \v 256 270  \0 314   $ 005 226   *   K  \r 
0000100 265 235 177   5 277   q   > 315 327 020   - 251   9   | 0000100 026 366 036   `   , 260 267 224   C 205 265 235 177 
0000120   ; 302 336   C 355   S   d 265   P   ]   ) 025   S   | 0000120   > 315 327 020   - 251   9  \f   ; 302 336   C 355 
0000140   r 374   } 327 353   q 313 004   H   U 367 330 201 2 | 0000140   P   ]   ) 025   S   b   & 201  \0   r 374   } 327 3
0000160 362 210 214 213 207   _ 226   /   _ 276 374 377 231   | 0000160 004   H   U 367 330 201 225 370 324 362 210 214 213 2

As you can see, the non-null/zero, non-LF data is present in both.  But there is padded data and linefeeds and such that is missing/added in the bad one.
The browser(Netscape4.72) incorrectly downloaded them as the wrong mime type.  base16(?) or 8-bit non-uuencoded(?) instead of base64. ???
People with a very small amount of file-handing experience will quickly notice that this is a pkzip compressed file.

I have no idea how to re-decode/encode this to be correct.  Every mention of this issue that I can find simply says how to fix the browser settings to re-download it.... not how to fix the corrupted file.


[Edit by Admin: truncated very long line, as it kills my ability to read the forum in recent posts mode.]

[quadz]

I have no idea how to re-decode/encode this to be correct.  Every mention of this issue that I can find simply says how to fix the browser settings to re-download it.... not how to fix the corrupted file.

Unfortunately, once those bytes have been stripped, information has been lost in a way that won't be recoverable algorithmically.


N.B. `hexdump -C` produces an alternate format that is also nice for viewing mixed ascii/binary data.


Regards,

quadz

[reaper]

The server is not sending the browser anything different, although HTTP will allow compression.  Netscape probably did something to it, but I don't think other browsers would.  Let's see the hex representation, and figure out what it did.

Is that the whole file?  I don't see 273 at a different location in the "failed" file.

[quadz]

Let's see the hex representation, and figure out what it did.

Is that the whole file?  I don't see 273 at a different location in the "failed" file.

He said it's the first 8 lines of the diff, however what may not be obvious is that the rows also truncated when sdiff put them side by side.

We can tell by the offsets it's dumping 32 bytes per line, though we're only seeing about 13 of those 32 bytes per line.

But in any case, it's clear that nulls and apparently most of the control characters apart from (surprisingly) \b {backspace} are being stripped:


0000000   P   K 003 004 024  \b   $   q   A   ) 273 275 375   |   0000000   P   K 003 004 024  \0  \0  \0  \b  \0   $   q   A 
0000020 001   6 225 003       V   A   P   _   S   I   N   2   |   0000020 375   > 036 033 001  \0   6 225 003  \0  \f  \0  \0 
0000040   P 304 375   k   s 034   G 226 256     256 270 314   |   0000040   P   _   S   I   N   2   .   D   M   2 304 375   k 
0000060   *   K       U 221 352 026 366 036   `   , 260 267 2 |   0000060 226 256  \v 256 270  \0 314   $ 005 226   *   K  \r 
0000100 265 235 177   5 277   q   > 315 327 020   - 251   9   |   0000100 026 366 036   `   , 260 267 224   C 205 265 235 177 
0000120   ; 302 336   C 355   S   d 265   P   ]   ) 025   S   |   0000120   > 315 327 020   - 251   9  \f   ; 302 336   C 355 
0000140   r 374   } 327 353   q 313 004   H   U 367 330 201 2 |   0000140   P   ]   ) 025   S   b   & 201  \0   r 374   } 327 3
0000160 362 210 214 213 207   _ 226   /   _ 276 374 377 231   |   0000160 004   H   U 367 330 201 225 370 324 362 210 214 213 2



No way in hell that's recoverable...

 

[yahoo]

im just curious.

can you edit a binary file using a hex editor and perhaps supply, change if not supply any missing varioables it may have?

[quadz]

can you edit a binary file using a hex editor and perhaps supply, change if not supply any missing varioables it may have?

Sure, if you know what's missing and precisely where to put it back.

In a binary file format like PK, generally if you drop even a single byte anywhere in the file it will be corrupt.

If something has just blindly stripped a whole range of byte values from the file, and you don't have the original to compare it with, you're pretty much f**ked.

[reaper]

Why would netscape remove characters from an HTML file?  Maybe it wants only ASCII characters like the alphabet?  But an HTML file could have any character and be different encodings.

I see mostly numbers, are most opcodes numbers when represented with ASCII?  So this browser took control ASCII characters, or bytes that weren't ASCII representations, and removed them?

I suppose you would have to fill in the gaps, so you'd have to start constructing the assembly source, but it'd be extremely difficult.

[quadz]

Why would netscape remove characters from an HTML file?  Maybe it wants only ASCII characters like the alphabet?  But an HTML file could have any character and be different encodings.

Common encodings like US-ASCII or ISO-8859-1 or UTF-8 aren't going use byte values lower than 32 in HTML, apart from whitespace like \n and \r and \t (10, 13, and 9).

It may not be polite of Netscape to assume it can strip those lower range values, but it's not entirely surprising.

I see mostly numbers, are most opcodes numbers when represented with ASCII?

It's a mixed ASCII/octal dump.  Where it outputs a single character, it's ASCII.  If it's three digits then it's an octal representation of the byte value.  If it's \x then it's a mnemonic for a particular control character.  ( http://msdn.microsoft.com/en-us/library/6aw8xdf2.aspx )

So this browser took control ASCII characters, or bytes that weren't ASCII representations, and removed them?

Yes, generally byte values less than 32.  (Apparently with a few exceptions like \b)

I suppose you would have to fill in the gaps, so you'd have to start constructing the assembly source, but it'd be extremely difficult.

Next to impossible, unless you knew what the compressed data stored in the PK was supposed to look like when uncompressed.

You might be able to salvage undamaged fragments of it, but lots of information has simply been erased.

[reaper]

couldn't you uncompress the code that is there, depending on the compression algorithm the exectuable format uses?

then maybe you could create some assembly source, and see what the program is attempting to do to recreate it?  And nice decoding of what the data is and what happened!

[yahoo]

i didnt know that netscape is still around.

[quadz]

couldn't you uncompress the code that is there, depending on the compression algorithm the exectuable format uses?

Not sure which 'code' you're referring to.  The original sample file appears to be a PKzip'd Q2 demo file.

The the bytes deleted by Netscape will have caused corruption in both the PK container format itself, as well as to the compressed Q2 demo data.

Even if you attempted to disregard the broken container format by manually trying to find the start of the compressed Q2 demo, the compressed data stream itself is corrupted.  If you tried to follow the zip decompression algorithm on the corrupted data, the output would be mostly garbage.