Syslinux - Jan 2014 - Cluster Size discrepancy between FAT32 and NTFS

On Tue, Jan 21, 2014 at 2:41 PM, mike setzer <qualityana at sbcglobal.net>
wrote:
> hello Gene,
>
> good to get your message
>
> I have not seen a link to reply to a particular thread,
> so I am replying to you right now
> hopefully the thread can be made to look right, also I double posted since
my connection was lost
> right during the first posting.
Sorry, I thought you were subscribed but this sounds as if you are not.
> On NTFS I have tested cluster sizes from 4096 up through 64K.
> Did not try smaller sizes than 4096.
Good to note.  The most likely answer is that the code either assumes
4096, assumes no more than 4096 OR there's actually a bug in your file
system.  I'll peek to see if I can tell from your VBR captures.
> Windows is bootable from these NTFS volumes having larger-than-4096
clusters, see further info below*.
> As for terminology details, 4096 is the default size for NTFS volumes.
> Standard sizes include only 512b, 1kb, 2kb, 4kb, 8kb, 16kb, 32kb, &
64kb.
> So I am not technically using larger than "standard", only larger
than default.
Good to know.  512/1024/2048 are also the default for smaller file systems.
> The experimentation is being done right now on the third partition of a
multiboot system,
>
> On this HDD the third partition is the third & final primary, 16GB
nominal partition size.
>
> NTFS formatting is then using either windows XP or the Windows 8.1 preview.
> This is when the cluster size is specified to begin with, so for NTFS
experimentation, the same NTFS partition is
> maintained after zeroing, only reformatted with varying cluster sizes then
reimaged from windows WIM files
> which handle the cluster differences well.
>
> Linux (distro which supports FAT and/or NTFS) is correctly placed on FAT32
or NTFS
> in the windows filesystem (whatever cluster size) by copying & pasting
using windows.  The bootable linux fileset is copied directly
> from the folders on the live linux CD or DVD, and in this way is bootable
not much differently than having the disk fileset on a
> bootable FAT or NTFS USB drive.
> Linux is not "installed", just the live disk behavior is
obtained.
> Later, using grub to address the proper bootsector, by chainloading either
to a windows NT bootsector or syslinux
> bootsector, either windows or linux are booted from the grub menu.
>
> in this simple application, syslinux is used as a direct substitute for
isolinux, apparently as designed
It took reading your instructions to see that you prepared your boot
sector properly.
> The larger cluster sizes provide noticably faster boot times.
Faster for what OS?
> *Windows does have a bug about the cluster size itself.
> The windows setup routine does not support over 4096,
> Windows filesets are deployed to larger-than-4096 volumes by extracting
from WIM images.
> Also the windows bootmanager is limited, where if the windows system files
(WINDOWS folder, etc.) are on a NTFS volume having clusters larger than 4096,
then the windows BOOT folder must be on a different volume having 4096 (or
less?) if NTFS,
> or up to 64K if FAT.
I interpret this statement as "The Windows bootloader is incabable of
directly booting from NTFS with 8k+ clusters" as you need a jump point
first.  I'd say this now becomes an even more rare occurrence to
encounter since you state even Windows can't boot like this.
> The syslinux limitation of apparently booting only to the volume containing
the target ldlinux.sys file I do consider ideal,
> but that feature precludes using syslinux as a direct substitute for NT
bootmanagers which have always allowed
> separate boot & system volumes.
> Grub legacy does address the separate boot & system volumes fine so I
see no need for that in syslinux.
multi-fs is coming.
> That is why it seems to me a good approach would be to correctly support
larger NTFS cluster sizes in syslinux
> similar to how they are acceptable in FAT.
>
> also, in all cases it can be seen that syslinuxing an NTFS volume fails to
replace the backup NT bootsector which
> is located at the final sector of the NTFS partition.  Syslinux only
prepares the working bootsector (first sector of the partition) as a linux VBR,
> apparently leaving the backup (final sector of the partition) unchanged.
This is an overall unusual set of circumstances.

Why do you want to use NTFS with 8k+ clusters?  Why not consider the
same workaround that Windows requires?

-- 
-Gene

A: Because it messes up the order in which people normally read text,
especially the archives of mailing lists.
Q: Why is Top-posting such a bad thing?
> here is the preparation log for the attached bootsectors:
>
> this is a traditional HDD having 512byte sectors
> on a traditional BIOS/MBR system
> using LBA
>
> existing partition created by windows 8.1
> -this is primary partition3 of a multipartitioned HDD
> -which displays same behavior as partition1 of a single-volume HDD
> bootsector in place at desired sector 19595264
> recognized by windows as volume Z:
>
> TEST OF 4096 cluster size on NTFS
>
> wrote zeros to partition
> dd if=/dev/zero of=/dev/sda3
> still recognized by windows as unformatted volume Z:
> -partition table in sector0 which defines partition & location is
unchanged by zeroing partition3 alone
>
> format using admin CMD prompt in W81 - "default" cluster size:
> FORMAT Z: /Q /FS:NTFS /A:4096 /V:NTFStest
> there is now a windows bootsector in place at 19595264 as a result of the
formatting process.
> save this new NT bootsector as NT3_4K.bin
> now intentionally overwrite with what should be a duplicate windows
bootsector:
> BOOTSECT /NT60 Z: /FORCE
> save this new NT bootsector as NT3B_4K.bin
> -if a bootable windows fileset with its BOOT folder were placed onto volume
Z: at this point,
> -it would boot normally if this partiton3 was made active
>
> using windows,
> copied live fileset from puppy linux Lucid Puppy 5.28 live CD,
> pasted onto NTFS volume Z:
> -this is a typical modern puppy live CD naturally supporting FAT &
NTFS.
> on volume Z:, rename isolinux.cfg to syslinux.cfg
>
> using windows and win32 syslinux version 4.07, overwrite the NT bootsector
with a linux bootsector
> SYSLINUX -F Z:
> -ldlinux.sys is created on volume Z: at this point
> save the new syslinux bootsector as SL3_4K.bin
>
> set partition3 active and boot to it from MBR or chainload (hd0,2)+1 from
grub legacy floppy
>
> PUPPY BOOTS as expected similar to a live CD, no differently than using a
FAT volume having up to 64K clusters
>
> TEST OF 64K cluster size on NTFS
>
> wrote zeros to partition
> dd if=/dev/zero of=/dev/sda3
> still recognized by windows as unformatted volume Z:
> -partition table in sector0 which defines partition & location is
unchanged by zeroing partition3 alone
>
> format using admin CMD prompt in W81 - largest cluster size
> FORMAT Z: /Q /FS:NTFS /A:64K /V:NTFStest
> there is now a windows bootsector in place at 19595264 as a result of the
formatting process.
> save this new NT bootsector as NT3_64K.bin
> now intentionally overwrite with what should be a duplicate windows
bootsector:
> BOOTSECT /NT60 Z: /FORCE
> save this new NT bootsector as NT3B_64K.bin
> -if a bootable windows fileset with its BOOT folder were placed onto volume
Z: at this point,
> -it would not actually boot if this partiton3 was made active, windows
bootmanager can not find all needed boot files
> -when the cluster size is greater than 4096 if formatted using NTFS.
> -instead it would require booting through a windows bootmanager (with BOOT
folder) residing on a different active volume,
> -one having cluster size up to 64K if FAT or only up to 4096 if NTFS, or
alternatively from a floppy
> -containing the windows bootmanager (with BOOT folder)
This in my mind is critical.  Linux can also have a separate /boot file system.
> using windows,
> copied live fileset from puppy linux Lucid Puppy 5.28 live CD,
> pasted onto NTFS volume Z:
> -this is a typical modern puppy live CD naturally supporting FAT &
NTFS.
> on volume Z:, rename isolinux.cfg to syslinux.cfg
>
> using windows and win32 syslinux version 4.07, overwrite the NT bootsector
with a linux bootsector
> SYSLINUX -F Z:
> -ldlinux.sys is created on volume Z: at this point
> save the new syslinux bootsector as SL3_64K.bin
>
> set partition3 active and boot to it from MBR or chainload (hd0,2)+1 from
grub legacy floppy
>
> PUPPY FAILS TO BOOT, syslinux stops after displaying a single line followed
by carriage return and blinking cursor:
> SYSLINUX 4.07 EDD 2013-07-25 Copyright (C) 1994-2013 H. Peter Anvin et al
> _
>
> at this point keyboard is now unresponsive
>
> repeating with progressively smaller NTFS cluster sizes continues to fail
until 4096 is reached
>
> --------------------------------------------
> On Sun, 1/19/14, Gene Cumm <gene.cumm at gmail.com> wrote:
>
>  Subject: Re: [syslinux] Cluster Size discrepancy between FAT32 and NTFS
>  To: "mike setzer" <qualityana at sbcglobal.net>, "For
discussion of Syslinux and tftp-hpa" <syslinux at zytor.com>
>  Date: Sunday, January 19, 2014, 10:57 AM
>
>  On Sun, Jan 19, 2014 at 9:44 AM, mike
>  setzer <qualityana at sbcglobal.net>
>  wrote:
>  > Hi,
>  >
>  > I am not an engineer or linux expert but I have been
>  using syslinux to boot
>  > live disk filesets (extracted from iso's) residing on
>  fat32 and NTFS volumes.
>  >
>  > In FAT32 there has been no problem going with larger
>  cluster sizes up to the nominal maximum of 64K
>  >
>  > however, with NTFS it has not been possible to exceed
>  the cluster size of 4096.
>  > With NTFS formatted using clusters of 8192 or larger,
>  > upon boot apparently the syslinux bootsector is
>  > read since the syslinux version is displayed, but
>  progress stops with no error message
>
>  Have you only tested 4096 and 8192?  Testing 16k+
>  probably won't be
>  useful if 8k fails but it may be good to know about
>  512/1024/2048
>  (eventually).
>
>  Are these NTFS volumes with non-standard cluster sizes
>  bootable by Windows?
>
>  How was the file system created?  Is there any way you
>  could export
>  the first sector of the partition to a file and either post
>  it
>  directly or a hex representation of it?  The first
>  sector should
>  specify things like sectors per cluster but it's possible
>  something is
>  being misinterpreted.
>
>  > this has been tested using real machines, not virtual
>  >
>  > ability to handle larger cluster sizes would seem to be
>  as useful for NTFS as in FAT32,
>  > if not more so.
>  >
>  > I stand by prepared for immediate testing of win32
>  syslinux versions as early as 4.06 which contain
>  > modifications to include this property
>
>  --
>  -Gene
>
>  search terms: 0.5 kiB, 1kiB, 2kiB, 4kiB 8kiB; half k, 1k,
>  2k, 4k, 8k;
>  0.5kB, 1kB, 2kB, 4kB, 8kB

On Tue, Jan 21, 2014 at 5:22 PM, Gene Cumm <gene.cumm at gmail.com>
wrote:
> On Tue, Jan 21, 2014 at 2:41 PM, mike setzer <qualityana at
sbcglobal.net> wrote:
>> hello Gene,
>>
>> good to get your message
>>
>> I have not seen a link to reply to a particular thread,
>> so I am replying to you right now
>> hopefully the thread can be made to look right, also I double posted
since my connection was lost
>> right during the first posting.
>
> Sorry, I thought you were subscribed but this sounds as if you are not.
>
>> On NTFS I have tested cluster sizes from 4096 up through 64K.
>> Did not try smaller sizes than 4096.
>
> Good to note.  The most likely answer is that the code either assumes
> 4096, assumes no more than 4096 OR there's actually a bug in your file
> system.  I'll peek to see if I can tell from your VBR captures.
Hexdumping the first 16B of each, it looks good.  0x08 for 4k and 0x80 for 64k.

  --NT3B_4K.bin
00000000  eb 52 90 4e 54 46 53 20  20 20 20 00 02 08 00 00  |.R.NTFS    .....|
  --NT3B_64K.bin
00000000  eb 52 90 4e 54 46 53 20  20 20 20 00 02 80 00 00  |.R.NTFS    .....|
  --SL3_4K.bin
00000000  eb 58 90 4e 54 46 53 20  20 20 20 00 02 08 00 00  |.X.NTFS    .....|
  --SL3_64K.bin
00000000  eb 58 90 4e 54 46 53 20  20 20 20 00 02 80 00 00  |.X.NTFS    .....|

-- 
-Gene