linux-ntfs-dev — Development discussions and bug reports.

[Linux-NTFS-Dev] ntfscloneimgdelta

[Kolja N. <ko...@no...>]

Hi,

I recently started using ntfsclone -s to create regular backups of 
multiple Windows machines in my home network. Storing multiple backup 
generations requires more space than I can bear, so I wanted to replace 
older images with delta files.

The binary diff tools which I tried where unable to efficiently handle 
such large files. rdiff for example consumed all my memory and finally 
crashed my kernel. xdelta3 worked somewhat, but was slow and produced 
suspiciously large deltas.

Therefore I wrote ntfscloneimgdelta, a simple program which reads two 
ntfsclone special image files, and creates a delta file with almost 
identical structure, containing only the header, the clusters and some 
commands necessary to reconstruct the second input file:

   ntfscloneimgdelta delta FILE1 FILE2 DELTA

To reconstruct FILE2 from FILE1 and DELTA:

   ntfscloneimgdelta patch FILE1 DELTA FILE2

Of course it's possible to dump the image and to create the delta in one 
step:

   mv curr.img prev.img
   ntfsclone -s -o - /dev/xyz | tee curr.img |
     ntfscloneimgdelta delta - prev.img prev.img.delta
   rm prev.img

And to reconstruct an image from a delta and to restore it in one step:

   ntfscloneimgdelta patch curr.img prev.img.delta |
     ntfsclone -r -O /dev/xyz -

Maybe my use case is very exotic and no one cares for deltas between 
ntfsclone images. But in case someone else finds this useful:

-> http://nowak2000.de/kolja/ntfscloneimgdelta.c

Greetings,
Kolja Nowak.

Re: [Linux-NTFS-Dev] attribute $UNKNOWN (0x110) from mft record

[Grant W <ka...@cz...>]

I will e-mail the developer to find out more info...

-G

On Sun, Jun 5, 2011 at 4:50 PM, Anton Altaparmakov <ai...@ca...> wrote:
> Hi,
>
> Thanks a lot for your report!
>
> On 5 Jun 2011, at 11:19, Grant W wrote:
>> I am doing my civic duty as a user, reporting malfunctions to the
>> developers, as requested. The attribute type 0x110 was pointing to
>> critical data.
>
> Where did this attribute come from?  It is not listed in the volume's $AttrDef system file thus it is not valid for this volume!
>
>> The faulty implementation of NTFS was unable to read a
>> valid sector pointer for non-resident data. Thus, linux-ntfs was
>> unable to understand a simple operation, and failed, causing a loss of
>> many man-hours.
>
> I am not sure what you are talking about.  First of all which Linux NTFS implementation are you talking about?  And second, if you are talking about ntfsprogs or ntfs-3g then neither is faulty and there are no problems with reading valide sector pointers for non-resident data.  What however is a problem is that this file has an invalid attribute.
>
>> Please add 0x110 as a alternate data stream, as
>> specified in the standard.
>
> What standard are you referring to?  NTFS is not documented and there certainly is no standard for it that I know of.
>
>> Attached... a well compressed ntfs disk image.
>
> Thanks, that is interesting!  This file (/dir60/key50883) was created on 17th May.  What did you use to create it?  Did you create the 0x110 attribute by hand?  What software did you use to create it or if you didn't create it then please give details of Windows version used and any special software you are running and in particular if any particular software owns this file.
>
> Thanks a lot in advance!
>
> Best regards,
>
>        Anton
>
>> ---------------------------------------------------------------
>> Dumping Inode 13446 (0x3486)
>> Upd. Seq. Array Off.:  48 (0x30)
>> Upd. Seq. Array Count:         3 (0x3)
>> Upd. Seq. Number:      8 (0x8)
>> LogFile Seq. Number:   0x0
>> MFT Record Seq. Numb.:         1 (0x1)
>> Number of Hard Links:  1 (0x1)
>> Attribute Offset:      56 (0x38)
>> MFT Record Flags:      IN_USE
>> Bytes Used:            512 (0x200) bytes
>> Bytes Allocated:       1024 (0x400) bytes
>> Next Attribute Instance: 5 (0x5)
>> MFT Padding:  00 00
>> Dumping attribute $STANDARD_INFORMATION (0x10) from mft record 13446 (0x3486)
>>       Resident:                Yes
>>       Attribute flags:         0x0000
>>       Attribute instance:      0 (0x0)
>>       Data size:               48 (0x30)
>>       Resident flags:          0x00
>>       File Creation Time:      Tue May 17 08:55:49 2011
>>       File Altered Time:       Tue May 17 08:55:49 2011
>>       MFT Changed Time:        Tue May 17 08:55:49 2011
>>       Last Accessed Time:      Sat Jun  4 23:48:17 2011
>>       File attributes:         ARCHIVE (0x00000000)
>> Dumping attribute $FILE_NAME (0x30) from mft record 13446 (0x3486)
>>       Resident:                Yes
>>       Attribute flags:         0x0000
>>       Attribute instance:      3 (0x3)
>>       Data size:               82 (0x52)
>>       Resident flags:          0x01
>>       Parent directory:        124 (0x7c)
>>       File Creation Time:      Tue May 17 08:55:49 2011
>>       File Altered Time:       Tue May 17 08:55:49 2011
>>       MFT Changed Time:        Tue May 17 08:55:49 2011
>>       Last Accessed Time:      Tue May 17 08:55:49 2011
>>       Allocated Size:          0 (0x0)
>>       Data Size:               0 (0x0)
>>       Filename Length:         8 (0x8)
>>       File attributes:         ARCHIVE (0x00000000)
>>       Namespace:               POSIX
>>       Filename:                'key50883'
>> Dumping attribute $SECURITY_DESCRIPTOR (0x50) from mft record 13446 (0x3486)
>>       Resident:                Yes
>>       Attribute flags:         0x0000
>>       Attribute instance:      1 (0x1)
>>       Data size:               80 (0x50)
>>       Resident flags:          0x00
>>       Revision:                1
>>       Control:                 0x8004
>>       Owner SID:               S-1-5-32-544
>>       Group SID:               S-1-5-32-544
>>       System ACL:              missing
>>       Discretionary ACL:
>>                       Revision         2
>>                       ACE:             type:allow  flags:0x3  access:0x1f01ff
>>                                        SID: S-1-1-0
>> Dumping attribute $DATA (0x80) from mft record 13446 (0x3486)
>>       Resident:                No
>>       Attribute flags:         0x0000
>>       Attribute instance:      2 (0x2)
>>       Compression unit:        0 (0x0)
>>       Data size:               1024 (0x400)
>>       Allocated size:          1024 (0x400)
>>       Initialized size:        1024 (0x400)
>> Dumping attribute $UNKNOWN (0x110) from mft record 13446 (0x3486)
>>       Resident:                No
>>       Attribute flags:         0x0000
>>       Attribute instance:      4 (0x4)
>>       Compression unit:        0 (0x0)
>>       Data size:               1024 (0x400)
>>       Allocated size:          1024 (0x400)
>>       Initialized size:        1024 (0x400)
>> =====  Please report this unknown attribute type to
>> lin...@li... =====
>> End of inode reached
>> <f200_a3c6a7f6a4b4f9511ee83>------------------------------------------------------------------------------
>> Simplify data backup and recovery for your virtual environment with vRanger.
>> Installation's a snap, and flexible recovery options mean your data is safe,
>> secure and there when you need it. Discover what all the cheering's about.
>> Get your free trial download today.
>> http://p.sf.net/sfu/quest-dev2dev2 _______________________________________________
>> Linux-NTFS-Dev mailing list
>> Lin...@li...
>> https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev
>
> Best regards,
>
>        Anton
> --
> Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
> Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
> Linux NTFS maintainer, http://www.linux-ntfs.org/
>
>

[Linux-NTFS-Dev] ntfscluster -i , Error reading inode

[Zejda L. <Lad...@s-...>]

Hello,

thank you for your work.

I run the command

ntfscluster -i $NTFS_PARTITION

and on some partittions I am getting messages "Error reading inode $NUM".
Should I worry about such partitions? 

Last time I see it just on new Windows 2008 R2 Server system disk:
ntfscluster -i /dev/sda2
Error reading inode 14053.
Error reading inode 19054.
Error reading inode 96851.
Error reading inode 96853.
Error reading inode 96947.
Error reading inode 103449.
Error reading inode 107636.
Error reading inode 107637.
Error reading inode 107639.
Error reading inode 107640.
Error reading inode 122266.
Error reading inode 122268.
Error reading inode 134925.
Error reading inode 139822.
Error reading inode 139829.
bytes per sector        : 512
bytes per cluster       : 4096
sectors per cluster     : 8
bytes per volume        : 42841665536
sectors per volume      : 10459391
clusters per volume     : 1307423
initialized mft records : 140032
mft records in use      : 56495
mft records percentage  : 40
bytes of free space     : 31237595136
sectors of free space   : 61010928
clusters of free space  : 7626366
percentage free space   : 72
bytes of user data      : 11389530112
sectors of user data    : 22245176
clusters of user data   : 2780647
percentage user data    : 26
bytes of metadata       : 214540288
sectors of metadata     : 419024
clusters of metadata    : 52378
percentage metadata     : 0

Thank you for you advice.

Best regards,
 
Ladislav Zejda
mailto: lz...@cs...






_____________
Tato zpr?va a v?echny p?ipojen? soubory jsou d?v?rn? a ur?en? v?lu?n? adres?tovi(-?m). Jestli?e nejste opr?vn?n?m adres?tem, je zak?z?no jak?koliv zve?ej?ov?n?, zprost?edkov?n? nebo jin? pou?it? t?chto informac?. Jestli?e jste tento mail dostali neopr?vn?n?, pros?m, uv?domte odesilatele a sma?te zpr?vu i p?ilo?en? soubory. Odesilatel nezodpov?d? za jak?koliv chyby nebo opomenut? zp?soben? t?mto p?enosem.
 
Jste si jisti, ?e opravdu pot?ebujete vytisknout tuto zpr?vu a/nebo jej? p??lohy? Myslete na p??rodu. 
 
 
This message and any attached files are confidential and intended solely for the addressee(s). Any publication, transmission or other use of the information by a person or entity other than the intended addressee is prohibited. If you receive this in error please contact the sender and delete the message as well as all attached documents. The sender does not accept liability for any errors or omissions as a result of the transmission.
 
Are you sure that you really need a print version of this message and/or its attachments? Think about nature. 

-.- --

[Linux-NTFS-Dev] [PATCH] ldm: Fix reassembly of extended VBLKs

[Ben H. <be...@de...>]

Extended VBLKs (those larger than the preset VBLK size) are divided
into fragments, each with its own VBLK header.  Our LDM implementation
generally assumes that each VBLK is contiguous in memory, so these
fragments must be assembled before further processing.

Currently the reassembly seems to be done quite wrongly - no VBLK
header is copied into the contiguous buffer, and the length of the
header is subtracted twice from each fragment.  Also the total
length of the reassembled VBLK is calculated incorrectly.

Signed-off-by: Ben Hutchings <be...@de...>
---
This is purely based on a little code review after seeing the patch for
CVE-2011-1017, and a quick look at the reverse-engineered documentation
of LDM.  I have no test case for it, but I suspect that you can force
Windows to create an extended VBLK by giving a partition a very long
name.

Ben.

 fs/partitions/ldm.c |   10 +++++++---
 1 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/fs/partitions/ldm.c b/fs/partitions/ldm.c
index af9fdf0..52271a6 100644
--- a/fs/partitions/ldm.c
+++ b/fs/partitions/ldm.c
@@ -1348,10 +1348,11 @@ found:
 
 	f->map |= (1 << rec);
 
+	if (rec == 0)
+		memcpy(f->data, data, VBLK_SIZE_HEAD);
 	data += VBLK_SIZE_HEAD;
 	size -= VBLK_SIZE_HEAD;
-
-	memcpy (f->data+rec*(size-VBLK_SIZE_HEAD)+VBLK_SIZE_HEAD, data, size);
+	memcpy(f->data + VBLK_SIZE_HEAD + rec * size, data, size);
 
 	return true;
 }
@@ -1401,7 +1402,10 @@ static bool ldm_frag_commit (struct list_head *frags, struct ldmdb *ldb)
 			return false;
 		}
 
-		if (!ldm_ldmdb_add (f->data, f->num*ldb->vm.vblk_size, ldb))
+		if (!ldm_ldmdb_add(f->data,
+				   VBLK_SIZE_HEAD + f->num *
+				   (ldb->vm.vblk_size - VBLK_SIZE_HEAD),
+				   ldb))
 			return false;		/* Already logged */
 	}
 	return true;
-- 
1.7.5.3

Re: [Linux-NTFS-Dev] attribute $UNKNOWN (0x110) from mft record

[Anton A. <ai...@ca...>]

Hi,

Thanks a lot for your report!

On 5 Jun 2011, at 11:19, Grant W wrote:
> I am doing my civic duty as a user, reporting malfunctions to the
> developers, as requested. The attribute type 0x110 was pointing to
> critical data.

Where did this attribute come from?  It is not listed in the volume's $AttrDef system file thus it is not valid for this volume!

> The faulty implementation of NTFS was unable to read a
> valid sector pointer for non-resident data. Thus, linux-ntfs was
> unable to understand a simple operation, and failed, causing a loss of
> many man-hours.

I am not sure what you are talking about.  First of all which Linux NTFS implementation are you talking about?  And second, if you are talking about ntfsprogs or ntfs-3g then neither is faulty and there are no problems with reading valide sector pointers for non-resident data.  What however is a problem is that this file has an invalid attribute.

> Please add 0x110 as a alternate data stream, as
> specified in the standard.

What standard are you referring to?  NTFS is not documented and there certainly is no standard for it that I know of.

> Attached... a well compressed ntfs disk image.

Thanks, that is interesting!  This file (/dir60/key50883) was created on 17th May.  What did you use to create it?  Did you create the 0x110 attribute by hand?  What software did you use to create it or if you didn't create it then please give details of Windows version used and any special software you are running and in particular if any particular software owns this file.

Thanks a lot in advance!

Best regards,

	Anton

> ---------------------------------------------------------------
> Dumping Inode 13446 (0x3486)
> Upd. Seq. Array Off.:	 48 (0x30)
> Upd. Seq. Array Count:	 3 (0x3)
> Upd. Seq. Number:	 8 (0x8)
> LogFile Seq. Number:	 0x0
> MFT Record Seq. Numb.:	 1 (0x1)
> Number of Hard Links:	 1 (0x1)
> Attribute Offset:	 56 (0x38)
> MFT Record Flags:	 IN_USE
> Bytes Used:		 512 (0x200) bytes
> Bytes Allocated:	 1024 (0x400) bytes
> Next Attribute Instance: 5 (0x5)
> MFT Padding:	00 00
> Dumping attribute $STANDARD_INFORMATION (0x10) from mft record 13446 (0x3486)
> 	Resident: 		 Yes
> 	Attribute flags:	 0x0000
> 	Attribute instance:	 0 (0x0)
> 	Data size:		 48 (0x30)
> 	Resident flags:		 0x00
> 	File Creation Time:	 Tue May 17 08:55:49 2011
> 	File Altered Time:	 Tue May 17 08:55:49 2011
> 	MFT Changed Time:	 Tue May 17 08:55:49 2011
> 	Last Accessed Time:	 Sat Jun  4 23:48:17 2011
> 	File attributes:	 ARCHIVE (0x00000000)
> Dumping attribute $FILE_NAME (0x30) from mft record 13446 (0x3486)
> 	Resident: 		 Yes
> 	Attribute flags:	 0x0000
> 	Attribute instance:	 3 (0x3)
> 	Data size:		 82 (0x52)
> 	Resident flags:		 0x01
> 	Parent directory:	 124 (0x7c)
> 	File Creation Time:	 Tue May 17 08:55:49 2011
> 	File Altered Time:	 Tue May 17 08:55:49 2011
> 	MFT Changed Time:	 Tue May 17 08:55:49 2011
> 	Last Accessed Time:	 Tue May 17 08:55:49 2011
> 	Allocated Size:		 0 (0x0)
> 	Data Size:		 0 (0x0)
> 	Filename Length:	 8 (0x8)
> 	File attributes:	 ARCHIVE (0x00000000)
> 	Namespace:		 POSIX
> 	Filename:		 'key50883'
> Dumping attribute $SECURITY_DESCRIPTOR (0x50) from mft record 13446 (0x3486)
> 	Resident: 		 Yes
> 	Attribute flags:	 0x0000
> 	Attribute instance:	 1 (0x1)
> 	Data size:		 80 (0x50)
> 	Resident flags:		 0x00
> 	Revision:		 1
> 	Control:		 0x8004
> 	Owner SID:		 S-1-5-32-544
> 	Group SID:		 S-1-5-32-544
> 	System ACL:		 missing
> 	Discretionary ACL:	
> 			Revision	 2
> 			ACE:		 type:allow  flags:0x3  access:0x1f01ff
> 					 SID: S-1-1-0
> Dumping attribute $DATA (0x80) from mft record 13446 (0x3486)
> 	Resident: 		 No
> 	Attribute flags:	 0x0000
> 	Attribute instance:	 2 (0x2)
> 	Compression unit:	 0 (0x0)
> 	Data size:		 1024 (0x400)
> 	Allocated size:		 1024 (0x400)
> 	Initialized size:	 1024 (0x400)
> Dumping attribute $UNKNOWN (0x110) from mft record 13446 (0x3486)
> 	Resident: 		 No
> 	Attribute flags:	 0x0000
> 	Attribute instance:	 4 (0x4)
> 	Compression unit:	 0 (0x0)
> 	Data size:		 1024 (0x400)
> 	Allocated size:		 1024 (0x400)
> 	Initialized size:	 1024 (0x400)
> =====  Please report this unknown attribute type to
> lin...@li... =====
> End of inode reached
> <f200_a3c6a7f6a4b4f9511ee83>------------------------------------------------------------------------------
> Simplify data backup and recovery for your virtual environment with vRanger.
> Installation's a snap, and flexible recovery options mean your data is safe,
> secure and there when you need it. Discover what all the cheering's about.
> Get your free trial download today. 
> http://p.sf.net/sfu/quest-dev2dev2 _______________________________________________
> Linux-NTFS-Dev mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

[Linux-NTFS-Dev] attribute $UNKNOWN (0x110) from mft record

[Grant W <ka...@cz...>]

I am doing my civic duty as a user, reporting malfunctions to the
developers, as requested. The attribute type 0x110 was pointing to
critical data. The faulty implementation of NTFS was unable to read a
valid sector pointer for non-resident data. Thus, linux-ntfs was
unable to understand a simple operation, and failed, causing a loss of
many man-hours. Please add 0x110 as a alternate data stream, as
specified in the standard.

Attached... a well compressed ntfs disk image.


---------------------------------------------------------------
Dumping Inode 13446 (0x3486)
Upd. Seq. Array Off.:	 48 (0x30)
Upd. Seq. Array Count:	 3 (0x3)
Upd. Seq. Number:	 8 (0x8)
LogFile Seq. Number:	 0x0
MFT Record Seq. Numb.:	 1 (0x1)
Number of Hard Links:	 1 (0x1)
Attribute Offset:	 56 (0x38)
MFT Record Flags:	 IN_USE
Bytes Used:		 512 (0x200) bytes
Bytes Allocated:	 1024 (0x400) bytes
Next Attribute Instance: 5 (0x5)
MFT Padding:	00 00
Dumping attribute $STANDARD_INFORMATION (0x10) from mft record 13446 (0x3486)
	Resident: 		 Yes
	Attribute flags:	 0x0000
	Attribute instance:	 0 (0x0)
	Data size:		 48 (0x30)
	Resident flags:		 0x00
	File Creation Time:	 Tue May 17 08:55:49 2011
	File Altered Time:	 Tue May 17 08:55:49 2011
	MFT Changed Time:	 Tue May 17 08:55:49 2011
	Last Accessed Time:	 Sat Jun  4 23:48:17 2011
	File attributes:	 ARCHIVE (0x00000000)
Dumping attribute $FILE_NAME (0x30) from mft record 13446 (0x3486)
	Resident: 		 Yes
	Attribute flags:	 0x0000
	Attribute instance:	 3 (0x3)
	Data size:		 82 (0x52)
	Resident flags:		 0x01
	Parent directory:	 124 (0x7c)
	File Creation Time:	 Tue May 17 08:55:49 2011
	File Altered Time:	 Tue May 17 08:55:49 2011
	MFT Changed Time:	 Tue May 17 08:55:49 2011
	Last Accessed Time:	 Tue May 17 08:55:49 2011
	Allocated Size:		 0 (0x0)
	Data Size:		 0 (0x0)
	Filename Length:	 8 (0x8)
	File attributes:	 ARCHIVE (0x00000000)
	Namespace:		 POSIX
	Filename:		 'key50883'
Dumping attribute $SECURITY_DESCRIPTOR (0x50) from mft record 13446 (0x3486)
	Resident: 		 Yes
	Attribute flags:	 0x0000
	Attribute instance:	 1 (0x1)
	Data size:		 80 (0x50)
	Resident flags:		 0x00
	Revision:		 1
	Control:		 0x8004
	Owner SID:		 S-1-5-32-544
	Group SID:		 S-1-5-32-544
	System ACL:		 missing
	Discretionary ACL:	
			Revision	 2
			ACE:		 type:allow  flags:0x3  access:0x1f01ff
					 SID: S-1-1-0
Dumping attribute $DATA (0x80) from mft record 13446 (0x3486)
	Resident: 		 No
	Attribute flags:	 0x0000
	Attribute instance:	 2 (0x2)
	Compression unit:	 0 (0x0)
	Data size:		 1024 (0x400)
	Allocated size:		 1024 (0x400)
	Initialized size:	 1024 (0x400)
Dumping attribute $UNKNOWN (0x110) from mft record 13446 (0x3486)
	Resident: 		 No
	Attribute flags:	 0x0000
	Attribute instance:	 4 (0x4)
	Compression unit:	 0 (0x0)
	Data size:		 1024 (0x400)
	Allocated size:		 1024 (0x400)
	Initialized size:	 1024 (0x400)
=====  Please report this unknown attribute type to
lin...@li... =====
End of inode reached