hoakley May 2, 2020 Macs, Technology

File Integrity 11 : Which RAID levels enable file recovery?

It’s good that more users are adopting RAID arrays for external storage, but essential to remember that the term RAID doesn’t necessarily bring any improvement in protection of file integrity. Across the seven standard levels of RAID, there is great variation in the protection afforded to the contents of the array, and even more when it comes to protecting the contents of files on it. This article looks at what each level means in practice.

For any storage system to protect the integrity of the files stored on it, it first needs to be able to detect whether a file can’t be read fully, or when reading it doesn’t deliver the expected data. The latter means keeping checksums or digests against which the current file can be compared. Should the data from any file not match its expected checksum/digest, the system then needs a means of repairing that copy, or of sourcing another copy which does match the checksum/digest. This is quite different from being able to recover the whole contents of the array in the event that one (or more) of its disks fails.

RAID 0, or striping, has no redundancy of data or error recovery at all. You can add file integrity checks, for example using Dintch or Fintch, but those will only indicate which files are damaged, not recover or repair them.

RAID 1, or mirroring, provides two or more complete copies of everything stored on the RAID system, which has inherent redundancy. In the event that one disk suffers failure, files can still be retrieved from the surviving disk until the RAID has been successfully rebuilt, normally by replacing the failed disk. Disk integrity checking is assumed to detect any errors, but to be confident that any given file is intact, file integrity checks are needed.

Care is needed to ensure that both storage media are drawn from different production batches, as hard disks (and possibly SSDs too) from the same batch tend to fail at about the same time. While double-disk failure may appear unlikely in theory, in practice it’s by no means unknown.

RAID 2 and 3 are very rarely encountered. Level 2 incorporates basic error correcting code (ECC), and 3 has a dedicated parity disk, so both should offer some level of detection and repair.

RAID 4, 5 and 6 appear very similar, and their differences might seem technical, but they are important when deciding which level to use.

RAID 4 uses block-level striping with parity data stored on a dedicated disk, which should support at least some degree of ECC. However, recovery depends on normal function of the parity disk, making this less popular.

RAID 5 uses block-level striping with parity data distributed across the disks, and is widely used as a good compromise between resilience and parity overhead (hence cost). In its most advanced form in ZFS, where it’s also known as RAID-Z1, it’s coupled with integrity checks and automatic recovery.

RAID 6 is similar to 5 but uses an additional parity block, which increases its resilience to disk failure at the expense of space efficiency. This too involves an ECC implementation, which is used in an advanced form in RAID-Z2 in ZFS.

Popular non-RAID designations for drive arrays include JBOD (Just a Bunch of Disks), Spanned or BIG, and Unraid. None of those is expected to incorporate any form of integrity checking or file recovery.

One important issue with all RAID systems is that they are designed to protect the integrity of the files stored on them, and don’t normally check that copying any files to or from the RAID storage preserves their integrity. For example, Time Machine backups don’t include a verification step in which backed up copies of files are checked against their originals. Any errors which occur during copying can therefore result in the RAID array preserving the integrity of corrupt files.

Another area of uncertainty is file system error. Although this should be extremely rare, damaged or corrupt file system data stored on a RAID array can have unpredictable effects on its contents.

Unless you pay extra for a hardware RAID system which implements RAID functions in its firmware, you’ll be relying on one of two software implementations:

AppleRAID supports only two RAID levels, 0 (striping) and 1 (mirroring), together with JBOD, according to Apple.
SoftRAID supports RAID levels 0 (striping), 1 (mirroring), 4 and 5 (both using striping with parity), and the combination of 1+0, according to its product page.

AppleRAID’s lack of support for any of the more sophisticated and resilient levels 4-6 is frustrating.

Specific RAID levels do provide often sophisticated mechanisms for preserving file integrity, including both detection and correction of corruption. However, unless they’re properly integrated into a system-wide scheme, their benefit is constrained.

Further detailed comparisons are given in Wikipedia.

8Comments

Add yours

1

Duncan on May 2, 2020 at 2:06 pm

“One important issue with all RAID systems is that they are designed to protect the integrity of the files stored on them, and don’t normally check that copying any files to or from the RAID storage preserves their integrity. For example, Time Machine backups don’t include a verification step in which backed up copies of files are checked against their originals. Any errors which occur during copying can therefore result in the RAID array preserving the integrity of corrupt files.”

Howard, I’m pretty sure you know this already but I’ll mention that Carbon Copy Cloner includes a feature which attempts to verify the copy/backup process using its “Find and replace corrupted files” option. Unfortunately it’s a bit of a black-box process, in that we can’t see *how* it is verifying the copy operation, nor is it complete because it only appears to verify the file integrity after the first copy has already been written. (It’s still better than nothing, but not enough to set my mind at ease.)

Regarding the verification process itself, CCC appears to calculate a checksum/digest on both the source and destination files, but then, as far as I can tell, it throws that checksum/digest away. (I’ll just call it a checksum now for simplicity.) Which is a shame because we end-users cannot then leverage that checksum for other uses. Bombich Software doesn’t seem to reveal any details about that, at least that I’ve been able to find.

Perhaps if you’re bored someday (hah!) you could dissect a test file that originates on your internal storage and compare all its attributes to the same file after it’s been copied using CCC to external storage, with the ‘Find and replace corrupted files’ option checked. But that might still not reveal the presence of an attached checksum if CCC saves those in a hidden proprietary database. Or it might not preserve them at all, forcing a complete re-calculation each time it’s run.

And regarding what CCC seems to do when implementing that option, as far as I can gather the *first* time the copy/clone operation runs there is no verification performed at all. My understanding is that the ‘Find and replace corrupted files’ feature only kicks in after the first copy has been made. In other words it attempts to preserve the integrity of the backup files once they’ve been created (which is a good thing) but you have to get there first. And there is no added provision for ECC to recover the original file from the copy.

I might be incorrect in some of that, and I’m not asking you to serve as Bombich tech support, but that’s about as close as I’ve seen any company address the copy operation in terms of file integrity. It’s just too bad they don’t go all the way and expose their checksum process for the user to manage beyond that, such as being able to run a separate checksum operation on the copied files independently and selectively.

LikeLiked by 1 person
- 2
  
  hoakley on May 2, 2020 at 6:29 pm
  
  Thank you, Duncan. That’s one reason why I didn’t try to generalise and claim that no backup system gives you the option to verify each backup. Time Machine certainly doesn’t, and when TM ‘verifies’ its backups it’s not checking file integrity, but file system integrity.
  Other backup systems may make this an option, but it appears not to be a general feature of those that I have looked at, or use here.
  In the context of backing up to a RAID system, I think that’s very important, as the RAID level may preserve its contents, which could of course contain corrupted files.
  Howard.
  
  LikeLike
  - 3
    
    Duncan on May 2, 2020 at 7:08 pm
    
    “In the context of backing up to a RAID system, I think that’s very important, as the RAID level may preserve its contents, which could of course contain corrupted files.”
    
    It seems like RAID-Z ‘does it all’, in terms of maintaining file integrity as well as disk-array integrity. I would think that any future file / storage-management system would aspire to that level of integration and completeness.
    
    (I’ve already expressed my disappointment how APFS fell short on both counts. I *really* hope that a backward-compatible APFS-2 corrects that, with proper Time Machine integration already addressed by that point. I think the entire domain of ‘data preservation’ needs to be architected holistically so that no one feature is left as an afterthought.)
    
    LikeLiked by 1 person
    - 4
      
      hoakley on May 2, 2020 at 7:19 pm
      
      Much as I agree with you, Duncan, I think there’s no chance of that happening at all. APFS is primarily if not overwhelmingly a file system for iOS. Apple’s complete withdrawal from the server market means that it has little interest in what it sees as specialist matters for sysadmins to address. The fact that macOS has no support for any archival storage system is another sign.
      The only way to address these is to add to macOS what we need.
      Howard.
      
      LikeLike
5

EcleX on May 2, 2020 at 4:15 pm

Thanks. Advantages and disadvantages would be also nice (maybe for other article). For instance, RAID 0 may increase sequential read/write speed twice (not random IOPS read/write, which is even more important, but most manufacturers do not even mention, mostly for slower disks, probably because most users do not even know about it!), but it may also increase failure rate if one of the disks or the controller fails.

LikeLiked by 1 person
- 6
  
  Duncan on May 2, 2020 at 4:47 pm
  
  “RAID 0… may also increase failure rate if one of the disks or the controller fails.”
  
  Actually, it guarantees failure rate if anything fails (not just tautologically, but in terms of complete data loss). But I suspect everyone reading these articles already understands that, or can at least follow the link to Wikipedia.
  
  LikeLiked by 1 person
- 7
  
  hoakley on May 2, 2020 at 6:30 pm
  
  Thank you.
  If you want to read an excellent review of all the other features, with comparisons between different RAID levels, that Wikipedia article is excellent. I wouldn’t attempt to cover the same ground.
  Howard.
  
  LikeLike
8

Raoul on May 4, 2020 at 12:55 am

Hi Duncan,
ZFS can work its magic even with a single disk. You can tell ZFS to write multiple copies (each with checksums) to a single physical disk if you really needed to. Unless you switch off checksumming (not advisable) All methods of ZFS RAID ensure file integrity (including striping of disks, although you have no redundancy). ZFS can even go further by offering triple or quadruple mirroring of disks and triple parity (RAIDZ3) to protect data should 3 disks in a pool fail.
When ZFS was first released, if it encountered data corruption along the data path that it could not correct, it would kernel panic the machine purposefully! That’s how important the Sun engineers regarded keeping files safe and consistent on disk. I know I keep saying it but ZFS is a wonderful thing.

LikeLiked by 1 person

·Comments are closed.

Share this:

Like this:

Related