hoakley April 8, 2020 Macs, Technology

File Integrity 3 : Where to store digests?

Having established that checking the integrity of files can be an important and useful thing to do, but that macOS and its standard file systems don’t support it, I then concluded that a file integrity checker should calculate the SHA256 digest on the data fork, which then needs to be associated somehow with that file, but probably not with a timestamp. This article considers how best to associate each file with its digest.

One of the most important considerations here is that the association must be general and as robust as possible. There’s no point in adopting a technique which only works with specific types of backup, and fails the moment the file is stored in, or passes through, iCloud or network storage.

The ideal location in which to store a file’s SHA256 digest is, of course, in its metadata, or attributes. Unfortunately, the standard attribute set for files on HFS+ and APFS volumes doesn’t include one for a digest, isn’t extensible to add an attribute for one, and there’s no existing attribute which could be repurposed for this. Furthermore, even if such an attribute were available, unless it was also supported by other file systems and cloud services, stored digests would be lost.

Three options remain: extended attributes, a separate associated file similar to the ._ mechanism used to store extended attributes in some other file systems (AppleDouble), and a hidden file in each folder/directory. I’ll take those in reverse order.

Folder-based digest lists

Maintaining a hidden file in each folder/directory in which the digests of other files in that folder are stored is a proven technique, used by diglloyd’s IntegrityChecker. So long as files remain inside that folder (in the sense that their relative path within that folder is fixed), this works well. However, it becomes complex to support moving and copying files beyond that folder.

You could in theory add a file’s digest and details to the hidden file in the destination folder, but that would exclude normal file operations performed using the Finder or at the command line. Folder storage of digests is therefore not suitable for many of the purposes for which we want integrity checking.

Flattened file formats

A family of file formats has been developed to preserve the additional metadata, including extended attributes, in macOS files. These include MacBinary, a binary format which combines data fork, extended attributes and additional metadata into a single binary file, and AppleDouble, in which the data fork is left unchanged and metadata combined into a shadow file of the same name prefixed with ._

MacBinary requires that its file contents are divided and reconstituted to their original form before they can be used, so has major drawbacks in this case. AppleDouble preserves the data fork for immediate use, which makes reconstitution unnecessary in many cases. However, the Finder and command tools don’t automatically recognise its shadow files on HFS+ and APFS file systems, so its use wouldn’t be transparent and would risk separation of the two components.

Extended attributes

Extended attributes (xattrs) are the macOS-supported mechanism for adding arbitrary metadata such as digests to files. They are fully supported on HFS+ and APFS file systems, and some other file systems such as FAT and its relatives use AppleDouble to preserve selected xattrs. Similarly, iCloud and most network transfer including AirDrop and supported file sharing (AFP, SMB) preserve selected xattrs, although some third-party cloud storage doesn’t. Xattrs are also stripped when burning to most removable media, unless the files are contained within a Disk Image.

Xattrs have acquired a reputation for fragility which has been largely founded on incomplete understanding of their behaviour. Xattrs have a flag system which determines whether they’re preserved or stripped during various operations. Appending #S to the name of a third-party xattr results in its being preserved during almost all file operations, including copying and duplicating, provided that the underlying file system supports xattrs.

Of the three options for storing digests for files, xattrs are the only one which ensures that, in the great majority of instances, the digest moves with its data, even across to different Macs, regardless of the location of the tagged file.

Xattrs and backups

One behaviour peculiar to xattrs for the storage of digests is that adding, changing or removing a xattr attached to a file doesn’t change the file’s modification datestamp, although the action is still recorded in the local FSEvents database. In some situations, this is advantageous, in others it isn’t helpful. These are best illustrated by looking at backups.

Most third-party backup systems for macOS rely on file modification dates to determine whether items need to be backed up. If a file is tagged with a xattr containing its current digest, that file’s modification date remains unchanged, and a fresh copy of the file (or even its xattrs) isn’t added to the backup.

One solution to this is for the user to tag items in a folder which is excluded from being backed up. Once they’re tagged, they can then be moved back to their normal locations, where they should be included in the next backup. This wouldn’t of course be feasible for the whole of a Home folder, but should prove straightforward for most important documents, whose integrity is essential.

This problem should also be reduced or eliminated by tagging documents with their digests immediately after editing their data so that they are backed up with those digests, rather than later in large batches.

Time Machine has the opposite problem. Because, in Catalina and earlier versions of macOS, it normally relies on the FSEvents database to determine which items are to be backed up afresh, and adding or changing xattrs is recorded as a change in FSEvents, every change made to stored digests in a xattr will result in the item being backed up in full. This can result in repeated full backups of files with very large data forks which haven’t changed, but a mere 32 bytes in a xattr have.

The best answers in both cases will only come with wider use of digests being saved to xattrs, and their accommodation in backup utilities. Time Machine’s behaviour already causes problems in other situations, for example when an app adds a quarantine flag to an existing file, which triggers the whole of that file to be copied into the next backup.

This leads us to store each SHA256 digest in a custom xattr attached to that file.

4Comments

Add yours

1

Bob on May 22, 2020 at 12:17 am

Using xattr is a clean solution if your goal during file copies is to preserve what the hash should have been, but not what it is. I fear there is no automated way out of this conundrum without a kext. The very fact that you copied the file should beg the question, shouldn’t you compute the hash again, just to be sure? And this very question drove me to define not simply a data structure, but a process around the motion of files. I chose the dot-file-in-each-directory method because I see it as the most portable method that remains entirely under my control and not at the whim of file system or utility behaviors. This of course condemns me to a self-managed process, but again, it’s completely under my control, and can be automated.

Consider a dot-file in each directory that contains a line for each file in that same directory, the file hash, the time-of-epoch date of the hash computation (“the last time I checked” date,) and the modification date of the file. Then it is up to a few scripts running out of cron to keep things organized. The tasks performed by these jobs in aggregate perform the following functions:

– [A process to register the hash] Scan the file system looking for files which don’t yet have a hash, and compute/store the hash for those new files.

– [A process to update the hash] Scan the dot-files looking for entries for which the file modification date has changed, and recompute the hash.

– [A process to remove the hash] Scan the dot-files looking for entries for files that no longer exist, and remove them.

– [A process to check the hash] Scan the dot-files, looking at the last-checked dates and if it is beyond a certain threshold, recompute the hash and compare it, just to be sure bit-rot hasn’t infiltrated the file. This is a LOT of data, so this process checkpoints itself, and each cycle checks a bit of the entire data store, as a continual process. If we set the age threshold carefully, it can run as a nearly continual process whereby every day, some of the data is checked, if one is that paranoid.

And additionally, a script to move a file. If it is in the same file system, then no hash computation is necessary, it will just mv the file and move the hash entry. If it is a different file system, then copy the file, flush the IO cache, compute the hash at the destination, compare to the stored value at the source, and if it matches, move the hash entry.

This is a solution for data at rest. Static files that won’t change, such as photos, videos, PDFs, and the like.

LikeLiked by 1 person
- 2
  
  hoakley on May 22, 2020 at 8:19 am
  
  Thank you.
  KEXTs are dead. Computing the hash again in the xattr-based solution is simple, and far less elaborate than storing it somewhere else. That’s what Dintch and Fintch already do. It’s surely far easier to have a background scrub running which simply works its way around your protected files comparing their saved hash against the currently-computed hash?
  And anything relying on changed file modification dates automatically fails to check for corruption which hasn’t occurred through normal file system calls, which are probably the more common now.
  Howard.
  
  LikeLike
3

Bob on May 22, 2020 at 11:33 am

“It’s surely far easier to have a background scrub running which simply works its way around your protected files comparing their saved hash against the currently-computed hash?”

I believe that’s what I’m doing. It is the last bullet in my list above.

“And anything relying on changed file modification dates automatically fails to check for corruption which hasn’t occurred through normal file system calls, which are probably the more common now.”

The last bullet checks for corruption on a time interval no matter what. I am only relying on the modification date to know that *I* changed the file and therefore the hash needs to be recomputed. If something else changed the file without touching the modification date, that would be considered a corruption by the scan, because the hash doesn’t match, yet the modification date didn’t change.

Perhaps the confusion is in my last statement. When I say it’s for data at rest / static files, I didn’t mean that in an absolute sense; poor wording on my part. The whole reason the process checks the modification date and updates the hash if necessary, is to accommodate file changes. However, in any scheme that computes hashes on files at an arbitrary time after they’re created, the hash is fairly useless because we cannot say that what the hash is “now” is what it should have been when we wrote the file. So I suppose I should have said that it is used for files that change infrequently, or not at all, because file changes are opportunities for undetected corruption due to the gap between when the file is written and when the hash is computed. And so this is only targeted at my data archives. Normally unchanging files that might change. This would include any end-product file derived from something else. For example, home movies must be converted to be loaded into iTunes. However, now that iTunes (and I assume TV) supports the h265 codec, I’m going back to video source and reconverting the old videos to h265 which is saving substantial space. Those changes would cause the hash to be recomputed.

In a perfect system, we would compute the hash at the moment of data creation, before it is even written to storage. This is the true hash value of the data (but even this value assumes no memory errors.) If you first write data to storage, then read it back to compute the hash, you have no guarantee that the hash is correct because you have no guarantee that the data was written and subsequently read properly, when you computed the hash. For this reason, I compute the hash at data creation, where I have that control. For example, video conversion:

$ converter | tee destination | hash computation > store

If I later have to copy this data (like for backups) the copy is compared against this baseline value. I mentioned previously that I use rsync for backups. The output from rsync is parsed, and the hashes computed for the copied data, and compared to the original baseilne hashes. Most of this is automated, and when I go to sleep at night, I know that not one bit has changed.

Now, when I initially offload home movies from the camera, I cannot escape the chance that the data was corrupted on the memory stick in the camera, but I’m as close to the source data as I can get, so I have to take this hash as baseline:

cat camera_file | tee destination | hash computation > store

I’m not a Mac developer. My reference to KEXT is the notion that certain system calls would have to be trapped in order to achieve instant, automatic hash computation that seamlessly follows the data. You mentioned “data fork” in your article. Is this a MacOS construct/facillity? I am familiar with the general concept, but am curious about MacOS having a built-in way to automatically perform the equivalent of ‘tee’ on file operations.

LikeLiked by 1 person
- 4
  
  hoakley on May 22, 2020 at 9:26 pm
  
  This actually needs to be done in the file system, as I have made clear several times. However, there’s no chance that APFS will ever do any integrity checking of this kind, so if we want to do it, we have to add it at a higher level.
  In doing so, the hash must be made part of the file itself, or it will become detached from the file. The way to do this in macOS is to make it part of the file metadata. As there’s no way to change the file attributes, that’s best done as an extended attribute, which is part of the file metadata.
  Traditional Mac (HFS) file had two forks – data and resource. Although they are now (in HFS+ and APFS) multi-forked, with support for a number of extended attributes, I and others still refer to the data part as the data fork, which I suppose is still correct.
  Howard.
  
  LikeLike

·Comments are closed.

Share this:

Related