hoakley April 17, 2020 Macs, Technology

How apps save changed files

I recently looked at how the Finder copies and moves files, and how it can create APFS clones. This doesn’t share the same process as when apps save their documents. This article examines that.

The simplest approach to saving a file in an app is to write out the changed contents to the original file, and let the file system handle any protection it can provide. As we currently can’t see inside APFS, it’s hard to know what that might be, and Apple’s current documentation on APFS doesn’t give any details either. This might involve cloning, but as that is currently transparent, and no information is left in macOS events logs, it’s impossible to know for sure. If you know of a tool which can cast light on this, please let me know. One tell-tale sign of this type of save is that the saved file has the same inode as the original.

For a great many apps, each time that you save a document, its file assumes a new inode, indicating that the old file has been replaced by a new one. This is because of the ‘atomic’ or ‘safe’ save which has taken place. My example here is my free Rich Text editor DelightEd, which relies on AppKit to perform its saving, and is here saving to the internal SSD of a T2 Mac running macOS 10.15.4, thus using APFS.

The original test document had a volfs reference of /.vol/16777232/103183697, which changed to /.vol/16777232/103184040 when a small change had been saved to it. This confirms that the saved file was created afresh for the purposes of the save process. Saving updated one of its extended attributes too: com.apple.lastuseddate#PS, and com.apple.macl may also change, depending on the app used to save the file.

When an open document is to be saved, the app first creates a temporary folder in /private/var/folders/sz/[long name]/T/TemporaryItems in which to keep its temporary files. It then creates a temporary file in that folder with the same name as the document, and makes that the destination for the save to take place. Once that save has successfullly completed, the new version in the temporary folder is ‘moved’ to the location of the original file by changing its path, and the temporary file and folder are deleted.

Because the updated file was originally created in the temporary folder, it has a new inode, but has copied across to it the same attributes and extended attributes as the original, subject to the LastUsedDate stored in com.apple.lastuseddate#PS being updated.

All isn’t complete, though, as AppKit provides full support for the macOS version system, which also needs to store a copy of the changed file. For that, macOS first creates a new folder and file in a staging folder inside that volume’s .DocumentRevisions-V100 folder. The updated file is copied there, and it’s marked as ‘dirty’ for the version management system to deal with.

safesave

There are some twists to this account. When you edit a file stored on an external volume, temporary files created for its safe save aren’t normally stored in /private/var/folders/sz/[long name]/T/TemporaryItems but locally on that volume. When you edit documents which are stored not in single files but folders which pose as files, such as RTFD, then only the file(s) which are actually changed are copied into the temporary folder, together with the folder which contains them. This should make folder-based file formats more efficient than those which wrap all the contents into single files.

What happens when a safe save goes wrong, then, which makes it any safer?

Writing the changed data to the original file is dangerous if any error occurs before that process completes. For example, if there is insufficient free disk space as the save is taking place, it will fail, leaving the file in an intermediate state, and probably corrupted. This can be mitigated by features such as copy on write, which APFS uses when it can.

If there is an error during a safe save, the original version of the file remains unscathed. Only when the new version of the file has been saved successfully is it swapped into place as the current version. As users of older versions of Microsoft Word will recall, this doesn’t always help: if you’re told that a save has failed, you want to quit the app, and you don’t want to do that until you’ve saved your work. But at least closing the app without saving the latest changes successfully doesn’t compromise the document’s existence.

Appendix: File system events transcribed from Crescendo

This is a simple save to an existing RTF file using DelightEd, which uses AppKit to handle its file saving using standard methods, in macOS 10.15.4 to the startup internal SSD (APFS).

File save
file::create path = "/private/var/folders/sz/qb23fyss56v96vmh60p8ft7r0000gn/T/TemporaryItems/(A Document Being Saved By DelightEd)"; file::create path = "/private/var/folders/sz/qb23fyss56v96vmh60p8ft7r0000gn/T/TemporaryItems/(A Document Being Saved By DelightEd)/COWtest1.rtf"; file::rename destdir = "/Users/hoakley/Documents/COWtest1.rtf"; destfile = ""; srcpath = "/private/var/folders/sz/qb23fyss56v96vmh60p8ft7r0000gn/T/TemporaryItems/(A Document Being Saved By DelightEd)/COWtest1.rtf"; file::rename destdir = "/private/var/folders/sz/qb23fyss56v96vmh60p8ft7r0000gn/T/TemporaryItems/(A Document Being Saved By DelightEd)/COWtest1.rtf"; destfile = ""; srcpath = "/Users/hoakley/Documents/COWtest1.rtf"; file::unlink dir = "/private/var/folders/sz/qb23fyss56v96vmh60p8ft7r0000gn/T/TemporaryItems/(A Document Being Saved By DelightEd)"; path = "/private/var/folders/sz/qb23fyss56v96vmh60p8ft7r0000gn/T/TemporaryItems/(A Document Being Saved By DelightEd)/COWtest1.rtf"; file::unlink dir = "/private/var/folders/sz/qb23fyss56v96vmh60p8ft7r0000gn/T/TemporaryItems"; path = "/private/var/folders/sz/qb23fyss56v96vmh60p8ft7r0000gn/T/TemporaryItems/(A Document Being Saved By DelightEd)";

Version save
file::create path = "/System/Volumes/Data/.DocumentRevisions-V100/staging/501-77493-zmUHkH2l/laAZPBi8"; file::create path = "/System/Volumes/Data/.DocumentRevisions-V100/staging/501-77493-zmUHkH2l/laAZPBi8/66D46788-4066-49D0-ABD4-62AB5BC769BF.rtf"; file::rename destdir = "/System/Volumes/Data/.DocumentRevisions-V100/PerUID/501/3373/com.apple.documentVersions"; destfile = "66D46788-4066-49D0-ABD4-62AB5BC769BF.rtf"; srcpath = "/System/Volumes/Data/.DocumentRevisions-V100/staging/501-77493-zmUHkH2l/laAZPBi8/66D46788-4066-49D0-ABD4-62AB5BC769BF.rtf"; file::unlink dir = "/System/Volumes/Data/.DocumentRevisions-V100/staging/501-77493-zmUHkH2l"; path = "/System/Volumes/Data/.DocumentRevisions-V100/staging/501-77493-zmUHkH2l/laAZPBi8"; file::create path = "/System/Volumes/Data/.DocumentRevisions-V100/.cs/ChunkStoreDirty"; file::unlink dir = "/System/Volumes/Data/.DocumentRevisions-V100/.cs"; path = "/System/Volumes/Data/.DocumentRevisions-V100/.cs/ChunkStoreDirty";

7Comments

Add yours

1

Dominic on April 17, 2020 at 7:58 am

Being a command-line person, I tried

$ cp -c index.html indexclone.html # index.html just happened to be my chosen victim file
$ ls -i index.html indexclone.html
5375817 index.html 6067134 indexclone.html
$ echo Change >> indexclone.html
$ ls -i index.html indexclone.html
5375817 index.html 6067134 indexclone.html

The inode of the clone is different from that of the original, and does not change when the clone is subsequently updated. Looking at the files with Precize (thank you!), the VolFS paths are, as expected /.vol/16777220/5375817 and /.vol/16777220/6067134. So what is it that differs between my test and yours?

(It strikes me that the “Biskus APFS Capture” forensics app, which you mentioned back in 2017, and which I do not have, might throw some light.)

LikeLiked by 1 person
- 2
  
  hoakley on April 17, 2020 at 8:06 am
  
  Thank you.
  That’s because this isn’t an app saving the file – cp -c creates a clone, which shares the data but unlike a hard link does have a different inode, because it’s a different file and not a link.
  If you were to edit that cloned HTML file in BBEdit, you’d also not see any change in inode, because BBEdit saves in place rather than using a regular ‘safe save’. Use an AppKit app which does use standard ‘safe save’, and the inode changes because of the mechanism I’ve detailed above.
  Confusing, isn’t it?
  Howard.
  
  LikeLike
3

Joss on April 17, 2020 at 8:35 am

You could add the sidenote that this atomic behavior, as I had to learn just recently, destroys the functionality of some Unix flags and ACEs (access control entries), or more precisely: makes them irrelevant for the most part. If you want to protect a file from deletion, but otherwise still want to be able to move/rename it, and modify its contents, you can’t do that anymore if you plan on using/editing the file in a GUI application, because that app needs to unlink the original file, when new contents are saved. That’s probably why Apple removed (never implemented in the first place?) support for FreeBSD’s uunlnk flag. It still exists on macOS, hidden away in the file system dungeon, but can’t be created with the `chflags` command, and can’t be read with stat/ls, and if you find a different way to create it, macOS will ignore it—and will remove it eventually, at the very latest when you try to save or modify such a file. Apple apparently couldn’t remove the sunlnk flag, so they restricted its use to very few system files, and you can only create & remove the flag from Recovery Mode (or single-user mode). But the append-only flags still exist (uappnd & sappnd), as does the ACL system where you can protect a file against deletion with the “deny delete” entry. If you apply one of those to a file, and then try to edit the file in a GUI app, macOS will tell you that it is “locked”. It’s not locked, of course, because locking a file means applying either the uchg or schg flag, but that’s what macOS tells you. It’s still fine to modify these kinds of files if you do it on the command line, but as soon as (most? all?) GUI apps come into play, it’s a no-go.

So you might think that this shows that the venerable Unix stuff isn’t all suited for modern operating systems, in this case when it comes to protecting user files from corruption. And that’s probably true. So what did Apple do? At least they chose to stick with the uchg flag in userland, which they call “Locked” in Finder (file info panel): locked is locked, not quasi-locked, so it’s probably the easiest to deal with. But they concocted a complex workaround, namely they let you unlock the file when you want to edit it (basically `chflags nouchg`). But once you’ve saved/closed the file, the uchg flag isn’t written back to the new version (new file); for that another piece of “workaround code” would be necessary, and to this date I’ve only found one application that re-adds the uchg flag to a previously locked file after macOS unlocked it.

So if Apple created this (at least partial) workaround for dealing with uchg in the GUI, why didn’t they do the same for the other “Unix stuff” mentioned above? Granted, handling append-only in a GUI app would be very complicated, so I understand why they didn’t do it in this case, but the uunlnk flag (or at least the “deny delete” ACE) could be easily dealt with: all macOS would need to do is re-add this flag or ACE to the new file after the user has edited its contents. Extended attributes are recreated for the new file, so why not this? Beats me. Maybe because “XNU”, because it’s “not Unix”? But in reality, there’s a whole Unix world underneath, no matter what Apple say, but for some reason they often choose to go against it instead of making it work in the modern world.

LikeLiked by 1 person
- 4
  
  hoakley on April 17, 2020 at 9:26 am
  
  Thank you.
  Well, it isn’t even that straightforward. Because Safe Saves aren’t part of the file system, they work differently in different apps. Above I have described what comes standard by default in AppKit SDK 10.15. Behaviour with iWork apps is different again, in that some extended attributes aren’t preserved in saved files, even when they have S flags attached, which should preserve them. So I’m not in the least bit surprised that you’re seeing differences elsewhere too.
  So the whole article should have the caveat that what’s described here for one app will work differently for others. But DelightEd does its saves by the book in AppKit, so is close to standard, if there is such a thing.
  Howard.
  
  LikeLike
  - 5
    
    Joss on April 17, 2020 at 9:42 am
    
    You’re correct. Microsoft Office applications use their own temp location, right? And the macOS screenshot utility seems to create the temp file as a dotfile in the dedicated screenshots folder. In the end I’m not super-keen on supporting all Unix functionality. I get that trying to make old stuff work today can create unnecessary complexity, so I’d rather go for something new, and nowadays that seems to be extended attributes: it would be great to have an XA (or a set of XAs) to tell macOS how to deal with files, with access, what not to do etc., something like a modern version of the old system of flags & ACLs, but one that works for all GUI apps, for the command-line, for Finder etc., XAs that are passed on across atomic writes and proprietary temp writes… Unix flags 2.0, if you want. Just shooting from the hip here, of course. 😉
    
    LikeLike
6

Frank on April 20, 2020 at 11:07 pm

Regarding saves to a network drive, I assume they would try to use the same “save local to that drive’ approach that you’ve described above for externals. But that seems to pose a problem because I simply can’t find a way to get TextEdit, DelightEd, Preview to do the save. Just errors every time, and now with your notes above I think I can see why. Only option I’ve found is to save to a new name then delete/rename. Wondering if you know of another approach? For all the benefits of versioning etc., not being able to update/save in place a basic 1 line plain text document is just, incredible.

LikeLiked by 1 person
- 7
  
  hoakley on April 20, 2020 at 11:12 pm
  
  Thank you.
  I don’t think that’s related to the issues above: I suspect that it’s a privacy issue, if you’re running Catalina, or permissions if it an earlier version of macOS. For Catalina, try adding the app to the Full Disk Access list in the Privacy tab.
  Howard.
  
  LikeLike

·Comments are closed.

Share this:

Like this:

Related