hoakley June 25, 2017 Macs, Technology

Last Week on My Mac: Testing trust in system upgrades and updates

The relationship of trust between an operating system supplier and the user is very complex. Apple makes it more so by its extraordinary secrecy, a policy which last week was brought into ridicule by the leaking of instructions about that policy.

I’m not here referring to the very reasonable desire to prevent competitors from knowing in advance of Apple’s plans, but to far simpler matters, such as what a system update actually changes and fixes.

Apple occasionally lets slip an insight into what is actually going on when you are waiting for an update to install – something the WWDC session on APFS revealed earlier this month.

Most of us only learned that iOS 10.3 had changed the file system on all our updated devices to Apple’s new APFS after we had updated. Maybe we should have waited a few days until the early adopters had discovered this, but if you took the trouble to read the official About this Update at the time, there was no mention of the change of file system. There were twenty other more important changes, such as Podcast shows or episodes are shareable to Messages with full playback support, but no mention of APFS.

Only if you happened to study the developers’ iOS SDK Release Notes for iOS 10.3 might you have seen that “When you update to iOS 10.3, your iOS device will update its file system to Apple File System (APFS).”

It was only in that WWDC session, though, that Apple’s Eric Tamura revealed that this was not the first update to iOS which had performed conversion of the file system to APFS. Apparently dry run tests had been included in the initial iOS 10.0 upgrade, and the 10.1 and 10.2 updates – in September, October, and December 2016, when many of us were still barely aware of APFS, and struggling to cope with iOS 10 and macOS Sierra.

According to Tamura, those dry runs involved performing the conversion of file system metadata exactly as was intended in the release conversion, but instead of then completing the process by writing out the new APFS ‘super block’ and then removing the old metadata, it reported back to Apple, then removed the converted metadata.

So far, Apple has not admitted to running any similar tests or dry runs during macOS updates; given the much greater scale of what would be involved in doing this with a macOS startup volume, I suspect that it probably hasn’t either. But as we were not told at the time – or even after those iOS updates – we simply have no way of knowing.

If you ignore the fact that hundreds of millions of iOS users have, it appears, unwittingly been alpha-testers for iOS 10.3, in which the conversion finally took place for real, what Apple did might appear a stroke of genius.

What Apple actually seems to have done was to unleash those alpha test file system conversion tools as part of what purported to be a final release quality system software update. It did not inform users beforehand or afterwards, nor did it give anyone the opportunity to decline to take part in its alpha-testing programme.

Now Apple would no doubt defend itself by arguing that the dry run conversion was designed so that it did not cause any problems: it only wrote to unused areas of storage, and cleaned up after itself. Well, that’s what they hoped it would do, and the whole purpose of carrying out such large-scale tests was to determine whether the conversion would work, and what percentage of iOS devices would suffer problems. Given that they were trying to determine those unknowns, Apple did not know prior to those dry runs that they might not trash a significant proportion of updated systems.

It would be equally vacuous to claim that users are always advised to ensure that they have full backups before all system updates. Apple knows full well that a great many users don’t, and that many users will trust Apple not to release software which has not been properly tested before release.

If you went to a medical practitioner and were treated for a condition, then several months later discovered that your treatment was in fact part of a trial, you would be rightly incensed, considering that your doctor had breached medical ethics. That would be as true if your doctor claimed that the drug used had been shown to be safe when used to treat other conditions, as it would if it was a completely experimental drug. Before you can take part in any trial of a drug or treatment, you must give fully informed consent as a genuine volunteer.

I felt uneasy from the first moment that I heard the story of these dry run tests on the iOS APFS conversion tool. The more that I think about how Apple seems to have treated those hundreds of millions of unsuspecting users, the more strongly that I feel that what Apple apparently did betrayed the trust that we put in our operating system supplier. If Apple wishes to bundle any more alpha tests in regular system software updates, then it surely must both warn users before they decide whether to update, and give us the option not to take part.

Knowingly and secretly installing such test software under cover of a series of system software updates would be a serious abuse of the trust between Apple and its users.

7Comments

Add yours

1

Sam J on June 26, 2017 at 6:08 pm

Your medical analogy is flawed. Studies which do not affect the actual care delivered, and are not personally identifiable, can be performed and reported without patient consent. This is often in the form of retroactive studies, they gather a bunch of records of patients who happened to get treatment A vs B and study the two. Another example is studies on body parts which are removed from you, which become property of the hospital, see Henrietta Lacks.

So Apple’s test, because it did not have the potential to affect your data outside normal filesystem operation, and the results were reported anonymously, would be acceptable from a medical ethics standpoint.

LikeLike
- 2
  
  hoakley on June 26, 2017 at 6:17 pm
  
  My medical analogy is very precisely measured. I wanted to undertake a formal trial to determine the effectiveness of a drug which was licensed for other related conditions, but not for the purpose for which we were already using it. For the trial to be sound, it had to be undertaken in a randomised double-blind controlled design. In order to do so, it required fully informed consent on the part of participants.
  You are referring to clinical audit, which is quite different, and cannot be used with treatments which have not already been proven safe and efficacious.
  Apple’s test did have the potential to affect data outside normal filesystem operation. If it had had bugs, it could readily have trashed the file system. One of the purposes in testing it was to determine whether it did so or not. That was not a known and established fact before the tests – it was one of the hoped-for outcomes.
  Apple did not even attempt to inform us, let alone give us an opt-out. That is at best paternalistic and at worst abusive of our trust.
  I am also amazed that no one else has even thought to debate the ethics of Apple’s actions. Perhaps we are also so blinded by faith that Apple knows best?
  Howard.
  
  LikeLike
  - 3
    
    Sam J on June 27, 2017 at 1:19 am
    
    Nope. I was personally involved in a study which utilized a new imaging system on tissue samples which had consent requirements waived by the IRB because it was judged too difficult to get the consent of the number of patients required versus the benefit of the study. A concern of the IRB was whether the technique could disturb or destroy the samples before actual analysis, but it was shown that this risk was negligible.
    
    It’s stated in the law, 45 CFR 46.116(d). It says informed consent can be waived if there is minimal risk to the subject.
    
    This is exactly analogous. You make unfounded assumptions of the risk of a trial conversion. Can you provide any evidence that the test conversion has any more risk than writing an extra file to the disk?
    
    Allocating a few ordinary files has negligible additional risk, given the thousands of files that are already touched during an ordinary update, and even the number of files that were touched visiting this website. How do you even know that the APFS structures were even written to disk and not dumped to /dev/null?
    
    LikeLike
    - 4
      
      hoakley on June 27, 2017 at 5:20 am
      
      The ethics of medicine is driven by the trust that a patient puts in the doctor to treat them in accordance with best practice for the sole purpose of addressing their medical condition. I argue from UK and European ethics, law, and practice (I’m afraid I would never try to defend US law or practice, which has resulted in some notoriously unethical studies). Minimum risk is never a permissible excuse here, nor is difficulty in obtaining consent, and quite rightly so, as it is the doctor who makes those assessments not the patients.
      In this case, Apple performed what is known in human research as a deceptive study. We trust Apple that operating system updates have gone through alpha and beta testing, and that Apple has established with reasonable confidence that the changes made are for our benefit. In this case, Apple deceived us by including components which had nothing to do with the update, but which were a mass experiment on our devices and our data.
      We have no idea how many devices had data corrupted or were trashed altogether as a result of these tests, as Apple does not even have the decency to inform us – it continues the deception. However, there were individuals who had to wipe and re-install when they upgraded to 10.0, and to 10.1 and 10.2 – you don’t have to look far to find them. Were there many more than in previous non-experimental updates? Only Apple knows and they’re keeping quiet.
      According to the account in that video (which you appear not to have viewed), the updates did write the new APFS metadata to device storage. There’s even a nice graphic to explain that.
      Why should Apple have done that without telling us before or afterwards? Because it knows that many users would not wish to have their devices used as test-beds in this way. In other words, it succeeded only by betraying the trust of those users. If we wanted to take part in Apple’s tests programmes, we could sign up to a beta-test programme. Apple secretly thrust us all into an alpha-test programme. That betrays that trust.
      Howard.
      
      LikeLike
5

Tony on July 1, 2017 at 4:28 pm

Firstly, thanks for the link to the video. I have been musing on your concerns about Apple’s dry runs but before commenting on that, there was some important information about a topic discussed here a little while ago.

Apple explained how iOS 11 will deal with Unicode normalisation. In brief, new devices (or those reinitialised/reloaded) will have file names normalised as they are written to the file system. In devices previously converted to APFS, unless and until they are reinitialised, the FS will search for files using a default normalisation and, if a file is not found, search again using the ‘other’ normalisation. [This starts at about 5m 30s into the video].

I don’t intend to split hairs here but this seems to me like the Unicode performance that most of us wanted/expected and represents quite a lot of Unicode understanding for a Unicode-insensitive FS; I think it is a good step forward (but I have always taken a pragmatic view of Unicode and HFS+). I found the video ambiguous on macOS but I can’t imagine that it won’t follow suit.

On to Apple’s conversion dry runs in iOS 10. What I *think* they did was to scan the volume, creating new (extra, different format) metadata on the drive. They then assessed whether this process was as expected, reported back the conclusions and erased the new metadata.

This process had potential for error, I agree. Information was sent to Apple, I agree. I deduce that what was sent was only metadata about the FS metadata (i.e. no user content) and I, at least, have already given them permission to report back diagnostic data. The FS (HFS+ or APFS) writes metadata all the time and I would not expect to be asked/told about changes to its format. The process that they used, which used part of the APFS conversion software, was presumably beta tested in the iOS beta programme before it was unleashed on the rest of us; I think it is unfair to accuse them of making us unwitting participants in a beta test and it certainly wasn’t an alpha test (assuming that we share a common terminology here).

So, I think I understand your reservations about this test. There is clearly a line beyond which Apple should not go without explicit permission but I don’t personally think that they overstepped it. I *was* concerned at the risks to my data from the conversion-in-place of the FS so I feel I should actually applaud deep testing of it before actually changing my data. I think they executed it well and I respect that it is not an easy thing to do.

LikeLike
- 6
  
  hoakley on July 1, 2017 at 4:41 pm
  
  Thanks for your thoughtful comments.
  My concerns are not about transmission of any personal data, although if Apple did backload metadata for analysis that is even more concerning as users were certainly not specifically informed about that. Apple is quite specific (as it must be under UK and EU law) about what data is gathered, and that certainly isn’t included in its list.
  10.0, 10.1 and 10.2 were released and installed last year – before the end of the year – and the 10.3 betas, which should have formally included APFS, were not available until early this year, as far as I am aware.
  So the metadata converter was pre-beta. It had certainly not been tested in any beta release before it was secretly unleashed on ordinary users as part of iOS 10.0 in September 2016.
  If this had been performed by any company operating under an appropriate ISO quality standard, it would have violated their procedures and the standard.
  I well understand the importance of thoroughly testing such components. But to do so secretly on users in this way is not, and never has been, part of any understanding between Apple and those users, has it? Maybe I missed the bit in the iOS licence which says that Apple can inflict on me software which has not undergone testing prior to general release, just because Apple thinks that it won’t mess anything up. And can do so without telling us beforehand, or even afterwards, except as an aside in a developer briefing.
  All Apple had to do was to fess up before the update, and giving us the option to participate would have treated us like valued customers, perhaps – hence my lead-in about its obsessive secrecy.
  Howard.
  
  LikeLike
  - 7
    
    Tony on July 1, 2017 at 5:04 pm
    
    I am assuming that the convert-in-place logic was beta-tested by virtue of being in 10.0 to 10.2 betas (I think it is wrong to assume that it is the same software as APFS proper – it will presumably ultimately disappear). Hence I assume that the dry-run code that we ran had been through the full test sequence. I also expect that the dry run code evolved over those releases as they digested the results – I doubt the 10.0 code was a full test. That’s how I would do it anyway.
    
    I also assume that what went to Apple was a summary of status and anomalies found by the dry run (hence my metadata about metadata comment) so would be very distant from user data.
    
    Once again, I understand your reservations and it is good to have the discussion. Ironically, I for one would have worried less about the update-in-place if Apple had told us that they had progressively tested the functionality before ‘arming’ it. I think it is the only time I have ever waited a fortnight, watching the reaction, before installing an iOS update!
    
    LikeLike

·Comments are closed.

Share this:

Like this:

Related