hoakley April 22, 2022 Macs, Technology

What performance to expect from SSDs

To bring to an end what’s turned out to be a week getting inside SSDs, this article provides representative performance figures for different classes of SSD when used with Apple’s current range of M1 models.

Methods

The two Macs I’ve been testing on are a MacBook Pro M1 Pro 16-inch, and a Mac Studio Max, each with 32 GB of memory and an internal 2 TB SSD which is currently lightly loaded. Both are running Monterey 12.3.1 with no relevant system extensions, in Full Security mode, and with FileVault turned on. Testing was performed using:

AmorphousDiskMark 4.0, with Test Count set to 5, and Test Size set to 1 GiB.
Blackmagic Disk Speed Test 3.3, with Stress set at 5 GB.
Stibium 1.0, writing 10 repeats of Write Series, and one of Read Folder, without a restart between them.

The three external SSDs used here are:

A Samsung X5 512 GB SSD, connected using a CalDigit TB4 passive cable of 0.8 m.
A Crucial 750 GB CT750MX300SS with a VIA Labs (VLI) interface and a certified Thunderbolt 3 cable of 0.8 m.
A Samsung 860 EVO 2 TB with an ASMedia Technologies interface and a Belkin certified Thunderbolt 3 cable of 0.8 m.

I also refer to the results which so many of you have kindly included in comments to my articles this week, which give a good idea of the ranges of performance you’re experiencing.

All disks were connected directly to a Thunderbolt port on the Mac being tested.

Results

Detailed results are given in the table below.

In broad terms, SATA-USB SSDs achieved 300-450 MB/s transfer rates. Interestingly, the slower rates shown in the table were achieved over an indicated connection speed of 10 Gb/s, with the Studio Max; significantly higher transfer rates were measured on the MacBook Pro, although its indicated connection speed was only 5 Gb/s. Even under ideal conditions, limitations of the SATA interface used in cheaper SSDs and cases mean that transfer speeds over 500 MB/s are unusual.

NVMe SSDs connected via Thunderbolt 3 delivered excellent read speeds typically between 2.5 and 3.0 GB/s, that’s more than five times the best speed you’re going to achieve from SATA-USB. However, their performance can be let down, as seen here, by thermal throttling during write testing. This was most obvious with the tests in Stibium, which write a total of 54 GB in 160 files. That caused performance to be limited to 700 MB/s, and the best performance attained less than 2.3 GB/s write speed.

Newer designs and SSDs can stave off thermal throttling better than the Samsung X5, but when sustained writes are likely, only an actively cooled enclosure can prevent throttling from occurring. Newer SSDs can also achieve slightly better performance, but generally top out at 3 GB/s.

Internal SSDs in T2 and M1 Macs comfortably exceed the performance of external SSDs, and in the latest models are in a class of their own. With both M1 Pro and Max chips, a 2 TB internal SSD should exceed 6.5 GB/s read, and is even faster at more than 7 GB/s when writing. Because the two M1 models tested both have active cooling systems, even Stibium’s demanding write tests failed to result in thermal throttling. The beefy fan system in the Mac Studio should be even more resistant than the MacBook Pro 16-inch.

Thus, Apple’s internal SSDs in these models are more than twice the speed of external NVMe/TB3 SSDs, and more than 13 times the speed of the best SATA-USB SSDs.

There’s more

I don’t have access to a Studio Ultra to measure the performance of its internal SSD. It’s also important to remember that the performance of SSDs improves with larger capacity, although I don’t know of any simple rule of thumb which can be used to predict how much faster a 4 TB unit would be than 2 TB.

It’s easy to forget that the comparisons above aren’t fair. Of the three classes tested, only one has FileVault turned on, the internal SSD. To retain the privacy and security of the data stored on an external SSD, it would also need to be encrypted, which is likely to impose overhead and reduce performance further. Even with encryption enabled, the risk of theft or loss of a compact external SSD is often greater than that of a whole Mac.

Separate enclosures with active cooling systems have to be powered separately, from the same UPS as the Mac. Unfortunately, that doesn’t work for a notebook, as Apple’s notebooks don’t have support for automatic shutdown under the control of a UPS, although it does hold good for a Mac Studio. I don’t know of any (reasonably priced) SSD enclosure which has support for such UPS control.

Summary

SATA-USB 0.3-0.5 GB/s,
NVMe-TB 2-3 GB/s, but beware thermal throttling with sustained writes,
M1 Pro/Max internal SSD 6.5-7 GB/s with FileVault.

101Comments

Add yours

1

Michele Galvagno on April 22, 2022 at 7:24 am

I couldn’t help but notice you have FileVault on, and I recall you once saying you normally didn’t have it activated.
Any reason for this change?

LikeLiked by 1 person
- 2
  
  hoakley on April 22, 2022 at 8:00 am
  
  Yes, before the advent of the T2, I didn’t normally use FileVault for many reasons, including its overhead.
  With the T2 encryption occurs anyway, so turning FileVault on for internal SSDs comes at no cost at all, and that’s what I do for all my T2 and M1 Macs now. Why ever would you not?
  Howard.
  
  LikeLike
  - 3
    
    Michele Galvagno on April 22, 2022 at 8:10 am
    
    We once talked about it being not necessary if you didn’t have “sensitive” data.
    If the internal SSD is already encrypted (by the user login password, right?), why would I need an extra layer? Should I remember yet another password every time I login?
    I admit never delving into this before, but is there any drawbacks to it nowadays?
    
    LikeLiked by 1 person
    - 4
      
      hoakley on April 22, 2022 at 8:20 am
      
      Ah. Perhaps I should explain.
      By default, all T2 and M1 Macs encrypt the Data volume. This is performed in the disk controller, which is part of the chip. You can’t turn that off.
      With FileVault turned off, the encryption is performed by a key contained in the Secure Enclave, to which you don’t have access.
      When you turn FileVault on, all that it means is that your admin user’s password is then used as additional protection for the existing encryption key. There’s no additional password – I only enter my normal admin user password once, and that unlocks the Data volume by granting access to the key in the SEP.
      So there’s no additional encryption step, no change to the encrypted data, it’s all about controlling access to the already-encrypted volume.
      The drawback is that if you forget your password, you can’t unlock the disk – but Apple offers iCloud-based recovery, and if you’re so forgetful you can’t remember your login password, it’s perhaps time to give up computers!
      Is that clearer?
      Howard.
      
      LikeLiked by 1 person
    - 5
      
      Michele Galvagno on April 22, 2022 at 8:22 am
      
      Much clearer!
      Thank you! I guess on 2016 MBP (T1?) it’s not going to be good, but whenever I’ll be able to get my hands on a new Mac, I’ll just do that!
      
      LikeLiked by 1 person
    - 6
      
      hoakley on April 22, 2022 at 8:23 am
      
      Yes, the T1 doesn’t contain the disk controller etc. to make this possible. It’s one of the major features brought with the T2, and designed into the M1 from the start.
      Howard.
      
      LikeLiked by 1 person
7

Walt on April 22, 2022 at 9:45 am

Howard, another great series of articles. My own testing on my Mac Studio base Ultra with an internal 2TB SSD option shows slightly higher (6983MB/s) reads and higher (7690MB/s) writes using AmorphousDiskMark 4.0 5/1GiB test parameters. Similar testing of a 1TB Crucial SATA SSD using a ASMedia USB-C 3.1 Gen 2 SATA carrier sled was when I connected it to my to my Ultra via my CalDigit Element USB-4 Hub. The results were 528MB/s reads and 512MB/s writes. Connection directly to an Ultra TB port could only muster reads slightly over 400MB/s and writes barely above the 390MB/s. It’s still somewhat disappointing that M1 Thunderbolt direct connected external disks speeds are still lacking and one needs to connect them via an additional Thunderbolt dock/hub to achieve respectable transfer speeds.

LikeLiked by 1 person
- 8
  
  hoakley on April 22, 2022 at 9:27 pm
  
  Thank you, Walt. I’m sorry you’re having problems with that SATA SSD.
  While you’re here, could you please help me?
  I would like to know how the Ultra structures its cluster(s) of E cores. As you know, it’s the only M1 with 2 x 2 E cores. Those could either be loaded as one cluster of two, then the other if required. Or as a single cluster of 4, like the P cores. This should be most evident soon after starting up, with heavyweight services like mds running, and should be visible in the CPU History window of Activity Monitor. Could you please check for me, so I can get it right for an article here?
  Thank you.
  Howard.
  
  LikeLike
  - 9
    
    Walt on April 23, 2022 at 9:12 am
    
    Howard, I’ll have a look at the the Ultra E core configuration.
    
    LikeLiked by 1 person
    - 10
      
      hoakley on April 23, 2022 at 9:32 am
      
      Thank you – that’s very kind.
      Howard.
      
      LikeLike
11

EcleX on April 22, 2022 at 1:40 pm

Thanks. If possible, it would be interesting to see also the random read/write benchmarks: Input/Output Operations Per Second (IOPS). Actually, unless you are dealing with large monolithic files like movies, random benchmarks are much important than sequential ones (MB/s). Both AmorphousDiskMark and ATTO Disk Benchmark show them on Mac.

LikeLiked by 1 person
- 12
  
  EcleX on April 22, 2022 at 4:18 pm
  
  On the other hand, how do you measure benchmarks with Blackmagic Disk Speed Test? I ask because it loops again and again for ever, and does not show average or end benchmarks, as other tests do. Thanks.
  
  LikeLiked by 1 person
  - 13
    
    hoakley on April 22, 2022 at 9:40 pm
    
    You have to watch the figures and extract one which is fairly consistent. Yes, it’s poor, but Blackmagic isn’t designed for this type of performance testing.
    Howard.
    
    LikeLike
- 14
  
  hoakley on April 22, 2022 at 9:35 pm
  
  Thank you.
  I don’t use IOPS, as few people understand them, and they’re more relevant to hard disks than SSDs.
  There’s a simple conversion from MB/s to IOPS: take the transfer rate in MB/s, multiply by 10^6, and divide by the block size in bytes. And that’s the catch with IOPS – it’s relative to the number of blocks transferred rather than the amount of data.
  Of the apps which could do that, there are only Amorphous and my own Stibium. ATTO doesn’t run native on M1 Macs, and in any case tends to generate rubbish on them.
  As very few readers here would know what to do with random read/write results in IOPS, I don’t give them. You assert that they’re more important, which simply isn’t correct. For users, transferring large files is usually the most readily perceived delay. And in SSDs, random read/write are far less revealing than on hard disks, where they were developed, primarily for servers, I think.
  Howard.
  
  LikeLike
  - 15
    
    EcleX on April 23, 2022 at 8:52 am
    
    Thanks. Random read/write (IOPS) are relevant when dealing with many small files. For instance, when booting or in general Mac routines and workflows, because most times the Mac does that (handle many small files all the time). The difference may be quite relevant. For instance, two versus 10 seconds to boot in the absence of third-party installations that may delay booting, of course.
    
    That explains why you do not notice performance difference when booting a Mac like the iMac 27-inch 5K retina (Mac18,3; mid 2017) from its internal faster 2 TB SSD (2,912 MB/s read & 2,126 MB/s write sequential), as compared when you boot it from a much slower (526 MB/s read & 491 MB/s write sequential) USB external one like the Samsung Portable SSD T5 2 TB (when connected to the thunderbolt port, not the USB port, albeit the disk is USB, as said, which generates 436 & 428 MB/s, and has slower performance). That may sound shocking and unbelievable to many users, but is what happens (I experience it all the time for years when booting that way). See to believe. Unless you are dealing with large monolithic files like movies, in which sequential speeds make a difference.
    
    There are huge differences in IOPS between SSD and manufacturers hide them except in high-end SSD, because they are usually low. The highest IOPS can be found in Intel Xpoint Optane and Samsung Z-SSD. Yet, they are much more expensive and consume much more electricity, generating also more heat and having smaller sizes than other “standard” SSD. So much, that they are not used in Macs, as far as I know.
    
    LikeLiked by 1 person
    - 16
      
      hoakley on April 23, 2022 at 9:44 am
      
      Thank you.
      So, a simple question for you: what random read/write in IOPS would you expect a T2 or M1 Mac to achieve to be ‘good’ in your experience?
      I’m still not quoting IOPS for any tests here, as so few people use them. It would be as useful as measuring something in ferkins per square sundance. And without other comparable measures, it’s useless.
      In my extensive experience of trying to measure performance in small files on SSDs, any figure claimed is pure fiction. SSDs are built with large buffers which can’t be disabled during normal benchmarking, and most of your measurements are therefore to the buffer memory rather than to storage. There’s huge variation seen when performing such measurements on the same SSD, which means that any figure you try to give is dominated by noise and essentially unreproducible.
      You may not have noticed, but in Big Sur and later macOS changed its boot process, and it now boots from a snapshot. That means that most of the boot process proceeds without random writes, and you’d need to measure random reads not from a normal file system, but from a mounted sealed and signed snapshot, which can’t be done by normal benchmarking tools (although Stibium should be able to).
      Of course, unless you’re running a recent T2 or M1 Mac, you probably won’t notice much difference in booting times, because the process isn’t as disk-bound as you seem to think. But I can assure you that booting an M1 Mac from an external SSD is significantly slower than from its internal SSD, and transfer speed does play a part in that.
      So trying to use an unreliable measure of SSD performance expressed in obscure units isn’t useful in gaining insight into storage.
      Howard.
      
      LikeLike
    - 17
      
      EcleX on April 23, 2022 at 12:42 pm
      
      Thanks. Booting from external SSD takes much longer than booting from the internal one in my example abobe. That is why I understand that the difference is due to random read/write (or just read with the new Macs, as you said), since the internal is much faster than the external in sequential read or sequential write. What is similar is performance on the routine work with the Mac, except when dealing with large monolithic files, where sequential read or sequential write makes a difference.
      
      LikeLiked by 1 person
    - 18
      
      hoakley on April 23, 2022 at 1:38 pm
      
      Thank you.
      I’m sorry, but earlier you wrote:
      “That explains why you do not notice performance difference when booting a Mac like the iMac 27-inch 5K retina (Mac18,3; mid 2017) from its internal faster 2 TB SSD (2,912 MB/s read & 2,126 MB/s write sequential), as compared when you boot it from a much slower (526 MB/s read & 491 MB/s write sequential) USB external one like the Samsung Portable SSD T5 2 TB (when connected to the thunderbolt port, not the USB port, albeit the disk is USB, as said, which generates 436 & 428 MB/s, and has slower performance).”
      So do you notice a difference in performance, or don’t you? What times from pushing the Power button to the appearance of the login window (assuming all else is equal)?
      Howard.
      
      LikeLike
    - 19
      
      EcleX on April 23, 2022 at 3:09 pm
      
      Thanks. Sorry for not making it clear. I meant performance working all-day-long when booting from one disk or the other (which feel alike to the user, except for large monolithic files); not the time taken to boot, except when I indicate such time to boot specifically (which is much longer when booting from external disks when both are SSD; faster from external if internal is mechanical rotational, as expected). That is based on experience for many years using different Macs and booting each single day (all Macs turned off overnight).
      
      LikeLiked by 1 person
    - 20
      
      hoakley on April 23, 2022 at 3:49 pm
      
      Thank you. I think I’m still lost here. Do you have any figures which illustrate the point you’re trying to make, please? They may be a bit clearer.
      Howard.
      
      LikeLike
    - 21
      
      EcleX on April 23, 2022 at 6:17 pm
      
      These are approximate times, as far as I recall:
      
      iMac 5K boots from internal 2TB in 15 s without third party soft and 22 s with it.
      iMac 5K boots from external Samsung Portable SSD T5 2TB in 45 s without third party soft & 1 min 18 s with it
      
      LikeLiked by 1 person
    - 22
      
      hoakley on April 23, 2022 at 8:24 pm
      
      45-78 seconds to boot? That’s from pressing the Power button to the Login Window? I think I used to get better times with a hard disk.
      Howard.
      
      LikeLike
    - 23
      
      EcleX on April 24, 2022 at 5:29 pm
      
      Here are figures, as tested today with third-party software installed (from pressing power button to Finder showing Desktop and its icons; no password required to boot) on macOS 12.3.1 (21E258) Monterey:
      
      Mac 5K boots from internal 2TB in 44 seconds.
      iMac 5K boots from external Samsung Portable SSD T5 2TB in 2 minutes.
      
      Yet, as said, no performance difference when working in one or the other all day long, except when dealing with large monolithic files like movies, in which case the internal SSD shows that it is faster. When many files are handled (which may represent 99% of work for many if not most users), the random read/write (IOPS) makes the difference.
      
      Note: when not third party software is installed, the boot time is much faster, as expected.
      
      LikeLiked by 1 person
    - 24
      
      hoakley on April 24, 2022 at 6:44 pm
      
      Thank you.
      So the only numbers that you have demonstrate that the ‘faster’ SSD results in a faster boot time. You don’t give random r/w figures in IOPS for the two SSDs, and you don’t give any benchmark for your claim that there’s “no performance difference”.
      You also repeat your unsubstantiated claim that the only difference is “with large monolithic files like movies” despite my telling you that Stibium measures performance on files between 2 MB and 2 GB in size, which falls far short of your claim.
      Howard.
      
      LikeLike
    - 25
      
      EcleX on April 24, 2022 at 7:16 pm
      
      I am just reporting my experience that the user does note notice difference in performance when working all day long after booting from a fast internal SSD or a slower external SSD. That is surprising, yet due to the fact that work of most users involve small files for which random read/write is the key factor instead of sequential speed. Additionally, slower SATA SSD consume less power, so if you want to boot a MacBook from it to work all day (all your stuff in your external SSD is useful in some workflows), battery will last longer than when booting from an external faster NVMe drive. Others report the same:
      
      [copied material and links removed]
      
      That is what I meant. So, what is the best portable external SSD? It depends on your needs. On what you want to do with it. Because faster sequential speed might not mean faster performance, depending on what you do with it, as said, being more expensive and reducing battery life when used to boot a MacBook.
      
      LikeLiked by 1 person
    - 26
      
      hoakley on April 24, 2022 at 7:24 pm
      
      I have warned you before: please do not post long quotes and links to other sites here, particularly when it has nothing to do with this article, which isn’t about choosing the best portable external SSD, is it?
      Have you ever heard of confirmation bias?
      Howard.
      
      LikeLike
    - 27
      
      EcleX on April 24, 2022 at 7:41 pm
      
      Sorry about that. I posted the three paragraphs with their links because in the past I have just indicated the information without links and you said that you could not find such information, so I posted the links and that was OK. If you want, I could post the paragraphs without links if you want, so that the readers have the full information. I think that it is good to have references from different people, so that readers can reach their own conclusions. Thanks in any case and twice for this great site. Honestly!
      
      LikeLiked by 1 person
    - 28
      
      hoakley on April 24, 2022 at 7:45 pm
      
      The content that I removed had nothing whatsoever to do with this article.
      You have insisted repeatedly that measurements of random r/w performance in IOPS are very important. What measurements have you made of such important performance indicators, then?
      Several of us have posted figures that we have measured, and the numbers appear meaningless. Do you have any figures that you have measured using Amorphous, and can explain to us how they lead you to conclude that they’re such important indicators of real-world performance on modern Macs?
      Howard.
      
      LikeLike
- 29
  
  Ric Ford on April 23, 2022 at 4:32 pm
  
  For what it’s worth, I happened to have IOPS results from an 8GB/256GB M1 Mini as well as a Samsung 970 EVO in a TekQ Cube enclosure connected by Thunderbolt to the same machine.
  
  256GB M1 Mini internal SSD:
  
  Rnd4K QD64 (R/W): 262991/23166
  
  Rnd4K QD1 (R/W): 16088/9868
  
  TekQ/970 EVO:
  
  Rnd4K QD64 (R/W): 330440/3188
  
  Rnd4K QD1 (R/W): 16062/3248
  
  I’d be curious to know if you can derive any useful information from those.
  
  LikeLiked by 1 person
  - 30
    
    Ric Ford on April 23, 2022 at 4:33 pm
    
    “For what it’s worth, I happened to have IOPS results …”
    
    from AmorphousDiskMark
    
    LikeLiked by 1 person
  - 31
    
    Ric Ford on April 23, 2022 at 4:55 pm
    
    AmorphousDiskMark IOPS results from a 2017 iMac 5K (FileVault disabled):
    
    Internal SSD (Apple SSD 1024L):
    
    Rnd4K QD64 (R/W): 325868/83708
    
    Rnd4K QD1 (R/W): 12432/111062
    
    TekQ Cube with 2TB Samsung 980 PRO
    
    Rnd4K QD64 (R/W): 358612/84018
    
    Rnd4K QD1 (R/W): 17055/109842
    
    LikeLiked by 1 person
    - 32
      
      Ric Ford on April 23, 2022 at 4:57 pm
      
      “results from a 2017 iMac 5K”
      
      with 40 GB of RAM
      
      LikeLiked by 1 person
  - 33
    
    hoakley on April 23, 2022 at 8:18 pm
    
    Here are IOPS from my Mac Studio Max (2 TB internal):
    RND4K QD64 read 164528 write 49773
    RND4K QD1 read 12505 write 8329
    They look exciting, but I’m not sure that I can make any sense of them, particularly in comparison with yours.
    Maybe I should add my shoe size?
    Howard.
    
    LikeLiked by 1 person
34

Simon on April 22, 2022 at 1:56 pm

Has anybody ever seen a Mac push ~1 GB/s over a gen-2 connection? In principle that should be possible and on Win/Linux I see such performance using appropriately cooled external flash with decent gen-2 bridges. But never on a Mac. The best here seems to be around 500 MB/s for which plain old vanilla USB3 or gen-1 would be plenty. Why can Macs not benefit form gen 2? I have to wonder if Apple screwed up something with their USB implementation or drivers.

LikeLiked by 1 person
- 35
  
  hoakley on April 22, 2022 at 9:36 pm
  
  Thank you. I’ve just looked at exactly that question, in this article.
  Howard.
  
  LikeLike
  - 36
    
    Simon on April 22, 2022 at 10:37 pm
    
    I did see that article. And it’s fine. But, and no offense intended here, while it raises the same questions, it still doesn’t deliver the answers I’m afraid. I would hope to eventually learn from somebody within Apple or at least with Apple-inside knowledge, what the underlying issue is.
    
    I admit, I don’t care about USB-only ports because I don’t have a Mac Studio nor am I interested in getting one. I do have a pretty new M1 Pro MBP which offers three identical TB4 ports that all claim to support 3.1 gen 2. And indeed, I have several SATA and nVME bridges that support gen2 and they do connect at 10 Gbps (as per System profiler). The problem is that I never see throughput on the NVMe storage beyond about 500 MB/s, which means they are connecting no faster than if they were attached to an old 5 Gbps USB3 (or gen 1) port. That is unsatisfactory for a top-$ latest and greatest pro portable Mac. And to add insult to injury, I know for a fact this is a Mac-specific problem, since the same bridges connected to the same NVMe flash and the very same high-quality cables show almost 1 GB/s when connected to a cheap Linux PC’s gen 2 ports. SO we know the external hardware could do it. It’s the Mac (or macOS) that’s the problem.
    
    I don’t mean to minimize the value of your posts, Howard. And I remain highly appreciative of your testing and the summaries you publish on these topics. I guess I’m just tooting the same horn and hoping that eventually we will learn some hard facts that help us understand the oddities we’re observing.
    
    LikeLiked by 1 person
  - 37
    
    joevt on April 22, 2022 at 11:28 pm
    
    USB 3.0 = 5 Gbps with 8b/10b encoding = 4 Gbps of data = 500 MB/s but there’s protocol overhead which limits that to ≈463 MB/s (AmorphousDiskMark, Intel Mac mini 2018 with USB NVMe connected to 5 Gbps USB-A port).
    
    A benchmark of 500 MB/s is greater than USB 3.0 which means you have a 10 Gbps connection.
    
    OWC’s Rocket Yard blog has a couple posts about how certain configurations may improve USB speed of M1 Macs.
    “Get Up to 67% Faster External Drive Performance With Your M1 Mac!”
    “More Ways to Get Faster External Drive Speed With Your M1 Mac”
    
    LikeLiked by 2 people
38

Ric Ford on April 22, 2022 at 2:42 pm

Hi, Howard – I’d like to thank you publicly for the astonishing amount of great work you’re doing for the Mac community; it’s a tremendous service! I can confirm many of your observations here, including the awful performance of the Samsung X5 for anything more than a few short transfers. (Placing it on a cooling fan can help for extended transfers, but that’s pretty ridiculous.) Other Thunderbolt compact enclosures and storage devices offer far faster extended performance, but another critical factor besides thermal throttling is the speed of the flash storage chips themselves, which is limited and far below the numbers you’re seeing in your internal M1 Mac storage benchmarks. My own testing with an entry-level M1 Mac Mini (8GB/256GB) showed the same stellar speed for its internal SSD as you’re seeing in your M1 Macs for modest file sizes, but transfer performance dropped dramatically when using 16GB files for SSD performance benchmarks (e.g. in AJA System Test Lite).

I see the same effect with an OWC Envoy Pro FX connected via Thunderbolt (on an iMac 5K): performance starts off relatively fast but drops dramatically as the internal cache fills for a large file (e.g. 16GB). These design issues are discussed e.g. in a review of Samsung’s brilliantly fast 980 PRO SSD here:
https://www.anandtech.com/show/16087/the-samsung-980-pro-pcie-4-ssd-review/3

Transfer rates around 7000 MB/s. indicate RAM caching, which is consistent with test parameters of 32GB RAM and 2GB file sizes. That said, the M1 Mac storage *system* (i.e. processors, RAM, software, and SSD) certainly performs as well as you suggest and is a joy to experience in everyday operation even at the entry level when not pushing huge files or RAM demands.

A quick note on FileVault encryption: As you explained in comments, the latest Macs have no penalty for using FileVault with internal storage, and I don’t believe there’s a noticeable performance penalty even for external storage. Years ago, Intel added hardware to its processors that greatly speeded up FileVault-type encryption, making the penalty negligible and I think the M1/neural hardware is far faster.

Lastly, I’d like to highlight a huge issue involving interfaces, which has some confounding elements. I’ve recently been testing the Samsung 980 PRO NVMe SSD with a variety of enclosures, computers and interfaces. In an Aukmax Thunderbolt enclosure, I see transfer rates around 2500 MB/s. on an iMac 5K. In a TekQ 10Gbps USB-C enclosure, that speed drops to around 970 MB/s. to the iMac’s USB-C port. Attach it to the iMac’s USB-A port, and speed drops to about 450 MB/s. Shockingly, when I moved the same device to a 2015 MacBook Pro with an ostensibly identical USB-A port, speeds dropped to around 350 MB/s. (The same thing happened with an OWC Envoy Pro FX, which an NVMe SSD storage device that works with either USB or Thunderbolt.)

I then tried testing on the 2015 MacBook Pro with a CalDigit TS3+ dock (connected via Apple’s Thunderbolt 3-Thunderbolt 2 adapter), which offers a 10Gbps USB-C port, as well as 5Gbps ports. Bizarrely, connecting the TekQ-980PRO device to the 10Gbps port at startup limited performance to about 350 MB/s. but attaching it only after booting raised speeds to the 700 MB/s. level – there must be different drivers loaded depending on when the device is attached with wildly different performance.

In summary, as you’ve helped explain, our storage performance clearly depends on several critical factors, including thermal throttling, caching, flash chip limitations, and critical interface and driver issues.

Thanks again for all your work!

LikeLiked by 2 people
- 39
  
  hoakley on April 22, 2022 at 9:39 pm
  
  Thank you, Ric, that’s very kind, especially considering your own long and distinguished service to the community.
  Funnily enough, tomorrow’s explainer is a basic account of FileVault. The latest estimate I have of its overhead on pre-T2 Intel Macs is 3%, although on slower, older models with hard disks, that’s likely to be a minimum.
  Howard.
  
  LikeLike
- 40
  
  joevt on April 22, 2022 at 11:52 pm
  
  For the drop on the 2015 MacBook Pro, the USB-A port is a chipset port? I would check the USB controller and the PCIe link rate.
  
  For the CalDigit TS3+, it has 4 USB controllers:
  – Two Fresco Logic FL1100 (5 Gbps -> 4 Gbps) connected at 5 GT/s (4 Gbps)
  – One ASMedia ASM1142 (10 Gbps -> 9.7 Gbps) connected at 8 GT/s (7.877 Gbps)
  – One Alpine Ridge (10 Gbps -> 9.7 Gbps)
  
  Connecting a USB 3.0 device to FL1100 is slower than connecting it to ASM1142 or Alpine Ridge because of the 5 GT/s PCIe link (USB 3.0 and 5 GT/s both do 4 Gbps of data but PCI overhead is greater than USB overhead).
  
  The ASM1142 may have negotiation problems which causes your 350 MB/s number at boot. If you connect it after boot, it may report a 5 Gbps link but actually be working at 10 Gbps (because of a bug in its firmware not setting the status register) but it’s limited by the 8 GT/s link (7.877 Gbps) so you get 700 MB/s instead of 970 MB/s.
  
  For 970 MB/s, you need to use the downstream Thunderbolt port of the CalDigit TS3+.
  
  LikeLiked by 2 people
  - 41
    
    Ric Ford on April 23, 2022 at 12:27 am
    
    “… For 970 MB/s, you need to use the downstream Thunderbolt port of the CalDigit TS3+….”
    
    Excellent info – thanks!
    
    I just confirmed this, connecting a 10Gbps NVMe enclosure (TekQ USB-C + Samsung 970 EVO) first to the CalDigit TS3 dock’s Thunderbolt port, which produced about 940 MB/s. while the dock’s 10Gpbs port only produced about 700 MB/s.
    
    [I’m reposting this because of initial confusion about how to attach it to the correct comment here.]
    
    LikeLiked by 1 person
42

joevt on April 23, 2022 at 12:15 am

Regarding NVMe in Thunderbolt enclosures: It may be that the slow writes are not caused by thermal throttling. AmorphousDiskMark tests are quick enough to not cause thermal throttling.

It is known that some NVMe do not perform properly (slow write speeds) in a Thunderbolt enclosure. You need to test them in a Mac or PC with a PCI 3.0 x4 slot to know the performance they can achieve with a non-Thunderbolt connection. I have some PCIe gen 4 NVMe that perform well in Thunderbolt enclosures (HFS+: 2727/2769, FAT:1947/2034 MB/s read/write, Mac mini 2018; and another at 2817/2738) but people have found some gen 4 NVMe that do not perform well in Thunderbolt.

I haven’t had any problem with any NVMe in a USB enclosure (no M1 Mac yet). They usually all can do ≈1000 MB/s, even an Intel 660p. I don’t do tests for cache size or thermal throttling or IOPS or random read/writes.

LikeLiked by 1 person
43

Ric Ford on April 23, 2022 at 12:25 am

“… For 970 MB/s, you need to use the downstream Thunderbolt port of the CalDigit TS3+….”

Excellent info – thanks!

I just confirmed this, connecting a 10Gbps NVMe enclosure (TekQ USB-C + Samsung 970 EVO) first to the CalDigit TS3 dock’s Thunderbolt port, which produced about 940 MB/s. while the dock’s 10Gpbs port only produced about 700 MB/s.

LikeLiked by 1 person
44

Ric Ford on April 23, 2022 at 12:37 am

I may have missed it, but I didn’t see mention here of a major issue with Thunderbolt performance on M1 Macs, where *not* having a display connected drastically reduces Thunderbolt performance.

https://valleau.art/blog/2022/04/07/m1-mac-external-drive-write-it-way-too-slow/

(I’m also collecting notes about M1 port problems here:
https://www.macintouch.com/post/23506/m1-mac-port-problems/ )

LikeLiked by 1 person
- 45
  
  hoakley on April 23, 2022 at 8:47 am
  
  Thank you, Ric.
  So next week I’ll turn to looking at the performance of Thunderbolt 3. I’ve ordered my second test system for that, and will be running tests to look at both transfer rates over different ports and at thermal throttling.
  And there was I hoping I could move on from external storage, although I am about to do a major group test for Mac|Life and MacFormat anyway, so there’s no peace for the wicked!
  Howard.
  
  LikeLike
  - 46
    
    Ric Ford on April 23, 2022 at 11:14 am
    
    “So next week I’ll turn to looking at the performance of Thunderbolt 3. I’ve ordered my second test system for that, and will be running tests to look at both transfer rates over different ports and at thermal throttling.”
    
    An interesting challenge is finding the best external storage that can do *both* USB and Thunderbolt, a rare capability so far. OWC’s Envoy Pro FX can do that, but its maximum speed is slower than I’ve seen for Samsung’s 980 Pro in a Thunderbolt enclosure, and it slows dramatically when its internal cache fills, but it runs quite cool and I don’t think thermal throttling is a problem. (It’s also expensive.) With a 2015 MacBook Pro 15″, USB performance was disappointing (about 350 MB/s. in Stibium or AJA System Test Lite) but still very usable.
    
    I just bought an Aukmax enclosure (off Amazon) that’s supposed to be “USB4” and connect to both Thunderbolt and (older) USB ports, but it failed to function at all in any USB system I tried, so I’m returning it.
    
    I’ll be interested in your upcoming report(s)!
    
    LikeLiked by 1 person
    - 47
      
      hoakley on April 23, 2022 at 1:46 pm
      
      Thank you, Ric. I’ll hold off on that additional invitation for the moment, if you don’t mind!
      Howard.
      
      LikeLike
  - 48
    
    joevt on April 30, 2022 at 9:31 pm
    
    For Thunderbolt 3, I would consider mostly NVMe drives that can exceed 3000 MB/s read/write in a PCIe 3.0 slot. The point is to discover the Thunderbolt bandwidth limit, not the NVMe limit so you need an NVMe that can fill the Thunderbolt bandwidth. Then I would group them into NVMe’s that can do > 2700 MB/s in a Thunderbolt enclosure and those that cannot (AmorphousDiskMark sequential read/write). Some NVMe have poor performance ( 2700 MB/s R/W) : Mushkin 4TB, Sabrent Rocket 4.0 2TB.
    Less good NVMe (2500 MB/s read, < 1500 MB/s writes): Adata XPG SX8200 Pro, Samsung 950 Pro 512GB (950 Pro doesn't exceed Thunderbolt limit in a PCIe slot).
    Bad NVMe ( 2700 MB/s read in Thunderbolt but < 1000 MB/s write)
    
    These were Intel Mac tests which are useful for seeing how an M1 Mac should perform. If you find that an M1 has much less performance, then see if you can reproduce OWC's observation that connecting a display or dock may improve performance.
    
    Some NVMe manufactures may change the controller of the NVMe without changing the product name/model, so NVMe of the same model can have different performance.
    
    LikeLiked by 1 person
    - 49
      
      hoakley on April 30, 2022 at 9:40 pm
      
      Thank you.
      “The point is to discover the Thunderbolt bandwidth limit”: that might be your point, and you’re welcome to try to assess that, but it’s not mine. We know that no Thunderbolt 4 storage can reach 3 GB/s, and that the absolute limit is around 2.7 GB/s. However, most TB3 cases and enclosures don’t approach that.
      I used a repeated measures design as before, which is the most efficient way to determine whether any given port or combination is performing below what is expected. And you can already read the results: without considering combinations with displays etc., there was no difference, which was the original question.
      Howard.
      
      LikeLike
    - 50
      
      joevt on May 1, 2022 at 7:27 am
      
      It’s not the point but it’s a point. There are some Thunderbolt 3 or 4 benchmarks that do exceed 3000 MB/s (read). See the “Thunderbolt 3 M.2 NVMe SSD Enclosures” thread at MacRumors, starting page 16 post #386. One of them is 3158 MB/s read which is slightly faster than 25 Gbps. There are several others between 24 and 25 Gbps. The benchmark is AmorphousDiskMark – maybe it has some caching issue on M1 Macs that cause it to report higher than normal speeds but I don’t think so. It could be related to the difference between HFS+ and APFS, or encrypted and non-encrypted. I did some tests in post #273 in the same thread. I usually do HFS+ non-encrypted.
      
      What is your definition of “expected”? If every M1 Mac you test reports 2400 MB/s then I guess you can expect other M1 Macs to do the same. But if there are Macs or PCs that do 2800 MB/s, then the expected value of 2400 MB/s of M1 Macs has a problem.
      
      LikeLiked by 1 person
    - 51
      
      hoakley on May 1, 2022 at 10:58 am
      
      Thank you. I’m afraid that your link got lost.
      Exactly what test do you think provided such an exceptionally high result? I don’t support it was the standard 5 repetitions of a 1 GB transfer? If so, your evidence for such outliers is flimsy in the extreme. Before you even accept such results, you need to be meticulous about the quality of the evidence, and most disk benchmarks are prone to substantial error, as we saw when they were used to test the first M1 Macs.
      You also need to know the variation in results across different tests before you consider whether 2.4 GB/s is significantly different from 2.8 GB/s. Given the paucity of high transfer rates, I’m not sure you have sufficient information there.
      In my testing, because it’s direct comparison across different systems, the expected values are from the reference system, in this case an iMac Pro with TB3.
      Howard.
      
      LikeLike
    - 52
      
      joevt on May 1, 2022 at 11:41 pm
      
      I did not include a link because blogs have anti-spam that sometimes cause my replies to never be posted. Googling the quoted thread name should be able to find it. The screenshots in the the thread show the standard 5 repetitions of a 1 GB transfer. My post #273 is on page 11. I suppose you need to check the dates of the posts to see what version of macOS they could be using at the time. The tests that exceed 3000 MB/s were made on Intel and M1 Macs.
      
      The 2400 vs 2800 MB/s example was hypothetical. Choose numbers that would indicate a problem (I thought a difference of 400 MB/s would be sufficient to indicate a problem). Since your tests include an Intel iMac Pro, I am satisfied that you may be able to detect a problem in the M1 Thunderbolt implementation if one exists. Your tests don’t include a non-Thunderbolt connection so you can’t see Thunderbolt problems such as NVMe with 1000 MB/s writes that should be able to do 3000+ MB/s with a non-Thunderbolt connection. Well, maybe the published specs of the NVMe is sufficient for detecting a Thunderbolt problem.
      
      LikeLiked by 1 person
    - 53
      
      hoakley on May 2, 2022 at 8:33 pm
      
      Thank you.
      You are always very welcome to provide links here. This site does use Akismet, but I fairly frequently check those comments which it has classified as spam, and quickly ensure they are published. I do check all links to ensure they’re not commercial/advertising or malicious. I value your comments, and those of others, and do my best to ensure they appear as you wish.
      Howard.
      
      LikeLike
54

EcleX on April 23, 2022 at 12:43 pm

I replied to your post above, yet my message shows here below.

LikeLiked by 1 person
55

Semi regular lurker on April 23, 2022 at 11:23 pm

Just a general observation/query here Howard, I am wondering if you can do anything about the comments section rendering replies with ever deeper levels of nesting resulting in every narrower columns of text. This rapidly gets to be unreadable for me — using iPadOS Safari on an iPad Pro if that makes any difference. The columns of text get so narrow that I literally see only 2-3 words per line so a single sentence takes many lines and it is comically bad — WordPress or the authors of whatever software you are using should be embarrassed if this is truly how their software was intended to work.

On a side note so much of the comments you get here are from people who simply do not consider just how different Apple Silicon based devices are from the classic/legacy X86 and traditional storage world’s architectures and implementations. I continue to observe and claim that architectures and all the subtle details matter except very rarely when they do not… engineering and “technology products” are all about making tradeoffs in a high dimensional space of the currently possible governed eventually by economics and the net complexity that is manageable or not.

LikeLiked by 1 person
- 56
  
  hoakley on April 24, 2022 at 5:38 am
  
  Thank you.
  I control the depth of threading or nesting of comments, and for a long time limited that to 3. At that level, those making comments repeatedly complained that they couldn’t reply to existing comments without their comment being put at the top level again. Eventually, I increased the depth to 5, and since then no one has complained.
  On a Mac with a browser opened up to allow a decent window width, this article’s comments remain thoroughly readable. When I last checked on an iPad in landscape orientation, that was true there too.
  If I reduce depth again, I won’t be able to please those who want this depth level. If I leave it set at 5, then I won’t be able to please those with narrower browser windows. I’m happy to change the depth to whatever level users want, but I know that will displease others.
  As an experiment, I have therefore reduced the depth to 4. I await the complaints.
  Howard.
  
  LikeLike
57

EcleX on April 25, 2022 at 9:24 am

I have just posted a new message indicating our practical experience in the comments at

Last Week on My Mac: Opening Pandora’s SSD
https://eclecticlight.co/2022/04/24/last-week-on-my-mac-opening-pandoras-ssd

I am not posting that here to prevent duplication.

LikeLiked by 1 person
- 58
  
  EcleX on April 25, 2022 at 9:25 am
  
  I replied to your post above, yet my message shows here below.
  
  LikeLiked by 1 person
  - 59
    
    hoakley on April 25, 2022 at 12:28 pm
    
    I’m afraid that may be due to the threading of replies. A reader recently complained that 5 levels was too deep to read, so I have cut it back to 4 for the moment. I’m sorry about that.
    Howard.
    
    LikeLike
60

Ric Ford on April 25, 2022 at 8:16 pm

Hi, Howard –

I have just done some storage/interface testing with a CalDigit Thunderbolt dock and a 2015 Intel MacBook Pro that might be interesting, because it demonstrates how a Pandora’s jar of Thunderbolt / USB / performance problems can arise even in a pure Intel environment, but some of these may also affect M1 systems, such as a Thunderbolt cable triggering ultra-slow USB 2 performance, for example. (This looks like your Mac Studio front-port experience.)

I can certainly understand how you’d like to wash your hands of this whole mess and just see Apple and storage vendors solve the problems, and I feel pretty much the same way!

Ric

Test Setup:

• 2015 MacBook Pro 15″ (Model 11,4 with integrated graphics, 2.2GHz Core i7)
• CalDigit TS3+ Thunderbolt dock
• Apple Thunderbolt 3-Thunderbolt 2 adapter
• 2TB OWC Envoy Pro FX with one port for Intel Titan Ridge (Thunderbolt) + Realtek RTL9210 (USB)
• OWC Thunderbolt 3 cable with tethered USB-C to USB Type-A adapter
• Samsung 10Gbps USB-C cable
• Samsung 5Gbps USB-C-USB-A cable
• macOS 10.14 Mojave with System Profiler (About This Mac)
• AJA System Test Lite w/ 4GB test file (similar results to Stibium)

NORMAL:

Connection: Envoy Pro to OWC Thunderbolt cable to Caldigit TB3 connector

Performance: around 1000 MB/sec – (varies from run to run, apparently due to Envoy Pro FX limits and issues

System Profiler/Notes: Envoy Pro at 40Gbps; Caldigit dock at 20 Gbps (2015 MacBook Pro Thunderbolt 2 maximum)

Connection: Envoy Pro to 10Gbps USB-C cable to CalDigit Thunderbolt port

Performance: about 575 MB/s

System Profiler/Notes: 10 Gbps

Connection: Envoy Pro to Samsung USB-A–USB-C cable to USB-A port on MacBook Pro

Performance: about 350 MB/sec.

System Profiler/Notes: 5 Gbps

PROBLEMATIC:

Connection: Envoy Pro to OWC Thunderbolt 4 cable + OWC USB-C-USB-A adapter to MacBook Pro USB-A port

*** Performance: about 38 MBps!!

*** System Profiler/Notes: 480 Mbps! – listed under “USB 3.0 Bus” shared with Bluetooth, and internal keyboard & trackpad but USB *2* speed

Connection: Envoy Pro to OWC Thunderbolt 4 cable + OWC USB-C-USB-A adapter to Caldigit USB-A

*** Performance: 37 MB/sec.!!

*** System Profiler/Notes: “USB 3.0 bus” (shared with MacBook Pro card reader): 480 Mbps USB *2* speed!

Connection: Envoy Pro **connected at boot** to OWC Thunderbolt 4 cable to Caldigit 10Gbps USB-C port –

** Performance: about 350 MB/sec. – half expected speed!!

System Profiler/Notes: [not recorded]

Connection: Envoy Pro *connected AFTER boot* to OWC Thunderbolt 4 cable to Caldigit 10Gbps USB-C port

** Performance: around 400-500 MB/sec. – better but well below expected speed!

System Profiler/Notes: [not recorded]

LikeLiked by 1 person
- 61
  
  Ric Ford on April 25, 2022 at 9:23 pm
  
  “Test Setup:”…
  
  I forgot to mention that the MacBook Pro was also connected to a 2K display via its other Thunderbolt 2/DisplayPort port. (This may have affected performance, but I didn’t think to disconnect it for further testing.)
  
  LikeLiked by 1 person
- 62
  
  hoakley on April 25, 2022 at 10:17 pm
  
  Thank you. I will return to the USB issues in a while, but I need to get my head around TB first, to check whether it’s any better. A breather is also very attractive, as I’ve just being doing my quarterly tax accounts, which were even more painful than USB.
  Howard.
  
  LikeLike
63

a passerby on May 18, 2022 at 6:01 pm

My own tests, carried out with the AmourphousDiskMark software:

– Mac Studio Max 32GB/1TB:

QD8: Read/Write: 6706/6737
QD1: Read/Write: 2921/5070

– Samsung 980 Pro 1 TB + Acasis USB 4/TB3 enclosure (model TBU401) and formatted in APFS:

QD8: Read/Write: 3105/2931
QD1: Read/Write: 2589/2490

LikeLiked by 1 person
- 64
  
  hoakley on May 18, 2022 at 6:29 pm
  
  Thank you.
  Howard.
  
  LikeLike
- 65
  
  Ric Ford on May 18, 2022 at 6:35 pm
  
  Passerby, thanks for posting your results, which are interesting as I’ve just now been testing the 980 Pro, too.
  
  Two questions:
  
  • Can you use the Acasis enclosure with a USB-C (not Thunderbolt) cable? If so, I’d be very interested in hearing what the results are.
  
  • If possible, could you post random 4K results, as well? What I’m seeing here is higher performance (in Amorphous Disk Mark) in Random 4K QD64 for the 980 Pro in a TekQ Thunderbolt enclosure than I see for the internal 1TB Apple SSD in an M1 MBP 14 (though the internal SSD blows away the external drive for sequential performance).
  
  Ric
  
  LikeLiked by 1 person
  - 66
    
    a passerby on May 21, 2022 at 10:15 am
    
    For sure Ric.
    
    – Connected the Acasis enclosure with a USB-C cable to thunderbolt port, the SSD is recognized. Here are the results obtained:
    
    QD8: Read/Write: 979/1019
    QD1: Read/Write: 769/792
    4K QD64: Read/Write: 141/17
    4K QD1: Read/Write: 39/21
    
    – Connected to the USB port of the Mac Studio Max, the SSD is not recognized.
    
    – The results in TB3 for random 4K in AmorphousDiskMark:
    
    4K QD64: Read/Write: 1171/292
    4K QD1: Read/Write: 64/38
    
    LikeLiked by 1 person
  - 67
    
    Ric Ford on May 21, 2022 at 4:43 pm
    
    Connected the Acasis enclosure with a USB-C cable to thunderbolt port, the SSD is recognized. Here are the results obtained:
    
    QD8: Read/Write: 979/1019
    QD1: Read/Write: 769/792
    4K QD64: Read/Write: 141/17
    4K QD1: Read/Write: 39/21
    
    Thanks, Passerby. It looks like your Mac Studio Max and Acasis enclosure are both performing very well, while I’m seeing performance problems with 10Gbps USB-C on an M1 MBP 14 (macOS Monterey) that do not occur on an iMac 5K (macOS Mojave or Catalina).
    
    Amorphous Disk Mark, 5GiB file, 4 or 5 iterations
    Samsung 970 EVO*
    TekQ 583 10Gbps** USB-C enclosure
    APFS (unencrypted)
    
    M1 Pro MBP 14*** (macOS 12.4)
    (MBps)
    
    QD8: Read/Write: 793/759
    QD1: Read/Write: 766/745
    4K QD64: Read/Write: 24/11
    4K QD1: Read/Write: 30/13
    
    2017 iMac 5K (macOS 10.14 or 10.15)
    (MBps)
    
    QD8: Read/Write: 1033/1023
    QD1: Read/Write: 835/865
    4K QD64: Read/Write: 141/68
    4K QD1: Read/Write: 27/109
    
    Notes:
    
    * The same Samsung 970 EVO in a TekQ Cube Thunderbolt enclosure hit sequential read/write speeds around 2500 / 1700 MBps
    
    ** After disconnecting and reconnecting the TekQ 583 at one point with the same cable and configuration, the connection was made at 5Gbps instead of 10Gbps, and performance dropped to less than half:
    
    QD8: Read/Write: 359 / 378
    
    *** Tested in clamshell mode with a Caldigit TS3+ dock connected to the right side Thunderbolt port with a 4K display attached – performance decreased slightly when the dock was disconnected and the laptop display was used.
    
    LikeLiked by 1 person
    - 68
      
      a passerby on May 21, 2022 at 5:58 pm
      
      It took me some time before choosing this Acasis enclosure with Samsung 980 Pro by reading many forum pages, articles and their comments, watching youtube video.
      I’m satisfied with it for now*.
      
      *Note that I had received a first Acasis enclosure which broke down less than 24 hours after receipt. I don’t have any problems with his replacement so far.
      
      LikeLiked by 1 person
    - 69
      
      hoakley on May 21, 2022 at 9:08 pm
      
      I looked at that enclosure, as I have a Samsung 980 PRO 2 TB looking for a home. In the UK at present, that enclosure costs the same as that 2 TB SSD!
      I don’t think I’ll be buying one anytime soon.
      Howard.
      
      LikeLike
    - 70
      
      Ric Ford on May 21, 2022 at 6:25 pm
      
      “After disconnecting and reconnecting the TekQ 583 at one point with the same cable and configuration, the connection was made at 5Gbps…”
      
      “Tested in clamshell mode with a Caldigit TS3+ dock connected…”
      
      I tested several variables with no noticeable effect and could not replicate the 5Gbps connection glitch.
      
      What didn’t matter:
      
      • Thunderbolt cable vs. USB-C (not-Thunderbolt) cable
      • connecting USB-C cable one side up or the other
      • which Thunderbolt/USB-C port I connected to
      • whether or not a display was connected to the HDMI port
      
      Ric
      
      LikeLiked by 1 person
    - 71
      
      a passerby on May 22, 2022 at 8:37 am
      
      Wow.
      Thunderbolt’s enclosures for Nvme SSD can be very expensive, but I wasn’t thinking at this point.
      I bought mine around 130 euros on Amazon and the official store (ANYOYO Store).
      
      Some resellers can indeed increase the prices very significantly compared to the manufacturer’s price.
      Maybe check with another seller on Amazon.
      
      PS: The model I bought is £109.99 this Sunday morning :
      
      LikeLiked by 1 person
    - 72
      
      hoakley on May 22, 2022 at 3:14 pm
      
      Thank you. Amazon is a disaster!
      Howard.
      
      LikeLike
    - 73
      
      a passerby on May 22, 2022 at 9:09 pm
      
      Good customer service, but yes, Amazon can be a disaster.
      Other information that may be of interest. The temperature of the SSD enclosure is around 42°C in operation (too high temperatures can have an impact, i believe, on the performance and lifespan of the SSD).
      
      LikeLiked by 1 person
- 74
  
  Ric Ford on May 18, 2022 at 6:50 pm
  
  For comparison, in Amorphous Disk Mark with 4GiB file, 5 iterations, APFS format, TekQ Cube enclosure, 2TB Samsung 980 Pro SSD, M1 MBP 14 (8-core).
  
  980 Pro / TekQ Cube:
  
  QD8: Read/Write: 2842 / 2796
  QD1: Read/Write: 2293 / 2229
  
  RND4K/QD64: 950 / 280 MBps
  RND4K/QD64: 59 / 36 MBps
  
  1TB MacBook Pro 14:
  
  QD8: Read/Write: 6774 / 6236
  QD1: Read/Write: 3073 / 4917
  
  RND4K/QD64: 650 / 164 MBps
  RND4K/QD64: 57 / 35 MBps
  
  LikeLiked by 1 person
75

Sean on June 29, 2022 at 2:14 am

I”m on an Ultra with 128GB ram, 48-core GPU, and 2TB SSD
My scores are similar to the Max posts here with a small % slower on all measurements except my RND4k QD64 is 66% slower. Bad SSD? https://i.imgur.com/K8DYzWt.jpg

LikeLiked by 1 person
- 76
  
  hoakley on June 29, 2022 at 6:08 am
  
  Those look good to me. I don’t see any reason to suspect your Mac’s SSD has any problem.
  Howard.
  
  LikeLike
77

Ric Ford on June 29, 2022 at 2:33 pm

I”m on an Ultra with 128GB ram, 48-core GPU, and 2TB SSD
My scores are similar to the Max posts here with a small % slower on all measurements except my RND4k QD64 is 66% slower. Bad SSD? https://i.imgur.com/K8DYzWt.jpg

Interesting… Here for comparison is an M1 MBP Pro 14 1TB/16GB, which shows much faster RND4k QD64:

View post on imgur.com

I wonder if you had some other processes competing with the benchmark program (e.g. Spotlight reindexing or some other demanding task in the background.)

LikeLiked by 1 person
- 78
  
  hoakley on June 29, 2022 at 4:25 pm
  
  Thank you, Ric. I expect you’d also see that on a well-used SSD, although I’m still not sure what the RND figures really mean. Stibium, for example, randomises all its test ordering. Does that mean they’re random reads and writes too?
  Howard.
  
  LikeLike
- 79
  
  naestyoh on June 29, 2022 at 5:34 pm
  
  My Mac Studio Ultra is about 4 days old. I’ve had it idle for most of those 4 days (24h on) so I’d think that any spotlighting of my sparse OS data would be done by now. I have an external USB4 nvme enclosure with an older WD Blue 1TB. It blows my Apple SSD away in random reads by 5x margin:
  
  View post on imgur.com
  
  This morning, I restarted the computer, waited a bit and ran the test again on the internal 2TB ssd. Same results. Might not seem like much but the random reads, i’m guessing, are what the OS deals with on the day to day mostly. It would be seen as what makes the computer snappy when opening and running apps. Correct? If so, then I’m seeing 33% of what other users are getting.
  
  LikeLiked by 1 person
  - 80
    
    Ric Ford on June 29, 2022 at 5:48 pm
    
    I have an external USB4 nvme enclosure with an older WD Blue 1TB. It blows my Apple SSD away in random reads by 5x margin:
    
    Here’s a Samsung 980 Pro 2TB in an external Thunderbolt 3 enclosure (TekQ Cube) for comparison, showing Random 4K performance easily exceeding that of the internal SSD (in spite of the Thunderbolt bottleneck):
    
    View post on imgur.com
    
    LikeLiked by 1 person
    - 81
      
      Sean on June 29, 2022 at 6:46 pm
      
      I just fired up my old 2017 mac I migrated from a few days ago. The random QD64 reads on this older computer are >> 9x <> 17x faster <<.
      
      Apps with database and other larger data load 2-3x faster on my old mac. What is going on?
      
      View post on imgur.com
      
      LikeLiked by 1 person
    - 82
      
      hoakley on June 29, 2022 at 7:00 pm
      
      Well, app load times are influenced by many factors. If you’re saying that loading large files takes longer, then that flies in the face of the other benchmark figures, doesn’t it?
      If you think there’s a problem with your Mac, then you should raise it with Apple. It might have a defect, although looking at the other figures, I suspect not.
      Howard.
      
      LikeLike
    - 83
      
      hoakley on June 29, 2022 at 6:57 pm
      
      Do we know of any correlation between those values and anything in the real world, or is this just an arbitrary figure?
      Howard.
      
      LikeLike
84

Ric Ford on June 29, 2022 at 8:35 pm

I just fired up my old 2017 mac I migrated from a few days ago. The random QD64 reads on this older computer are >> 9x 17x faster <<.

I see similar results for a 2017 iMac 5K:

View post on imgur.com

(There’s a little slowdown in macOS 12 vs. macOS 10.15.)

LikeLiked by 1 person
- 85
  
  naestyoh on June 29, 2022 at 8:42 pm
  
  I do a ton of very small read/writes with Lightroom’s database and sidecar XMP. I’m going to bring it to the Apple store nearby and see if they can swap the SSD.
  
  LikeLiked by 1 person
86

Ric Ford on June 29, 2022 at 8:46 pm

Do we know of any correlation between those values and anything in the real world,

Presumably these synthetic benchmarks do correlate to real-world results to some extent. Here’s an excellent and relevant article:

https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/

The most important of these is 4K random I/O, which we discussed at length above. 4K random is where the pain lives—it’s the reason your nice fast computer with a conventional hard drive suddenly sounds like it’s grinding coffee and makes you want to defenestrate it in frustration.

Certainly, real-world tests would be useful, and I haven’t had time enough to do a suite of those for storage, but I’ve been thinking about possible protocols for that, which involves considering the effects of caching and of APFS’s tricky features. One possibility would be to make a clean backup of an internal volume to a fast external drive using CCC, which maintains statistics on file transfers, then to reboot and to restore that backup to a different internal volume from the external. It might also be possible to test backing up an internal volume on one partition to another internal volume on a different partition, eliminating the Thunderbolt bottleneck. Another variation would be using the Finder to do the duplication, and yet another would be using Terminal commands to do it.

LikeLiked by 1 person
- 87
  
  hoakley on June 29, 2022 at 8:58 pm
  
  Ric,
  We went through all this over a year ago when I was developing Stibium. Look back – anything smaller than 2 MB file size many of the read and write times reflect transfer from memory to memory, and not to/from storage. They’re well within the caches operating at both ends of the transfer, and are simply nonsense. That’s the first reason that I don’t believe any figures for files smaller than 2 MB. I spent weeks working at this, and have plenty of graphs demonstrating it.
  Until someone shows me a clear correlation between the ‘benchmark’ figures being produced by Mac apps and anything in the real world, I’m treating them like inside leg measurements – useful for tailors, but they don’t tell much else about a person.
  For 2 MB to 2 GB (beyond if you really wish), there are good correlations with things like Finder transfer times, and you’ve seen the interesting relationship with M1 boot times.
  Howard.
  
  LikeLike
- 88
  
  hoakley on June 29, 2022 at 9:16 pm
  
  Ric,
  You might find this article explains some of the variation. I note that the command tool recommended in that article uses fsync as a recommended option. Well, it doesn’t work the way that they think it does, after all.
  Howard.
  
  LikeLike
89

Ric Ford on June 29, 2022 at 9:14 pm

Until someone shows me a clear correlation between the ‘benchmark’ figures being produced by Mac apps and anything in the real world, I’m treating them like inside leg measurements – useful for tailors, but they don’t tell much else about a person.

Yes, there’s clearly a need to assess the synthetic benchmarks vs. real-world tests, wihch involve many confusing and confounding factors. In addition to the tests suggested above, I might look at speed of, say, importing a great number of photos into a catalog (trying to avoid any thumbnail processing bottlenecks), or perform large database imports/exports. Perhaps you could assess the speed of some I/O-dependent XCode jobs, an area I’m not familiar with.

LikeLiked by 1 person
- 90
  
  hoakley on June 29, 2022 at 9:21 pm
  
  That’s exactly what Stibium does. It uses the same calls to read and write that apps do. It pulls no tricks or punches, and puts the reads and writes in a pseudo-random order every time. And I found when the file size fell much below 2 MB, many of the transfers occurred at indicated speeds of 20 GB/s and more, no matter what I tried in the way of disabling caching.
  Another factor that article doesn’t take into account, which is considered seriously elsewhere, is SLC write cache, something readily assessed in Stibium, although it’s only likely to be relevant in most cases (Samsung X5 excepted!) over sustained high-volume writes.
  Howard.
  
  LikeLike
- 91
  
  Ric Ford on June 29, 2022 at 9:34 pm
  
  Here’s a useful Terminal command construction for timing a large copy, for anyone interested in doing some real-world tests (modify as desired for different folders, drives, etc.):
  
  time ditto ~/ /Volumes/TestDrive/TestFolder
  
  real 1m32.729s
  user 0m1.181s
  sys 1m0.094s
  
  LikeLike
  - 92
    
    naestyoh on June 29, 2022 at 10:06 pm
    
    That cmd line dups my home folder? If so, our times will be wildly different, no? Regardless of all of this, Howard’s leading image in this article says 654. Mine says 180. That’s a 3.7x difference.
    
    I’m a pro photographer and I deal with both medium (25-50 MB ea) and a ton of small meta files (1-2kb) as well as several 20GB-sized databases. I’ll take every micro ounce I can get for the six grand I paid.
    
    LikeLiked by 1 person
    - 93
      
      hoakley on June 29, 2022 at 10:07 pm
      
      I wish you success.
      Howard
      
      LikeLike
    - 94
      
      Ric Ford on June 29, 2022 at 10:43 pm
      
      “That cmd line dups my home folder? If so, our times will be wildly different, no?”
      
      Yes, that was just an example; you might want to time the copying of a photo folder or something else.
      
      I haven’t yet come up with a test folder that meets the criteria of being easily reproducible for multiple people across multiple systems, representative of broad real-world macOS usage, easily obtainable and non-proprietary, and static/unchanging over time.
      
      LikeLiked by 1 person
    - 95
      
      hoakley on June 30, 2022 at 4:47 pm
      
      Ric,
      “I haven’t yet come up with a test folder”…
      Stibium can create one containing files from 2 MB to 2 GB dead easily for you, as it does when you run its write test.
      Howard.
      
      LikeLike
    - 96
      
      Ric Ford on June 30, 2022 at 5:23 pm
      
      Howard said, “Stibium can create one containing files from 2 MB to 2 GB…”
      
      CCC logged a recent batch of changed files to backup:
      
      20,515 Files (30.81 GB)
      
      which I think averages about 1.5 MB per file? For a test folder, I’d like to have something similar, though I don’t see an easy way to show the file size distribution and reproduce that in a test set.
      
      Meanwhile, I’ve done one quick test of importing photos into a catalog with Capture One on the M1 MBP 14 internal 1TB SSD vs. the Samsung 980 Pro 2TB SSD in a Thunderbolt enclosure (attached to a TB hub):
      
      980 Pro TB3: 12.9 sec.
      MBP internal: 11.8 sec.
      
      The 980 Pro volume is encrypted and the MBP volume isn’t – I don’t know how much that might affect results.
      
      Ric
      
      LikeLiked by 1 person
    - 97
      
      hoakley on June 30, 2022 at 7:27 pm
      
      Thank you.
      Your M1 MBP’s internal SSD most certainly is encrypted, well at least the Data volume is, even if you’ve not turned FileVault on (which I strongly recommend, as it comes completely free of any additional overhead, free as in lunch). However, its encryption is performed in hardware, in the disk controller, it’s thought. So it may have a slight performance advantage over software encryption for an external disk with FileVault turned on.
      As I’m sure you’ve guessed, distributing a 30 GB test suite isn’t likely to be popular! Hence, the need to generate the test files in software, where a couple of MB of app can create them fresh every time.
      Howard.
      
      LikeLike
    - 98
      
      Ric Ford on June 30, 2022 at 8:53 pm
      
      Howard wrote, “However, its encryption is performed in hardware, in the disk controller, it’s thought. So it may have a slight performance advantage over software encryption for an external disk with FileVault turned on.”
      
      Yes, that was the configuration: FileVault enabled for the external drive and not for the internal test volume, and I didn’t know if that would make any difference at all (clearly not a large difference, in any case).
      
      “As I’m sure you’ve guessed, distributing a 30 GB test suite isn’t likely to be popular! Hence, the need to generate the test files in software, where a couple of MB of app can create them fresh every time.”
      
      My point was about file size distribution mirroring real-world usage rather than the total size of the file set, but you’re right, having a standard app (like Stibium) to generate the files makes lots of sense. Ideally, the user could choose how large a set to build and maybe have some control of size distribution. I really have no idea what sort of file-size distribution might approximate broad/average real-world Mac usage (and, of course, it varies greatly among different individuals).
      
      Ric
      
      LikeLiked by 1 person
    - 99
      
      hoakley on June 30, 2022 at 11:07 pm
      
      Thank you, Ric.
      It’s interesting that you mention size “distribution”, and earlier referred to an average size, as I suspect distributions are far from normal, and averages are very poor indicators as a result.
      I also suspect that no one has clue of distribution of file read and write sizes during their typical work.
      Stibium offers two different distributions, both within the 2 MB to 2 GB range – the normal one is bottom-heavy, with mostly lower MB sizes; the other is purely random, which should be evenly distributed. It wouldn’t be hard to extend that range, but given the lack of knowledge, I’m not sure that would serve any useful purpose.
      Yes, and if you recall they mustn’t be smaller than 2 MB, otherwise all you see is memory-to-memory transfer rates, which would only poison any analysis. So whatever might happen in the ‘real world’, looking at small file transfer rates is pointless.
      Howard.
      
      LikeLike
100

Ric Ford on June 29, 2022 at 9:39 pm

Another factor that article doesn’t take into account, which is considered seriously elsewhere, is SLC write cache…

Howard, it sounds like you missed the excellent description of exactly that in the article:

Some models of SSD have an additional non-volatile write cache on the disk itself, made of a faster and more expensive type of flash media. For example, a TLC or QLC (Quad Layer Cell) SSD might have a few gigabytes of MLC (Multi-Layer Cell) media to use as a buffer for writes; this enables the SSD to keep up with the writes demanded by a typical desktop workload using the faster MLC buffer—but if presented with sustained heavy writes for too long a time, the fast MLC buffer fills, and throughput drops to what the slower TLC or QLC media can manage. This can frequently be a “fall off the cliff” type scenario, since the slower media will typically not only have to sustain ongoing writes, but do so while continuing to stream out the already-accepted writes from the fast MLC buffer.

LikeLiked by 1 person
- 101
  
  hoakley on June 29, 2022 at 9:45 pm
  
  Thank you. I did see that. On many 2 TB SSDs, it’s SLC and over 100 GB, so I don’t recognise the description!
  It’s also taken from the main pool of memory, but used in single rather than multi mode, which means it’s very quick but doesn’t achieve the same storage density.
  So that seems to be referring to something quite different.
  Howard
  
  LikeLike