rsync - Feb 2018 - [Bug 13266] New: Writing of files to Apple's new "AP" controller model SSD's takes 4x longer.

https://bugzilla.samba.org/show_bug.cgi?id=13266

            Bug ID: 13266
           Summary: Writing of files to Apple's new "AP"
controller model
                    SSD's takes 4x longer.
           Product: rsync
           Version: 3.1.3
          Hardware: x64
                OS: Mac OS X
            Status: NEW
          Severity: critical
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: info at danny-s.com
        QA Contact: rsync-qa at samba.org

Created attachment 13953
  --> https://bugzilla.samba.org/attachment.cgi?id=13953&action=edit
2017 MacBook Pro made in 2018 showing "AP" model drive.

I regularly use rsync to deploy roughly 40gb of data to new Macs that come in.
For the past few years this has taken roughly 8-10 minutes to transfer to SSD
models.

A few weeks ago this time suddenly jumped to over 40 minutes with no other
exchanges to hardware.

I've deeply troubleshooted this performance issue and found only one change
has
occurred. Originally blaming APFS, i tested with both APFS and HFS read and
write sources. It occurred on both an iMac Pro and two MacBook Pros. macOS has
been forked for the iMac Pro and I considered that perhaps it's build (which
my
admin OS was running) had a different SSD kext - however I tried both 10.13.3
17D47 and the iMac Pro build 10.13.3 17D2047 and confirmed the performance
issue on both.

We've been recreating this bug for faster testing by copying the
Applications
folder from the source to the destination (with a destination where the
Applications folder doesn't exist, just to get raw copy speed). The folder
is
25gb. With Finder, including the lengthy time it takes to pre-scan files, it
copies in 6 minutes, with rsync it's nearly 25 minutes, again about 1 minute
per gb.

I tested on my 2017 MacBook Pro and observed that both Finder and rsync could
copy the data in 6 minutes - same source, destination of my MacBook Pro was
APFS formatted too.

I had seen other MacBook Pros and SSD equipped iMacs come in and all copy with
the correct speed and deploy within 10 minutes.

Finally upon inspecting what is different about the recent MacBook Pros and the
iMac Pro, in the SSD NVME tab they have listed an "AP" model for the
SSD
(Presumably Apple with their own controller) my 2017 MacBook Pro that
doesn't
show this poor performance has an "SM" model (Samsung).

The only consistent thing on all the systems showing this 4x slower speed with
rsync is that they have AP model SSDs. The MacBook Pro has AP0512J, the iMac
Pro has AP1024M.
My MacBook Pro which works fine and was made just after the refresh is SM1024L.

So I assume Apple has quietly changed to the AP drives (and controller?) in
recent weeks for the MacBook Pro (some "SM" are still coming through
which work
fine with rsync) and of course the iMac Pro is shipping with it.

I'll be getting a number of these systems in, so any testing, logs or
anything
I can do to get to the bottom of this should be no problem - I have as well
filed a Rdar with Apple as it could be something they need to fix in their own
firmware.

Attached an SSD model number.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13266

--- Comment #1 from Daniel Shepherd <info at danny-s.com> ---
Created attachment 13954
  --> https://bugzilla.samba.org/attachment.cgi?id=13954&action=edit
Early 2017 MacBook Pro (mine) with "SM" model drive.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13266

--- Comment #2 from Daniel Shepherd <info at danny-s.com> ---
Created attachment 13955
  --> https://bugzilla.samba.org/attachment.cgi?id=13955&action=edit
iMac Pro with "AP" drive.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13266

--- Comment #3 from Daniel Shepherd <info at danny-s.com> ---
Have tested this with a 2016 MacBook that also has the "AP" SSD
controller and
can confirm the same performance issue.

So it's not just new machines, it's any Mac with an "AP" SSD
formatted with
APFS.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13266

--- Comment #4 from Chris Tipper <chris.tipper at gmail.com> ---
Model: iMac 4K (2015)
Platform: macOS 10.13.3
Storage: APFS running on APPLE SSD SM0512G 

Symptom: 5% throughput over Gigabit LAN compared to rsync 3.1.2 (4Mb/s vs
100Mb/s)
The target machine is FreeBSD 11.1 running rsync 3.1.3

I don't have any diagnostic but I have downgraded to rsync 3.1.2 from 3.1.3
and
seen a 20-fold increase in throughput. It seems unlikely that it is to do with
the specific model of SSD as mine is not one of the type listed in the bug
report. I suggest looking into APFS. I would also note that I installed rsync
from Macports.
https://github.com/macports/macports-ports/tree/master/net/rsync

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13266

--- Comment #5 from Chris Tipper <chris.tipper at live.co.uk> ---
I think I ought to report that the problem has resolved itself on my setup, I
didn't change anything but after the initial sync it returned to previous
throughput. I am using the --whole-files switch if that provides any insight,
it seemed to have an initial problem with archives produced on a previous run
with rsync 3.1.2. These are virtual machine images.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13266

--- Comment #6 from Daniel Shepherd <info at danny-s.com> ---
Respectfully Chris you didn't have this specific bug, as I said it only
affects
AP controllers. Whatever you were experiencing has nothing to do with this.
I've verified the AP controller bug on MacBook, MacBook Pro and iMac Pro.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13266

--- Comment #7 from Chris Tipper <chris.tipper at live.co.uk> ---
Sorry for hijacking your bug but I had an issue and did not want to create a
separate report. And you may have verified your bug to your own satisfaction
but not provided any diagnostic. So I don't see how it can be addressed in
its
current state.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.