R SIG Fedora - Jun 2023 - Problem with R, staged installation for packages, and samba share

Thanks again, unfortunately it still fails when mounting the drive on a
windows machine (even using nolease) on our end.
We will prepare a fully reproducible example using a virtual machine. This
could take us a day or two.
++

On Tue, 13 Jun 2023 at 11:05, I?aki Ucar <iucar at fedoraproject.org>
wrote:
> On Tue, 13 Jun 2023 at 10:35, Alexandre Courtiol
> <alexandre.courtiol at gmail.com> wrote:
> >
> > Thanks a lot for persevering.
> >
> > I?aki, that sounds very promising, could you please tell me what
> software your SMB share is running on (i.e. what does the server side look
> like?), and if it's Samba, then what version did you use?
>
> There was a Windows 10 machine around, so I created a test folder and
> hit "share".
>
> > (and yes, I am aware of your excellent cran2copr project).
> >
> > Tom, yes it also created the same issue with older RPMs. Please also
> find the configure log attached to this email.
>
> I see no significant differences, so this supports the hypothesis that
> the issue is in the mount.
>
> I?aki
>
> >
> > Best,
> >
> > Alex
> >
> >
> >
> > On Mon, 12 Jun 2023 at 21:08, I?aki Ucar <iucar at
fedoraproject.org>
> wrote:
> >>
> >> Update:
> >>
> >> On Mon, 12 Jun 2023 at 17:40, I?aki Ucar <iucar at
fedoraproject.org>
> wrote:
> >> >
> >> > Unfortunately, I cannot reproduce the issue here. :( I
mounted a smb
> >> > share, created a symlink from ~/R, and installed dplyr
without any
> >> > issue (apart from the installation process taking ages).
Fortunately,
> >> > this rules out our binary R package as responsible for the
issue you
> >> > are experiencing, unless I'm missing something.
> >>
> >> The experiment above was done using kio-fuse to mount the folder.
Now
> >> I've been able to reproduce the error using a regular CIFS
mount. I
> >> reproduced this both with Fedora and Debian R binaries. Then I
> >> executed the installation step by step, and I checked that, when
the
> >> mv command is invoked, manually running mv in the console does
trigger
> >> the error too. So again, I don't think there's any issue
with R.
> >>
> >> However, I've also managed to fix the issue by adding the
nolease
> >> option to the mount command. Hope it helps.
> >>
> >> I?aki
> >>
> >> >
> >> > Issue aside, may I ask what is the purpose of putting the
library in a
> >> > smb share? May I suggest [1]?
> >> >
> >> > [1]
https://cran.r-project.org/bin/linux/fedora/#additional-packages
> >> >
> >> > I?aki
> >> >
> >> > On Mon, 12 Jun 2023 at 16:48, Tom Callaway <spotrh at
gmail.com> wrote:
> >> > >
> >> > > The log of you running configure on your instance allows
me to
> compare it to the log of when we build it as a package (you don't need
to
> provide that one).
> >> > >
> >> > > Honestly, I have no idea right now _why_ pre-built R
would fail to
> move files to a cifs share when a non-root user can do it. I'm hoping
maybe
> something will jump out as to how it configures on your setup. We
aren't
> modifying R's source code at all, nor are we configuring it in a way
that
> would affect this as far as I know.
> >> > >
> >> > > Does this also happen with the older R binary RPMs? If
you go back
> to 4.2.3 or 4.2.2, does it happen?
> >> > >
> >> > > ~spot
> >> > >
> >> > >
> >> > > On Mon, Jun 12, 2023 at 10:38?AM Alexandre Courtiol <
> alexandre.courtiol at gmail.com> wrote:
> >> > >>
> >> > >> "is the staged installation enabled too in the
cases you don't see
> this issue?"
> >> > >> -> Yes, it should since we stick to default
config.
> >> > >>
> >> > >> "Can you provide a log of the output (including
the commandline
> with any options passed) for configure?"
> >> > >> -> If we compile locally there is no issue
despite not using any
> option, and if we install R from binary, there is no configure log AFAIK,
> so I am not sure how to meet this request.
> >> > >>
> >> > >> "can you provide the specific mount options for
the cifs share?"
> >> > >> ->
>
sec=krb5,multiuser,mfsymlinks,user=smbuser,domain=localdomain,_netdev,noauto,x-systemd.automount
> >> > >>
> >> > >> ++
> >> > >>
> >> > >> On Mon, 12 Jun 2023 at 16:11, Tom Callaway
<spotrh at gmail.com>
> wrote:
> >> > >>>
> >> > >>> Okay, so if it's not SELinux... perhaps
something about how R is
> being configured is different?
> >> > >>>
> >> > >>> Can you provide a log of the output (including
the commandline
> with any options passed) for configure?
> >> > >>>
> >> > >>> Also, can you provide the specific mount options
for the cifs
> share?
> >> > >>>
> >> > >>> ~spot
> >> > >>>
> >> > >>> On Mon, Jun 12, 2023 at 9:12?AM Alexandre
Courtiol <
> alexandre.courtiol at gmail.com> wrote:
> >> > >>>>
> >> > >>>> Thanks for the tip but nope: SELinux is off
on that system...
> >> > >>>>
> >> > >>>> On Mon, 12 Jun 2023 at 13:24, I?aki Ucar
<
> iucar at fedoraproject.org> wrote:
> >> > >>>>>
> >> > >>>>> On Mon, 12 Jun 2023 at 13:19, Tom
Callaway <spotrh at gmail.com>
> wrote:
> >> > >>>>> >
> >> > >>>>> > Hmm, that's a weird one. Is
SELinux on and enforcing on that
> setup?
> >> > >>>>>
> >> > >>>>> I was going to bet on the same thing. :)
But let me add: if the
> answer
> >> > >>>>> is affirmative, and this doesn't
happen with SELinux disabled,
> then
> >> > >>>>> *the answer is NOT to disable SELinux*.
Instead, let's figure
> this
> >> > >>>>> out, because the answer is to set the
proper labels or
> permissions.
> >> > >>>>>
> >> > >>>>> I?aki
> >> > >>>>>
> >> > >>>>> >
> >> > >>>>> > ~spot
> >> > >>>>> >
> >> > >>>>> > On Mon, Jun 12, 2023, 6:55 AM
Alexandre Courtiol <
> >> > >>>>> > alexandre.courtiol at gmail.com>
wrote:
> >> > >>>>> >
> >> > >>>>> > > Dear Fedora-R enthusiasts,
> >> > >>>>> > >
> >> > >>>>> > > We are experiencing some
issues with the binary releases of
> R 4.3 for
> >> > >>>>> > > Fedora 37 & 38 when
running it on our infrastructure.
> >> > >>>>> > >
> >> > >>>>> > > The issue is that packages
won't install, unless the
> (default) staged
> >> > >>>>> > > installation process for
packages is switched off
> (--no-staged-install).
> >> > >>>>> > >
> >> > >>>>> > > It seems to be related to the
fact that we are storing R
> libraries on a
> >> > >>>>> > > drive mounted via samba share.
> >> > >>>>> > >
> >> > >>>>> > > Interestingly, installing R
from sources does not cause the
> issue.
> >> > >>>>> > > When using other OS (e.g.
Arch) on the same infrastructure,
> the problem
> >> > >>>>> > > also disappears.
> >> > >>>>> > >
> >> > >>>>> > > We are looking for a way out
that would allow users to
> install R packages
> >> > >>>>> > > as usual and that would
ideally not force the admin to
> install R from
> >> > >>>>> > > sources.
> >> > >>>>> > >
> >> > >>>>> > > The issue should be
reproducible as follows:
> >> > >>>>> > >
> >> > >>>>> > > 1. Mount the share:
> >> > >>>>> > >
> >> > >>>>> > > $ mount -t cifs //server/share
/mnt/share -o ...options...
> >> > >>>>> > >
> >> > >>>>> > > 2. Put R library onto the
share:
> >> > >>>>> > >
> >> > >>>>> > > $ mkdir /mnt/share/R
> >> > >>>>> > > $ ln -s /mnt/share/R ~/R
> >> > >>>>> > >
> >> > >>>>> > > 3. Install a package that has
dependencies (It doesn't
> matter whether the
> >> > >>>>> > > dependencies actually have to
be installed or not. E.g. if
> you install all
> >> > >>>>> > > of the dependencies of
'dplyr' and then install 'dplyr'
> itself in an extra
> >> > >>>>> > > call to
'install.packages', the problem will still be
> triggered.):
> >> > >>>>> > >
> >> > >>>>> > > $ Rscript -e
"install.packages('dplyr', repos='
> https://cloud.r-project.org
> >> > >>>>> > > ')"
> >> > >>>>> > >
> >> > >>>>> > > 4. The installation will fail
while trying to move the
> package to its final
> >> > >>>>> > > location:
> >> > >>>>> > >
> >> > >>>>> > > mv: cannot move
> >> > >>>>> > >
>
'/mnt/share/R/x86_64-redhat-linux-gnu-library/4.3/00LOCK-dplyr/00new/dplyr'
> >> > >>>>> > > to
> '/mnt/share/R/x86_64-redhat-linux-gnu-library/4.3/dplyr':
Permission
> >> > >>>>> > > denied
> >> > >>>>> > > ERROR:   moving to final
location failed
> >> > >>>>> > >
> >> > >>>>> > > 5. Moving the folder as
described in the error manually
> from the shell
> >> > >>>>> > > succeeds:
> >> > >>>>> > >
> >> > >>>>> > > $ mv
> >> > >>>>> > >
> /mnt/share/R/x86_64-redhat-linux-gnu-library/4.3/00LOCK-dplyr/00new/dplyr
> >> > >>>>> > >
/mnt/share/R/x86_64-redhat-linux-gnu-library/4.3/dplyr
> >> > >>>>> > > $ # => works
> >> > >>>>> > >
> >> > >>>>> > > And if R is either compiled
from source (taking the usual
> >> > >>>>> > > "configure/make/make
install" route) or installed from
> Conda, the
> >> > >>>>> > > permission error at step 4
doesn't happen. We also tested
> creating the
> >> > >>>>> > > binary RPM package on the same
machine, and the result was
> the same as
> >> > >>>>> > > installing the binary package
from the repo (i.e. error).
> >> > >>>>> > >
> >> > >>>>> > > ++
> >> > >>>>> > >
> >> > >>>>> > >
> >> > >>>>> > > --
> >> > >>>>> > > Alexandre Courtiol,
www.datazoogang.de
> >> > >>>>> > >
> >> > >>>>> > >         [[alternative HTML
version deleted]]
> >> > >>>>> > >
> >> > >>>>> > >
_______________________________________________
> >> > >>>>> > > R-SIG-Fedora mailing list
> >> > >>>>> > > R-SIG-Fedora at r-project.org
> >> > >>>>> > >
https://stat.ethz.ch/mailman/listinfo/r-sig-fedora
> >> > >>>>> > >
> >> > >>>>> >
> >> > >>>>> >         [[alternative HTML version
deleted]]
> >> > >>>>> >
> >> > >>>>> >
_______________________________________________
> >> > >>>>> > R-SIG-Fedora mailing list
> >> > >>>>> > R-SIG-Fedora at r-project.org
> >> > >>>>> >
https://stat.ethz.ch/mailman/listinfo/r-sig-fedora
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> --
> >> > >>>>> I?aki ?car
> >> > >>>>
> >> > >>>>
> >> > >>>>
> >> > >>>> --
> >> > >>>> Alexandre Courtiol, www.datazoogang.de
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Alexandre Courtiol, www.datazoogang.de
> >> >
> >> >
> >> >
> >> > --
> >> > I?aki ?car
> >>
> >>
> >>
> >> --
> >> I?aki ?car
> >
> >
> >
> > --
> > Alexandre Courtiol, www.datazoogang.de
>
>
>
> --
> I?aki ?car
>

-- 
Alexandre Courtiol, www.datazoogang.de

	[[alternative HTML version deleted]]

Hi all,

The admin from our setup created a script that should help you reproduce
our problem (see below).
Of course, it may be that it is not an RPM-related issue, but we hope you
will have an idea on how to solve the issue all the same.

# 1. Prepare VM and set up Samba

- install Fedora 38 Workstation
(Fedora-Workstation-Live-x86_64-38-1.6.iso). Username: 'user'
- disable SELinux (edit /etc/selinux/config and set
"SELINUX=disabled") and
reboot. Alternatively set up SELinux as described here:
https://docs.fedoraproject.org/en-US/quick-docs/samba/

```
sudo dnf install samba
mkdir ~/share
sudo mkdir /mnt/share
sudo smbpasswd -a user
sudo cat << 'EOF' >> /etc/samba/smb.conf
[share]
  path = /home/user/share
  writable = yes
  browsable = yes
  valid users = user
  create mask = 0660
  directory mask = 0770
EOF
sudo systemctl restart smb
sudo mount -t cifs //localhost/share /mnt/share -o username=user,uid=user
```


# 2. Install R with dnf

```
sudo dnf install R
mkdir /mnt/share/R
ln -s /mnt/share/R ~/R
```
- start R and install a package with dependencies (e.g.
"install.packages('dplyr')")
- installation should fail (it does so on our side)


# 3. Compile R from sources

```
sudo dnf erase R
sudo dnf builddep R
sudo dnf install java-17-openjdk-devel
curl -Ls https://cran.r-project.org/src/base/R-4/R-4.3.0.tar.gz | tar xzf -
cd R-4.3.0
./configure
make
sudo make install
```
- installing a package with dependencies should succeed now (on our side at
least)


Many thanks, for having another look at this knot.
Best,
Alex

On Wed, 14 Jun 2023 at 10:50, Alexandre Courtiol <
alexandre.courtiol at gmail.com> wrote:
> Thanks again, unfortunately it still fails when mounting the drive on a
> windows machine (even using nolease) on our end.
> We will prepare a fully reproducible example using a virtual machine. This
> could take us a day or two.
> ++
>
> On Tue, 13 Jun 2023 at 11:05, I?aki Ucar <iucar at fedoraproject.org>
wrote:
>
>> On Tue, 13 Jun 2023 at 10:35, Alexandre Courtiol
>> <alexandre.courtiol at gmail.com> wrote:
>> >
>> > Thanks a lot for persevering.
>> >
>> > I?aki, that sounds very promising, could you please tell me what
>> software your SMB share is running on (i.e. what does the server side
look
>> like?), and if it's Samba, then what version did you use?
>>
>> There was a Windows 10 machine around, so I created a test folder and
>> hit "share".
>>
>> > (and yes, I am aware of your excellent cran2copr project).
>> >
>> > Tom, yes it also created the same issue with older RPMs. Please
also
>> find the configure log attached to this email.
>>
>> I see no significant differences, so this supports the hypothesis that
>> the issue is in the mount.
>>
>> I?aki
>>
>> >
>> > Best,
>> >
>> > Alex
>> >
>> >
>> >
>> > On Mon, 12 Jun 2023 at 21:08, I?aki Ucar <iucar at
fedoraproject.org>
>> wrote:
>> >>
>> >> Update:
>> >>
>> >> On Mon, 12 Jun 2023 at 17:40, I?aki Ucar <iucar at
fedoraproject.org>
>> wrote:
>> >> >
>> >> > Unfortunately, I cannot reproduce the issue here. :( I
mounted a smb
>> >> > share, created a symlink from ~/R, and installed dplyr
without any
>> >> > issue (apart from the installation process taking ages).
Fortunately,
>> >> > this rules out our binary R package as responsible for
the issue you
>> >> > are experiencing, unless I'm missing something.
>> >>
>> >> The experiment above was done using kio-fuse to mount the
folder. Now
>> >> I've been able to reproduce the error using a regular CIFS
mount. I
>> >> reproduced this both with Fedora and Debian R binaries. Then I
>> >> executed the installation step by step, and I checked that,
when the
>> >> mv command is invoked, manually running mv in the console does
trigger
>> >> the error too. So again, I don't think there's any
issue with R.
>> >>
>> >> However, I've also managed to fix the issue by adding the
nolease
>> >> option to the mount command. Hope it helps.
>> >>
>> >> I?aki
>> >>
>> >> >
>> >> > Issue aside, may I ask what is the purpose of putting the
library in
>> a
>> >> > smb share? May I suggest [1]?
>> >> >
>> >> > [1]
https://cran.r-project.org/bin/linux/fedora/#additional-packages
>> >> >
>> >> > I?aki
>> >> >
>> >> > On Mon, 12 Jun 2023 at 16:48, Tom Callaway <spotrh at
gmail.com> wrote:
>> >> > >
>> >> > > The log of you running configure on your instance
allows me to
>> compare it to the log of when we build it as a package (you don't
need to
>> provide that one).
>> >> > >
>> >> > > Honestly, I have no idea right now _why_ pre-built R
would fail to
>> move files to a cifs share when a non-root user can do it. I'm
hoping maybe
>> something will jump out as to how it configures on your setup. We
aren't
>> modifying R's source code at all, nor are we configuring it in a
way that
>> would affect this as far as I know.
>> >> > >
>> >> > > Does this also happen with the older R binary RPMs?
If you go back
>> to 4.2.3 or 4.2.2, does it happen?
>> >> > >
>> >> > > ~spot
>> >> > >
>> >> > >
>> >> > > On Mon, Jun 12, 2023 at 10:38?AM Alexandre Courtiol
<
>> alexandre.courtiol at gmail.com> wrote:
>> >> > >>
>> >> > >> "is the staged installation enabled too in
the cases you don't
>> see this issue?"
>> >> > >> -> Yes, it should since we stick to default
config.
>> >> > >>
>> >> > >> "Can you provide a log of the output
(including the commandline
>> with any options passed) for configure?"
>> >> > >> -> If we compile locally there is no issue
despite not using any
>> option, and if we install R from binary, there is no configure log
AFAIK,
>> so I am not sure how to meet this request.
>> >> > >>
>> >> > >> "can you provide the specific mount options
for the cifs share?"
>> >> > >> ->
>>
sec=krb5,multiuser,mfsymlinks,user=smbuser,domain=localdomain,_netdev,noauto,x-systemd.automount
>> >> > >>
>> >> > >> ++
>> >> > >>
>> >> > >> On Mon, 12 Jun 2023 at 16:11, Tom Callaway
<spotrh at gmail.com>
>> wrote:
>> >> > >>>
>> >> > >>> Okay, so if it's not SELinux... perhaps
something about how R is
>> being configured is different?
>> >> > >>>
>> >> > >>> Can you provide a log of the output
(including the commandline
>> with any options passed) for configure?
>> >> > >>>
>> >> > >>> Also, can you provide the specific mount
options for the cifs
>> share?
>> >> > >>>
>> >> > >>> ~spot
>> >> > >>>
>> >> > >>> On Mon, Jun 12, 2023 at 9:12?AM Alexandre
Courtiol <
>> alexandre.courtiol at gmail.com> wrote:
>> >> > >>>>
>> >> > >>>> Thanks for the tip but nope: SELinux is
off on that system...
>> >> > >>>>
>> >> > >>>> On Mon, 12 Jun 2023 at 13:24, I?aki Ucar
<
>> iucar at fedoraproject.org> wrote:
>> >> > >>>>>
>> >> > >>>>> On Mon, 12 Jun 2023 at 13:19, Tom
Callaway <spotrh at gmail.com>
>> wrote:
>> >> > >>>>> >
>> >> > >>>>> > Hmm, that's a weird one. Is
SELinux on and enforcing on that
>> setup?
>> >> > >>>>>
>> >> > >>>>> I was going to bet on the same
thing. :) But let me add: if
>> the answer
>> >> > >>>>> is affirmative, and this doesn't
happen with SELinux disabled,
>> then
>> >> > >>>>> *the answer is NOT to disable
SELinux*. Instead, let's figure
>> this
>> >> > >>>>> out, because the answer is to set
the proper labels or
>> permissions.
>> >> > >>>>>
>> >> > >>>>> I?aki
>> >> > >>>>>
>> >> > >>>>> >
>> >> > >>>>> > ~spot
>> >> > >>>>> >
>> >> > >>>>> > On Mon, Jun 12, 2023, 6:55 AM
Alexandre Courtiol <
>> >> > >>>>> > alexandre.courtiol at
gmail.com> wrote:
>> >> > >>>>> >
>> >> > >>>>> > > Dear Fedora-R enthusiasts,
>> >> > >>>>> > >
>> >> > >>>>> > > We are experiencing some
issues with the binary releases
>> of R 4.3 for
>> >> > >>>>> > > Fedora 37 & 38 when
running it on our infrastructure.
>> >> > >>>>> > >
>> >> > >>>>> > > The issue is that packages
won't install, unless the
>> (default) staged
>> >> > >>>>> > > installation process for
packages is switched off
>> (--no-staged-install).
>> >> > >>>>> > >
>> >> > >>>>> > > It seems to be related to
the fact that we are storing R
>> libraries on a
>> >> > >>>>> > > drive mounted via samba
share.
>> >> > >>>>> > >
>> >> > >>>>> > > Interestingly, installing
R from sources does not cause
>> the issue.
>> >> > >>>>> > > When using other OS (e.g.
Arch) on the same
>> infrastructure, the problem
>> >> > >>>>> > > also disappears.
>> >> > >>>>> > >
>> >> > >>>>> > > We are looking for a way
out that would allow users to
>> install R packages
>> >> > >>>>> > > as usual and that would
ideally not force the admin to
>> install R from
>> >> > >>>>> > > sources.
>> >> > >>>>> > >
>> >> > >>>>> > > The issue should be
reproducible as follows:
>> >> > >>>>> > >
>> >> > >>>>> > > 1. Mount the share:
>> >> > >>>>> > >
>> >> > >>>>> > > $ mount -t cifs
//server/share /mnt/share -o ...options...
>> >> > >>>>> > >
>> >> > >>>>> > > 2. Put R library onto the
share:
>> >> > >>>>> > >
>> >> > >>>>> > > $ mkdir /mnt/share/R
>> >> > >>>>> > > $ ln -s /mnt/share/R ~/R
>> >> > >>>>> > >
>> >> > >>>>> > > 3. Install a package that
has dependencies (It doesn't
>> matter whether the
>> >> > >>>>> > > dependencies actually have
to be installed or not. E.g. if
>> you install all
>> >> > >>>>> > > of the dependencies of
'dplyr' and then install 'dplyr'
>> itself in an extra
>> >> > >>>>> > > call to
'install.packages', the problem will still be
>> triggered.):
>> >> > >>>>> > >
>> >> > >>>>> > > $ Rscript -e
"install.packages('dplyr', repos='
>> https://cloud.r-project.org
>> >> > >>>>> > > ')"
>> >> > >>>>> > >
>> >> > >>>>> > > 4. The installation will
fail while trying to move the
>> package to its final
>> >> > >>>>> > > location:
>> >> > >>>>> > >
>> >> > >>>>> > > mv: cannot move
>> >> > >>>>> > >
>>
'/mnt/share/R/x86_64-redhat-linux-gnu-library/4.3/00LOCK-dplyr/00new/dplyr'
>> >> > >>>>> > > to
>> '/mnt/share/R/x86_64-redhat-linux-gnu-library/4.3/dplyr':
Permission
>> >> > >>>>> > > denied
>> >> > >>>>> > > ERROR:   moving to final
location failed
>> >> > >>>>> > >
>> >> > >>>>> > > 5. Moving the folder as
described in the error manually
>> from the shell
>> >> > >>>>> > > succeeds:
>> >> > >>>>> > >
>> >> > >>>>> > > $ mv
>> >> > >>>>> > >
>>
/mnt/share/R/x86_64-redhat-linux-gnu-library/4.3/00LOCK-dplyr/00new/dplyr
>> >> > >>>>> > >
/mnt/share/R/x86_64-redhat-linux-gnu-library/4.3/dplyr
>> >> > >>>>> > > $ # => works
>> >> > >>>>> > >
>> >> > >>>>> > > And if R is either
compiled from source (taking the usual
>> >> > >>>>> > > "configure/make/make
install" route) or installed from
>> Conda, the
>> >> > >>>>> > > permission error at step 4
doesn't happen. We also tested
>> creating the
>> >> > >>>>> > > binary RPM package on the
same machine, and the result was
>> the same as
>> >> > >>>>> > > installing the binary
package from the repo (i.e. error).
>> >> > >>>>> > >
>> >> > >>>>> > > ++
>> >> > >>>>> > >
>> >> > >>>>> > >
>> >> > >>>>> > > --
>> >> > >>>>> > > Alexandre Courtiol,
www.datazoogang.de
>> >> > >>>>> > >
>> >> > >>>>> > >         [[alternative HTML
version deleted]]
>> >> > >>>>> > >
>> >> > >>>>> > >
_______________________________________________
>> >> > >>>>> > > R-SIG-Fedora mailing list
>> >> > >>>>> > > R-SIG-Fedora at
r-project.org
>> >> > >>>>> > >
https://stat.ethz.ch/mailman/listinfo/r-sig-fedora
>> >> > >>>>> > >
>> >> > >>>>> >
>> >> > >>>>> >         [[alternative HTML
version deleted]]
>> >> > >>>>> >
>> >> > >>>>> >
_______________________________________________
>> >> > >>>>> > R-SIG-Fedora mailing list
>> >> > >>>>> > R-SIG-Fedora at r-project.org
>> >> > >>>>> >
https://stat.ethz.ch/mailman/listinfo/r-sig-fedora
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>> --
>> >> > >>>>> I?aki ?car
>> >> > >>>>
>> >> > >>>>
>> >> > >>>>
>> >> > >>>> --
>> >> > >>>> Alexandre Courtiol, www.datazoogang.de
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >> --
>> >> > >> Alexandre Courtiol, www.datazoogang.de
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > I?aki ?car
>> >>
>> >>
>> >>
>> >> --
>> >> I?aki ?car
>> >
>> >
>> >
>> > --
>> > Alexandre Courtiol, www.datazoogang.de
>>
>>
>>
>> --
>> I?aki ?car
>>
>
>
> --
> Alexandre Courtiol, www.datazoogang.de
>

-- 
Alexandre Courtiol, www.datazoogang.de

	[[alternative HTML version deleted]]