Answers inline
----- Original Message -----> From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> To: "Ankireddypalle Reddy" <areddy at commvault.com>,
"Vijay Bellur" <vbellur at redhat.com>, gluster-users at
gluster.org,
> "Shyam" <srangana at redhat.com>, "Niels de Vos"
<ndevos at redhat.com>
> Sent: Wednesday, December 16, 2015 1:14:35 PM
> Subject: Re: [Gluster-users] libgfapi access
>
>
>
> On 12/16/2015 01:51 AM, Ankireddypalle Reddy wrote:
> > Thanks for the explanation. Valgrind profiling shows multiple
memcpy's
> > being invoked for each write through libgfapi. Is there a way to avoid
> > these memcpy's?. Also is there a limit on the number of glfs_t*
instances
For every buffer passed by application to libgfapi, libgfapi does a memcopy,
for various reasons. As of yet there is no way to limit this, there were some
dicussions
on adding capabilities for libgfapi to provide buffer so that memcopies can be
avoided.
> > that can be allocated at a given point of time. I've encountered
cases
> > where if more than 8 glfs_t* instances are being allocated then
glfs_init
> > fails.
>
Currently there is no way to limit the number of glfs_t instances for a process.
But it is quite easy to implement this in the application itself. What
application
are using libgfapi for?
> Including maintainers of gfapi.
>
> Pranith
>
> >
> >
> > Thanks and Regards,
> > Ram
> >
> > -----Original Message-----
> > From: Pranith Kumar Karampuri [mailto:pkarampu at redhat.com]
> > Sent: Monday, December 14, 2015 11:13 PM
> > To: Ankireddypalle Reddy; Vijay Bellur; gluster-users at gluster.org
> > Subject: Re: [Gluster-users] libgfapi access
> >
> >
> >
> > On 12/11/2015 08:58 PM, Ankireddypalle Reddy wrote:
> >> Pranith,
> >> Thanks for checking this. Though the time taken
to run
> >> was 18 seconds if you look at the time consumed
in
> >> user land as well as kernel land for executing
the
> >> command then it is evident that fuse took almost
half
> >> the time as libgfapi. Also from the collected
profiles
> >> it is evident that the average latency for the
write
> >> command is less for fuse than for libgfapi. Are
there
> >> any recommendations for I/O through libgfapi for
> >> disperse volumes. Is there any way to avoid the
extra
> >> memcpy's that are being made when performing
I/O
> >> through libgfapi.
> > hi Ankireddy,
> > Oh this is not a problem. If we use fuse, the system call
'write'
> > from ./GlusterFuseTest will go through fuse-kernel, fuse
kernel
> > sends the write operation to glusterfs mount process which
is a
> > user process. Time taken to complete that call from then on
is
> > computed against the glusterfs mount process until it
responds
> > to the fuse-kernel, not against the ./GlusterFuseTest
process.
> > If we use gfapi, there is no system call over head, instead
> > ./GlusterFuseTest process directly makes calls with the
bricks
> > through gfapi library. So all the time that the process
spends
> > communicating with the bricks and getting the response is
> > counted against ./GlusterFuseTest. That is the reason you
see
> > more 'user' time.
> >
> > So again, There are quite a few workloads where gfapi has proven to
give
> > better response times than fuse mounts because we avoid the context
switch
> > costs of ./GlusterFuseTest -> fuse-kernel -> glusterfs-mount
->
> > fuse-kernel (for response)-> ./GlusterFuseTest (for response to
'write')
> >
> > Hope that helps. Sorry for the delay in response, was in too many
meetings
> > yesterday.
> >
> > Pranith
> >> Thanks and Regards,
> >> Ram
> >>
> >> -----Original Message-----
> >> From: Pranith Kumar Karampuri [mailto:pkarampu at redhat.com]
> >> Sent: Thursday, December 10, 2015 10:57 PM
> >> To: Ankireddypalle Reddy; Vijay Bellur; gluster-users at
gluster.org
> >> Subject: Re: [Gluster-users] libgfapi access
> >>
> >>
> >>
> >> On 12/10/2015 07:15 PM, Ankireddypalle Reddy wrote:
> >>> Hi,
> >>> Please let me know in case you need any more details.
Even for
> >>> only write operations fuse seems to outperform
libgfapi. Is it
> >>> because of disperse volumes?. Also I noticed a lot of
data loss
> >>> in case I use libgfapi asyn I/O for disperse volumes.
> >> Fuse and gfapi seem to take same amount of time to complete the
run, i.e.
> >> 18 seconds. Could you let me know what you mean by fuse
outperforming
> >> gfapi?
> >>
> >> Pranith
> >>> Thanks and Regards,
> >>> Ram
> >>>
> >>> -----Original Message-----
> >>> From: Ankireddypalle Reddy
> >>> Sent: Wednesday, December 09, 2015 5:01 PM
> >>> To: 'Pranith Kumar Karampuri'; Vijay Bellur;
> >>> gluster-users at gluster.org
> >>> Subject: RE: [Gluster-users] libgfapi access
> >>>
> >>> Hi,
> >>> I upgraded my setup to gluster 3.7.3. I tested
writes by
> >>> performing writes through fuse and through
libgfapi.
> >>> Attached are the profiles generated from fuse and
libgfapi.
> >>> The test programs essentially writes 10000 blocks
each of
> >>> 128K.
> >>>
> >>> [root at santest2 Base]# time ./GlusterFuseTest /ws/glus
131072 10000 Mount
> >>> path: /ws/glus Block size: 131072 Num of blocks: 10000 Will
perform
> >>> write test on mount path : /ws/glus Succesfully created file
> >>> /ws/glus/1449697583.glfs Successfully filled file
> >>> /ws/glus/1449697583.glfs Write test succeeded Write test
succeeded.
> >>>
> >>> real 0m18.722s
> >>> user 0m3.913s
> >>> sys 0m1.126s
> >>>
> >>> [root at santest2 Base]# time ./GlusterLibGFApiTest
dispersevol santest2
> >>> 24007 131072 10000 Host name: santest2
> >>> Volume: dispersevol
> >>> Port: 24007
> >>> Block size: 131072
> >>> Num of blocks: 10000
> >>> Will perform write test on volume: dispersevol Successfully
filled file
> >>> 1449697651.glfs Write test succeeded Write test succeeded.
> >>>
> >>> real 0m18.630s
> >>> user 0m8.804s
> >>> sys 0m1.870s
> >>>
> >>> Thanks and Regards,
> >>> Ram
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Pranith Kumar Karampuri [mailto:pkarampu at redhat.com]
> >>> Sent: Wednesday, December 09, 2015 1:39 AM
> >>> To: Ankireddypalle Reddy; Vijay Bellur; gluster-users at
gluster.org
> >>> Subject: Re: [Gluster-users] libgfapi access
> >>>
> >>>
> >>>
> >>> On 12/08/2015 08:28 PM, Ankireddypalle Reddy wrote:
> >>>> Vijay,
> >>>> We are trying to write data backed up by
Commvault
> >>>> simpana to glusterfs volume. The data
being written
> >>>> is around 30 GB. Two kinds of write
requests happen.
> >>>> 1) 1MB requests
> >>>> 2) Small write requests of size 128 bytes. In case of
libgfapi access
> >>>> these are cached and a single 128KB write request is made
where as in
> >>>> case of FUSE the 128 byte write request is handled to
FUSE directly.
> >>>>
> >>>> glusterfs 3.6.5 built on Aug 24 2015 10:02:43
> >>>>
> >>>> Volume Name: dispersevol
> >>>> Type: Disperse
> >>>> Volume ID: c5d6ccf8-6fec-4912-ab2e-6a7701e4c4c0
> >>>> Status: Started
> >>>> Number of Bricks: 1 x (2 + 1) = 3
> >>>> Transport-type: tcp
> >>>> Bricks:
> >>>> Brick1: ssdtest:/mnt/ssdfs1/brick3
> >>>> Brick2: sanserver2:/data/brick3
> >>>> Brick3: santest2:/home/brick3
> >>>> Options Reconfigured:
> >>>> performance.cache-size: 512MB
> >>>> performance.write-behind-window-size: 8MB
> >>>> performance.io-thread-count: 32
> >>>> performance.flush-behind: on
> >>> hi,
> >>> Things look okay. May be we can find something using
profile
> >>> info.
> >>>
> >>> Could you post the results of the following operations:
> >>> 1) gluster volume profile <volname> start
> >>> 2) Run the fuse workload
> >>> 3) gluster volume profile <volname> info >
/path/to/file-1/to/send/us
> >>> 4) Run the libgfapi workload
> >>> 5)gluster volume profile <volname> info >
/path/to/file-2/to/send/us
> >>>
> >>> Send both these files to us to check what are the extra fops
if any that
> >>> are sent over network which may be causing the delay.
> >>>
> >>> I see that you are using disperse volume. If you are going to
use
> >>> disperse volume for production usecases, I suggest you use
3.7.x
> >>> preferably 3.7.3. We fixed a bug in releases from 3.7.4 till
3.7.6 which
> >>> will be released in 3.7.7.
> >>>
> >>> Pranith
> >>>> Thanks and Regards,
> >>>> Ram
> >>>>
> >>>>
> >>>> -----Original Message-----
> >>>> From: Vijay Bellur [mailto:vbellur at redhat.com]
> >>>> Sent: Monday, December 07, 2015 6:13 PM
> >>>> To: Ankireddypalle Reddy; gluster-users at gluster.org
> >>>> Subject: Re: [Gluster-users] libgfapi access
> >>>>
> >>>> On 12/07/2015 10:29 AM, Ankireddypalle Reddy wrote:
> >>>>> Hi,
> >>>>>
> >>>>> I am trying to use libgfapi interface to
access
> >>>>> gluster volume. What I noticed is that reads/writes to
the gluster
> >>>>> volume through libgfapi interface are slower than
FUSE. I was
> >>>>> expecting the contrary. Are there any
recommendations/settings
> >>>>> suggested to be used while using libgfapi interface.
> >>>>>
> >>>> Can you please provide more details about your tests?
Providing
> >>>> information like I/O block size, file size, throughput
would be
> >>>> helpful.
> >>>>
> >>>> Thanks,
> >>>> Vijay
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> ***************************Legal
> >>>> Disclaimer***************************
> >>>> "This communication may contain confidential and
privileged material
> >>>> for the sole use of the intended recipient. Any
unauthorized review,
> >>>> use or distribution by others is strictly prohibited. If
you have
> >>>> received the message by mistake, please advise the sender
by reply email
> >>>> and delete the message. Thank you."
> >>>>
********************************************************************
> >>>> *
> >>>> * _______________________________________________
> >>>> Gluster-users mailing list
> >>>> Gluster-users at gluster.org
> >>>> http://www.gluster.org/mailman/listinfo/gluster-users
> >>>
> >>> ***************************Legal
> >>> Disclaimer***************************
> >>> "This communication may contain confidential and
privileged material
> >>> for the sole use of the intended recipient. Any unauthorized
review,
> >>> use or distribution by others is strictly prohibited. If you
have
> >>> received the message by mistake, please advise the sender by
reply email
> >>> and delete the message. Thank you."
> >>>
*********************************************************************
> >>> *
> >>
> >>
> >> ***************************Legal
Disclaimer***************************
> >> "This communication may contain confidential and privileged
material
> >> for the sole use of the intended recipient. Any unauthorized
review,
> >> use or distribution by others is strictly prohibited. If you have
> >> received the message by mistake, please advise the sender by reply
email
> >> and delete the message. Thank you."
> >>
**********************************************************************
> >
> >
> >
> > ***************************Legal Disclaimer***************************
> > "This communication may contain confidential and privileged
material for
> > the
> > sole use of the intended recipient. Any unauthorized review, use or
> > distribution
> > by others is strictly prohibited. If you have received the message by
> > mistake,
> > please advise the sender by reply email and delete the message. Thank
you."
> > **********************************************************************
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>