R devel - Jun 2013 - How can C++ read the R object written into socket with saveRDS or save

Hi, all,

Recently, I met one issue when using socket between R & C++ to transmit R
object. Would you pls help give me some suggestions? Many thanks!

[Background]:
I create a socket connection between R & C++ binary first, and then, want
to use saveRDS() or save() in R to save the object into connection
directly. So that the C++ binary can read the object, and send it to
another remote R.

[What I did so far]:
1. I used socketConnection in R and listen/accept in C++, to establish one
blocking socket.
2. I used saveRDS to save the R object into socket directly
3. I want to use "recv()" in C++ to receive the R object.

[Issues I met]:
I found actually, the saveRDS writes the R object with XDR format. I could
not know how many bytes are sent into socket, when calling saveRDS to save
R object. So in the C++ binary, I could not know exactly how many bytes I
should receive from the socket. It is not safe for me, to always use a
pre-defined buffer size to read from the socket.

Any suggestions for this? Are there safe way for me to read the R object
from the socket?
I do not do any conversion with the received data, and only need to
transfer them into a remote R to do the function execution.

====================
Rong "Jessica", Li (ÀîÈÙ)
Platform Symphony TET, CSTL, IBM Systems &Technology Group, Development
Tel:86-10-82451010  Email:ronglli@cn.ibm.com
	[[alternative HTML version deleted]]

Hi, all,

Recently, I met one issue when using socket between R & C++ to transmit R
object. Would you pls help give me some suggestions? Many thanks!

[Background]:
I create a socket connection between R & C++ binary first, and then, want
to use saveRDS() or save() in R to save the object into connection
directly. So that the C++ binary can read the object, and send it to
another remote R.

[What I did so far]:
1. I used socketConnection in R and listen/accept in C++, to establish one
blocking socket.
2. I used saveRDS to save the R object into socket directly
3. I want to use "recv()" in C++ to receive the R object.

[Issues I met]:
I found actually, the saveRDS writes the R object with XDR format. I could
not know how many bytes are sent into socket, when calling saveRDS to save
R object. So in the C++ binary, I could not know exactly how many bytes I
should receive from the socket. It is not safe for me, to always use a
pre-defined buffer size to read from the socket.

Any suggestions for this? Are there safe way for me to read the R object
from the socket?
I do not do any conversion with the received data, and only need to
transfer them into a remote R to do the function execution.

====================
Rong "Jessica", Li (ÀîÈÙ)
Platform Symphony TET, CSTL, IBM Systems &Technology Group, Development
Tel:86-10-82451010  Email:ronglli@cn.ibm.com
	[[alternative HTML version deleted]]

On 25/06/2013 16:52, Rong lI Li wrote:
>
> Hi, all,
>
> Recently, I met one issue when using socket between R & C++ to transmit
R
> object. Would you pls help give me some suggestions? Many thanks!
R-help was really the wrong list: see the posting guide. And multiple 
posting is really discouraged.

But see package 'parallel' which has lots of examples of doing this.
> [Background]:
> I create a socket connection between R & C++ binary first, and then,
want
> to use saveRDS() or save() in R to save the object into connection
> directly. So that the C++ binary can read the object, and send it to
> another remote R.
>
> [What I did so far]:
> 1. I used socketConnection in R and listen/accept in C++, to establish one
> blocking socket.
> 2. I used saveRDS to save the R object into socket directly
> 3. I want to use "recv()" in C++ to receive the R object.
>
> [Issues I met]:
> I found actually, the saveRDS writes the R object with XDR format. I could
It may do ... depending how you call it.  You may prefer to serialize() 
to a raw vector whose size you can find, and transmit that.
> not know how many bytes are sent into socket, when calling saveRDS to save
> R object. So in the C++ binary, I could not know exactly how many bytes I
> should receive from the socket. It is not safe for me, to always use a
> pre-defined buffer size to read from the socket.
>
> Any suggestions for this? Are there safe way for me to read the R object
> from the socket?
> I do not do any conversion with the received data, and only need to
> transfer them into a remote R to do the function execution.
>
> ====================>
> Rong "Jessica", Li (????)
> Platform Symphony TET, CSTL, IBM Systems &Technology Group, Development
> Tel:86-10-82451010  Email:ronglli at cn.ibm.com
> 	[[alternative HTML version deleted]]
You were asked not to send HTML.
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

On Jun 25, 2013, at 11:52 AM, Rong lI Li wrote:
> 
> Hi, all,
> 
> Recently, I met one issue when using socket between R & C++ to transmit
R
> object. Would you pls help give me some suggestions? Many thanks!
> 
> [Background]:
> I create a socket connection between R & C++ binary first, and then,
want
> to use saveRDS() or save() in R to save the object into connection
> directly. So that the C++ binary can read the object, and send it to
> another remote R.
> 
> [What I did so far]:
> 1. I used socketConnection in R and listen/accept in C++, to establish one
> blocking socket.
> 2. I used saveRDS to save the R object into socket directly
> 3. I want to use "recv()" in C++ to receive the R object.
> 
> [Issues I met]:
> I found actually, the saveRDS writes the R object with XDR format. I could
> not know how many bytes are sent into socket, when calling saveRDS to save
> R object. So in the C++ binary, I could not know exactly how many bytes I
> should receive from the socket. It is not safe for me, to always use a
> pre-defined buffer size to read from the socket.
> 
> Any suggestions for this? Are there safe way for me to read the R object
> from the socket?
There is nothing preventing you from adding a frame with the size of the
serialization - that is what Rserve does (mentioned earlier by Dirk - it even
has a C++ client ...): it uses a fixed-size header that specifies the kind of
payload and its length, then the serialization follows in the payload. That way
you only need to read a fixed header.

Cheers,
Simon

> I do not do any conversion with the received data, and only need to
> transfer them into a remote R to do the function execution.
> 
> ====================> 
> Rong "Jessica", Li (ÀîÈÙ)
> Platform Symphony TET, CSTL, IBM Systems &Technology Group, Development
> Tel:86-10-82451010  Email:ronglli@cn.ibm.com
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

On Jun 25, 2013, at 11:52 AM, Rong lI Li wrote:
> 
> Hi, all,
> 
> Recently, I met one issue when using socket between R & C++ to transmit
R
> object. Would you pls help give me some suggestions? Many thanks!
> 
> [Background]:
> I create a socket connection between R & C++ binary first, and then,
want
> to use saveRDS() or save() in R to save the object into connection
> directly. So that the C++ binary can read the object, and send it to
> another remote R.
> 
> [What I did so far]:
> 1. I used socketConnection in R and listen/accept in C++, to establish one
> blocking socket.
> 2. I used saveRDS to save the R object into socket directly
> 3. I want to use "recv()" in C++ to receive the R object.
> 
> [Issues I met]:
> I found actually, the saveRDS writes the R object with XDR format. I could
> not know how many bytes are sent into socket, when calling saveRDS to save
> R object. So in the C++ binary, I could not know exactly how many bytes I
> should receive from the socket. It is not safe for me, to always use a
> pre-defined buffer size to read from the socket.
> 
> Any suggestions for this? Are there safe way for me to read the R object
> from the socket?
There is nothing preventing you from adding a frame with the size of the
serialization - that is what Rserve does (mentioned earlier by Dirk - it even
has a C++ client ...): it uses a fixed-size header that specifies the kind of
payload and its length, then the serialization follows in the payload. That way
you only need to read a fixed header.

Cheers,
Simon

> I do not do any conversion with the received data, and only need to
> transfer them into a remote R to do the function execution.
> 
> ====================> 
> Rong "Jessica", Li (????)
> Platform Symphony TET, CSTL, IBM Systems &Technology Group, Development
> Tel:86-10-82451010  Email:ronglli at cn.ibm.com
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

Hi, all,

Many thanks for your suggestions/infos!

For my previous question: "how to get the size of R object, so that the C++
process can know exact bytes to read, and receive the R objectvia socket
connection", for now, I used the following way to achieve this. It worked
for me. Pls feel free to let me know if there are anything I missed.

I sent the length of serialized raw vector into socket connection first,
and then send the raw vector into connection:
> a <- list(1,2,3)
> a_serial <- serialize(a, NULL)
> a_length <- length(a_serial)
> a_length
[1] 70
> writeBin(as.integer(a_length), connection, endian="big")
> serialize(a, connection)
In C++ process, I receive one int variable first to get the length, and
then read <length> bytes from the connection.

It worked for me now.

BTW, if there is anything I against the posting rule, pls feel free to
correct me.

====================
Rong "Jessica", Li
Platform Symphony TET, CSTL, IBM Systems &Technology Group, Development
Email:ronglli@cn.ibm.com
	[[alternative HTML version deleted]]