R help - May 2012 - Query about memory used in list and dataframe

Hi,

I had a query regarding which object, a list or a dataframe, consumes more
R memory. Let me clarify this:

For example, I have a df of 6 rows and 12 columns, say 'test'. I do
object.size() and find it uses 3.3 KB of memory.

I run a loop and make a list, say 'testlist', of 6 elements, each
element
being the above mentioned df 'test'. The size of this list is 19.9 KB,
understandably.

Now I combine this list into a dataframe using rbind. The df formed has 12
cols and 36 rows. The size of this df is just 5.8 KB, almost a 75%
reduction in memory.

I had to work with a much larger list, and I thought of using the same
method to convert my bigger list (62 dataframes, each having 4 cols and
close to 200,000 rows) into a single dataframe. The big list, sat LIST A,
had a size of 571 MB. But when I convert it into a dataframe, say DF A,
using rbind, the object size increases to 1.35 GB. This was in
contradiction to the earlier result.

What am I missing? Why a 75% reduction in size in one case and double size
in other? Anyone with any explanation?

Sorry for the verbose email, just wanted to make my case clear.

Thanks in advance,
Regards
Shivam

	[[alternative HTML version deleted]]

I think you really have to show use your exact code that you did along
with an 'str' of each intermediate data structure since my quick test
does not bear out what you were saying:
> test.df <- data.frame(a1= 1:6, a2= 1:6, a3 = 1:6, a4 = 1:6, a5 = 1:6, a6
= 1:6
+     , a7=1:6, a8 = 1:6, a9 = 1:6, a10 = 1:6, a11 = 1:6, a12 =
1:6)
> test.df
  a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12
1  1  1  1  1  1  1  1  1  1   1   1   1
2  2  2  2  2  2  2  2  2  2   2   2   2
3  3  3  3  3  3  3  3  3  3   3   3   3
4  4  4  4  4  4  4  4  4  4   4   4   4
5  5  5  5  5  5  5  5  5  5   5   5   5
6  6  6  6  6  6  6  6  6  6   6   6   6
> object.size(test.df)
2264 bytes
> x <- 1:6
> test.df1 <- data.frame(x,x,x,x,x,x,x,x,x,x,x,x)
> object.size(test.df1)
2264 bytes
> list.1 <- list(x,x,x,x,x,x,x,x,x,x,x,x)
> object.size(list.1)
1032 bytes
> list.2 <- NULL
> for (i in 1:12) list.2[[i]] <- x
> object.size(list.2)
1032 bytes
> test.df3 <- as.data.frame(do.call(cbind, list.2))
> object.size(test.df3)
2264 bytes
>
>
>

On Sat, May 5, 2012 at 5:50 PM, Shivam <shivamsingh at gmail.com>
wrote:
> Hi,
>
> I had a query regarding which object, a list or a dataframe, consumes more
> R memory. Let me clarify this:
>
> For example, I have a df of 6 rows and 12 columns, say 'test'. I do
> object.size() and find it uses 3.3 KB of memory.
>
> I run a loop and make a list, say 'testlist', of 6 elements, each
element
> being the above mentioned df 'test'. The size of this list is 19.9
KB,
> understandably.
>
> Now I combine this list into a dataframe using rbind. The df formed has 12
> cols and 36 rows. The size of this df is just 5.8 KB, almost a 75%
> reduction in memory.
>
> I had to work with a much larger list, and I thought of using the same
> method to convert my bigger list (62 dataframes, each having 4 cols and
> close to 200,000 rows) into a single dataframe. The big list, sat LIST A,
> had a size of 571 MB. But when I convert it into a dataframe, say DF A,
> using rbind, the object size increases to 1.35 GB. This was in
> contradiction to the earlier result.
>
> What am I missing? Why a 75% reduction in size in one case and double size
> in other? Anyone with any explanation?
>
> Sorry for the verbose email, just wanted to make my case clear.
>
> Thanks in advance,
> Regards
> Shivam
>
> ? ? ? ?[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.


-- 
Jim Holtman
Data Munger Guru

What is the problem that you are trying to solve?
Tell me what you want to do, not how you want to do it.