Hi everyone,
In relation to the RODBC odbcClose bug which was fixed back in the
changelog below:
Version: 1.2-3 (2008-01-24, released)
* Plug a memory leak in inRODBCClose (closing a connection),
reported by Stephan Henne.
* Use translateChar() on character data sent in.
Background:
I am running some data from a SQL Server database, through unixODBC
(freetds) into R via the RODBC package. If I make many calls to
odbcConnect, perform a query and then close the connection over a period
of time the memory attached appears never to be freed and the process
eventually - say 4 hours into the run - dies due to lack of memory.
Research:
I have created a dummy piece of code which shows the growing memory
allocation shown below. Each call brings back between 20-100 pieces of
data which fits into a one column vector - so not a lot of data. The
varible x is normally returns from a function and used subsequently but
for the purposes of this test I have kept it simple.
library("RODBC")
for( i in 1:100 ) {
conn <- odbcConnect(************) <-- anonymised
x <- sqlQuery(conn, "SELECT * FROM table_name"); <--
anonymised
odbcClose(conn);
}
quit()
I am not sure whether the leak is related to this previous bug or if it is
related to the way the garbage collection works, but either way if I run
this piece of code the memory attached to the database handle is never
freed so as I bring back more and more data over a large number of runs my
memory slowly fills up and the application crashes, unable to allocate any
more memory.
Just to make sure I was running the latest and greatest RODBC I installed
it again from source, but the initial installation was only done last
month so I presume .install.packages would have got this latest version
anyway.
Any advice on where to start searching to patch this as it is a real
pain. I have included below version information and the valgrind
output. If this is a new bug I will add it the bug tracker but I wanted
to make sure I was not missing anything blindingly obvious to someone in
the know.
Thanks for your help,
Tom
> version
_
platform i686-pc-linux-gnu
arch i686
os linux-gnu
system i686, linux-gnu
status
major 2
minor 7.0
year 2008
month 04
day 22
svn rev 45424
language R
version.string R version 2.7.0 (2008-04-22)
==8682== Memcheck, a memory error detector.
==8682== Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward et al.
==8682== Using LibVEX rev 1471, a library for dynamic binary translation.
==8682== Copyright (C) 2004-2005, and GNU GPL'd, by OpenWorks LLP.
==8682== Using valgrind-3.1.0, a dynamic binary instrumentation framework.
==8682== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et al.
==8682== For more details, rerun with: -v
==8682=
R version 2.7.0 (2008-04-22)
Copyright (C) 2008 The R Foundation for Statistical Computing
ISBN 3-900051-07-0
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> library("RODBC")
--8682-- WARNING: unhandled syscall: 311
--8682-- You may be able to write your own handler.
--8682-- Read the file README_MISSING_SYSCALL_OR_IOCTL.> for( i in 1:100 ) {
+ conn <- odbcConnect(************) <-- anonymised
+ x <- sqlQuery(conn, "SELECT * FROM table_name"); <--
anonymised
+ odbcClose(conn);
+ }> quit()
==8682===8682== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 255 from 1)
==8682== malloc/free: in use at exit: 13,470,816 bytes in 6,739 blocks.
==8682== malloc/free: 121,121 allocs, 114,382 frees, 134,084,145 bytes
allocated.
==8682== For counts of detected errors, rerun with: -v
==8682== searching for pointers to 6,739 not-freed blocks.
==8682== checked 13,332,140 bytes.
==8682===8682== 56 bytes in 1 blocks are possibly lost in loss record 15 of 55
==8682== at 0x40045EB: calloc (vg_replace_malloc.c:279)
==8682== by 0x805AA60: R_chk_calloc (memory.c:2368)
==8682== by 0x57C89D6: RODBCDriverConnect (RODBC.c:244)
==8682== by 0x8160A68: do_dotcall (dotcode.c:863)
==8682== by 0x81857CE: Rf_eval (eval.c:489)
==8682== by 0x81895B7: do_set (eval.c:1420)
==8682== by 0x81855B0: Rf_eval (eval.c:463)
==8682== by 0x818631E: do_begin (eval.c:1172)
==8682== by 0x81855B0: Rf_eval (eval.c:463)
==8682== by 0x81879B4: Rf_applyClosure (eval.c:669)
==8682== by 0x8185469: Rf_eval (eval.c:507)
==8682== by 0x818631E: do_begin (eval.c:1172)
==8682===8682===8682== 56 bytes in 14 blocks are definitely lost in loss record
16 of 55
==8682== at 0x40051F9: malloc (vg_replace_malloc.c:149)
==8682== by 0x4005271: realloc (vg_replace_malloc.c:306)
==8682== by 0x80B47EC: parse_expression (regex.c:5202)
==8682== by 0x80B4934: parse_branch (regex.c:4714)
==8682== by 0x80B49BF: parse_reg_exp (regex.c:4666)
==8682== by 0x80B5317: Rf_regcomp (regex.c:4635)
==8682== by 0x812A38F: do_gsub (character.c:1356)
==8682== by 0x806650A: do_internal (names.c:1129)
==8682== by 0x81855B0: Rf_eval (eval.c:463)
==8682== by 0x818631E: do_begin (eval.c:1172)
==8682== by 0x81855B0: Rf_eval (eval.c:463)
==8682== by 0x81879B4: Rf_applyClosure (eval.c:669)
==8682===8682===8682== 237 bytes in 9 blocks are definitely lost in loss record
27 of 55
==8682== at 0x40051F9: malloc (vg_replace_malloc.c:149)
==8682== by 0x4D6BFC51: xmalloc (in /usr/lib/libreadline.so.5.0)
==8682== by 0x4D6AA4B9: readline_internal_teardown (in
/usr/lib/libreadline.so.5.0)
==8682== by 0x4D6BC1DA: rl_callback_read_char (in
/usr/lib/libreadline.so.5.0)
==8682== by 0x80F9A8F: Rstd_ReadConsole (sys-std.c:902)
==8682== by 0x8058949: Rf_ReplIteration (main.c:206)
==8682== by 0x8058A89: R_ReplConsole (main.c:306)
==8682== by 0x8058D67: run_Rmainloop (main.c:967)
==8682== by 0x8056670: main (Rmain.c:35)
==8682===8682===8682== 5,544 bytes in 99 blocks are definitely lost in loss
record 46 of
55
==8682== at 0x40045EB: calloc (vg_replace_malloc.c:279)
==8682== by 0x805AA60: R_chk_calloc (memory.c:2368)
==8682== by 0x57C89D6: RODBCDriverConnect (RODBC.c:244)
==8682== by 0x8160A68: do_dotcall (dotcode.c:863)
==8682== by 0x81857CE: Rf_eval (eval.c:489)
==8682== by 0x81895B7: do_set (eval.c:1420)
==8682== by 0x81855B0: Rf_eval (eval.c:463)
==8682== by 0x818631E: do_begin (eval.c:1172)
==8682== by 0x81855B0: Rf_eval (eval.c:463)
==8682== by 0x81879B4: Rf_applyClosure (eval.c:669)
==8682== by 0x8185469: Rf_eval (eval.c:507)
==8682== by 0x818631E: do_begin (eval.c:1172)
==8682===8682== LEAK SUMMARY:
==8682== definitely lost: 5,837 bytes in 122 blocks.
==8682== possibly lost: 56 bytes in 1 blocks.
==8682== still reachable: 13,464,923 bytes in 6,616 blocks.
==8682== suppressed: 0 bytes in 0 blocks.
==8682== Reachable blocks (those to which a pointer was found) are not
shown.
==8682== To see them, rerun with: --show-reachable=yes
