Displaying 17 results from an estimated 17 matches similar to: "Cores hang when calling mcapply"
2024 Dec 12
1
Cores hang when calling mcapply
Hi Gregg.
Just wanted to follow up on the solution you proposed.
I had to make some adjustments to get exactly what I wanted, but it works, and takes about 15 minutes on our server configuration:
temp <-
??????open_dataset(
????????????sources = input_files,
????????????format = 'csv',
????????????unify_schema = TRUE,
????????????col_types = schema(
????????????"ID_Key"
2024 Dec 12
1
Cores hang when calling mcapply
Hi Thomas,
Glad to hear the suggestion helped, and that switching to a `data.table` approach reduced the processing time and memory overhead?15 minutes for one of the smaller datasets is certainly better! Sounds like the adjustments you devised, especially keeping the multicore approach for `make_clean_names()` and ensuring that `ID_Key` values remain intact, were the missing components you
2024 Dec 11
1
Cores hang when calling mcapply
How is the server configured to handle memory distribution for individual users. I see it has over 700GB of total system memory, but how much can be assigned it each individual user?
AAgain - just curious, and wondering how much memory was assigned to your instance when you were running R.
regards,
Gregg
On Wednesday, December 11th, 2024 at 9:49 AM, Deramus, Thomas Patrick <tderamus at
2024 Dec 11
1
Cores hang when calling mcapply
Hello Thomas,
Consider that the primary bottleneck may be tied to memory usage and the complexity of pivoting extremely large datasets into wide formats with tens of thousands of unique values per column. Extremely large expansions of columns inherently stress both memory and CPU, and splitting into 110k separate data frames before pivoting and combining them again is likely causing resource
2024 Dec 11
1
Cores hang when calling mcapply
About to try this implementation.
As a follow-up, this is the exact error:
Lost warning messages
Error: no more error handlers available (recursive errors?); invoking 'abort' restart
Execution halted
Error: cons memory exhausted (limit reached?)
Error: cons memory exhausted (limit reached?)
Error: cons memory exhausted (limit reached?)
Error: cons memory exhausted (limit reached?)
2024 Dec 11
1
Cores hang when calling mcapply
Thomas,
I'm curious - what OS are you running this on, and how much memory does the computer have??
Let me know if that code worked out as I hoped.
regards,
gregg
On Wednesday, December 11th, 2024 at 6:51 AM, Deramus, Thomas Patrick <tderamus at mgb.org> wrote:
> About to try this implementation.
>
> As a follow-up, this is the exact error:
>
> Lost warning
2024 Dec 11
1
Cores hang when calling mcapply
It's Redhat Enterprise Linux 9
Specifically:
OS Information:
NAME="Red Hat Enterprise Linux"
VERSION="9.3 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.3 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
2010 Apr 19
1
[PATCH matahari] Removes all code for the previous CPUWrapper class.
This class has been replaced by the ProcessorsAgent.
Signed-off-by: Darryl L. Pierce <dpierce at redhat.com>
---
src/Makefile.am | 4 -
src/cpu.cpp | 216 -------------------------------------------------------
src/cpu.h | 111 ----------------------------
src/host.cpp | 24 ------
src/host.h | 3 -
src/schema.xml | 16 ----
6 files changed, 0 insertions(+),
2018 Apr 07
0
Is it possible to clone an NT ACL from one file or dir to a totally different file or dir ?
On 04/05/2018 08:32 PM, Andrew Bartlett wrote:
> On Sat, 2018-03-24 at 20:38 -0400, Ken McDonald via samba wrote:
>>> By default, step 4 takes forever to run on large datasets because it
>>> uses named entities. My ultimate plan was to use the numeric
>>> user/group id's in that step instead of named ones so the Winbind cost
>>> is not incurred. Seems for
2009 Jul 29
3
yum update broke ovirt
I did a yum update on my management server and now the ovirt gui is no
longer accessable. Is it supposed to just work or do I need to do
something after the yum update?
Justin.
2018 Apr 06
2
Is it possible to clone an NT ACL from one file or dir to a totally different file or dir ?
On Sat, 2018-03-24 at 20:38 -0400, Ken McDonald via samba wrote:
>
> > By default, step 4 takes forever to run on large datasets because it
> > uses named entities. My ultimate plan was to use the numeric
> > user/group id's in that step instead of named ones so the Winbind cost
> > is not incurred. Seems for the whole process, the calls to Winbind to
> >
2009 Jul 10
2
[PATCH: server 0/3] Add host-register.rb (replaces host-browser.rb in part)
Removes node identification functionality from host-browser.rb and adds a new script,
host-register.rb, that takes over that functionality.
The chief difference is that host-browser used a simple TCP server setup to get data
from the node, while host-register uses the qpid bus to do so. Specifically, it
communicates with the matahari qmf agent added to the node in two related patchsets to
node
2010 Nov 12
3
Updating oVirt Server to the latest Matahari 0.4.0 schema
Hi all,
I looked at patching oVirt Server to work with the latest Matahari schema, from Matahari's 'next' branch, version 0.4.0.
ovirt-server depends on matahari version 0.0.5, which is currently latest in Fedora, and it is used during Node registration, to enumerate hardware information.
Usage extracted from src/host-browser/host-register.rb is as follows:
host_list =
2009 Jul 08
1
[PATCH: host-browser replacement 0/3] replacement of host-browser on ovirt-server
The purpose of this patch is to replace the identify function in
host-browser.rb with a new script, host-register.rb.
host-register.rb is a qmf ruby console that interfaces with the newly
added matahari qmf agent on the ovirt node. While it stores node data
in the database with the same behavior as the original host-browser
implementation, it acquires the data using the amqp protocol (and
2009 Nov 04
4
[PATCH server] Update daemons to use new QMF.
This patch updates dbomatic, taskomatic and host-register to use the
new C++ wrapped ruby QMF bindings. It also fixes a couple of bugs
along the way including the 0 cpu bug for host-register. This is a
compilation of work done by myself and Arjun Roy.
Signed-off-by: Ian Main <imain at redhat.com>
---
src/db-omatic/db_omatic.rb | 111 ++++++-------
2018 Nov 01
2
RFC: System (cache, etc.) model for LLVM
Hi,
thank you for sharing the system hierarchy model. IMHO it makes a lot
of sense, although I don't know which of today's passes would make use
of it. Here are my remarks.
I am wondering how one could model the following features using this
model, or whether they should be part of a performance model at all:
* ARM's big.LITTLE
* NUMA hierarchies (are the NUMA domains
2010 Mar 15
0
[PATCH] Updated the list of ignored files and fixed whitespace issues.
This patch changes no functionality. It merely adds items to the ignore
file, cleans up the formatting for source files and updates the prologs
for files to include the name of the file being licensed.
Signed-off-by: Darryl L. Pierce <dpierce at redhat.com>
---
.gitignore | 12 +++++++++++
AUTHORS | 3 +-
ChangeLog | 1 -
INSTALL | 1 -
Makefile.am