Task #6941

Build a multi-core benchmark for RStudio to test resource saving

Added by Gianpaolo Coro almost 3 years ago. Updated over 2 years ago.

Status:ClosedStart date:Feb 07, 2017
Priority:NormalDue date:
Assignee:Andrea Dell'Amico% Done:

100%

Category:Default
Sprint:RStudio elastic management
Infrastructure:
Milestones:
Duration:

Description

An R benchmark script will be built to test the resource saving watcher on the RStudio machines.

ParallelBenchmark.r (1.15 KB) Gianpaolo Coro, Feb 07, 2017 05:59 PM


Related issues

Related to BlueBRIDGE - Task #6938: Reinforce RStudio support to make it scalable and robust Closed Feb 07, 2017 Feb 13, 2017
Related to BlueBRIDGE - Project Task #688: T9.2 Data Analytics Facilities [Months: 1-29] Closed Dec 18, 2015 Jan 18, 2018

History

#1 Updated by Massimiliano Assante almost 3 years ago

  • Sprint changed from WP04 to RStudio elastic management

#2 Updated by Gianpaolo Coro almost 3 years ago

  • % Done changed from 0 to 50
  • Assignee changed from Gianpaolo Coro to Andrea Dell'Amico
  • Status changed from New to In Progress
  • File ParallelBenchmark.r added

I have prepared a process to test the occupation of memory and CPU (attached). Everyone can use it from RStudio. It uses two parameters, coresToUse (number of cores to use) and numberOfCycles (controls memory size and the duration of the execution). @andrea.dellamico@isti.cnr.it when you're ready we can do the measurements.

#3 Updated by Massimiliano Assante over 2 years ago

  • Parent task deleted (#6938)

#4 Updated by Massimiliano Assante over 2 years ago

  • Related to Task #6938: Reinforce RStudio support to make it scalable and robust added

#5 Updated by Andrea Dell'Amico over 2 years ago

  • % Done changed from 50 to 80

We now have a very rude script that kills a user processes - but not their session - if they go over any of the two thresholds: CPU number - 1, Total memory - 1GB

I'm going to deploy it its actual form on all the rstudio instances.

In future it could be extended to kill rsession processes older than N days, and it could output something usable by a nagios check for reporting.

#6 Updated by Andrea Dell'Amico over 2 years ago

  • % Done changed from 80 to 100
  • Status changed from In Progress to Feedback

The script has been installed on all the dev and prod rstudio server instances.

#7 Updated by Gianpaolo Coro over 2 years ago

which is the schedule of the script on the machines?

#8 Updated by Andrea Dell'Amico over 2 years ago

It runs every 5 minutes right now.

#9 Updated by Gianpaolo Coro over 2 years ago

OK, I will make tests in the production env. VREs.

#10 Updated by Gianpaolo Coro over 2 years ago

  • Status changed from Feedback to Closed

There is something strange. I have repeatedly run a process using 5 cores and "few" RAM and after a while I was shutdown. I have used rstudio.d4science.org, could you please see the reason?
The process does not have issues when it runs using very few RAM (few kB).

Let's check that there is no wrong setting of the maximum allowed RAM (the threshold should be MAX RAM - 1GB).

#11 Updated by Andrea Dell'Amico over 2 years ago

Your jobs were killed because there were too many running processes:

Jun  8 14:30:01 rstudio logger: /usr/local/bin/kill-rogue-jobs: user gianpaolo.coro is running too many processes
Jun  8 14:30:01 rstudio logger: /usr/local/bin/kill-rogue-jobs: find_rogue_processes for user gianpaolo.coro

That happened four times:

Jun  8 13:45:02 rstudio logger: /usr/local/bin/kill-rogue-jobs: user gianpaolo.coro is running too many processes
Jun  8 13:50:02 rstudio logger: /usr/local/bin/kill-rogue-jobs: user gianpaolo.coro is running too many processes
Jun  8 14:25:01 rstudio logger: /usr/local/bin/kill-rogue-jobs: user gianpaolo.coro is running too many processes
Jun  8 14:30:01 rstudio logger: /usr/local/bin/kill-rogue-jobs: user gianpaolo.coro is running too many processes

No other users were effected.

#12 Updated by Gianpaolo Coro over 2 years ago

Also available in: Atom PDF