server-spy

If you can't beat them, join them. Measure how much resources your experiment actually gets, and who is stealing it from you.

Install Features GitHub
crates.io version GitHub release CI status GPLv3 license
Features

Gain confidence about the conditions and comparability of your experiments.

Auto-detect your experiment runs

  • Set regex filter and find all processes related to your experment.
  • Monitor how server resource conditions change for each run and qunatify how much each run gets affected.

Clear and interpretable metrics

  • Congestion Score: CPU, Mem and I/O pressure in one value.
  • Sched. wait How long a process is queued vs. run.
  • Focusing only on the cores your experiments actually runs on.

Debug what slows you down

See at a glance what's the bottleneck (CPU, Mem, I/O) and how much they contribute to congestion.

Who is slowing you down

  • Find out which of your runs got affected the most.
  • Find out which users or processes contribute how much to the slowdown.

LaTeX paper-ready export

Export a statistics table and report template that quantifies the server environment and fairness of resources over all your experiments.

Detach & reattach

The daemon keeps recording in the background. Close the TUI, come back later, attach again.

What it's not What it is
  • doesn't render SLURM or Kubernetes irrelevant
  • no alternative to perf, bpftrace, ...
  • not for in-depth analysis of single-run experiments
  • for smaller busy academic servers that don't use SLURM etc.
  • give easy and fast interpetation on who and what is slowing you down
  • make multiple sequential experiments on the same server comparable
Install

Written in rust, no runtime dependencies, installable on user-level without sudo.

Quick install
$ curl -fsSL https://raw.githubusercontent.com/lennart-rth/server-spy/master/install.sh | sh
PlatformCommandDetails
Debian / Ubuntucurl -fsSL https://lennart-rth.github.io/server-spy/install-apt.sh | sudo shapt repository
Nix / NixOSnix run github:lennart-rth/server-spy
Fedora / RHELrpm -Uvh server-spy-<ver>-1.x86_64.rpmdownload from GitHub releases
crates.iocargo install server-spy
static binaryserver-spy-<ver>-x86_64-unknown-linux-musl.tar.gzdownload from GitHub releases
Quick start
  1. Select your experiment processes - press f, to filter for your experiment runs using regex or simple matching.
  2. Monitor - Watch who and what is slowing down your experiments.
  3. Detach - Detach the TUI. server-spy continues to collect the data in the background. You can connect again any time.
  4. Save/Export - Save as CSV report or export LaTeX ready tables and templates directly.
fupdate the worker filter (with live preview)
s / lsave / load snapshots to disk
qterminate - stop the background daemon and exit
ddetach - exit, the daemon keeps recording
rrestart recording
tstealth mode - rename the processes so ps/top show something innocuous for others