Skip to content

Latest commit

 

History

28 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

.,;,,,,,.
.*:,    .,:::                                 +---------+
,*;:::..  .  *;.                             /|        /|
 ;***;;,..    ,:,,,,..                      / |       / |
   :****;,:  ..,     .:.:,                 +---------+  |
     .;***;:;*,.  ; .    ,,*               |  + - - -|- +
       .,;*******;,.,.      *:.            | /       | /
          .  *******.       ..:;.          |/        |/
              :*** ,*          .,,         +---------+
               ,:;;..:           :*
               . ,;*; .           ,..,,,,.
               .    .                   ..;;,.      .. : ::;;
              .,                     . ,:,.. *,:; ,:,;***;;.;
               ;                     .,, :* :*;*********;***,
          . .:      .              , ,:;,*.*************,.
          : .       :,,.,:...      .  .,:;**********;.
        :       ..,,:****.       ;:;. . .,.*******,
      . ;      ,:********.      :,**:. .:  : :*;
    .,  ,., ..,*;:   :**;: ..  .,.****,.,, ...;;
   , : .,,..,**,    ..,*; ,,:   ,.*******;;.  .*
   ..,,,,,,**,         .:;  .    *;*;*****;**;;
 ::*,: .*;*:.           :   ,   ,*        .
 ,*:;,;:*;.             ;, .    ;;
  .;**::.               ;*,,    ;,
                        .;**.:;**
                          .:;,:,
Outstretched hand holding a periodic cubic unit cell

farhand

Chat-driven orchestration of a shared computational-chemistry workstation.

Remote hands on the bench: submit, monitor, arbitrate and post-process jobs on a single shared workstation from a phone, over a chat client, with no shell access to the machine.

Why

Small computational-chemistry groups often have one good workstation rather than a cluster allocation, and institutional networks often permit no inbound SSH. The usual workflow tools assume both a scheduler and a shell. farhand assumes neither: a thin chat bot writes trigger files, a watcher process owns all compute, and a lease arbiter lets engines that do not contend run concurrently.

The design is extracted from a system that has run unattended plane-wave DFT (Quantum ESPRESSO), classical MD (LAMMPS), molecular DFT (ORCA) and GCMC (RASPA) campaigns on one hybrid-core workstation, controlled entirely from a phone across a four-week absence.

Status

Early. The components marked released below are generalised, tested and usable on their own. The remaining layers run in production in a private system and are being rewritten for public release component by component; see the roadmap.

Component State
arena — resource lease arbiter released, tested
launch — per-engine job runner released, tested
watcher — poll loop, job adoption, signals in development
bot — chat front end in development
post-processing track planned

How farhand is organised

farhand separates three things that are usually tangled together. The boundary matters: it is what lets the tool be upgraded without touching your work, and what lets your work be archived and cited without dragging infrastructure into the deposit.

Layer Contains Lives in Public?
Tool arena.sh, launch.sh, watcher.sh, the bot this repository yes (MIT)
Config arena.conf, engines.conf, chat tokens and allowlists ~/.config/farhand/ never — machine-specific, secret-bearing
Work job inputs, structures, manifests, outputs, analysis scripts any directory you choose your call

Three consequences worth stating:

  • Upgrading is safe. Nothing you create lives inside this repository, so git pull to a new version can never conflict with your edits or overwrite your data.
  • Your work is archivable on its own. A paper's data deposit is a snapshot of the work directory. No scrubbing, and no need to explain which files came from the tool.
  • Secrets never reach a repository. Chat tokens live only in ~/.config/farhand/, which is outside both trees by construction.

farhand does not care whether your work directory is a git repository. If it is, farhand can update it on request; if not, that step is skipped.

arena

Mutual exclusion for compute engines sharing one machine. Each engine declares the resource tokens it needs; engines with disjoint token sets co-run by construction. No daemon, no scheduler, no priorities.

mkdir -p ~/.config/farhand
cp arena/arena.conf.example ~/.config/farhand/arena.conf

arena/arena.sh acquire qe       # ok
arena/arena.sh acquire orca     # busy (cpu_hi:qe)   -- shares a token
arena/arena.sh acquire lammps   # ok                 -- disjoint, co-runs
arena/arena.sh status
arena/arena.sh release qe

Config is one line per engine, <engine> <token>.... Token names are arbitrary strings; only set overlap matters:

qe        cpu_hi
orca      cpu_hi
raspa     cpu_hi
raspa_e   ecores
lammps    cpu_lo gpu

Every check-then-grab runs under a single flock, so two concurrent watchers cannot both conclude a token is free. Acquisition is all-or-nothing: an engine needing two tokens claims neither unless both are available, so a refused request cannot leak a lease. release only drops leases the calling engine actually holds, so a crashed and restarted process cannot release another engine's token. acquire is idempotent for the current holder, which is what lets a watcher re-adopt a running job on restart.

launch

Runs one engine's job: pinned to a CPU set, output captured, success judged by a per-engine rule.

cp launch/engines.conf.example ~/.config/farhand/engines.conf
launch/launch.sh qe mysystem
[qe]
pin = 2-15
run = mpirun -np 12 pw.x -in __JOB__.in
ok  = grep -q 'JOB DONE' run.log

pin is a CPU list for taskset. Pinning is not optional on hybrid-core CPUs: an MPI calculation advances at the speed of its slowest rank, so one rank scheduled onto an efficiency core slows the entire job to that core's pace.

run is the command; __JOB__ is replaced with the job name.

ok decides whether the job actually succeeded, and it is a command rather than a flag because scientific codes disagree about how to report failure. Some return a reliable exit code; some exit 0 having crashed; some report success only in their log; some require parsing the results block. ok = exit0 trusts the exit code, anything else is run in the job directory and judged by its own exit status. Changing a success criterion is a config edit, not a code change.

Tests

bash tests/test_arena.sh

Roadmap

v0.1 — released: arena, launch.

v0.2 — minimum viable system. Enough to run an unattended campaign from a phone.

  • watcher: poll loop, trigger protocol, job state directory, PID tracking
  • job adoption on restart, so restarting the watcher never kills a running calculation
  • signal handling: TERM cascade, and the interaction with KillMode=process
  • bot: chat front end with /run, /status, /tail, /kill
  • systemd units and an install script
  • a worked end-to-end example

v0.3 and beyond.

  • post-processing track: leaseless, runs alongside any calculation, output returned to the chat
  • pp-track status reported in /status, so a post-processing job that is running, crashed, or never started is distinguishable rather than silent
  • file retrieval and upload commands
  • multi-lane engines (one engine with both a full-speed and a co-running lane)
  • updating work repositories on request
  • richer engine configuration: working directory, analysis directory

Citing

If farhand contributes to published work, please cite the archived release - see CITATION.cff.

Licence

MIT. See LICENSE.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages