Configuration

Most users can run gflow without configuration. Use a config file (TOML) and/or environment variables when you need to change where the daemon listens, or restrict GPU usage.

Config File

Default location:

~/.config/gflow/gflow.toml

Generate one interactively:

gflowd init

Minimal example:

[daemon]
host = "localhost"
port = 59000
# gpus = [0, 2]
# gpu_allocation_strategy = "sequential" # or "random"
# gpu_poll_interval_secs = 10

All CLIs accept --config <path> to use a different file:

gflowd --config <path> up
ginfo --config <path>
gbatch --config <path> --gpus 1 python train.py

Starting the daemon with a custom config (gflowd up -c <path>, restart, reload, or service install) passes the path through to the daemon itself, so the daemon serves from the same config file the CLI validated against — including a custom host/port.

Daemon Settings

Host and Port

[daemon]
host = "localhost"
port = 59000

• Default: localhost:59000
• Use 0.0.0.0 only if you understand the security implications.

GPU Selection

Restrict which physical GPUs the scheduler is allowed to allocate.

Config file:

[daemon]
gpus = [0, 2]

GPU Allocation Strategy

Control how gflow picks GPU indices when multiple GPUs are available.

Config file:

[daemon]
gpu_allocation_strategy = "sequential" # default
# gpu_allocation_strategy = "random"

• sequential: deterministic, prefer lower GPU indices first.
• random: randomize GPU selection order each scheduling cycle.

Daemon CLI flag (overrides config):

gflowd up --gpu-allocation-strategy random
gflowd restart --gpu-allocation-strategy sequential

Daemon CLI flag (overrides config):

gflowd up --gpus 0,2
gflowd restart --gpus 0-3

GPU Poll Interval

Control how quickly gflow notices unmanaged GPU occupancy changes.

Config file:

[daemon]
gpu_poll_interval_secs = 3 # default: 10

• Lower values react faster to external GPU usage changes, but poll NVML more often.
• Value must be at least 1.

Daemon CLI flag (overrides config):

gflowd up --gpu-poll-interval-secs 3
gflowd reload --gpu-poll-interval-secs 1

Runtime control (affects new allocations only):

gctl set-gpus 0,2
gctl set-gpus all
gctl show-gpus

Supported specs: 0, 0,2,4, 0-3, 0-1,3,5-6.

Precedence (highest → lowest):

1. CLI flag (gflowd up --gpus ...)
2. Env var (GFLOW_DAEMON__GPUS=...)
3. Config file (daemon.gpus = [...])
4. Default: all detected GPUs

For allocation strategy:

1. CLI flag (gflowd up --gpu-allocation-strategy ...)
2. Env var (GFLOW_DAEMON__GPU_ALLOCATION_STRATEGY=...)
3. Config file (daemon.gpu_allocation_strategy = "...")
4. Default: sequential

For GPU poll interval:

1. CLI flag (gflowd up --gpu-poll-interval-secs ...)
2. Env var (GFLOW_DAEMON__GPU_POLL_INTERVAL_SECS=...)
3. Config file (daemon.gpu_poll_interval_secs = ...)
4. Default: 10

Fair-Share Scheduling

On a shared workstation, fair-share scheduling reorders queued jobs so that users who have consumed less GPU-time recently are scheduled sooner. It only reorders jobs within the same priority band — a higher priority always wins — and never overrides hard limits (group concurrency caps, reservations, memory/GPU availability).

Historical usage is tracked per user as GPU-seconds (gpus × runtime) and decays exponentially with a configurable half-life, so recent usage matters more than old usage. Usage accounting is persisted across daemon restarts.

Config file:

[daemon.fair_share]
enabled = true        # default: true
half_life_secs = 604800 # default: 7 days

• enabled: whether fair-share reordering influences scheduling. Usage is accounted regardless, so enabling it later already has history available.
• half_life_secs: decay half-life for historical usage. Smaller values react faster to recent usage; larger values smooth fairness over a longer window.

Environment variables (override config):

GFLOW_DAEMON__FAIR_SHARE__ENABLED=false
GFLOW_DAEMON__FAIR_SHARE__HALF_LIFE_SECS=86400

With a single user, fair-share has no observable effect.

Timezone

Configure timezone for displaying and parsing reservation times.

Config file:

timezone = "Asia/Shanghai"

Per-command override:

gctl reserve create --user alice --gpus 2 --start "2026-02-01 14:00" --duration "2h" --timezone "UTC"

Supported formats:

• IANA timezone names: "Asia/Shanghai", "America/Los_Angeles", "UTC"
• Time input: ISO8601 ("2026-02-01T14:00:00Z") or simple format ("2026-02-01 14:00")

Precedence (highest → lowest):

1. CLI flag (--timezone)
2. Config file (timezone = "...")
3. Default: local system timezone

Job Executor

Choose how job payloads are executed.

[executor]
type = "tmux" # or "process"

• tmux (default): each job runs in a detached tmux session with terminal key injection. Enables interactive inspection via gjob attach and gjob close-sessions.
• process (opt-in, no tmux required): each job runs as a detached child process group (setsid), with stdout/stderr redirected to logs/<job_id>.log. Cancellation SIGTERMs the whole process group and escalates to SIGKILL after a grace period; zombie detection uses real process liveness. Not yet considered stable.

The process executor applies --conda-env in the job shell. The daemon locates a Conda installation using $CONDA_EXE, conda on $PATH, $CONDA_PREFIX, and common locations such as ~/miniconda3 or /opt/conda, then sources its etc/profile.d/conda.sh before running conda activate <env>. If no installation is found, the job fails with an explanatory message in its log.

Project Tracking

Use project settings to standardize job ownership metadata across teams.

[projects]
known_projects = ["ml-research", "cv-team"]
require_project = false

• known_projects: allowed project codes. Empty means any non-empty code is allowed.
• require_project: when true, every submitted job must include a non-empty project.
• Project values are normalized (trimmed). Whitespace-only values are treated as unset.
• Project code length limit: 64 characters.
• If both settings are used, project must be present and in known_projects.

Related CLI usage:

gbatch --project ml-research python train.py
gqueue --project ml-research
gqueue --format JOBID,NAME,PROJECT,ST,TIME

Resource Quotas

Quotas cap how much of the machine one user or project can occupy, so a single heavy user cannot starve everyone else on a shared workstation. Limits apply to two subjects — users (submitted_by) and projects (--project) — and a job must satisfy both its user quota and its project quota.

[quota]
# Fallback limits for every user / project not listed below.
# Omit a field (or the whole table) to leave that dimension unlimited.
default_user = { max_running_jobs = 4, max_running_gpus = 2, max_queued_jobs = 50 }
default_project = { max_running_gpus = 6 }

[quota.users]
alice = { max_running_gpus = 4 }

[quota.projects]
cv-team = { max_running_gpus = 8, max_queued_jobs = 100 }

Limit fields (all optional; unset = unlimited):

• max_running_jobs: maximum concurrently running jobs.
• max_running_gpus: maximum concurrently allocated GPUs. A job that would push usage over the cap waits; a job requesting more GPUs than the cap will never start, so size caps with that in mind.
• max_queued_jobs: maximum pending (queued/held) jobs, enforced at submission — submissions over the cap are rejected rather than queued.

Named entries are merged over the matching default field-by-field: alice above keeps default_user.max_running_jobs = 4 and max_queued_jobs = 50 while overriding max_running_gpus to 4.

When a running job hits a max_running_* cap, the job stays queued with reason Quota (visible via gqueue) and starts automatically when usage drops.

Runtime overrides (persist across daemon restarts, take precedence over the config file):

gctl quota list                                   # usage vs. limits
gctl quota set --user alice --max-running-gpus 4
gctl quota set --default-user --max-queued-jobs 50
gctl quota remove --user alice                    # fall back to gflow.toml

Notifications

Use Notifications when you need webhook or email delivery for job and system events.

• notifications.emails is also the SMTP transport used by per-job flags such as gbatch --notify-email.
• Keep the daemon on localhost when possible if notification payloads contain sensitive job metadata.

Logging

• gflowd: use -v/--verbose (see gflowd --help).
• Client commands (gbatch, gqueue, ginfo, gjob, gctl): use RUST_LOG (e.g. RUST_LOG=info).

Environment Variables

Nested daemon keys use double underscores (__).

export GFLOW_DAEMON__HOST=localhost
export GFLOW_DAEMON__PORT=59000
export GFLOW_DAEMON__GPUS=0,2
export GFLOW_DAEMON__GPU_ALLOCATION_STRATEGY=random
export GFLOW_DAEMON__GPU_POLL_INTERVAL_SECS=3

Files and State

gflow follows the XDG Base Directory spec:

~/.config/gflow/gflow.toml
~/.local/share/gflow/state.msgpack  (or state.json for legacy)
~/.local/share/gflow/logs/<job_id>.log

State Persistence Format

Starting from version 0.4.11, gflowd uses MessagePack binary format for state persistence:

• New installations: State is saved to state.msgpack (binary format)
• Automatic migration: Existing state.json files are automatically migrated to state.msgpack on first load
• Backward compatibility: gflowd can still read old state.json files

Recovery mode (state file issues)

If the state file cannot be deserialized or migrated (e.g. after upgrading/downgrading versions), gflowd enters recovery mode:

• gflowd continues running, but does not overwrite the state file.
• State changes are persisted to a single-snapshot journal file: ~/.local/share/gflow/state.journal.jsonl (it is overwritten on each save).
• /health returns 200 with status: "recovery" and mode: "journal".
• A backup copy is created next to the state file (e.g. state.msgpack.backup.<timestamp> or state.msgpack.corrupt.<timestamp>).

When the state file becomes readable again, gflowd loads the latest journal snapshot, rewrites the state file, and truncates the journal.

If the journal file is not writable, gflowd falls back to read-only mode and mutating APIs return 503.

To recover, upgrade/downgrade to a version that can read/migrate your state, or restore from the backup file.

Troubleshooting

Config file not found

ls -la ~/.config/gflow/gflow.toml

Port already in use

Change the port:

[daemon]
port = 59001

See Also

• Installation - Initial setup
• Quick Start - Basic usage
• Multi-User Usage - Shared scheduler setup
• GPU Management - GPU allocation