In Containers (Kubernetes)#

A project can run in many containers at once, such as the pods of a Kubernetes Job, without Slurm. Each container runs one share of the receptors:

stilt run /project --task 3/20

--task I/N runs task I of N in this process: every N-th receptor of the project, starting at I (0 to N - 1). Every task splits the receptors the same way, whenever it starts. Receptors whose results exist are skipped, so running a task again only fills in what is missing.

--receptors ids.txt runs only the receptors listed in a file, one id per line. With --task too, the task takes its share of that list.

What every container needs#

  • PYSTILT installed (pip install pystilt).

  • The project folder (config.yaml and receptors.csv), the meteorology, and the output directory, on a volume all the containers share. Results are written there, one file per simulation, so the containers do not need to talk to each other. The output directory can instead be on an object store (output: s3://bucket/..., see Projects And Variants), with the containers given its credentials. stilt run --output s3://bucket/... uses another output directory than config.yaml names, without editing it.

  • A compute root, where each run gets its workdir. Set PYSTILT_COMPUTE_ROOT to a folder on fast local disk, such as an emptyDir.

Fetching the meteorology once#

With download set, a run downloads the met files its hours need into the met directory. Pods on one shared volume find the files the first pod fetched. Pods with their own disks do not: 50 pods whose receptors fall on the same days download, and crop, the same files 50 times.

stilt fetch gets the files once, before the Job starts:

stilt fetch /project

For each met it gets the files that cover the time of every simulation that has yet to run, cropped to the met’s subgrid. They are the files stilt run would ask for, so a run that follows finds them all and downloads nothing. Run it on one pod, with the met directory on the volume the Job’s pods then mount. Simulations that are finished need no met and are left out; --output is as for stilt run, so it reads the same results.

A met without download is only checked: stilt fetch reports the files it lacks, as stilt check does, and does not make them.

It exits with 0 when every file is there, 1 when some could not be had (the archive does not have the file; it prints the times of the runs that need them), 2 when the command line is wrong or a met directory cannot be read, and 3 when it was stopped (Ctrl-C, or SIGTERM, which Kubernetes and Slurm send) or the archive was busy or the connection failed (a throttled bucket, a timeout, a refused connection), which a retry may get past. Any other unexpected error is 4 (Exit codes). A stop removes the file it was downloading, and a second stop signal while it does is ignored. Run it again to try the missing files, or to finish after a stop: the ones it got are kept. A process killed with SIGKILL, which cannot be caught, leaves its hidden .<name>.<random>.partial file in the met directory; delete it.

Fetches can overlap. Several stilt fetch processes, or a fetch beside runs, can share one met directory: each file is written under a temporary name and renamed into place when it is whole, so no one reads half a file. Two that start the same file both download it, and the second to finish replaces the first with the same contents.

In Python, project.fetch() does the same and returns the files, one row each, with the times it could not get files for.

A Kubernetes indexed Job#

An indexed Job gives each pod its index in JOB_COMPLETION_INDEX, which is the task number:

apiVersion: batch/v1
kind: Job
metadata:
  name: stilt-my-project
spec:
  completionMode: Indexed
  completions: 20          # N, the number of tasks
  parallelism: 20          # pods running at once
  backoffLimit: 6
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: stilt
          image: my-registry/pystilt:latest   # any image with PYSTILT
          command: ["sh", "-c"]
          args: ["exec stilt run /project --task $JOB_COMPLETION_INDEX/20 --cpus 4"]
          env:
            - name: PYSTILT_COMPUTE_ROOT
              value: /scratch
          resources:
            requests:
              cpu: "4"
              memory: 8Gi
          volumeMounts:
            - {name: project, mountPath: /project}
            - {name: met, mountPath: /met, readOnly: true}
            - {name: scratch, mountPath: /scratch}
      volumes:
        - name: project
          persistentVolumeClaim: {claimName: stilt-project}
        - name: met
          persistentVolumeClaim: {claimName: hrrr-arl}
        - name: scratch
          emptyDir: {}

The met directory in config.yaml must be the path inside the container (/met here). completions and the N in --task must match. --cpus 4 runs four receptors at once in each pod. exec makes stilt the process that receives Kubernetes’ stop signal, so a stopped pod stops its runs cleanly and exits with code 3, within a few seconds of the signal and so inside Kubernetes’ default 30 s grace period. That holds for --cpus above 1 too: the pod’s workers and HYSPLIT runs stop with it.

Exit codes#

stilt run exits with:

Code

Meaning

0

Every simulation it ran is complete.

1

Some simulations failed. stilt status counts them by reason. A retry runs them again, which helps only when the cause is fixed.

2

The command line was wrong: an unknown option, a bad value, or a directory that is not a project. Or a met directory does not exist or holds no ARL file (stilt check lists it). Nothing ran. Do not retry.

3

Retry it. Some simulations did not finish, because the run was stopped (Ctrl-C, or a stop signal from Kubernetes or Slurm), or a connection or I/O error, such as an output bucket that could not be reached, stopped an unfinished simulation or ended the run. A retry continues where it stopped.

4

A bug: any other error the command did not expect. Its traceback is printed. Do not retry; a retry meets the same error.

Whether an error is a connection or I/O error is read from the error and what it was raised from, not from its being an OSError: s3fs turns many permanent answers of a bucket into a plain OSError. It is 3 when it is a refused, reset or timed-out connection or a name-resolution failure, an error of the cloud libraries that is not an OSError (such as botocore’s EndpointConnectionError and ReadTimeoutError), an FTP 4xx reply (“421 too many users”), or has the status of a busy or briefly unavailable service (408, 429, 500, 502, 503, 504, and the throttling codes such as SlowDown). When an error hides its cause, as fsspec’s HTTP filesystem does by raising FileNotFoundError for every failure, the cause decides. An OSError that nothing explains is 3 when its errno is input/output trouble that goes away (EIO, a stale NFS handle, a busy or interrupted call) and 4 otherwise, a full disk (ENOSPC) included. It is 4 when the status is any other 3xx, 4xx or 5xx (a bucket in another region, a bad request, a 403, a 404), an FTP 5xx reply, a file that is not there or may not be used, or anything else. Every stilt command ends this way on an error it did not expect, not only stilt run, except that a closed standard output (stilt status | head) ends quietly with 1, as it always has. stilt fetch follows the same table, except that 1 is a file the archive does not have (a 404, whatever library reports it).

The same rule applies to an error inside one simulation, such as a worker that cannot write its particles or footprint to the output bucket. An error that is 3 by the rule above is not a failure of the simulation: no failure record is written (an earlier attempt’s is removed), the simulation stays unfinished (pending, or interrupted once its particles run had started), the other simulations go on, and the run ends with 3 so that a retry picks it up. Any other error inside a simulation, a SimulationError such as a failed HYSPLIT run or missing met files included, a bucket answer that a retry repeats (access denied, no such key), or a full disk, is a failure record and exit 1. When both happen in one run it exits 3, and the retry, which runs only what did not finish, ends with 1 if the failure remains. A bucket that stays down keeps every try at 3, so a driver sets how many times it retries. If the output cannot be reached to remove an earlier failure record, that record stays until a run writes the result.

For a Kubernetes indexed Job, a podFailurePolicy that retries on 3 and fails the task on 1 and 4 follows from this table.

When the Job is done, stilt status /project counts the finished and failed simulations of the whole project. stilt status /project --json gives the same counts, per state and variant, as JSON for the program that drives the Job.