arlmet.index

arlmet.index(directory, *, recursive=False, cache_dir=None)

Index the ARL files in a directory by their headers.

Each file in directory is opened once, and what its index records say is kept: its source, grid, first and last valid time, and time step. The table is cached, so a later call reads again only the files that are new or changed (by size or modification time) and drops the files that are gone. Use DirectoryIndex.covering() to find the files for a time window.

Links to files are followed, and each real file is one row, shown by its real path (pathlib.Path.resolve()) however many links reach it. A link that cannot be followed (its target is gone, or links loop) is a row with the reason in error. Links to folders are not followed. Hidden files and folders (names starting with .) and backup copies (names ending in ~) are skipped. A file that is not ARL, or is damaged, is kept with the reason in error and is not used to cover a window.

Files are opened 16 at a time: on a network filesystem each one waits on a few small reads. The cache is saved every 500 files, so an interrupted first index of a large archive keeps what it read.

Parameters:

• directory (str | PathLike[str]) – The folder of ARL files.

• recursive (bool) – Also read every folder below directory, as a copy of NOAA’s YYYY/MM/ layout needs. By default only the files directly in directory are read.

• cache_dir (str | PathLike[str] | None) – Folder for the cache file. By default the cache is .arlmet-index.csv (.arlmet-index-recursive.csv with recursive) in directory when directory is writable, and otherwise a file named after the directory in $ARLMET_CACHE_DIR, or in the platform’s user cache folder (arlmet/index in $XDG_CACHE_HOME on Linux) when that variable is not set.

Return type:

DirectoryIndex

Raises:

• FileNotFoundError – If directory does not exist.

• OSError – If the cache cannot be written, or a folder cannot be listed.

Warns:

ARLFormatWarning – Whatever opening a file warns, such as a time step repeated byte for byte.

See also

DirectoryIndex.covering

The files for a time window.

File.check

Damage that opening a file does not catch.

Examples

>>> idx = arlmet.index("hrrr")
>>> idx.covering("2021-01-14 06:00", "2021-01-15 06:00")
[PosixPath('hrrr/20210114_06-11_hrrr'), ..., PosixPath('hrrr/20210115_06-11_hrrr')]