arlmet.archives.Archive.sync#

Archive.sync(dest_dir, *, start=None, end=None, mirror='s3', dry_run=False)[source]#

Keep the local folder dest_dir complete and checked against NOAA’s MD5 list.

Reads the product’s listing.md5.txt from the mirror, finds the local copy of each listed file anywhere under dest_dir, and downloads what is missing. Nothing is ever deleted, and files NOAA does not list are not touched. dest_dir is the product’s folder (for HRRR, the one that holds 2024/07/...), and new files go where the archive’s own path puts them.

Parameters:

  • dest_dir (Path | str) – The local mirror of the product. Created if absent.

  • start (Timestamp | str | None) – Sync only the files covering this time range (inclusive). By default, the whole archive: from its start_date to now.

  • end (Timestamp | str | None) – Sync only the files covering this time range (inclusive). By default, the whole archive: from its start_date to now.

  • mirror (Literal['s3', 'ftp', 'http']) – Which copy of the archive to read the list and the files from, as for fetch().

  • dry_run (bool) – Say what would be done without downloading, reading a local file, or writing anything (not even the lock or the manifest).

Returns:

The files downloaded, adopted, unchanged, mismatched and failed.

Return type:

SyncResult

Raises:

  • ImportError – If fsspec is not installed.

  • ValueError – If the range begins before the archive’s start_date, or the mirror’s list has no entries.

  • RuntimeError – If another sync holds the lock (see Notes).

Notes

Each listed file is dealt with by what the folder holds:

  • No local copy: downloaded to a hidden .partial file, hashed as it streams, and renamed into place only if its MD5 is NOAA’s; otherwise it is deleted and the file is in failed.

  • A copy the manifest records as NOAA’s (its size and modification time are as recorded): nothing is done and the file is not read.

  • A copy the manifest records as differing from NOAA’s, while NOAA’s MD5 is still the one recorded: it is in mismatched and left alone, without being read.

  • A copy NOAA has corrected (NOAA’s MD5 is not the one recorded when the copy was checked, and the copy is not NOAA’s new file): downloaded again the same way, replacing it where it is.

  • A copy the manifest does not list (a mirror made another way) or whose size or modification time changed: its MD5 is computed and recorded. If it is NOAA’s it is adopted; if not it is mismatched and left alone.

A copy is found by its name wherever it sits under dest_dir, so a folder in another layout (all files in one folder, say) is not downloaded again; if a name has several copies, the one at the archive’s own path is used, else the first by path. Folders whose names start with a dot are not searched.

The manifest, arlmet-manifest.tsv in dest_dir, keeps apart what the local file is and what NOAA’s copy was when it was checked. It has one line per file with five tab-separated fields: the file’s MD5, NOAA’s MD5 at the check, the file’s size in bytes, its modification time in nanoseconds, and its path relative to dest_dir (last, so it may hold spaces). A file is read only when it is not in the manifest or its size or modification time differ from the line; to have a file checked again, delete its line. The manifest is written as the run goes, so a killed run keeps what it checked.

A folder .arlmet-sync.lock made with mkdir (atomic on NFS) stops a second sync and names the host and start time of the first. It is removed at the end, also after an error; a run killed without a chance to clean up leaves it, and the next run then asks you to delete it.

A file that cannot be fetched or fails its check is logged and the run goes on to the next. A file that NOAA lists but this archive has no path for (another NAMS domain, say) is skipped.

Examples

>>> from arlmet.archives import HRRRArchive
>>> result = HRRRArchive().sync("./hrrr", dry_run=True)
>>> result.downloaded
[PosixPath('hrrr/2019/06/20190612_00-05_hrrr'), ...]