Download Data from ISTAT SDMX API

Downloads statistical data from the ISTAT (Istituto Nazionale di Statistica) SDMX API for a specified dataset. Uses centralized configuration for default values.

Usage

download_istat_data(
  dataset_id,
  filter = NULL,
  start_time = "",
  end_time = "",
  incremental = FALSE,
  timeout = NULL,
  verbose = TRUE,
  updated_after = NULL,
  return_result = FALSE,
  check_update = FALSE,
  cache_dir = .istatlab_cache_dir(),
  existing_data = NULL,
  api = getOption("istatlab.default_api", "legacy")
)

Arguments

dataset_id

Character string specifying the ISTAT dataset ID (e.g., "534_50")

filter

Character string specifying data filters. Default uses config value ("ALL")

start_time

Character string specifying the start period (e.g., "2019"). If empty, downloads all available data

end_time

Character string or Date specifying the end period (e.g., "2024", "2024-06", "2024-06-30"). If empty (default), no upper bound is applied. Accepts formats "YYYY", "YYYY-MM", or "YYYY-MM-DD".

incremental

Logical or Date/character. If FALSE (default), fetches all data. If a Date object or character string ("YYYY", "YYYY-MM", or "YYYY-MM-DD"), fetches only data from that period onwards using the SDMX startPeriod parameter. Takes precedence over start_time if both are provided.

timeout

Numeric timeout in seconds for the download operation. Default uses config value

verbose

Logical indicating whether to print status messages. Default is TRUE

updated_after

POSIXct timestamp. If provided, only data updated since this time will be retrieved. Used for incremental update detection.

return_result

Logical indicating whether to return full istat_result object instead of just the data.table. Default is FALSE for backward compatibility.

check_update

Logical indicating whether to check ISTAT's LAST_UPDATE timestamp before downloading. If TRUE and data hasn't changed since last download, returns NULL with a message. Default is FALSE for backward compatibility.

cache_dir

Character string specifying directory for download log cache. Defaults to the ISTATLAB_CACHE_DIR environment variable, or "meta" if unset.

existing_data

Optional data.table of previously downloaded data. When provided, the function determines the already-covered date range and downloads only non-overlapping periods, merging and deduplicating the result.

api

Character string specifying the API surface to use. One of "legacy" (default), "hvd_v1", or "hvd_v2". Can be set session-wide via options(istatlab.default_api = "hvd_v1").

Value

A data.table containing the downloaded data with an additional 'id' column, or NULL if the download fails or data is unchanged. If return_result = TRUE, returns an istat_result object with additional metadata (exit_code, md5, message, is_timeout).

Details

The default cache_dir is resolved from the ISTATLAB_CACHE_DIR environment variable (Sys.getenv("ISTATLAB_CACHE_DIR", unset = "meta")). Set this variable, for example in .Renviron, to share a single metadata cache across multiple projects; when unset, the local "meta" directory is used.

Examples

if (FALSE) { # \dontrun{
# Download all data for dataset 534_50
data <- download_istat_data("534_50")

# Download data from 2019 onwards
data <- download_istat_data("534_50", start_time = "2019")

# Download with specific filter
data <- download_istat_data("534_50", filter = "M..", start_time = "2020")

# Download data for a specific date interval
data <- download_istat_data("534_50", start_time = "2020", end_time = "2023")

# Download only data updated since a specific timestamp
timestamp <- as.POSIXct("2025-12-10 14:30:00", tz = "UTC")
data <- download_istat_data("534_50", updated_after = timestamp)

# Get full result with metadata
result <- download_istat_data("534_50", return_result = TRUE)
if (result$success) {
  print(paste("Downloaded", nrow(result$data), "rows, MD5:", result$md5))
}

# Check if data has been updated before downloading
data <- download_istat_data("534_50", check_update = TRUE)
# Returns NULL with message if data unchanged since last download
} # }