Skip to content

configuration-v1.md

Latest commit

 

History

History
292 lines (210 loc) · 7.29 KB

configuration-v1.md

File metadata and controls

292 lines (210 loc) · 7.29 KB
Raw

Configuration v1

This document specifies the target configuration format for OpenFisca Data Manager. It is intentionally separate from the current INI/JSON implementation so the migration can be implemented in small, testable steps.

Goals

  • Replace the historical config.ini + raw_data.ini pair with a single human-maintained YAML file.
  • Replace collection JSON files with YAML files that can still be generated by the tools.
  • Keep global user configuration separate from generated collection metadata.
  • Make paths, formats and per-survey overrides explicit.
  • Keep old INI/JSON files importable during the transition.

File Layout

Recommended layout:

~/.config/openfisca-data-manager/
├── data-manager.yaml
└── collections/
    ├── erfs.yaml
    └── bdf.yaml

data-manager.yaml is the user-maintained entrypoint. Files under collections/ describe collection metadata and may be generated or edited manually.

Global Config

Example data-manager.yaml:

version: 1

storage:
  collections_directory: /path/to/collections
  output_directory: /path/to/data
  tmp_directory: /path/to/tmp

defaults:
  source_format: sas
  store_format: parquet
  categorical_strategy: unique_labels

collections:
  erfs:
    label: Enquete revenus fiscaux et sociaux
    metadata: collections/erfs.yaml
    raw_surveys:
      "2019": /raw/erfs/2019
      "2020": /raw/erfs/2020

  bdf:
    label: Budget des familles
    metadata: collections/bdf.yaml
    source_format: parquet
    raw_surveys:
      "2017": /raw/bdf/2017

Global Fields

version is required and must be 1 for this format.

storage.collections_directory is the directory containing collection metadata files when relative metadata paths are used.

storage.output_directory is where transformed data is written.

storage.tmp_directory is where temporary files are written.

defaults is optional. Values apply to all collections and surveys unless overridden.

collections maps collection names to collection declarations.

Collection Declaration Fields

label is optional.

metadata is optional. If omitted, the default is <storage.collections_directory>/<collection_name>.yaml.

source_format, store_format and categorical_strategy are optional collection-level overrides.

raw_surveys replaces raw_data.ini. It maps survey suffixes, usually years, to raw input paths. Values may be either a string path or an object with overrides.

Example with per-survey overrides:

collections:
  erfs:
    metadata: collections/erfs.yaml
    raw_surveys:
      "2019": /raw/erfs/2019
      "2020":
        path: /raw/erfs/2020
        source_format: parquet
        store_format: parquet

Collection Metadata

Example collections/erfs.yaml:

version: 1

name: erfs
label: Enquete revenus fiscaux et sociaux

surveys:
  erfs_2019:
    label: ERFS 2019
    raw_path: /raw/erfs/2019
    source_format: sas
    store_format: parquet
    parquet_file_path: /path/to/data/erfs_2019
    tables:
      household:
        label: Household table
        source_format: parquet
        variables:
          - household_id
          - household_weight
        parquet_file: /path/to/data/erfs_2019/household.parquet
      person:
        label: Person table
        source_format: parquet
        parquet_file: /path/to/data/erfs_2019/person.parquet

Collection Metadata Fields

version is required and must be 1.

name is required and must match the key used in data-manager.yaml.

label is optional.

surveys maps survey names to survey metadata.

Survey Metadata Fields

label is optional.

raw_path is optional but recommended for traceability.

source_format, store_format and categorical_strategy are optional effective values used to build or rebuild the survey.

hdf5_file_path and parquet_file_path describe transformed storage. New builds should prefer parquet_file_path.

tables maps table names to table metadata.

Table Metadata Fields

label is optional.

source_format is optional if inherited from the survey.

variables is optional and contains the known variables for the table.

parquet_file is optional and points to the table parquet file when data is stored as parquet.

Additional reader-specific fields may be stored under reader_options later.

Mapping From Legacy Files

config.ini

Legacy:

[collections]
collections_directory = /path/to/collections
erfs = /path/to/collections/erfs.json

[data]
output_directory = /path/to/data
tmp_directory = /path/to/tmp

Maps to:

storage:
  collections_directory: /path/to/collections
  output_directory: /path/to/data
  tmp_directory: /path/to/tmp
collections:
  erfs:
    metadata: /path/to/collections/erfs.yaml

raw_data.ini

Legacy:

[erfs]
2019 = /raw/erfs/2019
2020 = /raw/erfs/2020

Maps to:

collections:
  erfs:
    raw_surveys:
      "2019": /raw/erfs/2019
      "2020": /raw/erfs/2020

Collection JSON

Legacy JSON:

{
  "name": "erfs",
  "label": "ERFS",
  "surveys": {
    "erfs_2019": {
      "name": "erfs_2019",
      "label": "ERFS 2019",
      "parquet_file_path": "/path/to/data/erfs_2019",
      "tables": {
        "person": {
          "source_format": "parquet",
          "variables": ["person_id"]
        }
      }
    }
  }
}

Maps to:

version: 1
name: erfs
label: ERFS
surveys:
  erfs_2019:
    label: ERFS 2019
    parquet_file_path: /path/to/data/erfs_2019
    tables:
      person:
        source_format: parquet
        variables:
          - person_id

The survey name field is redundant with the surveys key and should not be repeated in the YAML form unless a future use case requires aliases.

Path Rules

  • Absolute paths are kept as-is.
  • Relative paths in data-manager.yaml are resolved relative to the directory containing data-manager.yaml.
  • Relative metadata paths are resolved relative to storage.collections_directory if that field is absolute, otherwise relative to data-manager.yaml.
  • Relative paths in collection metadata are resolved relative to the collection metadata file.

Compatibility Strategy

The loader should support three combinations during the transition:

  • INI global config + JSON collection metadata.
  • YAML global config + JSON collection metadata.
  • YAML global config + YAML collection metadata.

The canonical format is YAML global config + YAML collection metadata.

Migration CLI

The migrate-survey-manager-config command supports both conservative INI migration and YAML migration:

migrate-survey-manager-config
migrate-survey-manager-config --format yaml
migrate-survey-manager-config --format yaml --collections

--format yaml creates data-manager.yaml from config.ini and raw_data.ini.

--format yaml --collections also converts referenced collection JSON files to collection YAML files.

The old files must not be deleted by the migration command.

Open Questions

  • Whether source_format should be mandatory once inferred from raw files.
  • Whether generated collection metadata should include all raw file paths or only transformed storage paths.
  • Whether collection metadata files should be treated as generated artifacts or user-editable configuration.
  • Whether data-manager.yaml should allow environment-variable expansion such as ${DATA_DIR}.