From Data Repositories to Production Data Pipelines: Bridging Hugging Face Datasets and Dagster with dagster-hf-datasets

[ParagEkbote]

With the milestone of 1M+ datasets on HF hub, I have contributed an library with hf datasets and dagster with the following features:

• Native support for Dataset and DatasetDict assets
• Automatic lineage tracking and schema observability
• Local dataset materialization with HFParquetIOManager
• Publish curated assets back to the Hub as part of your pipeline

Read the article on HF: https://huggingface.co/blog/AINovice2005/dagster-hf-datasets