Datagrepper Fetcher

Fetches Fedora Infrastructure message bus data from Datagrepper and saves each day as one file per day in Parquet format for use in additional processing, analytics, etc.

The files will be named fedora-{YYYYMMDD}.parquet and an additional file will be generated called fedora-{YYYYMMDD}.done which just is an empty file to note if the full parquet file is loaded.

The script will attempt to gather and write to the parquet file as it downloads data, meaning that your file may not be considered complete until all pages are loaded, and a .done file is created for that day. If you remove a set of files, for example, if you want to remove the month of June 2025 (rm fedora-202506*), the done files will be removed as well from your output and will be regathered.

By default, we go back 90 days, but check the done files to see if there is any missing days of data.

Usage

Build the container

docker build -t datagrepper-fetch .

Run the container

docker run --rm \
  -e START_DATE="$(date -d '90 days ago' +%F)" \
  -e END_DATE="$(date -d 'yesterday' +%F)" \
  -v ~/data/fedora/datagrepper-raw:/output \
  datagrepper-fetch

Parquet files will be written to:

~/data/fedora/datagrepper-raw

License

This project is licensed under the GNU General Public License v3.0.

Copyright © 2025 Robert Wright

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see https://www.gnu.org/licenses/.