QoL Data Pipeline Automation
Making pipelines to the quality of life explorer seamless
Background
For more than a decade, The Charlotte Urban Institute has partnered with the City of Charlotte and Mecklenburg County to establish and maintain the data and computing infrastructure of the Charlotte Quality of Life (QoL) Explorer. The QoL explorer has around 80 individual indicators and our team has built an extensive process of data collection, management and analyses of datasets that feeds to the application, but a large majority of the processes were still manually done that contributed to inefficiencies. In the Fall of 2021, we outlined this project to revamp the QoL data pipeline.
Objective(s) and Scope
This project aims to optimize and streamline the data acquisition, processing, and storage processes for maintaining the QoL explorer. The project focuses on improving the efficiency and reliability of the data pipelines via automation that ensures accuracy and consistency of deliverables.
To this end, the project will identify gaps and bottlenecks in current QoL workflows, develop scripts and routines for data acquisition, computing and review processes and develop a robust database management framework.
The project is a significant test-case and demonstration of the larger UI reproducibility project.
- Start Date: 08/08/2021
- Status: Active