Roadmap for co-creating the Fiscal Data Explorer

The Tech4Dev collaboration between CivicDataLab and Centre for Budget and Governance Accountability aims at developing a comprehensive dashboard called the Fiscal Data Explorer with Himachal Pradesh(HP) as the focus state. This dashboard will be a unique addition to the already existing data platform, Open Budgets India(OBI), in bringing together data on state budget allocations for Himachal and its spending data in a single tool.

CivicDataLab team planning the Fiscal Data Explorer roadmap in Himachal Pradesh

What makes state fiscal data so complex? 

Citizen’s access to data on government budgets in India diminishes drastically as we go deeper from the level of the Union Government to the sub-national levels. The low usability of state budget documents is also compounded by the fact that most of the detailed information is locked away in closed formats like PDF and hence cannot be read or consumed by machines. Further, the lack of metadata hinders the process of easy searchability for relevant fiscal data. 

When it comes to data on actual spending of these allocated funds at the district level, only a handful of states actually do bring out this information in the public domain. Even for this latter subset of the states, district spending data is associated with complicated coding structures making it difficult to comprehend for the uninitiated users. Needless to say, such hurdles at the state and district level, crucially restrict public engagement with fiscal data and in turn, constrain public participation with fiscal issues.

What is Fiscal Data Explorer?

Fiscal Data Explorer is a unique tool where citizens can explore both budgets and spending data of state governments in an easy to comprehend and simple to use manner. Such a dashboard can enable users to obtain answers to their budget-related queries in an interactive fashion. Some of the key features to assist the users in their exploration are downloadable data in machine-readable formats, easily searchable data tables, dynamic visualizations, etc.

What are we trying to address? 

The proposed project will aim at solving the dual problems of lack of easy accessibility to detailed information on state budgets and spending data in specific states and the lack of usability of such data due to the manner in which it is shared in the public domain. Further, with the introduction of comprehensive metadata, the Fiscal Data Explorer will help different user groups to curate information and insights in a more transparent and organized manner. With such an intervention, we expect to facilitate greater public engagement with both budgets at the state level and spending data at the district level. 

What will be the components/elements of Fiscal Data Explorer?

The following are the various components/elements which will be included in the dashboard:

  • Money Flow: This will provide aggregate and detailed information about the inflow and outflow of money in a state. 
  • Overall Budget (Budget Highlights): This will provide a brief representation of the budget on an aggregate level and information like major schemes, announcements, etc.
  • Expenditure: This will be divided into the following parts for easy dissemination of expenditure information as follows:
    • Expenditure Summary: This section will give an overall gist of how the state has performed in terms of its expenditure across years and among sectors.
    • Detailed Expenditure: The objective of this part will be to dig deeper and give granular insights into each department, commonly classified as different demand numbers or grant numbers.
    • Expenditure Tracker: This will help the user to track how money is being spent on a daily and monthly basis across various districts and expenditure heads.
  • Receipts: This will be divided into the following parts for easy dissemination of expenditure information as follows:
    • Detailed Receipts: Structured to provide both aggregates as well as granular insights on different sources of receipts like tax, non-tax revenue receipts, etc.
    • Receipts Tracker: This will enable the user to track how receipts are being collected on a daily and monthly basis across various districts and receipts heads. 
  • Schemes: This will provide detailed information related to various schemes across the years. 
  • Sectors: This will provide sectoral information and comprehensive representation of how individual sectors have performed across the years. It will also enable inter-sectoral comparisons in the state and its various districts.

What has been our roadmap for developing the dashboard?

The unique feature of this project has been the close association between public finance knowledge and the sound application of technological solutions to it. Needless to say, the technical, research and design roadmaps towards developing this dashboard are dependent on one another and naturally do have overlapping segments.

On the technical front, the following are the steps are being planned out


Technical Roadmap
Components HP Budgets HP Spending
Data Scraping
  • Expenditure: POST request, budget code mapping
  • Revenue: Based on the availability of machine-readable files from advocacy efforts

Expenditure & Receipts: Collecting datasets from DDOs using Scrapy framework. GET requests to fetch dataset HTML

Data Extraction (Note: This is subject to change based on our advocacy efforts)

  • Expenditure: Extracting datasets from HTML and pdf to machine-readable format
  • Other documents: Explore PDFs & extract data to machine-readable format

Expenditure & Receipts : Extracting datasets from HTML to csv using the python csv module 

Data Cleaning and Validation

Data cleaning and validation using different tools (GoodTables) and custom validation logic(s)

Process Manager

Setting up Apache Airflow process manager

Exploratory Data Analysis

Exploring and analysing data using different tools/technologies such as Apache Superset, Pandas

Data Modelling & Storage

The data model for the budget will be read heavy, low volume with categorical data

Further steps for data modelling:

  • Identify Query Patterns 
  • Database selection
  • Schema planning & development

The data model for spending data will be read and write heavy, high volume with time-series and geospatial data

Further steps for data modelling:

  • Identify Query Patterns 
  • Database selection
  • Schema planning & development
Platform Building

Steps of Platform building are: 

  • Developing a basic platform
  • User Interface and Experience Design
  • Compiling a list of features
  • Planning dynamic data Visualizations: Both Static & Dynamic
  • Feedback Loops

Publishing different blogs/Stories/Case-Studies on online platforms

On the research side of the project, the following roadmap is being followed: 

Research Roadmap
Components HP Budget HP Treasury
Research on State/sector related issues 

Budget Speech scan and newspaper articles, sectoral issues reports for HP

Civil Society Organizations and newspaper reports on fund utilization issues 

Scheme selection
  • National level schemes for HPs (prominent Centrally Sponsored Schemes)
  • State schemes (based on highest budget share in 2019-20)

Track those schemes in HP treasury

Data Mapping

Mapping based on budget codes and/or budget head descriptions

Trend Analysis
  • Analysis of trends in allocations and spending to build an analytical section on the dashboard
  • Identification of sectors with low fund utilisation and how the trend changes over a fiscal year
Content Development for dashboard
  • Based on different components
  • The textual content on budget highlights (capturing new announcements on schemes, programmes, focus area by the government)
  • Curated content on sectoral budgets and focus areas

 Blogs on OBI’s Medium platform


HP State Departments (Finance, select Line departments) 

  • State level organisations working on sectoral issues in HP
  • National and International organisations like World Bank, HAQ, etc.
  • Educational Institutes


In terms of design and design research activities we have the following roadmap:

Design Roadmap
Components HP Budgets HP Spending
Stakeholder Identification

Which user groups primarily refer to budget data? Provide examples of usage.

Which user groups primarily refer to spending data? Provide examples of usage.

(Pre)Survey – Send it out and collect and collate responses

Refer to the survey done for the Assam project

Usability Eval using Heuristic Analysis(HA)

HA of eBudget

HA of HimKosh

Usability Eval using Heuristic Analysis

HA of Assam Budget Explorer

Stakeholder interviews in Delhi and Himachal Pradesh

CPR, CBGA, NewsLaundry, HowIndiaLives, World Bank, HP Govt.

Analysis of Stakeholder Interviews

Document motivations, use cases, workflows and challenges faced.

Generation of User Stories

Convert insights from the previous stage into clear and concise stories that can inform future design decisions of the platform. Follows format described in agile methodology.

Identify and develop feedback loops for users 

Look into some potential feedback mechanisms based on preliminary understanding of the needs of the stakeholders.

Outreach & Dissemination  for HP Fiscal Data Explorer

Plan for disseminating information about the platform and its usefulness to some targeted groups on channels like social media, newsletters, etc.

Develop Fiscal Knowledge sharing mechanisms

Find spaces in the user flow to introduce more know-how related information to grow fiscal literacy of budget curious users of the platform

What do we expect this tool to achieve? 

We expect this tool to facilitate strengthened engagement with openly accessible public finance data at the state and district level by researchers, journalists and other stakeholder groups.  A deeper understanding of the priorities of the state government and the performances of the states in raising their revenues and spending the allocated funds have the potential to build stronger advocacy routes with the concerned government authorities by the civil society organizations. In doing so, the fiscal data on this proposed dashboard can be made extensive use of and therein lies its potential. 

About the Authors

This blog has been jointly written Gaurav Godhwani, Shreya Agrawal, Simonti Chakraborty and Thomson Muriyadan working on the Open Budgets India Initiative.

Published by

Leave a Reply

%d bloggers like this: