• Introduction
• Introduction
• Data-driven research
• Data literacy
• Introducing data
• Types of spending data
• Reference data
• Machine-readable and useable data
• Working with data
• The data processing pipeline
• Getting data
• Cleaning data
• Analysis
• Presentation and engagement
• Framing: topic, objective, audience
• Engaging the public online
• Case study: BudgIT
• Working with others
• Why collaborate?
• Using technology in your work
• Countering arguments against publishing data
• Appendices
• About this book
• Privacy guide
• Resources
• Glossary

Resources

This chapter outlines a set of staple tools for data-driven research, explaining what it’s useful for and what the barrier to entry is.

Key to levels

• Basic. An off-the-shelf tool that can be learned and used independently within a day. No advanced installation (e.g. installation on servers) required.
• Intermediate. Between one day and one week to master basic functionality. May require tweaking of code.
• Advanced. Requires programming ability.

Stage 1: Extracting and getting data

Stage 2: Cleaning, Working with and Analyzing Data

Note on SPSS and R: It’s our impression that interviewees seemed largely to have been trained to use SPSS. R is important to mention, however, as it offers free access to a broad section of the same models, though based on a programming interface.

A few examples of analysis of spending data, which can be done with statistical software such as SPSS or R:

a) Hidden Markov Models: Hidden Markov Models were originally developed for finding patterns in bioinformatics, but they have turned out to be useful for predicting fraudulent and corrupt behaviour as well. They were, for example, used to analyse spending data from 50 mio transactions in the Slovenian platform Supervizor. Using Hidden Markov Models requires high quality data.

b) Benford’s law: Benford's law checks the distribution of figures in your data against how it should actually look. Diversions from the normal distribution can help detect fraudulent reporting (eg. if companies tend to report ernings less than $500 mio. in order to fit a particular regulation, Benford’s law could be a tool to detect that). Check out this example using Benford’s law to test the release of all Danish corporate tax filings, and check out this R blog post on the topic.

Though SPSS is fairly easy to get started using, it can be difficult to collaborate around, as it applies its own SPSS data format. Some models might also be unavailable from the basic SPSS package. R is a free alternative where all extensions are accessible and where community support and code samples are widely available. One possible compromise bridging the convenience of SPSS and the wide usability of R is the proprietary software R Revolution.

Stage 3: Presenting Data

Publishing Data