Home Product Data Sources Customers Company Jobs Resources SQL School Playbook Sign In

Handy Python Libraries for Formatting and Cleaning Data

The real world is messy, and so too is its data. So messy, that a recent survey reported data scientists spend 60% of their time cleaning data. Unfortunately, 57% of them also find it to be the least enjoyable aspect of their job.

Cleaning data may be time-consuming, but lots of tools have cropped up to make this crucial duty a little more bearable. The Python community offers a host of libraries for making data orderly and legible—from styling DataFrames to anonymizing datasets.

Let us know which libraries you find useful—we’re always looking to prioritize which libraries to add to Mode Python Notebooks.

Scrub that Data Too bad cleaning isn’t as fun for data scientists as it is for this little guy.

Dora

Dora is designed for exploratory analysis; specifically, automating the most painful parts of it, like feature selection and extraction, visualization, and—you guessed it—data cleaning. Cleansing functions include:

  • Reading data with missing and poorly scaled values
  • Imputing missing values
  • Scaling values of input variables

Created by: Nathan Epstein
Where to learn more: https://github.com/NathanEpstein/Dora

datacleaner

Surprise, surprise, datacleaner cleans your data—but only once it’s in a pandas DataFrame. From creator Randy Olson: “datacleaner is not magic, and it won’t take an unorganized blob of text and automagically parse it out for you.”

It will, however, drop rows with missing values, replace missing values with the mode or median on a column-by-column basis, and encode non-numeric variables with numerical equivalents. This library is fairly new, but since DataFrames are fundamental to analysis in Python, it’s worth checking out.

Created by: Randy Olson
Where to learn more: https://github.com/rhiever/datacleaner

PrettyPandas

DataFrames are powerful, but they don’t produce the kind of tables you’d want to show your boss. PrettyPandas makes use of the pandas Style API to transform DataFrames into presentation-worthy tables. Create summaries, add styling, and format numbers, columns, and rows. Added bonus: robust, easy-to-read documentation.

Created by: Henry Hammond
Where to learn more: https://github.com/HHammond/PrettyPandas

tabulate

tabulate lets you print small, nice-looking tables with just one function call. It’s handy for making tables more readable with column alignment by decimal, number formatting, headers, and more.

One of the coolest features is the ability to output data in a variety of formats like HTML, PHP, or Markdown Extra, so you can continue working with your tabular data in another tool or language.

Created by: Sergey Astanin
Where to learn more: https://pypi.python.org/pypi/tabulate

scrubadub

Data scientists in fields like healthcare and finance regularly have to anonymize datasets. scrubadub removes personally identifiable information (PII) from free text, such as:

  • Names (proper nouns)
  • Email addresses
  • URLs
  • Phone numbers
  • username/password combinations
  • Skype usernames
  • Social security numbers

The documentation does a good job of showing ways in which you might want to customize scrubadub’s behavior, like defining new PII types or excluding certain kinds of PII from being scrubbed.

Created by: Datascope Analytics
Where to learn more: http://scrubadub.readthedocs.io/en/stable/index.html

Arrow

Let’s be honest: working with dates and times in Python is a pain. Local timezones aren’t automatically recognized. It takes several lines of unpleasant code to convert timezones and timestamps.

Arrow aims to fix these problems and plug functionality gaps to help you handle dates and times with less code and fewer imports. Unlike Python’s standard library, Arrow is time-zone aware and UTC by default. You can convert timezones or parse strings using one line of code.

Created by: Chris Smith
Where to learn more: http://arrow.readthedocs.io/en/latest/

Beautifier

Beautifier’s mission is simple: clean and prettify URLs and email addresses. You can parse emails by domain and username; URLs by domain and parameters (e.g. UTMs or tokens).

Created by: Sachin Philip Mathew
Where to learn more: https://github.com/sachinvettithanam/beautifier

ftfy

ftfy (fixes text for you) takes in bad Unicode outputs good Unicode. Basically, it fixes all the junk characters. “quotesâ€\x9d becomes "quotes"; ü becomes ü; &lt;3 becomes <3. If you work with text on a daily basis, this library is, as one user says, “a handy piece of magic.”

Created by: Luminoso
Where to learn more: https://github.com/LuminosoInsight/python-ftfy

Further resources for wrangling data

Here are a couple of our favorite reads on munging/wrangling/cleansing data.