About 7 min read

Previously we corrected a couple of data entry issues that found their way into the Enron dataset and discussed how this may have otherwise affected the predicting power of our model. This post builds on this work and continues the data exploratory section of this project by addressing missing and incomplete data.

As seen in a previous posts this dataset is plagued with missing values. Using the same Pandas DataFrame we can print out a list of all the features in the dataset alongside the number of values missing for that feature.

missing_data = enron_data.isna().sum().to_frame(name='# missing values')
missing_data
  # missing values
salary 51
to_messages 60
deferral_payments 108
total_payments 21
exercised_stock_options 45
bonus 64
restricted_stock 35
shared_receipt_with_poi 60
restricted_stock_deferred 128
total_stock_value 20
expenses 49
loan_advances 142
from_messages 60
other 54
from_this_person_to_poi 60
poi 0
director_fees 130
deferred_income 96
long_term_incentive 80
email_address 35
from_poi_to_this_person 60

From the above we can see that every example in the dataset has a value for poi, indicating whether it represents a person of interest. However no other features can be considered complete. This will have the effect of causing difficulty for our predictive model to discover patterns from this data. The feature of largest concern is loan_advances which has only 4 values present in the entire dataset (~3%). Lets visualise the extent of the missing values in the form of a bar graph.

missing_data.plot(kind='bar', figsize=(15,5));
Missing Values by Attribute
Missing Values by Attribute

Due to the high number of missing values there may be many examples not offering anything useful to our predictive model. I’ve chosen, somewhat arbitrarily, a threshold of 85% missing values to mean those examples we may be interested in removing from the dataset.

We can retrieve these examples using the following snippet (thank you stackoverflow and Wen).

threshold = .85

# Return the rows that have a percentage of their data missing.
#
# `enron_data.isna().sum(1)` gives the number of missing values for each person.
# `enron_data.shape[1]` is equivalent to `len(enron_data.columns)`
enron_data[(enron_data.isna().sum(1) / enron_data.shape[1]) > threshold]
  salary to_messages deferral_payments total_payments exercised_stock_options bonus restricted_stock shared_receipt_with_poi restricted_stock_deferred total_stock_value … loan_advances from_messages other from_this_person_to_poi poi director_fees deferred_income long_term_incentive email_address from_poi_to_this_person
GRAMM WENDY L nan nan nan 119292.00 nan nan nan nan nan nan … nan nan nan nan False 119292.00 nan nan NaN nan
LOCKHART EUGENE E nan nan nan nan nan nan nan nan nan nan … nan nan nan nan False nan nan nan NaN nan
THE TRAVEL AGENCY IN THE PARK nan nan nan 362096.00 nan nan nan nan nan nan … nan nan 362096.00 nan False nan nan nan NaN nan
WHALEY DAVID A nan nan nan nan 98718.00 nan nan nan nan 98718.00 … nan nan nan nan False nan nan nan NaN nan
WROBEL BRUCE nan nan nan nan 139130.00 nan nan nan nan 139130.00 … nan nan nan nan False nan nan nan NaN nan

5 rows x 21 columns

These really are adding little value and none of them are persons of interest. In fact THE TRAVEL AGENCY IN THE PARK does not represent an individual employee and may have a negative impact on our predictive model. Lets just drop them from the dataset.

threshold = 1 - threshold

# Return the rows that have a percentage of their data present.
#
# `enron_data.shape[1]` is equivalent to `len(enron_data.columns)`
enron_data = enron_data.dropna(thresh=enron_data.shape[1] * threshold)

For various reasons, many real world datasets contain missing values, often encoded as blanks, or in the case of this dataset np.nan values. Such datasets however are incompatible with the estimators we’ll be using in later sections which assume that all values in an array are numerical, and that all have and hold meaning. A basic strategy to use incomplete datasets is to discard entire rows and/or columns containing missing values. However, this comes at the price of losing data which may be valuable (even though incomplete) and in the case of this dataset will remove every available feature. A better strategy is to impute the missing values, i.e. to infer them from the known part of the data.

There are a few statistical methods often employed to impute missing values that focus around central tendency. Each have their own qualities:

  • mean: The mean value is very sensitive to outliers. That is, one or two extreme values can change the mean a lot,
  • mode: Continuous data is unlikely going to have a single value that is most frequent. Therefore this is best used for categorical data,
  • median: The median is far less sensitive to outliers making it more robust.

The majority of the features in this dataset are concerned with financial data. According to the FindLaw report, where a value appears missing (has a “-” in place of a numerical value) this is equivalent to the value being 0. This we can imagine would be true for such features as director_fees and loan_advances since the majority of Enron employees would not be directors of the company and are unlikely to be receiving loan advances. In light of this let’s fill in the np.nan values for financial features with a 0.

financial_features = payment_features + stock_value_features
enron_data[financial_features] = enron_data[financial_features].fillna(0)

As for the other type of feature concerning numbers of emails being sent to and from Enron employees, np.nan values indicate that the data is unknown. For this type of feature I want to infer the values from the available data, most likely using the median value given it is less sensitive to outliers. However imputing the data this early may be a mistake; something that is discussed further in a later post concerning validation leakage.

Despite the risk of imputing data this early I am forced to do so anyway because the test_classifier method I will later use to evaluate my classifier does not allow for the validation data (after being split) to be imputed, or manipulated in any way.

Because the behaviour of a person of interest is likely to differ from somebody we’re not interested in I will separately impute the values for these two classes. I feel this will provide a more accurate and fair assessment of each classes distribution of email values.

from sklearn.preprocessing import Imputer

email_features = ['from_messages', 'from_poi_to_this_person', 'from_this_person_to_poi',
                  'shared_receipt_with_poi', 'to_messages']

poi_email_features = enron_data.loc[enron_data.poi == True, email_features]
non_poi_email_features = enron_data.loc[enron_data.poi == False, email_features]

# Simple imputer using the median value for each feature.
median_imputer = Imputer(missing_values=np.nan, strategy='median')

enron_data.loc[enron_data.poi == True, email_features] = median_imputer.fit_transform(poi_email_features)
enron_data.loc[enron_data.poi == False, email_features] = median_imputer.fit_transform(non_poi_email_features)

The one exception within the email features is email_address which is a text value where perhaps np.nan values should be treated as empty strings. However since the sklearn estimators require values to be numerical I will just remove the email_address feature entirely.

enron_data = enron_data.drop(columns=['email_address'])

Hopefully it is evident why dealing with incomplete data is an important step in data preparation, and the challenges it may present. Currently the dataset is as “complete” as I am able to achieve and will be used for the remainder of this project.