About 7 min read

In previous sections I’ve cleaned data that exhibited issues with data entry and missing values. Now I would like to discover the outliers. This part of the data exploration has deliberately been left until last. The reason being that this step is destructive. That is up until now we have been correcting the data so that it is true to the source, or using sensible data where it was otherwise not available. In this section I will remove data that will not be useful to the investigation.

Conveniently the definition of an outlier can be described as a point which falls more than 1.5 times the interquartile range above the third quartile or below the first quartile.

original_data = enron_data.drop(columns=['poi']) # poi is not a feature and should not be considered

first_quantile = original_data.quantile(.25)
third_quantile = original_data.quantile(.75)

median = original_data.quantile(.5)
interquantile_range = third_quantile - first_quantile

outlier_data = median + 1.5 * interquantile_range
outliers = (original_data > outlier_data).sum(axis=1).to_frame(name='# of outliers') \
    .sort_values('# of outliers', ascending=False)

five_percent_of_sample = int(len(enron_data) * 0.06)

# Merge the outliers with the original enron data.
outliers = outliers.merge(right=enron_data, how='inner', left_index=True, right_index=True) \
  .loc[:, ['# of outliers', 'poi']]
outliers.head(five_percent_of_sample)
  # outliers poi
FREVERT MARK A 13 False
LAY KENNETH L 13 True
SKILLING JEFFREY K 12 True
TOTAL 12 False
WHALLEY LAWRENCE G 12 False
LAVORATO JOHN J 10 False
HAEDICKE MARK E 8 False
BELDEN TIMOTHY N 8 True

Our data set is really small, so I’m going to consider just 5% of the samples with the largest number of outliers:

  • Mark Frevert and Lawrence ‘Greg’ Whalley are lesser known employees but still top managers of Enron (the latter being President and COO of Enron) who also represent valuable examples for our predictive model.
  • Kenneth Lay (Enron chairman and CEO) and Jeffrey Skilling (former Enron president and CEO) are very well known persons from Enron and heavily involved in the scandal. They represent important examples in our dataset and falsely flagged as true outliers.
  • The example TOTAL is the total value of financial payments from the FindLaw dataset. This clearly is not an individual person and will unbalance the distribution across several features in the dataset. Therefore it should be removed.
  • John Lavorato, another senior Enron executive, has little media exposure and there are no reports of him being involved in the scandal. Since his data contains many outliers that could potentially bias our predictive model I’ll remove him from the dataset.
  • Timothy ‘Norris’ Belden is the former head of trading in Enron Energy Services. He is considered a key player in the Enron scandal constituting a good example for our predictive model.

From the above considered 8 examples I will exclude only 2 of them. However before doing so I will visualise some of the features, hoping they will make the outliers and errors immediately stand out. This will convince me further that these examples should be removed from the dataset.

from ggplot import *

ggplot(enron_data, aes(x='salary', y='bonus', color='poi')) + \
  geom_point(size=40.0)
Salary v Bonus Original
Salary v Bonus Original

The above graph diagrams the salaries each Enron employee earned against the bonuses they received. It is evident that there is some discrepancy with this data where one employee received both a salary and bonus at a size well beyond everyone else. Let’s take a look at this data point more closely.

enron_data[enron_data['salary'] > 1e7]
  salary to_messages deferral_payments total_payments exercised_stock_options bonus restricted_stock shared_receipt_with_poi restricted_stock_deferred total_stock_values expenses loan_advances from_messages other from_this_person_to_poi poi directors_fees deferred_income long_term_incentive from_poi_to_this_person
TOTAL 26704229.00 944.00 32083396.00 309886585.00 311764000.00 97343619.00 130322299.00 594.00 -7576788.00 434509511.00 5235198.00 83925000.00 41.00 42667589.00 6.00 False 1398517.00 -27992891.00 48521928.00 26.50

This data point corresponds to the TOTAL flagged previously as an outlier. Let’s just remove this and make the same plot to see how things have improved.

enron_data = enron_data.drop('TOTAL')

ggplot(enron_data, aes(x='salary', y='bonus', color='poi')) + \
  geom_point(size=40.0)
Salary v Bonus Without TOTAL
Salary v Bonus Without TOTAL

You may conclude that there are four more outliers in the data. However considering the top 2 earners are Jeffrey Skilling and Kenneth Lay, both persons of interest, these should remain in the dataset. Similarly we can conclude there to be a single outlier within the bonus data. We can list the highest receivers to see just who this is.

enron_data.sort_values(by='bonus', ascending=False).head().loc[:, ['salary', 'bonus', 'poi']]
  salary bonus poi
LAVORATO JOHN J 339288.00 8000000.00 False
LAY KENNETH L 1072321.00 7000000.00 True
SKILLING JEFFREY K 1111258.00 5600000.00 True
BELDEN TIMOTHY N 213999.00 5249999.00 True
ALLEN PHILLIP K 201955.00 4175000.00 False

The person receiving the highest bonus is John Lavorato, somebody we have already identified above as a potential outlier. Given this name has appeared a couple times as an outlier I will remove it from the dataset.

enron_data = enron_data.drop('LAVORATO JOHN J')

ggplot(enron_data, aes(x='salary', y='bonus', color='poi')) + \
  geom_point(size=40.0)
Salary v Bonus Without Outlier
Salary v Bonus Without Outlier

Now that we’ve cleaned up the dataset, lets see how many examples and features remain.

enron_data.info()

<class 'pandas.core.frame.DataFrame'>
Index: 139 entries, ALLEN PHILLIP K to YEAP SOON
Data columns (total 20 columns):
salary                       139 non-null float64
to_messages                  139 non-null float64
deferral_payments            139 non-null float64
total_payments               139 non-null float64
exercised_stock_options      139 non-null float64
bonus                        139 non-null float64
restricted_stock             139 non-null float64
shared_receipt_with_poi      139 non-null float64
restricted_stock_deferred    139 non-null float64
total_stock_value            139 non-null float64
expenses                     139 non-null float64
loan_advances                139 non-null float64
from_messages                139 non-null float64
other                        139 non-null float64
from_this_person_to_poi      139 non-null float64
poi                          139 non-null bool
director_fees                139 non-null float64
deferred_income              139 non-null float64
long_term_incentive          139 non-null float64
from_poi_to_this_person      139 non-null float64
dtypes: bool(1), float64(19)
memory usage: 21.9+ KB

In total I have:

  • removed 7 examples,
  • removed 1 feature,
  • corrected inconsistent data,
  • filled in missing financial data with a zero value,
  • imputed missing email data using the median value from each class.

Let’s write this cleaned dataset to disk to be used for the later evaluation.

# Save the cleaned dictionary containing the dataset.
with open('final_prject_dataset_cleaned.pkl', 'wb') as data_file:
    pickle.dump(enron_data.to_dict(orient='index'), data_file, protocol=pickle.HIGHEST_PROTOCOL)

With the dataset cleaned up it feels about the right time to investigate how machine learning can help identify persons of interest, starting with how the results of the investigation are to be measured, to ensure consistent and fair comparison between alternative approaches. This will be the focus of the next post on theis topic.