In 2000, Enron Corporation was an American energy, commodities, and services company based in Houston, Texas. It was founded in 1985 as a merger between Houston Natural Gas and InterNorth, both relatively small regional companies. Before its bankruptcy on December 2, 2001, Enron employed approximately 20,000 staff and was a major electricity, natural gas, communications and pulp and paper company, with claimed revenues of nearly $101 billion during 2000. Fortune named Enron “America’s Most Innovative Company” for six consecutive years.
At the end of 2001, it was revealed that Enron’s reported financial condition was sustained by institutionalised, systematic, and creatively planned accounting fraud, known since as the Enron scandal. Enron has since become a well-known example of wilful corporate fraud and corruption. The scandal also brought into question the accounting practices and activities of many corporations in the United States and was a factor in the enactment of the Sarbanes–Oxley Act of 2002. The scandal also affected the greater business world by causing the dissolution of the Arthur Andersen accounting firm, which had been Enron’s main auditor for years.
The following series of articles detail the final project of the Udacity machine learning course which I completed the end of 2018. The objective: to implement and optimise a predictive model using the financial and email data provided by the Enron dataset, with the capacity to identify persons on interest (POIs). Specifically the project intends to demonstrate working with a real world dataset, performing data cleansing tasks and preparing the data; using feature selection and engineering techniques; possible dimensionality reduction; supervised classification and hyperparameter tuning; and validation.
This series of posts is a write up of a jupyter notebook used to conduce this experiment. The original notebook can be found on GitHub at #tomasbasham/ud120-projects.
To help guide the implementation of this project and address some key points we’ll attempt to follow the rubric provided by the Udacity team. Moreover we’ll be using some typical statistical and visualisation tools during this project. Since these are under very active development it may be necessary to use the same versions detailed in this post if wanting to replicate the results.
from __future__ import print_function
# scipy
import scipy
print('scipy: %s' % scipy.__version__)
# scikit-learn
import sklearn
print('sklearn: %s' % sklearn.__version__)
# numpy
import numpy
print('numpy: %s' % numpy.__version__)
# matplotlib
import matplotlib
print('matplotlib: %s' % matplotlib.__version__)
# pandas
import pandas
print('pandas: %s' % pandas.__version__)
scipy: 1.0.0
sklearn: 0.19.1
numpy: 1.14.0
matplotlib: 2.1.2
pandas: 0.22.0
Data Structure
The Enron dataset is a collection of key/value pairs associating Enron employees (the keys) with a collection of labelled features (the values). The labelled features for each employee are represented by a dictionary and include the following data:
- Financial features (all units in US dollars):
bonusdeferral_paymentsdeferred_incomedirector_feesexercised_stock_optionsexpensesloan_advanceslong_term_incentiveotherrestricted_stockrestricted_stock_deferredsalarytotal_paymentstotal_stock_value
- Email features (most units are integers with exception to
email_addresswhich is a string)email_addressfrom_messagesfrom_poi_to_this_personfrom_this_person_to_poishared_receipt_with_poito_messages
- Labels (boolean, represented as an integer)
poi
Data Exploration
This first post examines the data that we have to work with. It is very often the case that a dataset will contain errors, and it is imperative that these errors are resolved before making any further analysis of the data at hand. If the errors are not resolved then the final predictive model may not generalise well to any new data it receives.
When examining the data we are trying to answer the following:
- Are there any errors in the data?
- Are there any outliers in the data?
- Are there any missing values in the data?
- Does the data represent a realistic range of values?
Lets start by reading the data from the provided pkl file. This extension
expresses that the data is stored in a Pickle file, used to serialise and
deserialise Python data objects into a sequence of raw bytes that may be more
efficiently stored on disk.
import pickle
# Load the dictionary containing the dataset.
with open('final_project_dataset.pkl', 'r') as data_file:
data_dict = pickle.load(data_file)
# Sanity check
assert type(data_dict) is dict
By the lack of error we may conclude we have a dictionary. Dictionaries are a primitive data structure in python; but whilst they provide a simple interface through which data may be manipulated they are not flexible enough to perform many statistical operations. Instead this data can be loaded into a Pandas DataFrame: A two-dimensional size-mutable, tabular data structure with labelled axes (rows and columns).
import pandas as pd
# Use 2 decimal place output instead of scientific notation. It is far easier to read and compare.
pd.set_option('display.float_format', lambda x: '%.2f' % x)
# Transpose the data so the names are used as indices.
enron_data = pd.DataFrame.from_dict(data_dict, orient='index')
enron_data.head()
| salary | to_messages | deferral_payments | total_payments | exercised_stock_options | bonus | restricted_stock | shared_receipt_with_poi | restricted_stock_deferred | total_stock_value | … | loan_advances | from_messages | other | from_this_person_to_poi | poi | director_fees | deferred_income | long_term_incentive | email_address | from_poi_to_this_person | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ALLEN PHILLIP K | 201955 | 2902 | 2869717 | 4484442 | 1729541 | 4175000 | 126027 | 1407 | -126027 | 1729541 | … | NaN | 2195 | 152 | 65 | False | NaN | -3081055 | 304805 | [email protected] | 47 |
| BADUM JAMES P | NaN | NaN | 178980 | 182466 | 257817 | NaN | NaN | NaN | NaN | 257817 | … | NaN | NaN | NaN | NaN | False | NaN | NaN | NaN | NaN | NaN |
| BANNANTINE JAMES M | 477 | 566 | NaN | 916197 | 4046157 | NaN | 1757552 | 465 | -560222 | 5243487 | … | NaN | 29 | 864523 | 0 | False | NaN | -5104 | NaN | [email protected] | 39 |
| BAXTER JOHN C | 267102 | NaN | 1295738 | 5634343 | 6680544 | 1200000 | 3942714 | NaN | NaN | 10623258 | … | NaN | NaN | 2660303 | NaN | False | NaN | -1386055 | 1586055 | NaN | NaN |
| BAY FRANKLIN R | 239671 | NaN | 260455 | 827696 | NaN | 400000 | 145796 | NaN | -82782 | 63014 | … | NaN | NaN | 69 | NaN | False | NaN | -201641 | NaN | [email protected] | NaN |
5 rows × 21 columns
We’re in luck! The data could be read into a Pandas DataFrame. This will make it far easier to explore in the following sections and allows for any changes made to be re-exported as a simple dictionary.
The indexes are the names of Enron employees. There are several familiar names in the expanded list closely linked to the Enron scandal, including Kennth Lay, Jeffry Skilling, Andrew Fastow, and Cliff Baxter. There are also many lesser known names in the list that are hopefully equally useful in describing the patterns we’ll later discover in the data.
Additionally there is an entry for “The Travel Agency in the Park” that is noted in the footnotes of the original FindLaw document and states that payments were made to this company by Enron and that this business was co-owned by the sister of Enron’s former Chairman. From an accounting perspective this information is very relevant but I suspect this will impede our predictive model. Therefore this entry is a good candidate to be omitted in later sections.
The columns describe the features available in the dataset. They are expressive enough that we may understand what each value represents but unfortunately they do not indicate their unit of measurement. Luckily we’re told how each value is represented in the original FindLaw document so this will not be an issue.
One of the first things of note is there seems to be a lot of missing data
scattered throughout the dataset. This is indicated by the NaN string. Python
will not be able to interpret this as missing data and may present issues when
plotting features and performing any data manipulation. Instead we can tell
Pandas to replace missing values by specifying our missing value marker.
import numpy as np
# NumPy uses the IEEE Standard for Binary Floating-Point for Arithmetic (IEEE 754).
# This means that Not a Number is not equivalent to infinity.
enron_data = enron_data.replace('NaN', np.nan)
type(enron_data.loc['ALLEN PHILLIP K'].director_fees)
numpy.float64
Great! Now that Pandas is aware of missing data and we have some sensible data to work with lets print out a few statistics.
# Print information about the data frame.
enron_data.info()
print()
# Justify the output of statistics.
print_stat = lambda t, stat: print('{}:'.format(t).ljust(28), stat)
# Print a few statistics not included in the above.
print_stat('Number of data points', enron_data.shape[0] * enron_data.shape[1])
print_stat('Number of missing values', enron_data.isna().sum().sum())
print_stat('Number of POIs', len(enron_data[enron_data['poi'] == True]))
<class 'pandas.core.frame.DataFrame'>
Index: 146 entries, ALLEN PHILLIP K to YEAP SOON
Data columns (total 21 columns):
salary 95 non-null float64
to_messages 86 non-null float64
deferral_payments 39 non-null float64
total_payments 125 non-null float64
exercised_stock_options 102 non-null float64
bonus 82 non-null float64
restricted_stock 110 non-null float64
shared_receipt_with_poi 86 non-null float64
restricted_stock_deferred 18 non-null float64
total_stock_value 126 non-null float64
expenses 95 non-null float64
loan_advances 4 non-null float64
from_messages 86 non-null float64
other 93 non-null float64
from_this_person_to_poi 86 non-null float64
poi 146 non-null bool
director_fees 17 non-null float64
deferred_income 49 non-null float64
long_term_incentive 66 non-null float64
email_address 111 non-null object
from_poi_to_this_person 86 non-null float64
dtypes: bool(1), float64(19), object(1)
memory usage: 29.1+ KB
Number of data points: 3066
Number of missing values: 1358
Number of POIs: 18
It looks like we have lots of integers, and two single columns of strings and
booleans. The boolean column, poi indicates whether the person is a “Person of
Interest”. This is the column we’ll be trying to predict using the other data.
Additionally the above shows the following:
- There are 146 examples each containing 21 values,
- Of the 21 values there are 19 numerical features, 1 text feature (“email”) and 1 binary classification (“poi”),
- There are 3066 data points, of which 1358 are missing,
- 18 of the 146 examples are known persons of interest.
A significant proportion (44.3%) of the data is missing. From the table we can
see this is particularly prevalent in the features deferral_payments,
deferred_income, director_fees, loan_advances and
restricted_stock_deferred. At this time I will not remove any of these
features just in case they are still useful when attempting to derive patterns
in the data.
In addition to the summary statistics we can print some aggregate statistics over each features in the dataset. This allows us to see some typical statistical values that may unearth some useful information.
enron_data.describe()
| salary | to_messages | deferral_payments | total_payments | exercised_stock_options | bonus | restricted_stock | shared_receipt_with_poi | restricted_stock_deferred | total_stock_value | expenses | loan_advances | from_messages | other | from_this_person_to_poi | director_fees | deferred_income | long_term_incentive | from_poi_to_this_person | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| count | 95.00 | 86.00 | 39.00 | 125.00 | 102.00 | 82.00 | 110.00 | 86.00 | 18.00 | 126.00 | 95.00 | 4.00 | 86.00 | 93.00 | 86.00 | 17.00 | 49.00 | 66.00 | 86.00 |
| mean | 562194.29 | 2073.86 | 1642674.15 | 5081526.49 | 5987053.77 | 2374234.61 | 2321741.14 | 1176.47 | 166410.56 | 6773957.45 | 108728.92 | 41962500.00 | 608.79 | 919064.97 | 41.23 | 166804.88 | -1140475.14 | 1470361.45 | 64.90 |
| std | 2716369.15 | 2582.70 | 5161929.97 | 29061716.40 | 31062006.57 | 10713327.97 | 12518278.18 | 1178.32 | 4201494.31 | 38957772.73 | 533534.81 | 47083208.70 | 1841.03 | 4589252.91 | 100.07 | 319891.41 | 4025406.38 | 5942759.32 | 86.98 |
| min | 477.00 | 57.00 | -102500.00 | 148.00 | 3285.00 | 70000.00 | -2604490.00 | 2.00 | -7576788.00 | -44093.00 | 148.00 | 400000.00 | 12.00 | 2.00 | 0.00 | 3285.00 | -27992891.00 | 69223.00 | 0.00 |
| 25% | 211816.00 | 541.25 | 81573.00 | 394475.00 | 527886.25 | 431250.00 | 254018.00 | 249.75 | -389621.75 | 494510.25 | 22614.00 | 1600000.00 | 22.75 | 1215.00 | 1.00 | 98784.00 | -694862.00 | 281250.00 | 10.00 |
| 50% | 259996.00 | 1211.00 | 227449.00 | 1101393.00 | 1310813.50 | 769375.00 | 451740.00 | 740.50 | -146975.00 | 1102872.50 | 46950.00 | 41762500.00 | 41.00 | 52382.00 | 8.00 | 108579.00 | -159792.00 | 442035.00 | 35.00 |
| 75% | 312117.00 | 2634.75 | 1002671.50 | 2093263.00 | 2547724.00 | 1200000.00 | 1002369.75 | 1888.25 | -75009.75 | 2949846.75 | 79952.50 | 82125000.00 | 145.50 | 362096.00 | 24.75 | 113784.00 | -38346.00 | 938672.00 | 72.25 |
| max | 26704229.00 | 15149.00 | 32083396.00 | 309886585.00 | 311764000.00 | 97343619.00 | 130322299.00 | 5521.00 | 15456290.00 | 434509511.00 | 5235198.00 | 83925000.00 | 14368.00 | 42667589.00 | 609.00 | 1398517.00 | -833.00 | 48521928.00 | 528.00 |
Without a statistical background, further analysis is not particularly easy unless we know that our data should fall in a particular range. It’s usually better to visualize the data in some way. Visualization makes outliers and errors immediately stand out, whereas they might go unnoticed in a large table of numbers. I’ll investigate this further in the next post that will be on the topic of data cleansing.