One of the things I like most about IPython notebooks is the amazing accesability of documentation. If you hit the tab key while typing, a great autocompletion window comes up. If you pause once the word you are typing is complete, you receive documentation on the finction in a popup window. In newer versions it seems the same functionality is acheived by hitting shift then tab instead of tab. Hitting the tab key gives even more documentation in the popup window, and hitting it once more will give all the documentation on the word you have typed in a window below the current input cell. Another great feature is the re-evaluatable cell, which is great for making mistakes and learning from them. It allows you to type something, view the output immediately, and rapidly and iteratively change the input until you finally get the desired output (if you make tons of small mistakes like me). The built in documentation features, in combination with re-evaluatable cells are extremely helpful when using a new tool or library (like Pandas).
%matplotlib inline
import pandas as pd
Code written using the Pandas Library, when compared to Python Lists or similar native container data types, is much more compact and often times can avoid loops in places where Python can't (at least in pretty or compact ways). Some of the most useful methods include reading csv files readcsv() and excel files, plotting .plot(), statistics .median(), bar graphs, slicing by columns, conditional returning of data (like SQL query), adding new columns, groupby and aggregate (like SQL versions).
Pandas is a Python library for data exploration, manipulation, and analysis. Primarily it includes data tables and time series data structures, along with methods for manipulating them. Dataframes are the central objects in the library, which are similar to excel spreadsheets. Pandas was inspired largely by a statistical programming language called R, and maintains many of the basic data structures and methods included in R, such as data frames. The creators of Pandas are Financial Data Analysts. One of the reasons some prefer Pandas to R is the fact you're coding in Python. Another reason is that it interfaces very well with other great python packages (matplotlib, numpy, etc...); in many cases you can just drop a Pandas dataframe into your favorite matplotlib or numpy functions and get the desired behavior, which is great! It's alwo quite fast on many tasks, as it uses numpy for many basic dataframe methods. Dataframe's ability to handle missing data makes it an invaluable tool when dealing with any kind of data.
Here are some resources for getting started:
Using pip or anaconda to install pandas is the safest/best way to get started:
http://pandas.pydata.org/pandas-docs/stable/install.html
Pandas Website:
Quick Pandas tutorial video:
Julia Evans PyCon Pandas Tutorial for Data Analysis:
https://www.youtube.com/watch?v=rEalbu8UGeo
Tons of IPython Notebooks:
http://nbviewer.ipython.org/github/jvns/talks/blob/master/pyconca2013/pistes-cyclables.ipynb
It is very easy to read data (CSV, Excel, SQL, JSON, etc...) into a pandas dataframe and get started exploring. I'm going to do this for this City of Boston employee earnings dataset: https://data.cityofboston.gov/City-Services/Closed-Pothole-Cases/wivc-syw7
df = pd.read_csv("Employee_Earnings_Report_2014.csv")
df
NAME | TITLE | DEPARTMENT NAME | REGULAR | RETRO | OTHER | OVERTIME | INJURED | DETAIL | QUINN | TOTAL EARNINGS | ZIP | |
---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | Adario,Anthony J | Supvising Claims Agent (Asd) | ASD Human Resources | $98538.32 | $0.00 | $1842.87 | $0.00 | $0.00 | $0.00 | $0.00 | $100381.19 | 02132 |
1 | Baker,Katherine E. | Nurse Case Manager | ASD Human Resources | $83942.21 | $0.00 | $225.00 | $0.00 | $0.00 | $0.00 | $0.00 | $84167.21 | 02136 |
2 | Braga,Ann Hess | HumanResourcesComplianceOffcr | ASD Human Resources | $98538.34 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $98538.34 | 02126 |
3 | Calobrisi,Jane Ellen | Personnel Analyst | ASD Human Resources | $70073.81 | $0.00 | $225.00 | $0.00 | $0.00 | $0.00 | $0.00 | $70298.81 | 02090 |
4 | Castillo-Cook,Wendolyn M. | Alcholism Coord I | ASD Human Resources | $79253.11 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $79253.11 | 02121 |
5 | Curran,Liam Patrick | Asst Corp Counsel III | ASD Human Resources | $69266.69 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $69266.69 | 02122 |
6 | De Jesus,Winnie Yam | Affirm Action Monitor | ASD Human Resources | $42061.21 | $0.00 | $487.21 | $0.00 | $0.00 | $0.00 | $0.00 | $42548.42 | 02458 |
7 | Donnelly,Jennifer P. | Prin Clerk | ASD Human Resources | $44080.61 | $0.00 | $300.00 | $0.00 | $0.00 | $0.00 | $0.00 | $44380.61 | 02129 |
8 | Donovan,Carol Anne | Prin Admin Analyst | ASD Human Resources | $73547.25 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $73547.25 | 02127 |
9 | Donovan,John J | Exec Asst (EAP/OHR) | ASD Human Resources | $15019.39 | $0.00 | $10250.05 | $0.00 | $0.00 | $0.00 | $0.00 | $25269.44 | 02132 |
10 | Ettingoff,Kimberly | Intern I | ASD Human Resources | $4960.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $4960.00 | 02144 |
11 | Figueroa Ortiz,Irene | Intern I | ASD Human Resources | $4960.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $4960.00 | 02138 |
12 | Firicano,Frank P. | Sr Admin Asst (OHR) | ASD Human Resources | $91743.12 | $0.00 | $225.00 | $0.00 | $0.00 | $0.00 | $0.00 | $91968.12 | 02189 |
13 | Flemming,Teresia K | Head Account Clerk | ASD Human Resources | $47455.39 | $0.00 | $856.78 | $0.00 | $0.00 | $0.00 | $0.00 | $48312.17 | 02128 |
14 | Gee,Xing-Bing | Data Proc Sys Analyst | ASD Human Resources | $76393.46 | $0.00 | $3082.44 | $0.00 | $0.00 | $0.00 | $0.00 | $79475.90 | 02130 |
15 | Grant,Michael Scott | Sr Data Proc Sys Anl | ASD Human Resources | $106594.16 | $0.00 | $2620.06 | $0.00 | $0.00 | $0.00 | $0.00 | $109214.22 | 02136 |
16 | Green,Allison Camille | Intern I | ASD Human Resources | $4960.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $4960.00 | 02143 |
17 | Green,Kathleen M | Health Insurance Coordinator | ASD Human Resources | $91938.06 | $0.00 | $4765.38 | $0.00 | $0.00 | $0.00 | $0.00 | $96703.44 | 02129 |
18 | Guante,Celiana | Admin Asst | ASD Human Resources | $23934.49 | $0.00 | $998.34 | $0.00 | $0.00 | $0.00 | $0.00 | $24932.83 | 02121 |
19 | Hanson,Sheila A | Head Clerk | ASD Human Resources | $42311.33 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $42311.33 | 02136 |
20 | Jackson-Rodgers,Jacqueline N. | Sr Admin Asst (OHR) | ASD Human Resources | $91744.59 | $0.00 | $627.00 | $0.00 | $0.00 | $0.00 | $0.00 | $92371.59 | 02126 |
21 | Kessler,William | Asst Dir (Human Resources) | ASD Human Resources | $117411.06 | $0.00 | $4391.66 | $0.00 | $0.00 | $0.00 | $0.00 | $121802.72 | 02132 |
22 | Killarney,Bernard | Director of Operations | ASD Human Resources | $9608.45 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $9608.45 | 02136 |
23 | Lanchester,Diane | Office Manager | ASD Human Resources | $65169.15 | $0.00 | $787.25 | $0.00 | $0.00 | $0.00 | $0.00 | $65956.40 | 02131 |
24 | Lee,Winnie | Sr Admin Asst | ASD Human Resources | $76393.49 | $0.00 | $1653.72 | $0.00 | $0.00 | $0.00 | $0.00 | $78047.21 | 02170 |
25 | Leonard,Vivian | Supv of Personnel | ASD Human Resources | $115982.88 | $0.00 | $2230.44 | $0.00 | $0.00 | $0.00 | $0.00 | $118213.32 | 02136 |
26 | Marshall,Leah A. | Human Resources Representative | ASD Human Resources | $46423.61 | $0.00 | $300.00 | $0.00 | $0.00 | $0.00 | $0.00 | $46723.61 | 02126 |
27 | McNeil,Jean M | Supv-Management Svcs | ASD Human Resources | $71139.91 | $0.00 | $865.37 | $0.00 | $0.00 | $0.00 | $0.00 | $72005.28 | 02136 |
28 | Melle,Laura | Intern I | ASD Human Resources | $5580.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $5580.00 | 02139 |
29 | Morant,Darius | Prin Clerk | ASD Human Resources | $5710.80 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $5710.80 | 02119 |
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
22203 | Wilson,Jeffrey | Fire Fighter-AdvanceTechnician | Boston Fire Department | $88662.18 | $12890.60 | $550.00 | $12730.82 | $6633.81 | $128.00 | $0.00 | $121595.41 | 02136 |
22204 | Wilson,Keith Michael | Fire Fighter | Boston Fire Department | $53235.90 | $2999.38 | $550.00 | $5352.20 | $0.00 | $0.00 | $0.00 | $62137.48 | 02132 |
22205 | Wilson,Marlon | Fire Fighter | Boston Fire Department | $88461.02 | $13602.82 | $550.00 | $13775.17 | $4855.79 | $8326.75 | $0.00 | $129571.55 | 02136 |
22206 | Wilson,Ryan Jon | Fire Fighter | Boston Fire Department | $21364.38 | $0.00 | $275.00 | $0.00 | $0.00 | $0.00 | $0.00 | $21639.38 | 02132 |
22207 | Wilson,Sean E | Fire Fighter | Boston Fire Department | $78538.07 | $7593.22 | $550.00 | $6989.46 | $49.23 | $8462.25 | $0.00 | $102182.23 | 02136 |
22208 | Wilson,Wayne A | Fire Fighter | Boston Fire Department | $96178.88 | $10010.18 | $2157.24 | $24629.94 | $4337.00 | $561.00 | $0.00 | $137874.24 | 02026 |
22209 | Wirtz,Eric M | Fire Fighter | Boston Fire Department | $71911.66 | $7198.69 | $550.00 | $3892.15 | $7678.90 | $0.00 | $0.00 | $91231.40 | 02122 |
22210 | Wong,Connie S | Dep Comm-Labor & Legal | Boston Fire Department | $28750.02 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $28750.02 | 02111 |
22211 | Wong,Robert | Fire Fighter | Boston Fire Department | $15670.19 | $8210.82 | $11065.36 | $2127.66 | $84109.32 | $0.00 | $0.00 | $121183.35 | 02118 |
22212 | Wong,William | Asst Prin accountant | Boston Fire Department | $39974.95 | $0.00 | $747.72 | $38029.14 | $0.00 | $0.00 | $0.00 | $78751.81 | 02132 |
22213 | Woodbury,Dennis A | Fire Fighter | Boston Fire Department | $96764.06 | $14814.32 | $2157.24 | $21890.55 | $718.99 | $2401.50 | $0.00 | $138746.66 | 02124 |
22214 | Woods,Patrick J | Fire Lieutenant | Boston Fire Department | $110597.16 | $16537.28 | $2451.66 | $18038.47 | $0.00 | $0.00 | $0.00 | $147624.57 | 02129 |
22215 | Wornum,Dwight H | Fire Fighter | Boston Fire Department | $149.99 | $6261.44 | $3063.63 | $0.00 | $100310.76 | $0.00 | $0.00 | $109785.82 | 02136 |
22216 | Wornum,John W | FF (FPDInspLevl1Certification) | Boston Fire Department | $98109.51 | $13524.14 | $6243.64 | $4866.68 | $0.00 | $0.00 | $0.00 | $122743.97 | 02121 |
22217 | Wornum,Phillip B. | FF LEP Title 3 Inspector - ADR | Boston Fire Department | $1195.32 | $1188.70 | $85430.84 | $0.00 | $103564.81 | $0.00 | $0.00 | $191379.67 | 02186 |
22218 | Worrell,David E | Fire Fighter | Boston Fire Department | $0.00 | $1013.60 | $0.00 | $0.00 | $1425.72 | $0.00 | $0.00 | $2439.32 | 02121 |
22219 | Wosny,Joseph Andrew | Fire Fighter | Boston Fire Department | $21364.38 | $0.00 | $275.00 | $0.00 | $0.00 | $0.00 | $0.00 | $21639.38 | 02132 |
22220 | Wright,Alexander E | Fire Fighter-Technician | Boston Fire Department | $94499.10 | $14911.41 | $2151.24 | $19095.59 | $0.00 | $5132.75 | $0.00 | $135790.09 | 02072 |
22221 | Wright,Ballin L | Fire Fighter-AdvanceTechnician | Boston Fire Department | $84536.18 | $14204.17 | $3721.05 | $18540.90 | $13082.71 | $1488.00 | $0.00 | $135573.01 | 02125 |
22222 | Wright,Christopher D | Fire Fighter-Technician | Boston Fire Department | $93427.63 | $14077.81 | $550.00 | $17740.79 | $448.65 | $0.00 | $0.00 | $126244.88 | 02132 |
22223 | Wright,James | Fire Fighter | Boston Fire Department | $0.00 | $6499.01 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $6499.01 | 02368 |
22224 | Wuschke,Barbara | Fire Alarm Operator | Boston Fire Department | $95154.82 | $13596.87 | $550.00 | $9325.59 | $0.00 | $0.00 | $0.00 | $118627.28 | 02124 |
22225 | Wyman,George L | District Fire Chief Tech-ADR | Boston Fire Department | $0.00 | $3699.29 | $0.00 | $0.00 | $2845.81 | $0.00 | $0.00 | $6545.10 | 02360 |
22226 | Wyzard,Keith T | Fire Fighter | Boston Fire Department | $94460.92 | $14055.44 | $1466.23 | $11700.05 | $0.00 | $0.00 | $0.00 | $121682.64 | 02368 |
22227 | Yanosick,Justin Timothy | Fire Fighter | Boston Fire Department | $92646.90 | $14250.84 | $550.00 | $28269.86 | $0.00 | $0.00 | $0.00 | $135717.60 | 02132 |
22228 | Yee,Jacob | Fire Fighter | Boston Fire Department | $80073.63 | $13529.71 | $550.00 | $15560.09 | $13353.47 | $7245.75 | $0.00 | $130312.65 | 02136 |
22229 | Yetman,Patrick | Fire Fighter | Boston Fire Department | $69886.18 | $5191.64 | $550.00 | $8227.25 | $0.00 | $0.00 | $0.00 | $83855.07 | 02127 |
22230 | Young,Allan M | Fire Fighter | Boston Fire Department | $97234.17 | $14091.91 | $3050.00 | $6193.76 | $0.00 | $4464.00 | $0.00 | $125033.84 | 02136 |
22231 | Young,Michael L | Fire Fighter | Boston Fire Department | $95694.84 | $13973.38 | $550.00 | $10678.12 | $0.00 | $132.00 | $0.00 | $121028.34 | 02121 |
22232 | Zagami,Anthony J | Fire Alarm Operator | Boston Fire Department | $0.00 | $5494.93 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $5494.93 | 02124 |
22233 rows × 12 columns
We can now explore the dataset quickly and easily using the countless built in methods for dataframes. For example, which Boston employee makes the most money?
df.sort('TOTAL EARNINGS',ascending=0)
NAME | TITLE | DEPARTMENT NAME | REGULAR | RETRO | OTHER | OVERTIME | INJURED | DETAIL | QUINN | TOTAL EARNINGS | ZIP | |
---|---|---|---|---|---|---|---|---|---|---|---|---|
16758 | Linnenbank,Kristen Laura | Director (D) (non-ac) | UP Academy Dorchester | $99999.90 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99999.90 | 01970 |
13686 | Iatrou,Kyriaki | Teacher | Curley K-8 | $99393.58 | $0.00 | $600.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99993.58 | 02167 |
8769 | Rice,Juli | Teacher | BPS Manning Elementary | $99992.65 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99992.65 | 02130 |
6432 | Cartagena,Angel L. | Teacher | BPS East Boston High | $95554.68 | $0.00 | $4436.50 | $0.00 | $0.00 | $0.00 | $0.00 | $99991.18 | 02149 |
16246 | Gantt,Danielle | Coordinator 7 | Teaching & Learning | $99989.77 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99989.77 | 02121 |
8179 | Lonergan,Mark J | Teacher | BPS Human Resource Team | $99989.33 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99989.33 | 02131 |
9637 | Pinkney-Chase,LaTrelle Natosha | Teacher | BPS MPH\Health Academy | $94444.78 | $0.00 | $5537.60 | $0.00 | $0.00 | $0.00 | $0.00 | $99982.38 | 02132 |
5938 | O'Connell,Bebhinn | Teacher | BPS Cl#10 Lee Academy | $98979.47 | $0.00 | $1000.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99979.47 | 02062 |
13786 | McIsaac,Stephen J. | Teacher | Dorchester Academy | $98778.83 | $0.00 | $1200.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99978.83 | 02155 |
13480 | Torres-Vivas,Odalis | Teacher | BPS Winship Elementary | $97960.72 | $0.00 | $2015.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99975.72 | 02135 |
15797 | Tippett,Jodi L | Nurse | Lyon K-8 | $99971.35 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99971.35 | 02421 |
9063 | Holland,Mary F | Teacher | BPS McCormack Middle | $99952.22 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99952.22 | 02072 |
5939 | Purple,Keri L | Specialist (BTU) | BPS Cl#10 Lee Academy | $99429.05 | $0.00 | $500.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99929.05 | 02131 |
6249 | Ward-Wilson,Stephanie A | School Psychologist(1) | BPS Counseling Service | $98778.83 | $0.00 | $1150.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99928.83 | 02043 |
17515 | Burns,Nolan R | Police Officer | Boston Police Department | $69251.35 | $2936.57 | $807.33 | $12722.97 | $0.00 | $14207.00 | $0.00 | $99925.22 | 02129 |
16865 | Peddie,Jabari | Principal Elementary | UP Academy Holland | $99924.65 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99924.65 | 02126 |
6644 | Galvin,Janice M. | Teacher | BPS Ellis Elementary | $99322.59 | $0.00 | $600.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99922.59 | 02132 |
12560 | Gomes,Idalina | Teacher | BPS Sumner Elementary | $99405.43 | $0.00 | $500.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99905.43 | 02189 |
10478 | Donovan,Amy M | Teacher | BPS Rogers Middle | $98901.87 | $0.00 | $1000.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99901.87 | 02184 |
4204 | McCabe,James B | Workers Comp Job Classificatn | Workers Compensation Service | $0.00 | $0.00 | $0.00 | $0.00 | $999.31 | $0.00 | $0.00 | $999.31 | 02132 |
4775 | Hart,Allyson N. | Director (Basas 10B) | BPS Blackstone Elementary | $99898.99 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99898.99 | 02062 |
10948 | Couture,Aaron | SpEd Support Svcs Supervisor | BPS Special Education | $99898.99 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99898.99 | 02081 |
7506 | Yee,James | Sen Supv Schl Bldg Electrical | BPS Facility Mgmt/A & R | $83106.38 | $663.16 | $3842.00 | $12280.31 | $0.00 | $0.00 | $0.00 | $99891.85 | 02467 |
4651 | Gangemi,Josephine C. | Teacher | BPS Bates Elementary | $99886.57 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99886.57 | 02131 |
13448 | Feuerstein,Tia Marie | Teacher | BPS Winship Elementary | $99886.57 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99886.57 | 02124 |
21153 | Hardy,Brett R. | Fire Fighter-Technician | Boston Fire Department | $82685.69 | $8492.91 | $550.00 | $8128.65 | $27.31 | $0.00 | $0.00 | $99884.56 | 02122 |
16087 | Chang,Sarah | Teacher | Quincy Upper School | $96884.16 | $0.00 | $3000.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99884.16 | 02118 |
5272 | Amoroso,Rita A | Nurse | BPS Bradley Elementary | $99883.34 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99883.34 | 02151 |
15985 | Adam,S.Farhan | Teacher | Newcomers Academy | $98778.83 | $0.00 | $1102.80 | $0.00 | $0.00 | $0.00 | $0.00 | $99881.63 | 02124 |
15315 | Gallivan,Ellen E | Teacher | Jackson/Mann K-8 | $99378.91 | $0.00 | $500.00 | $0.00 | $0.00 | $0.00 | $0.00 | $99878.91 | 01746 |
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
4855 | Gonzalez,Ramiro G | Teacher | BPS Boston Arts Academy | $70885.98 | $0.00 | $29248.02 | $0.00 | $0.00 | $0.00 | $0.00 | $100134.00 | 02908 |
21347 | Kelly,Michael Joseph | Fire Fighter | Boston Fire Department | $65265.15 | $3402.52 | $550.00 | $9819.90 | $765.59 | $20330.75 | $0.00 | $100133.91 | 02124 |
6091 | Petruccelli,Carol A | Teacher | BPS Condon Elementary | $99378.91 | $0.00 | $755.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100133.91 | 02115 |
4648 | Fitzgerald,Christine L. | Teacher | BPS Bates Elementary | $100128.45 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100128.45 | 02760 |
16089 | Chen,Yuyu | Teacher | Quincy Upper School | $87374.84 | $0.00 | $12750.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100124.84 | 02155 |
18397 | Goodband,Clifford | Prin Admin Asst | Boston Police Department | $73768.58 | $0.00 | $1520.58 | $24831.84 | $0.00 | $0.00 | $0.00 | $100121.00 | 02132 |
15474 | Moody,Kathyann R | Teacher | Kennedy, JF Elementary | $97716.46 | $0.00 | $2401.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100117.46 | 02126 |
3428 | Pham,Tan V | Sr Civil Engineer | Public Works Department | $82568.29 | $0.00 | $4124.83 | $13412.71 | $0.00 | $0.00 | $0.00 | $100105.83 | 02124 |
4660 | Kelley,Noreen M | Teacher | BPS Bates Elementary | $100101.93 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100101.93 | 02124 |
18941 | Lewis,Janet P. | Police Officer | Boston Police Department | $80225.99 | $10171.54 | $5164.95 | $4348.43 | $189.00 | $0.00 | $0.00 | $100099.91 | 02131 |
16910 | Capone-Cameron,Teresa | Teacher | Warren/Prescott K-8 | $99348.35 | $0.00 | $750.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100098.35 | 02155 |
8283 | Galac,Mitchelle A. | Teacher | BPS Irving Middle | $99833.53 | $0.00 | $250.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100083.53 | 02170 |
6668 | Murad,Lynda J. | Teacher | BPS Ellis Elementary | $99833.53 | $0.00 | $250.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100083.53 | 02131 |
2820 | Towle,David | Summer Youth Worker | Parks Department | $10008.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $10008.00 | 02131 |
4358 | Jar-Curran,Lena | Teacher | Baldwin ELC | $98078.91 | $0.00 | $2000.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100078.91 | 02141 |
21668 | Moran,Patrick | Fire Fighter-Technician | Boston Fire Department | $82668.55 | $8595.18 | $550.00 | $8255.26 | $0.00 | $0.00 | $0.00 | $100068.99 | 02132 |
1899 | Alvarez,Indira C | Exec Asst (ISD) | Inspectional Services Dept | $99414.58 | $0.00 | $0.00 | $641.96 | $0.00 | $0.00 | $0.00 | $100056.54 | 02121 |
13768 | Fitzpatrick,Marcia E | SpEd Support Svcs Coordinator | Dorchester Academy | $93744.08 | $0.00 | $6310.80 | $0.00 | $0.00 | $0.00 | $0.00 | $100054.88 | 02122 |
8167 | Kelley,Daniel W | Teacher | BPS Human Resource Team | $100052.06 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100052.06 | 02122 |
16987 | Erilus,Yves F. | Teacher | West Roxbury Academy | $100048.89 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100048.89 | 02760 |
5134 | Freeman,Judi H | Teacher | BPS Boston Latin | $100048.89 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100048.89 | 02118 |
13942 | Gothie,Megan M | Teacher | Edison K-8 | $100048.89 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100048.89 | 02494 |
13978 | O'Rourke,Kerry A | Teacher | Edison K-8 | $100044.60 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100044.60 | 02122 |
7841 | Jean,Maria T | Teacher | BPS Hennigan Elementary | $98593.76 | $0.00 | $1438.98 | $0.00 | $0.00 | $0.00 | $0.00 | $100032.74 | 02136 |
10523 | Smith,Maya S. | School Disciplinary Officer | BPS Rogers Middle | $96574.76 | $0.00 | $3450.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100024.76 | 02131 |
9987 | Johnson,Anita | Attendant | BPS Office Of Budget Mgmt | $7692.21 | $0.00 | $0.00 | $0.00 | $2310.00 | $0.00 | $0.00 | $10002.21 | 02126 |
8023 | Chace,Jennifer A | Teacher | BPS Horace Mann | $100012.37 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100012.37 | 02135 |
5600 | Scott,Tarek | Over-Time Stipend Other | BPS Business Service | $0.00 | $0.00 | $1000.00 | $0.00 | $0.00 | $0.00 | $0.00 | $1000.00 | 02119 |
14664 | Morin,Kristen Marie | Community Field Coordinator | Greenwood, E Leadership Acad | $0.00 | $0.00 | $100.00 | $0.00 | $0.00 | $0.00 | $0.00 | $100.00 | 02130 |
6762 | Larievy,Arthur F. | Program Director (Basas 9A) | BPS English High | $0.00 | $10.82 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $10.82 | 02118 |
22233 rows × 12 columns
Cleaning the data to allow for statistical analysis is also not too bad!
df['TOTAL EARNINGS'] = df['TOTAL EARNINGS'].replace('[\$]', '', regex=True).astype(float)
df
NAME | TITLE | DEPARTMENT NAME | REGULAR | RETRO | OTHER | OVERTIME | INJURED | DETAIL | QUINN | TOTAL EARNINGS | ZIP | |
---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | Adario,Anthony J | Supvising Claims Agent (Asd) | ASD Human Resources | $98538.32 | $0.00 | $1842.87 | $0.00 | $0.00 | $0.00 | $0.00 | 100381.19 | 02132 |
1 | Baker,Katherine E. | Nurse Case Manager | ASD Human Resources | $83942.21 | $0.00 | $225.00 | $0.00 | $0.00 | $0.00 | $0.00 | 84167.21 | 02136 |
2 | Braga,Ann Hess | HumanResourcesComplianceOffcr | ASD Human Resources | $98538.34 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | 98538.34 | 02126 |
3 | Calobrisi,Jane Ellen | Personnel Analyst | ASD Human Resources | $70073.81 | $0.00 | $225.00 | $0.00 | $0.00 | $0.00 | $0.00 | 70298.81 | 02090 |
4 | Castillo-Cook,Wendolyn M. | Alcholism Coord I | ASD Human Resources | $79253.11 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | 79253.11 | 02121 |
5 | Curran,Liam Patrick | Asst Corp Counsel III | ASD Human Resources | $69266.69 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | 69266.69 | 02122 |
6 | De Jesus,Winnie Yam | Affirm Action Monitor | ASD Human Resources | $42061.21 | $0.00 | $487.21 | $0.00 | $0.00 | $0.00 | $0.00 | 42548.42 | 02458 |
7 | Donnelly,Jennifer P. | Prin Clerk | ASD Human Resources | $44080.61 | $0.00 | $300.00 | $0.00 | $0.00 | $0.00 | $0.00 | 44380.61 | 02129 |
8 | Donovan,Carol Anne | Prin Admin Analyst | ASD Human Resources | $73547.25 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | 73547.25 | 02127 |
9 | Donovan,John J | Exec Asst (EAP/OHR) | ASD Human Resources | $15019.39 | $0.00 | $10250.05 | $0.00 | $0.00 | $0.00 | $0.00 | 25269.44 | 02132 |
10 | Ettingoff,Kimberly | Intern I | ASD Human Resources | $4960.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | 4960.00 | 02144 |
11 | Figueroa Ortiz,Irene | Intern I | ASD Human Resources | $4960.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | 4960.00 | 02138 |
12 | Firicano,Frank P. | Sr Admin Asst (OHR) | ASD Human Resources | $91743.12 | $0.00 | $225.00 | $0.00 | $0.00 | $0.00 | $0.00 | 91968.12 | 02189 |
13 | Flemming,Teresia K | Head Account Clerk | ASD Human Resources | $47455.39 | $0.00 | $856.78 | $0.00 | $0.00 | $0.00 | $0.00 | 48312.17 | 02128 |
14 | Gee,Xing-Bing | Data Proc Sys Analyst | ASD Human Resources | $76393.46 | $0.00 | $3082.44 | $0.00 | $0.00 | $0.00 | $0.00 | 79475.90 | 02130 |
15 | Grant,Michael Scott | Sr Data Proc Sys Anl | ASD Human Resources | $106594.16 | $0.00 | $2620.06 | $0.00 | $0.00 | $0.00 | $0.00 | 109214.22 | 02136 |
16 | Green,Allison Camille | Intern I | ASD Human Resources | $4960.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | 4960.00 | 02143 |
17 | Green,Kathleen M | Health Insurance Coordinator | ASD Human Resources | $91938.06 | $0.00 | $4765.38 | $0.00 | $0.00 | $0.00 | $0.00 | 96703.44 | 02129 |
18 | Guante,Celiana | Admin Asst | ASD Human Resources | $23934.49 | $0.00 | $998.34 | $0.00 | $0.00 | $0.00 | $0.00 | 24932.83 | 02121 |
19 | Hanson,Sheila A | Head Clerk | ASD Human Resources | $42311.33 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | 42311.33 | 02136 |
20 | Jackson-Rodgers,Jacqueline N. | Sr Admin Asst (OHR) | ASD Human Resources | $91744.59 | $0.00 | $627.00 | $0.00 | $0.00 | $0.00 | $0.00 | 92371.59 | 02126 |
21 | Kessler,William | Asst Dir (Human Resources) | ASD Human Resources | $117411.06 | $0.00 | $4391.66 | $0.00 | $0.00 | $0.00 | $0.00 | 121802.72 | 02132 |
22 | Killarney,Bernard | Director of Operations | ASD Human Resources | $9608.45 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | 9608.45 | 02136 |
23 | Lanchester,Diane | Office Manager | ASD Human Resources | $65169.15 | $0.00 | $787.25 | $0.00 | $0.00 | $0.00 | $0.00 | 65956.40 | 02131 |
24 | Lee,Winnie | Sr Admin Asst | ASD Human Resources | $76393.49 | $0.00 | $1653.72 | $0.00 | $0.00 | $0.00 | $0.00 | 78047.21 | 02170 |
25 | Leonard,Vivian | Supv of Personnel | ASD Human Resources | $115982.88 | $0.00 | $2230.44 | $0.00 | $0.00 | $0.00 | $0.00 | 118213.32 | 02136 |
26 | Marshall,Leah A. | Human Resources Representative | ASD Human Resources | $46423.61 | $0.00 | $300.00 | $0.00 | $0.00 | $0.00 | $0.00 | 46723.61 | 02126 |
27 | McNeil,Jean M | Supv-Management Svcs | ASD Human Resources | $71139.91 | $0.00 | $865.37 | $0.00 | $0.00 | $0.00 | $0.00 | 72005.28 | 02136 |
28 | Melle,Laura | Intern I | ASD Human Resources | $5580.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | 5580.00 | 02139 |
29 | Morant,Darius | Prin Clerk | ASD Human Resources | $5710.80 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | 5710.80 | 02119 |
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
22203 | Wilson,Jeffrey | Fire Fighter-AdvanceTechnician | Boston Fire Department | $88662.18 | $12890.60 | $550.00 | $12730.82 | $6633.81 | $128.00 | $0.00 | 121595.41 | 02136 |
22204 | Wilson,Keith Michael | Fire Fighter | Boston Fire Department | $53235.90 | $2999.38 | $550.00 | $5352.20 | $0.00 | $0.00 | $0.00 | 62137.48 | 02132 |
22205 | Wilson,Marlon | Fire Fighter | Boston Fire Department | $88461.02 | $13602.82 | $550.00 | $13775.17 | $4855.79 | $8326.75 | $0.00 | 129571.55 | 02136 |
22206 | Wilson,Ryan Jon | Fire Fighter | Boston Fire Department | $21364.38 | $0.00 | $275.00 | $0.00 | $0.00 | $0.00 | $0.00 | 21639.38 | 02132 |
22207 | Wilson,Sean E | Fire Fighter | Boston Fire Department | $78538.07 | $7593.22 | $550.00 | $6989.46 | $49.23 | $8462.25 | $0.00 | 102182.23 | 02136 |
22208 | Wilson,Wayne A | Fire Fighter | Boston Fire Department | $96178.88 | $10010.18 | $2157.24 | $24629.94 | $4337.00 | $561.00 | $0.00 | 137874.24 | 02026 |
22209 | Wirtz,Eric M | Fire Fighter | Boston Fire Department | $71911.66 | $7198.69 | $550.00 | $3892.15 | $7678.90 | $0.00 | $0.00 | 91231.40 | 02122 |
22210 | Wong,Connie S | Dep Comm-Labor & Legal | Boston Fire Department | $28750.02 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | 28750.02 | 02111 |
22211 | Wong,Robert | Fire Fighter | Boston Fire Department | $15670.19 | $8210.82 | $11065.36 | $2127.66 | $84109.32 | $0.00 | $0.00 | 121183.35 | 02118 |
22212 | Wong,William | Asst Prin accountant | Boston Fire Department | $39974.95 | $0.00 | $747.72 | $38029.14 | $0.00 | $0.00 | $0.00 | 78751.81 | 02132 |
22213 | Woodbury,Dennis A | Fire Fighter | Boston Fire Department | $96764.06 | $14814.32 | $2157.24 | $21890.55 | $718.99 | $2401.50 | $0.00 | 138746.66 | 02124 |
22214 | Woods,Patrick J | Fire Lieutenant | Boston Fire Department | $110597.16 | $16537.28 | $2451.66 | $18038.47 | $0.00 | $0.00 | $0.00 | 147624.57 | 02129 |
22215 | Wornum,Dwight H | Fire Fighter | Boston Fire Department | $149.99 | $6261.44 | $3063.63 | $0.00 | $100310.76 | $0.00 | $0.00 | 109785.82 | 02136 |
22216 | Wornum,John W | FF (FPDInspLevl1Certification) | Boston Fire Department | $98109.51 | $13524.14 | $6243.64 | $4866.68 | $0.00 | $0.00 | $0.00 | 122743.97 | 02121 |
22217 | Wornum,Phillip B. | FF LEP Title 3 Inspector - ADR | Boston Fire Department | $1195.32 | $1188.70 | $85430.84 | $0.00 | $103564.81 | $0.00 | $0.00 | 191379.67 | 02186 |
22218 | Worrell,David E | Fire Fighter | Boston Fire Department | $0.00 | $1013.60 | $0.00 | $0.00 | $1425.72 | $0.00 | $0.00 | 2439.32 | 02121 |
22219 | Wosny,Joseph Andrew | Fire Fighter | Boston Fire Department | $21364.38 | $0.00 | $275.00 | $0.00 | $0.00 | $0.00 | $0.00 | 21639.38 | 02132 |
22220 | Wright,Alexander E | Fire Fighter-Technician | Boston Fire Department | $94499.10 | $14911.41 | $2151.24 | $19095.59 | $0.00 | $5132.75 | $0.00 | 135790.09 | 02072 |
22221 | Wright,Ballin L | Fire Fighter-AdvanceTechnician | Boston Fire Department | $84536.18 | $14204.17 | $3721.05 | $18540.90 | $13082.71 | $1488.00 | $0.00 | 135573.01 | 02125 |
22222 | Wright,Christopher D | Fire Fighter-Technician | Boston Fire Department | $93427.63 | $14077.81 | $550.00 | $17740.79 | $448.65 | $0.00 | $0.00 | 126244.88 | 02132 |
22223 | Wright,James | Fire Fighter | Boston Fire Department | $0.00 | $6499.01 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | 6499.01 | 02368 |
22224 | Wuschke,Barbara | Fire Alarm Operator | Boston Fire Department | $95154.82 | $13596.87 | $550.00 | $9325.59 | $0.00 | $0.00 | $0.00 | 118627.28 | 02124 |
22225 | Wyman,George L | District Fire Chief Tech-ADR | Boston Fire Department | $0.00 | $3699.29 | $0.00 | $0.00 | $2845.81 | $0.00 | $0.00 | 6545.10 | 02360 |
22226 | Wyzard,Keith T | Fire Fighter | Boston Fire Department | $94460.92 | $14055.44 | $1466.23 | $11700.05 | $0.00 | $0.00 | $0.00 | 121682.64 | 02368 |
22227 | Yanosick,Justin Timothy | Fire Fighter | Boston Fire Department | $92646.90 | $14250.84 | $550.00 | $28269.86 | $0.00 | $0.00 | $0.00 | 135717.60 | 02132 |
22228 | Yee,Jacob | Fire Fighter | Boston Fire Department | $80073.63 | $13529.71 | $550.00 | $15560.09 | $13353.47 | $7245.75 | $0.00 | 130312.65 | 02136 |
22229 | Yetman,Patrick | Fire Fighter | Boston Fire Department | $69886.18 | $5191.64 | $550.00 | $8227.25 | $0.00 | $0.00 | $0.00 | 83855.07 | 02127 |
22230 | Young,Allan M | Fire Fighter | Boston Fire Department | $97234.17 | $14091.91 | $3050.00 | $6193.76 | $0.00 | $4464.00 | $0.00 | 125033.84 | 02136 |
22231 | Young,Michael L | Fire Fighter | Boston Fire Department | $95694.84 | $13973.38 | $550.00 | $10678.12 | $0.00 | $132.00 | $0.00 | 121028.34 | 02121 |
22232 | Zagami,Anthony J | Fire Alarm Operator | Boston Fire Department | $0.00 | $5494.93 | $0.00 | $0.00 | $0.00 | $0.00 | $0.00 | 5494.93 | 02124 |
22233 rows × 12 columns
There are boat loads of statistical analysis methods:
df['TOTAL EARNINGS'].median(),df['TOTAL EARNINGS'].mean()
(61714.87, 66920.917950344083)
df['TOTAL EARNINGS'].hist()
<matplotlib.axes._subplots.AxesSubplot at 0x1111dfed0>
If you are familiar with python, its pretty much just picking up a new library. It might take a while to get comfortable with the dataframe data structure (Pandas documentation is not super helpful), but once you've gotten the hang of its indexing notation and the oodles of incredibly useful methods for wrangling data its a very easy and powerful tool.
If my friend knew Python or needed to do some more advanced data analysis, then yes. In my opinion, the main thing Pandas has going for it is that it is a Python library. It is easy-ish to use because Python is easy-ish to use. Its main advantages just come from its proximity to Python, and the fact that it's the only popular data analysis package out there. I'm not really a Python programmer, but I use the DataFrames and DataArrays packages in Julia for statistical analysis relatively frequently so I imagine I will use Pandas for some of the data cleaning and analysis for my final data story as well.