Pandas (software)
Pandas (styled as pandas) is a software library written for the Python programming language for data manipulation and analysis.
≈ 8 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
Pandas (styled as pandas) is a software library written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series. It is free software released under the three-clause BSD license.
The name is derived from the term "panel data", an econometrics term for data sets that include observations over multiple time periods for the same individuals, as well as a play on the phrase "Python data analysis". Wes McKinney started building what would become Pandas at AQR Capital while he was a researcher there from 2007 to 2010.
The development of Pandas introduced into Python many comparable features of working with DataFrames that were established in the R programming language. The library is built upon another library, NumPy.
History
Developer Wes McKinney started working on Pandas in 2008 while at AQR Capital Management out of the need for a high performance, flexible tool to perform quantitative analysis on financial data. Before leaving AQR, he was able to convince management to allow him to open source the library in 2009.
Another AQR employee, Chang She, joined the effort in 2012 as the second major contributor to the library. In 2015, Pandas signed on as a fiscally sponsored project of NumFOCUS, a 501(c)(3) nonprofit charity in the United States.
Data model
Pandas is built around data structures called Series and DataFrames. Data for these collections can be imported from various file formats such as comma-separated values, JSON, Parquet, SQL database tables or queries, and Microsoft Excel. import pandas as pd ser = pd.Series(['a', 'b', 'c'], index=["x", "y", "z"]) To access a value or list of values from a Series, use its index or list of indices: ser['x'] ser'x', 'z' Series can be used arithmetically, as in the statement series_3 = series_1 + series_2 .
This will align data points with corresponding index values in series_1 and series_2 (similar to a join in relational algebra), then add them together to produce new values in series_3 . A Series has various attributes, such as name (Series name), dtype (data type of values), shape (number of rows), values , and index .
They can be used in many of the same operations as NumPy arrays, with additional methods for reindexing, label-based selection, and handling missing data. df1 = pd.Series(['A', 'B', 'C']).to_frame() df2 = pd.DataFrame({"grade": ["A", "B", "C"], "score": [100, 80, 60]}) df3 = pd.read_csv('path/classgrades.csv') To retrieve a DataFrame column as a Series, use either 1) the index (dict-like notation) or 2) the name of column if the name is a valid Python identifier (attribute-like access). DataFrames support operations such as column assignment, row and column deletion, label-based indexing with loc , position-based indexing with iloc , reshaping, grouping, and joining.
Merge operations implement a subset of relational algebra and allow one-to-one, many-to-one, and many-to-many joins. Some common attributes of a DataFrame include dtypes (data type of each column), shape (dimensions of the DataFrame returned as a tuple with form (number of rows, number of columns) ), index / columns (labels of the DataFrame's rows/columns, respectively, returned as an Index object), values (data in the DataFrame returned as a 2D array), and empty (returns True if the DataFrame is empty).
Criticisms
Pandas has been criticized for its inefficiency. The entire dataset must be loaded in RAM, and the library does not optimize query plans or support parallel computing across multiple cores. Wes McKinney, the creator of Pandas, has recommended Apache Arrow as an alternative to address these performance concerns and other limitations.
Otherwise, he says, "my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset".
Examples
Pandas is customarily imported as . import numpy as np import pandas as pd Here's a fake dataset on the nutritional value of various food items: df = pd.DataFrame({ "food": ["Apple", "Banana", "Almonds", "Broccoli", "Salmon", "Oatmeal"], "calories": [95, 105, 164, 55, 208, 158], "protein_g": [0.5, 1.3, 6.0, 3.7, 22.0, 6.0], "carb_g": [25.1, 27.0, 6.4, 11.2, 0.5, 29.3], "fat_g": [0.3, 0.4, 14.0, 0.6, 13.0, 3.2], "fiber_g": [4.4, 3.1, 3.5, 2.4, 0.0, 4.0], "category": ["Fruit", "Fruit", "Nuts", "Vegetable", "Meat", "Grain"] }) df food calories protein_g carb_g fat_g fiber_g category 0 Apple 95 0.5 25.1 0.3 4.4 Fruit 1 Banana 105 1.3 27.0 0.4 3.1 Fruit 2 Almonds 164 6.0 6.4 14.0 3.5 Nuts 3 Broccoli 55 3.7 11.2 0.6 2.4 Vegetable 4 Salmon 208 22.0 0.5 13.0 0.0 Meat 5 Oatmeal 158 6.0 29.3 3.2 4.0 Grain Some possible manipulations and analyses that can be performed: Compute descriptive statistics: df.describe() This provides summaries across numeric columns, including count, mean, standard deviation, and min/max values. Select specific columns: df"food", "calories" Sort foods by calories (descending order): df.sort_values("calories", ascending=False) Note: Any missing values are automatically sorted to the end of the Series.
Find nutrient-dense foods (> 3g protein + > 3g fat): df[(df["protein_g"] > 3) & (df["fat_g"] > 3)] Group items by category and calculate average macros: df.groupby("category")"carb_g", "fat_g", "protein_g".mean() This returns a GroupBy object that is analogous to a collection of DataFrames. Find the highest-calorie food per category: df.loc[df.groupby("category")["protein_g"].idxmax()] idxmax() returns the index value of the maximum value. idxmin() performs an analogous function with the minimum value.
Create new columns (macronutrient ratio as a percentage of total calories): df["carb_pct"] = df["carb_g"] * 4 / df["calories"] * 100 df["fat_pct"] = df["fat_g"] * 9 / df["calories"] * 100 df["protein_pct"] = df["protein_g"] * 4 / df["calories"] * 100 Handle missing data: df.loc[2, "fiber_g"] = None # Simulate a missing value df["fiber_g"].fillna(df["fiber_g"].mean()) This replaces the None value with NaN . Alternatively, we can also drop rows containing missing values by calling df.dropna() .
Merge DataFrames (combine with grocery prices) prices = pd.DataFrame({ "food": ["Apple", "Banana", "Almonds", "Broccoli", "Salmon", "Oatmeal"], "price": [0.7, 0.5, 4.2, 2.0, 10.5, 3.4] }) merged = df.merge(prices, on="food") The on keyword specifies the name of the key column by which the two DataFrames will be merged. To merge datasets with different key column names, use the parameters left_on and right_on instead.
Categorize food by calories: df["calorie_level"] = pd.cut( df["calories"], bins=[0,100,200,300], labels=["Low", "Medium", "High"] ) This splits numeric data into separate "bins" or intervals, thereby allowing continuous measurements to be analyzed as discrete categories.
Example 2: Resampling
Create example time series data, daily: periods = 30 days = pd.date_range(start='1 June 2019', periods=periods) np.random.seed(0) # Seed the random number generator (RNG) values = np.random.rand(periods) s_daily = pd.Series(values, index=days) print(s_daily) 2019-06-01 0.548814 2019-06-02 0.715189 2019-06-03 0.602763 ... 2019-06-28 0.944669 2019-06-29 0.521848 2019-06-30 0.414662 Freq: D, Length: 30, dtype: float64 Resample to weekly ending Monday: s_weekly = s_daily.resample('W-Mon').sum() print(s_weekly) 2019-06-03 1.866766 2019-06-10 4.290897 2019-06-17 2.992645 2019-06-24 5.500574 2019-07-01 2.782728 Freq: W-MON, dtype: float64
See also
matplotlib Dask SciPy Polars dplyr scikit-learn GeoPandas List of numerical analysis software List of data science software
Text derived from https://en.wikipedia.org/w/index.php?oldid=1375023062 under CC BY-SA 4.0; modifications have been made.