(For more resources related to this topic, see here.)
Getting ready
A good understanding of indexes in pandas is crucial to quickly move the data around. From a business intelligence perspective, they create a distinction similar to that of metrics and dimensions in an OLAP cube. To illustrate this point, this recipe walks through getting stock data out of pandas, combining it, then reindexing it for easy chomping.
How to do it…
- Use the DataReader object to transfer stock price information into a DataFrame and to explore the basic axis of Panel.
> from pandas.i git push -u origin master o.data import DataReader > tickers = [‘gs’, ‘ibm’, ‘f’, ‘ba’, ‘axp’] > dfs = {} > for ticker in tickers: dfs[ticker] = DataReader(ticker, “yahoo”, ‘2006-01-01’) # a yet undiscussed data structure, in the same way the a # DataFrame is a collection of Series, a Panel is a collection of # DataFrames > pan = pd.Panel(dfs) > pan <class ‘pandas.core.panel.Panel’> Dimensions: 5 (items) x 1764 (major_axis) x 6 (minor_axis)Items axis: axp to ibm Major_axis axis: 2006-01-03 00:00:00 to 2013-01-04 00:00:00 Minor_axis axis: Open to Adj Close > pan.items Index([axp, ba, f, gs, ibm], dtype=object) > pan.minor_axis Index([Open, High, Low, Close, Volume, Adj Close], dtype=object) > pan.major_axis <class ‘pandas.tseries.index.DatetimeIndex’>[2006-01-03 00:00:00, …, 2013-01-04 00:00:00] Length: 1764, Freq: None, Timezone: None
- Use the axis selectors to easily compute different sets of summary statistics.
> pan.minor_xs(‘Open’).mean() axp 46.227466 ba 70.746451 f 9.135794 gs 151.655091 ibm 129.570969 # major axis is sliceable as well > day_slice = pan.major_axis[1] > pan.major_xs(day_slice)[[‘gs’, ‘ba’]] ba gs Open 70.08 127.35 High 71.27 128.91 Low 69.86 126.38 Close 71.17 127.09 Volume 3165000.00 4861600.00 Adj Close 60.43 118.12 Convert the Panel to a DataFrame. > dfs = [] > for df in pan: idx = pan.major_axis idx = pd.MultiIndex.from_tuples(zip([df]*len(idx), idx)) idx.names = [‘ticker’, ‘timestamp’] dfs.append(pd.DataFrame(pan[df].values, index=idx, columns=pan.minor_axis)) > df = pd.concat(dfs) > df Data columns: Open 8820 non-null values High 8820 non-null values Low 8820 non-null values Close 8820 non-null values Volume 8820 non-null values Adj Close 8820 non-null values dtypes: float64(6)
- Perform the analogous operations as in the preceding examples on the newly created DataFrame.
# selecting from a MultiIndex isn’t much different than the Panel # (output muted) > df.ix[‘gs’:’ibm’] > df[‘Open’]
How it works…
The previous example was certainly contrived, but when indexing and statistical techniques are incorporated, the power of pandas begins to come through. Statistics will be covered in an upcoming recipe.
pandas’ indexes by themselves can be thought of as descriptors of a certain point in the DataFrame. When ticker and timestamp are the only indexes in a DataFrame, then the point is individualized by the ticker, timestamp, and column name. After the point is individualized, it’s more convenient for aggregation and analysis.
There’s more…
Indexes show up all over the place in pandas so it’s worthwhile to see some other use cases as well.
Advanced header indexes
Hierarchical indexing isn’t limited to rows. Headers can also be represented by MultiIndex, as shown in the following command line:
> header_top = [‘Price’, ‘Price’, ‘Price’, ‘Price’, ‘Volume’, ‘Price’] > df.columns = pd.MultiIndex.from_tuples(zip(header_top, df.columns)
Performing aggregate operations with indexes
As a prelude to the following sections, we’ll do a single groupby function here since they work with indexes so well.
> df.groupby(level=[‘tickers’, ‘day’])[‘Volume’].mean()
This answers the question for each ticker and for each day (not date), that is, what was the mean volume over the life of the data.
Summary
This article talks about the use and importance of indexes in pandas. It also talks about different operations that can be done with indexes.
Resources for Article :
Further resources on this subject:
- Installing Panda3D [Article]
- Setting Up Panda3D and Configuring Development Tools [Article]
- Collision Detection and Physics in Panda3D Game Development [Article]