pandas: powerful Python data analysis toolkit

pandas 文档中文翻译

pandas 版本: 0.19.0

pipy: http://pypi.python.org/pypi/pandas

源码库: http://github.com/pydata/pandas

Issues & Ideas: https://github.com/pydata/pandas/issues

Q&A Support: http://stackoverflow.com/questions/tagged/pandas

Developer Mailing List: http://groups.google.com/group/pydata

pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. It is already well on its way toward this goal.

pandas is well suited for many different kinds of data:

  • Tabular data with heterogeneously-typed columns, as in an SQL table or Excel spreadsheet
  • Ordered and unordered (not necessarily fixed-frequency) time series data.
  • Arbitrary matrix data (homogeneously typed or heterogeneous) with row and column labels
  • Any other form of observational / statistical data sets. The data actually need not be labeled at all to be placed into a pandas data structure

The two primary data structures of pandas, Series (1-dimensional) and DataFrame (2-dimensional), handle the vast majority of typical use cases in finance, statistics, social science, and many areas of engineering. For R users, DataFrame provides everything that R’s data.frame provides and much more. pandas is built on top of NumPy and is intended to integrate well within a scientific computing environment with many other 3rd party libraries.

Here are just a few of the things that pandas does well:

  • Easy handling of missing data (represented as NaN) in floating point as well as non-floating point data
  • Size mutability: columns can be inserted and deleted from DataFrame and higher dimensional objects
  • Automatic and explicit data alignment: objects can be explicitly aligned to a set of labels, or the user can simply ignore the labels and let Series, DataFrame, etc. automatically align the data for you in computations
  • Powerful, flexible group by functionality to perform split-apply-combine operations on data sets, for both aggregating and transforming data
  • Make it easy to convert ragged, differently-indexed data in other Python and NumPy data structures into DataFrame objects
  • Intelligent label-based slicing, fancy indexing, and subsetting of large data sets
  • Intuitive merging and joining data sets
  • Flexible reshaping and pivoting of data sets
  • Hierarchical labeling of axes (possible to have multiple labels per tick)
  • Robust IO tools for loading data from flat files (CSV and delimited), Excel files, databases, and saving / loading data from the ultrafast HDF5 format
  • Time series-specific functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging, etc.

Many of these principles are here to address the shortcomings frequently experienced using other languages / scientific research environments. For data scientists, working with data is typically divided into multiple stages: munging and cleaning data, analyzing / modeling it, then organizing the results of the analysis into a form suitable for plotting or tabular display. pandas is the ideal tool for all of these tasks.

Some other notes

  • pandas is fast. Many of the low-level algorithmic bits have been extensively tweaked in Cython code. However, as with anything else generalization usually sacrifices performance. So if you focus on one feature for your application you may be able to create a faster specialized tool.
  • pandas is a dependency of statsmodels, making it an important part of the statistical computing ecosystem in Python.
  • pandas has been used extensively in production in financial applications.

注解

This documentation assumes general familiarity with NumPy. If you haven’t used NumPy much or at all, do invest some time in learning about NumPy first.

See the package overview for more detail about what’s in the library.

pandas 文档中文翻译

翻译流程

  1. 因为在官方文档中还有很多自动生成的 API 文档，这些 API 文档作为查阅资料并不需要翻译
  2. 将官方文档中( https://github.com/pandas-dev/pandas/tree/master/doc )的需要翻译的章节文档原始 rst 文件整理到本项目中
  3. 翻译人员即在本项目中的 rst 文件开始文档翻译工作
  4. 本项目的文件版本使用 git 进行管理，版本库托管在 github 上
  5. 协作方式按照通常的 fork、pull-request、merge 方式进行

自动发布流程

因为本项目基于 Sphinx （ http://www.sphinx-doc.org/ ） 构建，并且已经关联了 ReadtheDocs （ https://readthedocs.org/ ） 在线服务，所以在每次代码库有变动之后，文档就会在 ReadtheDocs 自动构建，输出友好阅读版本（ 地址： http://pandas-docs-zh-cn.rtfd.io/ ）

文档版本

依照 pandas v0.19.0 的文档 （ https://github.com/pandas-dev/pandas/tree/v0.19.0/doc ）

协作交流

QQ群： 84616803

TODO

  1. 整理官方文档 rst 文件到本项目代码库中
  2. 去掉官方文档中的自动生成的文档，以及具体 API 调用说明文档，这些资料直接查阅英文文档更合适
  3. 去掉官方文中的类库模块相关引用，以及非标准标记语句
  4. 召集人员进行翻译工作

最新进展

这里记录了 pandas 的最新特性与改进。

对于其他历史版本的信息可以参阅这里： http://pandas.pydata.org/pandas-docs/stable/whatsnew.html

版本 v0.19.0 ( 发布于2016年10月2日)

This is a major release from 0.18.1 and includes number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version. rst Highlights include:

  • merge_asof() for asof-style time-series joining, see here
  • .rolling() is now time-series aware, see here
  • read_csv() now supports parsing Categorical data, see here
  • A function union_categorical() has been added for combining categoricals, see here
  • PeriodIndex now has its own period dtype, and changed to be more consistent with other Index classes. See here
  • Sparse data structures gained enhanced support of int and bool dtypes, see here
  • Comparison operations with Series no longer ignores the index, see here for an overview of the API changes.
  • Introduction of a pandas development API for utility functions, see here.
  • Deprecation of Panel4D and PanelND. We recommend to represent these types of n-dimensional data with the xarray package.
  • Removal of the previously deprecated modules pandas.io.data, pandas.io.wb, pandas.tools.rplot.

警告

pandas >= 0.19.0 will no longer silence numpy ufunc warnings upon import, see here.

New features

merge_asof for asof-style time-series joining

A long-time requested feature has been added through the merge_asof() function, to support asof style joining of time-series (:issue:`1870`, :issue:`13695`, :issue:`13709`, :issue:`13902`). Full documentation is here.

The merge_asof() performs an asof merge, which is similar to a left-join except that we match on nearest key rather than equal keys.

We typically want to match exactly when possible, and use the most recent value otherwise.

We can also match rows ONLY with prior data, and not an exact match.

In a typical time-series example, we have trades and quotes and we want to asof-join them. This also illustrates using the by parameter to group data before merging.

An asof merge joins on the on, typically a datetimelike field, which is ordered, and in this case we are using a grouper in the by field. This is like a left-outer join, except that forward filling happens automatically taking the most recent non-NaN value.

This returns a merged DataFrame with the entries in the same order as the original left passed DataFrame (trades in this case), with the fields of the quotes merged.

.rolling() is now time-series aware

.rolling() objects are now time-series aware and can accept a time-series offset (or convertible) for the window argument (:issue:`13327`, :issue:`12995`). See the full documentation here.

This is a regular frequency index. Using an integer window parameter works to roll along the window frequency.

Specifying an offset allows a more intuitive specification of the rolling frequency.

Using a non-regular, but still monotonic index, rolling with an integer window does not impart any special calculation.

Using the time-specification generates variable windows for this sparse data.

Furthermore, we now allow an optional on parameter to specify a column (rather than the default of the index) in a DataFrame.

read_csv has improved support for duplicate column names

Duplicate column names are now supported in read_csv() whether they are in the file or passed in as the names parameter (:issue:`7160`, :issue:`9424`)

Previous behavior:

In [2]: pd.read_csv(StringIO(data), names=names)
Out[2]:
   a  b  a
0  2  1  2
1  5  4  5

The first a column contained the same data as the second a column, when it should have contained the values [0, 3].

New behavior:

read_csv supports parsing Categorical directly

The read_csv() function now supports parsing a Categorical column when specified as a dtype (:issue:`10153`). Depending on the structure of the data, this can result in a faster parse time and lower memory usage compared to converting to Categorical after parsing. See the io docs here.

Individual columns can be parsed as a Categorical using a dict specification

注解

The resulting categories will always be parsed as strings (object dtype). If the categories are numeric they can be converted using the to_numeric() function, or as appropriate, another converter such as to_datetime().

Categorical Concatenation

  • A function union_categoricals() has been added for combining categoricals, see Unioning Categoricals (:issue:`13361`, :issue:`:13763`, issue:13846, :issue:`14173`)

  • concat and append now can concat category dtypes with different categories as object dtype (:issue:`13524`)

    Previous behavior:

    In [1]: pd.concat([s1, s2])
ValueError: incompatible categories in categorical concat

    New behavior:

Semi-Month Offsets

Pandas has gained new frequency offsets, SemiMonthEnd (‘SM’) and SemiMonthBegin (‘SMS’). These provide date offsets anchored (by default) to the 15th and end of month, and 15th and 1st of month respectively. (:issue:`1543`)

SemiMonthEnd:

SemiMonthBegin:

Using the anchoring suffix, you can also specify the day of month to use instead of the 15th.

New Index methods

The following methods and options are added to Index, to be more consistent with the Series and DataFrame API.

Index now supports the .where() function for same shape indexing (:issue:`13170`)

Index now supports .dropna() to exclude missing values (:issue:`6194`)

For MultiIndex, values are dropped if any level is missing by default. Specifying how='all' only drops values where all levels are missing.

Index now supports .str.extractall() which returns a DataFrame, see the docs here (:issue:`10008`, :issue:`13156`)

Index.astype() now accepts an optional boolean argument copy, which allows optional copying if the requirements on dtype are satisfied (:issue:`13209`)

Google BigQuery Enhancements
  • The read_gbq() method has gained the dialect argument to allow users to specify whether to use BigQuery’s legacy SQL or BigQuery’s standard SQL. See the docs for more details (:issue:`13615`).
  • The to_gbq() method now allows the DataFrame column order to differ from the destination table schema (:issue:`11359`).
Fine-grained numpy errstate

Previous versions of pandas would permanently silence numpy’s ufunc error handling when pandas was imported. Pandas did this in order to silence the warnings that would arise from using numpy ufuncs on missing data, which are usually represented as NaN s. Unfortunately, this silenced legitimate warnings arising in non-pandas code in the application. Starting with 0.19.0, pandas will use the numpy.errstate context manager to silence these warnings in a more fine-grained manner, only around where these operations are actually used in the pandas codebase. (:issue:`13109`, :issue:`13145`)

After upgrading pandas, you may see new RuntimeWarnings being issued from your code. These are likely legitimate, and the underlying cause likely existed in the code when using previous versions of pandas that simply silenced the warning. Use numpy.errstate around the source of the RuntimeWarning to control how these conditions are handled.

get_dummies now returns integer dtypes

The pd.get_dummies function now returns dummy-encoded columns as small integers, rather than floats (:issue:`8725`). This should provide an improved memory footprint.

Previous behavior:

In [1]: pd.get_dummies(['a', 'b', 'a', 'c']).dtypes

Out[1]:
a    float64
b    float64
c    float64
dtype: object

New behavior:

Downcast values to smallest possible dtype in to_numeric

pd.to_numeric() now accepts a downcast parameter, which will downcast the data if possible to smallest specified numerical dtype (:issue:`13352`)

pandas development API

As part of making pandas API more uniform and accessible in the future, we have created a standard sub-package of pandas, pandas.api to hold public API’s. We are starting by exposing type introspection functions in pandas.api.types. More sub-packages and officially sanctioned API’s will be published in future versions of pandas (:issue:`13147`, :issue:`13634`)

The following are now part of this API:

注解

Calling these functions from the internal module pandas.core.common will now show a DeprecationWarning (:issue:`13990`)

Other enhancements
  • Timestamp can now accept positional and keyword parameters similar to datetime.datetime() (:issue:`10758`, :issue:`11630`)
  • The .resample() function now accepts a on= or level= parameter for resampling on a datetimelike column or MultiIndex level (:issue:`13500`)
  • The .get_credentials() method of GbqConnector can now first try to fetch the application default credentials. See the docs for more details (:issue:`13577`).
  • The .tz_localize() method of DatetimeIndex and Timestamp has gained the errors keyword, so you can potentially coerce nonexistent timestamps to NaT. The default behavior remains to raising a NonExistentTimeError (:issue:`13057`)
  • .to_hdf/read_hdf() now accept path objects (e.g. pathlib.Path, py.path.local) for the file path (:issue:`11773`)
  • The pd.read_csv() with engine='python' has gained support for the decimal (:issue:`12933`), na_filter (:issue:`13321`) and the memory_map option (:issue:`13381`).
  • Consistent with the Python API, pd.read_csv() will now interpret +inf as positive infinity (:issue:`13274`)
  • The pd.read_html() has gained support for the na_values, converters, keep_default_na options (:issue:`13461`)
  • Categorical.astype() now accepts an optional boolean argument copy, effective when dtype is categorical (:issue:`13209`)
  • DataFrame has gained the .asof() method to return the last non-NaN values according to the selected subset (:issue:`13358`)
  • The DataFrame constructor will now respect key ordering if a list of OrderedDict objects are passed in (:issue:`13304`)
  • pd.read_html() has gained support for the decimal option (:issue:`12907`)
  • Series has gained the properties .is_monotonic, .is_monotonic_increasing, .is_monotonic_decreasing, similar to Index (:issue:`13336`)
  • DataFrame.to_sql() now allows a single value as the SQL type for all columns (:issue:`11886`).
  • Series.append now supports the ignore_index option (:issue:`13677`)
  • .to_stata() and StataWriter can now write variable labels to Stata dta files using a dictionary to make column names to labels (:issue:`13535`, :issue:`13536`)
  • .to_stata() and StataWriter will automatically convert datetime64[ns] columns to Stata format %tc, rather than raising a ValueError (:issue:`12259`)
  • read_stata() and StataReader raise with a more explicit error message when reading Stata files with repeated value labels when convert_categoricals=True (:issue:`13923`)
  • DataFrame.style will now render sparsified MultiIndexes (:issue:`11655`)
  • DataFrame.style will now show column level names (e.g. DataFrame.columns.names) (:issue:`13775`)
  • DataFrame has gained support to re-order the columns based on the values in a row using df.sort_values(by='...', axis=1) (:issue:`10806`)
  • Added documentation to I/O regarding the perils of reading in columns with mixed dtypes and how to handle it (:issue:`13746`)
  • to_html() now has a border argument to control the value in the opening <table> tag. The default is the value of the html.border option, which defaults to 1. This also affects the notebook HTML repr, but since Jupyter’s CSS includes a border-width attribute, the visual effect is the same. (:issue:`11563`).
  • Raise ImportError in the sql functions when sqlalchemy is not installed and a connection string is used (:issue:`11920`).
  • Compatibility with matplotlib 2.0. Older versions of pandas should also work with matplotlib 2.0 (:issue:`13333`)
  • Timestamp, Period, DatetimeIndex, PeriodIndex and .dt accessor have gained a .is_leap_year property to check whether the date belongs to a leap year. (:issue:`13727`)
  • astype() will now accept a dict of column name to data types mapping as the dtype argument. (:issue:`12086`)
  • The pd.read_json and DataFrame.to_json has gained support for reading and writing json lines with lines option see Line delimited json (:issue:`9180`)
  • read_excel() now supports the true_values and false_values keyword arguments (:issue:`13347`)
  • groupby() will now accept a scalar and a single-element list for specifying level on a non-MultiIndex grouper. (:issue:`13907`)
  • Non-convertible dates in an excel date column will be returned without conversion and the column will be object dtype, rather than raising an exception (:issue:`10001`).
  • pd.Timedelta(None) is now accepted and will return NaT, mirroring pd.Timestamp (:issue:`13687`)
  • pd.read_stata() can now handle some format 111 files, which are produced by SAS when generating Stata dta files (:issue:`11526`)
  • Series and Index now support divmod which will return a tuple of series or indices. This behaves like a standard binary operator with regards to broadcasting rules (:issue:`14208`).

API changes

Series.tolist() will now return Python types

Series.tolist() will now return Python types in the output, mimicking NumPy .tolist() behavior (:issue:`10904`)

Previous behavior:

In [7]: type(s.tolist()[0])
Out[7]:
 <class 'numpy.int64'>

New behavior:

Series operators for different indexes

Following Series operators have been changed to make all operators consistent, including DataFrame (:issue:`1134`, :issue:`4581`, :issue:`13538`)

  • Series comparison operators now raise ValueError when index are different.
  • Series logical operators align both index of left and right hand side.

警告

Until 0.18.1, comparing Series with the same length, would succeed even if the .index are different (the result ignores .index). As of 0.19.0, this will raises ValueError to be more strict. This section also describes how to keep previous behavior or align different indexes, using the flexible comparison methods like .eq.

As a result, Series and DataFrame operators behave as below:

Arithmetic operators

Arithmetic operators align both index (no changes).

Comparison operators

Comparison operators raise ValueError when .index are different.

Previous Behavior (Series):

Series compared values ignoring the .index as long as both had the same length:

In [1]: s1 == s2
Out[1]:
A    False
B     True
C    False
dtype: bool

New behavior (Series):

In [2]: s1 == s2
Out[2]:
ValueError: Can only compare identically-labeled Series objects

注解

To achieve the same result as previous versions (compare values based on locations ignoring .index), compare both .values.

If you want to compare Series aligning its .index, see flexible comparison methods section below:

Current Behavior (DataFrame, no change):

In [3]: df1 == df2
Out[3]:
ValueError: Can only compare identically-labeled DataFrame objects
Logical operators

Logical operators align both .index of left and right hand side.

Previous behavior (Series), only left hand side index was kept:

In [4]: s1 = pd.Series([True, False, True], index=list('ABC'))
In [5]: s2 = pd.Series([True, True, True], index=list('ABD'))
In [6]: s1 & s2
Out[6]:
A     True
B    False
C    False
dtype: bool

New behavior (Series):

注解

Series logical operators fill a NaN result with False.

注解

To achieve the same result as previous versions (compare values based on only left hand side index), you can use reindex_like:

Current Behavior (DataFrame, no change):

Flexible comparison methods

Series flexible comparison methods like eq, ne, le, lt, ge and gt now align both index. Use these operators if you want to compare two Series which has the different index.

Previously, this worked the same as comparison operators (see above).

Series type promotion on assignment

A Series will now correctly promote its dtype for assignment with incompat values to the current dtype (:issue:`13234`)

Previous behavior:

In [2]: s["a"] = pd.Timestamp("2016-01-01")

In [3]: s["b"] = 3.0
TypeError: invalid type promotion

New behavior:

.to_datetime() changes

Previously if .to_datetime() encountered mixed integers/floats and strings, but no datetimes with errors='coerce' it would convert all to NaT.

Previous behavior:

In [2]: pd.to_datetime([1, 'foo'], errors='coerce')
Out[2]: DatetimeIndex(['NaT', 'NaT'], dtype='datetime64[ns]', freq=None)

Current behavior:

This will now convert integers/floats with the default unit of ns.

Bug fixes related to .to_datetime():

  • Bug in pd.to_datetime() when passing integers or floats, and no unit and errors='coerce' (:issue:`13180`).
  • Bug in pd.to_datetime() when passing invalid datatypes (e.g. bool); will now respect the errors keyword (:issue:`13176`)
  • Bug in pd.to_datetime() which overflowed on int8, and int16 dtypes (:issue:`13451`)
  • Bug in pd.to_datetime() raise AttributeError with NaN and the other string is not valid when errors='ignore' (:issue:`12424`)
  • Bug in pd.to_datetime() did not cast floats correctly when unit was specified, resulting in truncated datetime (:issue:`13834`)
Merging changes

Merging will now preserve the dtype of the join keys (:issue:`8596`)

Previous behavior:

In [5]: pd.merge(df1, df2, how='outer')
Out[5]:
   key    v1
0  1.0  10.0
1  1.0  20.0
2  2.0  30.0

In [6]: pd.merge(df1, df2, how='outer').dtypes
Out[6]:
key    float64
v1     float64
dtype: object

New behavior:

We are able to preserve the join keys

Of course if you have missing values that are introduced, then the resulting dtype will be upcast, which is unchanged from previous.

.describe() changes

Percentile identifiers in the index of a .describe() output will now be rounded to the least precision that keeps them distinct (:issue:`13104`)

Previous behavior:

The percentiles were rounded to at most one decimal place, which could raise ValueError for a data frame if the percentiles were duplicated.

In [3]: s.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.9999])
Out[3]:
count     5.000000
mean      2.000000
std       1.581139
min       0.000000
0.0%      0.000400
0.1%      0.002000
0.1%      0.004000
50%       2.000000
99.9%     3.996000
100.0%    3.998000
100.0%    3.999600
max       4.000000
dtype: float64

In [4]: df.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.9999])
Out[4]:
...
ValueError: cannot reindex from a duplicate axis

New behavior:

Furthermore:

  • Passing duplicated percentiles will now raise a ValueError.
  • Bug in .describe() on a DataFrame with a mixed-dtype column index, which would previously raise a TypeError (:issue:`13288`)
Period changes
PeriodIndex now has period dtype

PeriodIndex now has its own period dtype. The period dtype is a pandas extension dtype like category or the timezone aware dtype (datetime64[ns, tz]) (:issue:`13941`). As a consequence of this change, PeriodIndex no longer has an integer dtype:

Previous behavior:

In [1]: pi = pd.PeriodIndex(['2016-08-01'], freq='D')

In [2]: pi
Out[2]: PeriodIndex(['2016-08-01'], dtype='int64', freq='D')

In [3]: pd.api.types.is_integer_dtype(pi)
Out[3]: True

In [4]: pi.dtype
Out[4]: dtype('int64')

New behavior:

Period('NaT') now returns pd.NaT

Previously, Period has its own Period('NaT') representation different from pd.NaT. Now Period('NaT') has been changed to return pd.NaT. (:issue:`12759`, :issue:`13582`)

Previous behavior:

In [5]: pd.Period('NaT', freq='D')
Out[5]: Period('NaT', 'D')

New behavior:

These result in pd.NaT without providing freq option.

To be compatible with Period addition and subtraction, pd.NaT now supports addition and subtraction with int. Previously it raised ValueError.

Previous behavior:

In [5]: pd.NaT + 1
...
ValueError: Cannot add integral value to Timestamp without freq.

New behavior:

PeriodIndex.values now returns array of Period object

.values is changed to return an array of Period objects, rather than an array of integers (:issue:`13988`).

Previous behavior:

In [6]: pi = pd.PeriodIndex(['2011-01', '2011-02'], freq='M')
In [7]: pi.values
array([492, 493])

New behavior:

Index + / - no longer used for set operations

Addition and subtraction of the base Index type and of DatetimeIndex (not the numeric index types) previously performed set operations (set union and difference). This behavior was already deprecated since 0.15.0 (in favor using the specific .union() and .difference() methods), and is now disabled. When possible, + and - are now used for element-wise operations, for example for concatenating strings or subtracting datetimes (:issue:`8227`, :issue:`14127`).

Previous behavior:

In [1]: pd.Index(['a', 'b']) + pd.Index(['a', 'c'])
FutureWarning: using '+' to provide set union with Indexes is deprecated, use '|' or .union()
Out[1]: Index(['a', 'b', 'c'], dtype='object')

New behavior: the same operation will now perform element-wise addition:

Note that numeric Index objects already performed element-wise operations. For example, the behavior of adding two integer Indexes is unchanged. The base Index is now made consistent with this behavior.

Further, because of this change, it is now possible to subtract two DatetimeIndex objects resulting in a TimedeltaIndex:

Previous behavior:

In [1]: pd.DatetimeIndex(['2016-01-01', '2016-01-02']) - pd.DatetimeIndex(['2016-01-02', '2016-01-03'])
FutureWarning: using '-' to provide set differences with datetimelike Indexes is deprecated, use .difference()
Out[1]: DatetimeIndex(['2016-01-01'], dtype='datetime64[ns]', freq=None)

New behavior:

Index.difference and .symmetric_difference changes

Index.difference and Index.symmetric_difference will now, more consistently, treat NaN values as any other values. (:issue:`13514`)

Previous behavior:

In [3]: idx1.difference(idx2)
Out[3]: Float64Index([nan, 2.0, 3.0], dtype='float64')

In [4]: idx1.symmetric_difference(idx2)
Out[4]: Float64Index([0.0, nan, 2.0, 3.0], dtype='float64')

New behavior:

Index.unique consistently returns Index

Index.unique() now returns unique values as an Index of the appropriate dtype. (:issue:`13395`). Previously, most Index classes returned np.ndarray, and DatetimeIndex, TimedeltaIndex and PeriodIndex returned Index to keep metadata like timezone.

Previous behavior:

In [1]: pd.Index([1, 2, 3]).unique()
Out[1]: array([1, 2, 3])

In [2]: pd.DatetimeIndex(['2011-01-01', '2011-01-02', '2011-01-03'], tz='Asia/Tokyo').unique()
Out[2]:
DatetimeIndex(['2011-01-01 00:00:00+09:00', '2011-01-02 00:00:00+09:00',
               '2011-01-03 00:00:00+09:00'],
              dtype='datetime64[ns, Asia/Tokyo]', freq=None)

New behavior:

MultiIndex constructors, groupby and set_index preserve categorical dtypes

MultiIndex.from_arrays and MultiIndex.from_product will now preserve categorical dtype in MultiIndex levels (:issue:`13743`, :issue:`13854`).

Previous behavior:

In [4]: midx.levels[0]
Out[4]: Index(['b', 'a', 'c'], dtype='object')

In [5]: midx.get_level_values[0]
Out[5]: Index(['a', 'b'], dtype='object')

New behavior: the single level is now a CategoricalIndex:

An analogous change has been made to MultiIndex.from_product. As a consequence, groupby and set_index also preserve categorical dtypes in indexes

Previous behavior:

In [11]: df_grouped.index.levels[1]
Out[11]: Index(['b', 'a', 'c'], dtype='object', name='C')
In [12]: df_grouped.reset_index().dtypes
Out[12]:
A      int64
C     object
B    float64
dtype: object

In [13]: df_set_idx.index.levels[1]
Out[13]: Index(['b', 'a', 'c'], dtype='object', name='C')
In [14]: df_set_idx.reset_index().dtypes
Out[14]:
A      int64
C     object
B      int64
dtype: object

New behavior:

read_csv will progressively enumerate chunks

When read_csv() is called with chunksize=n and without specifying an index, each chunk used to have an independently generated index from 0 to n-1. They are now given instead a progressive index, starting from 0 for the first chunk, from n for the second, and so on, so that, when concatenated, they are identical to the result of calling read_csv() without the chunksize= argument (:issue:`12185`).

Previous behavior:

In [2]: pd.concat(pd.read_csv(StringIO(data), chunksize=2))
Out[2]:
   A  B
0  0  1
1  2  3
0  4  5
1  6  7

New behavior:

Sparse Changes

These changes allow pandas to handle sparse data with more dtypes, and for work to make a smoother experience with data handling.

int64 and bool support enhancements

Sparse data structures now gained enhanced support of int64 and bool dtype (:issue:`667`, :issue:`13849`).

Previously, sparse data were float64 dtype by default, even if all inputs were of int or bool dtype. You had to specify dtype explicitly to create sparse data with int64 dtype. Also, fill_value had to be specified explicitly because the default was np.nan which doesn’t appear in int64 or bool data.

In [1]: pd.SparseArray([1, 2, 0, 0])
Out[1]:
[1.0, 2.0, 0.0, 0.0]
Fill: nan
IntIndex
Indices: array([0, 1, 2, 3], dtype=int32)

# specifying int64 dtype, but all values are stored in sp_values because
# fill_value default is np.nan
In [2]: pd.SparseArray([1, 2, 0, 0], dtype=np.int64)
Out[2]:
[1, 2, 0, 0]
Fill: nan
IntIndex
Indices: array([0, 1, 2, 3], dtype=int32)

In [3]: pd.SparseArray([1, 2, 0, 0], dtype=np.int64, fill_value=0)
Out[3]:
[1, 2, 0, 0]
Fill: 0
IntIndex
Indices: array([0, 1], dtype=int32)

As of v0.19.0, sparse data keeps the input dtype, and uses more appropriate fill_value defaults (0 for int64 dtype, False for bool dtype).

See the docs for more details.

Operators now preserve dtypes

  • Sparse data structure now can preserve dtype after arithmetic ops (:issue:`13848`)

  • Sparse data structure now support astype to convert internal dtype (:issue:`13900`)

    astype fails if data contains values which cannot be converted to specified dtype. Note that the limitation is applied to fill_value which default is np.nan.

    In [7]: pd.SparseSeries([1., np.nan, 2., np.nan], fill_value=np.nan).astype(np.int64)
Out[7]:
ValueError: unable to coerce current fill_value nan to int64 dtype
Other sparse fixes
  • Subclassed SparseDataFrame and SparseSeries now preserve class types when slicing or transposing. (:issue:`13787`)
  • SparseArray with bool dtype now supports logical (bool) operators (:issue:`14000`)
  • Bug in SparseSeries with MultiIndex [] indexing may raise IndexError (:issue:`13144`)
  • Bug in SparseSeries with MultiIndex [] indexing result may have normal Index (:issue:`13144`)
  • Bug in SparseDataFrame in which axis=None did not default to axis=0 (:issue:`13048`)
  • Bug in SparseSeries and SparseDataFrame creation with object dtype may raise TypeError (:issue:`11633`)
  • Bug in SparseDataFrame doesn’t respect passed SparseArray or SparseSeries ‘s dtype and fill_value (:issue:`13866`)
  • Bug in SparseArray and SparseSeries don’t apply ufunc to fill_value (:issue:`13853`)
  • Bug in SparseSeries.abs incorrectly keeps negative fill_value (:issue:`13853`)
  • Bug in single row slicing on multi-type SparseDataFrame s, types were previously forced to float (:issue:`13917`)
  • Bug in SparseSeries slicing changes integer dtype to float (:issue:`8292`)
  • Bug in SparseDataFarme comparison ops may raise TypeError (:issue:`13001`)
  • Bug in SparseDataFarme.isnull raises ValueError (:issue:`8276`)
  • Bug in SparseSeries representation with bool dtype may raise IndexError (:issue:`13110`)
  • Bug in SparseSeries and SparseDataFrame of bool or int64 dtype may display its values like float64 dtype (:issue:`13110`)
  • Bug in sparse indexing using SparseArray with bool dtype may return incorrect result (:issue:`13985`)
  • Bug in SparseArray created from SparseSeries may lose dtype (:issue:`13999`)
  • Bug in SparseSeries comparison with dense returns normal Series rather than SparseSeries (:issue:`13999`)
Indexer dtype changes

注解

This change only affects 64 bit python running on Windows, and only affects relatively advanced indexing operations

Methods such as Index.get_indexer that return an indexer array, coerce that array to a “platform int”, so that it can be directly used in 3rd party library operations like numpy.take. Previously, a platform int was defined as np.int_ which corresponds to a C integer, but the correct type, and what is being used now, is np.intp, which corresponds to the C integer size that can hold a pointer (:issue:`3033`, :issue:`13972`).

These types are the same on many platform, but for 64 bit python on Windows, np.int_ is 32 bits, and np.intp is 64 bits. Changing this behavior improves performance for many operations on that platform.

Previous behavior:

In [1]: i = pd.Index(['a', 'b', 'c'])

In [2]: i.get_indexer(['b', 'b', 'c']).dtype
Out[2]: dtype('int32')

New behavior:

In [1]: i = pd.Index(['a', 'b', 'c'])

In [2]: i.get_indexer(['b', 'b', 'c']).dtype
Out[2]: dtype('int64')
Other API Changes
  • Timestamp.to_pydatetime will issue a UserWarning when warn=True, and the instance has a non-zero number of nanoseconds, previously this would print a message to stdout (:issue:`14101`).
  • Series.unique() with datetime and timezone now returns return array of Timestamp with timezone (:issue:`13565`).
  • Panel.to_sparse() will raise a NotImplementedError exception when called (:issue:`13778`).
  • Index.reshape() will raise a NotImplementedError exception when called (:issue:`12882`).
  • .filter() enforces mutual exclusion of the keyword arguments (:issue:`12399`).
  • eval‘s upcasting rules for float32 types have been updated to be more consistent with NumPy’s rules. New behavior will not upcast to float64 if you multiply a pandas float32 object by a scalar float64 (:issue:`12388`).
  • An UnsupportedFunctionCall error is now raised if NumPy ufuncs like np.mean are called on groupby or resample objects (:issue:`12811`).
  • __setitem__ will no longer apply a callable rhs as a function instead of storing it. Call where directly to get the previous behavior (:issue:`13299`).
  • Calls to .sample() will respect the random seed set via numpy.random.seed(n) (:issue:`13161`)
  • Styler.apply is now more strict about the outputs your function must return. For axis=0 or axis=1, the output shape must be identical. For axis=None, the output must be a DataFrame with identical columns and index labels (:issue:`13222`).
  • Float64Index.astype(int) will now raise ValueError if Float64Index contains NaN values (:issue:`13149`)
  • TimedeltaIndex.astype(int) and DatetimeIndex.astype(int) will now return Int64Index instead of np.array (:issue:`13209`)
  • Passing Period with multiple frequencies to normal Index now returns Index with object dtype (:issue:`13664`)
  • PeriodIndex.fillna with Period has different freq now coerces to object dtype (:issue:`13664`)
  • Faceted boxplots from DataFrame.boxplot(by=col) now return a Series when return_type is not None. Previously these returned an OrderedDict. Note that when return_type=None, the default, these still return a 2-D NumPy array (:issue:`12216`, :issue:`7096`).
  • pd.read_hdf will now raise a ValueError instead of KeyError, if a mode other than r, r+ and a is supplied. (:issue:`13623`)
  • pd.read_csv(), pd.read_table(), and pd.read_hdf() raise the builtin FileNotFoundError exception for Python 3.x when called on a nonexistent file; this is back-ported as IOError in Python 2.x (:issue:`14086`)
  • More informative exceptions are passed through the csv parser. The exception type would now be the original exception type instead of CParserError (:issue:`13652`).
  • pd.read_csv() in the C engine will now issue a ParserWarning or raise a ValueError when sep encoded is more than one character long (:issue:`14065`)
  • DataFrame.values will now return float64 with a DataFrame of mixed int64 and uint64 dtypes, conforming to np.find_common_type (:issue:`10364`, :issue:`13917`)
  • .groupby.groups will now return a dictionary of Index objects, rather than a dictionary of np.ndarray or lists (:issue:`14293`)

Deprecations

  • Series.reshape and Categorical.reshape have been deprecated and will be removed in a subsequent release (:issue:`12882`, :issue:`12882`)
  • PeriodIndex.to_datetime has been deprecated in favor of PeriodIndex.to_timestamp (:issue:`8254`)
  • Timestamp.to_datetime has been deprecated in favor of Timestamp.to_pydatetime (:issue:`8254`)
  • Index.to_datetime and DatetimeIndex.to_datetime have been deprecated in favor of pd.to_datetime (:issue:`8254`)
  • pandas.core.datetools module has been deprecated and will be removed in a subsequent release (:issue:`14094`)
  • SparseList has been deprecated and will be removed in a future version (:issue:`13784`)
  • DataFrame.to_html() and DataFrame.to_latex() have dropped the colSpace parameter in favor of col_space (:issue:`13857`)
  • DataFrame.to_sql() has deprecated the flavor parameter, as it is superfluous when SQLAlchemy is not installed (:issue:`13611`)
  • Deprecated read_csv keywords:
    • compact_ints and use_unsigned have been deprecated and will be removed in a future version (:issue:`13320`)
    • buffer_lines has been deprecated and will be removed in a future version (:issue:`13360`)
    • as_recarray has been deprecated and will be removed in a future version (:issue:`13373`)
    • skip_footer has been deprecated in favor of skipfooter and will be removed in a future version (:issue:`13349`)
  • top-level pd.ordered_merge() has been renamed to pd.merge_ordered() and the original name will be removed in a future version (:issue:`13358`)
  • Timestamp.offset property (and named arg in the constructor), has been deprecated in favor of freq (:issue:`12160`)
  • pd.tseries.util.pivot_annual is deprecated. Use pivot_table as alternative, an example is here (:issue:`736`)
  • pd.tseries.util.isleapyear has been deprecated and will be removed in a subsequent release. Datetime-likes now have a .is_leap_year property (:issue:`13727`)
  • Panel4D and PanelND constructors are deprecated and will be removed in a future version. The recommended way to represent these types of n-dimensional data are with the xarray package. Pandas provides a to_xarray() method to automate this conversion (:issue:`13564`).
  • pandas.tseries.frequencies.get_standard_freq is deprecated. Use pandas.tseries.frequencies.to_offset(freq).rule_code instead (:issue:`13874`)
  • pandas.tseries.frequencies.to_offset‘s freqstr keyword is deprecated in favor of freq (:issue:`13874`)
  • Categorical.from_array has been deprecated and will be removed in a future version (:issue:`13854`)

Removal of prior version deprecations/changes

  • The SparsePanel class has been removed (:issue:`13778`)
  • The pd.sandbox module has been removed in favor of the external library pandas-qt (:issue:`13670`)
  • The pandas.io.data and pandas.io.wb modules are removed in favor of the pandas-datareader package (:issue:`13724`).
  • The pandas.tools.rplot module has been removed in favor of the seaborn package (:issue:`13855`)
  • DataFrame.to_csv() has dropped the engine parameter, as was deprecated in 0.17.1 (:issue:`11274`, :issue:`13419`)
  • DataFrame.to_dict() has dropped the outtype parameter in favor of orient (:issue:`13627`, :issue:`8486`)
  • pd.Categorical has dropped setting of the ordered attribute directly in favor of the set_ordered method (:issue:`13671`)
  • pd.Categorical has dropped the levels attribute in favor of categories (:issue:`8376`)
  • DataFrame.to_sql() has dropped the mysql option for the flavor parameter (:issue:`13611`)
  • Panel.shift() has dropped the lags parameter in favor of periods (:issue:`14041`)
  • pd.Index has dropped the diff method in favor of difference (:issue:`13669`)
  • pd.DataFrame has dropped the to_wide method in favor of to_panel (:issue:`14039`)
  • Series.to_csv has dropped the nanRep parameter in favor of na_rep (:issue:`13804`)
  • Series.xs, DataFrame.xs, Panel.xs, Panel.major_xs, and Panel.minor_xs have dropped the copy parameter (:issue:`13781`)
  • str.split has dropped the return_type parameter in favor of expand (:issue:`13701`)
  • Removal of the legacy time rules (offset aliases), deprecated since 0.17.0 (this has been alias since 0.8.0) (:issue:`13590`, :issue:`13868`). Now legacy time rules raises ValueError. For the list of currently supported offsets, see here.
  • The default value for the return_type parameter for DataFrame.plot.box and DataFrame.boxplot changed from None to "axes". These methods will now return a matplotlib axes by default instead of a dictionary of artists. See here (:issue:`6581`).
  • The tquery and uquery functions in the pandas.io.sql module are removed (:issue:`5950`).

Performance Improvements

  • Improved performance of sparse IntIndex.intersect (:issue:`13082`)
  • Improved performance of sparse arithmetic with BlockIndex when the number of blocks are large, though recommended to use IntIndex in such cases (:issue:`13082`)
  • Improved performance of DataFrame.quantile() as it now operates per-block (:issue:`11623`)
  • Improved performance of float64 hash table operations, fixing some very slow indexing and groupby operations in python 3 (:issue:`13166`, :issue:`13334`)
  • Improved performance of DataFrameGroupBy.transform (:issue:`12737`)
  • Improved performance of Index and Series .duplicated (:issue:`10235`)
  • Improved performance of Index.difference (:issue:`12044`)
  • Improved performance of RangeIndex.is_monotonic_increasing and is_monotonic_decreasing (:issue:`13749`)
  • Improved performance of datetime string parsing in DatetimeIndex (:issue:`13692`)
  • Improved performance of hashing Period (:issue:`12817`)
  • Improved performance of factorize of datetime with timezone (:issue:`13750`)
  • Improved performance of by lazily creating indexing hashtables on larger Indexes (:issue:`14266`)
  • Improved performance of groupby.groups (:issue:`14293`)
  • Unecessary materializing of a MultiIndex when introspecting for memory usage (:issue:`14308`)

Bug Fixes

  • Bug in groupby().shift(), which could cause a segfault or corruption in rare circumstances when grouping by columns with missing values (:issue:`13813`)
  • Bug in groupby().cumsum() calculating cumprod when axis=1. (:issue:`13994`)
  • Bug in pd.to_timedelta() in which the errors parameter was not being respected (:issue:`13613`)
  • Bug in io.json.json_normalize(), where non-ascii keys raised an exception (:issue:`13213`)
  • Bug when passing a not-default-indexed Series as xerr or yerr in .plot() (:issue:`11858`)
  • Bug in area plot draws legend incorrectly if subplot is enabled or legend is moved after plot (matplotlib 1.5.0 is required to draw area plot legend properly) (:issue:`9161`, :issue:`13544`)
  • Bug in DataFrame assignment with an object-dtyped Index where the resultant column is mutable to the original object. (:issue:`13522`)
  • Bug in matplotlib AutoDataFormatter; this restores the second scaled formatting and re-adds micro-second scaled formatting (:issue:`13131`)
  • Bug in selection from a HDFStore with a fixed format and start and/or stop specified will now return the selected range (:issue:`8287`)
  • Bug in Categorical.from_codes() where an unhelpful error was raised when an invalid ordered parameter was passed in (:issue:`14058`)
  • Bug in Series construction from a tuple of integers on windows not returning default dtype (int64) (:issue:`13646`)
  • Bug in TimedeltaIndex addition with a Datetime-like object where addition overflow was not being caught (:issue:`14068`)
  • Bug in .groupby(..).resample(..) when the same object is called multiple times (:issue:`13174`)
  • Bug in .to_records() when index name is a unicode string (:issue:`13172`)
  • Bug in calling .memory_usage() on object which doesn’t implement (:issue:`12924`)
  • Regression in Series.quantile with nans (also shows up in .median() and .describe() ); furthermore now names the Series with the quantile (:issue:`13098`, :issue:`13146`)
  • Bug in SeriesGroupBy.transform with datetime values and missing groups (:issue:`13191`)
  • Bug where empty Series were incorrectly coerced in datetime-like numeric operations (:issue:`13844`)
  • Bug in Categorical constructor when passed a Categorical containing datetimes with timezones (:issue:`14190`)
  • Bug in Series.str.extractall() with str index raises ValueError (:issue:`13156`)
  • Bug in Series.str.extractall() with single group and quantifier (:issue:`13382`)
  • Bug in DatetimeIndex and Period subtraction raises ValueError or AttributeError rather than TypeError (:issue:`13078`)
  • Bug in Index and Series created with NaN and NaT mixed data may not have datetime64 dtype (:issue:`13324`)
  • Bug in Index and Series may ignore np.datetime64('nat') and np.timdelta64('nat') to infer dtype (:issue:`13324`)
  • Bug in PeriodIndex and Period subtraction raises AttributeError (:issue:`13071`)
  • Bug in PeriodIndex construction returning a float64 index in some circumstances (:issue:`13067`)
  • Bug in .resample(..) with a PeriodIndex not changing its freq appropriately when empty (:issue:`13067`)
  • Bug in .resample(..) with a PeriodIndex not retaining its type or name with an empty DataFrame appropriately when empty (:issue:`13212`)
  • Bug in groupby(..).apply(..) when the passed function returns scalar values per group (:issue:`13468`).
  • Bug in groupby(..).resample(..) where passing some keywords would raise an exception (:issue:`13235`)
  • Bug in .tz_convert on a tz-aware DateTimeIndex that relied on index being sorted for correct results (:issue:`13306`)
  • Bug in .tz_localize with dateutil.tz.tzlocal may return incorrect result (:issue:`13583`)
  • Bug in DatetimeTZDtype dtype with dateutil.tz.tzlocal cannot be regarded as valid dtype (:issue:`13583`)
  • Bug in pd.read_hdf() where attempting to load an HDF file with a single dataset, that had one or more categorical columns, failed unless the key argument was set to the name of the dataset. (:issue:`13231`)
  • Bug in .rolling() that allowed a negative integer window in contruction of the Rolling() object, but would later fail on aggregation (:issue:`13383`)
  • Bug in Series indexing with tuple-valued data and a numeric index (:issue:`13509`)
  • Bug in printing pd.DataFrame where unusual elements with the object dtype were causing segfaults (:issue:`13717`)
  • Bug in ranking Series which could result in segfaults (:issue:`13445`)
  • Bug in various index types, which did not propagate the name of passed index (:issue:`12309`)
  • Bug in DatetimeIndex, which did not honour the copy=True (:issue:`13205`)
  • Bug in DatetimeIndex.is_normalized returns incorrectly for normalized date_range in case of local timezones (:issue:`13459`)
  • Bug in pd.concat and .append may coerces datetime64 and timedelta to object dtype containing python built-in datetime or timedelta rather than Timestamp or Timedelta (:issue:`13626`)
  • Bug in PeriodIndex.append may raises AttributeError when the result is object dtype (:issue:`13221`)
  • Bug in CategoricalIndex.append may accept normal list (:issue:`13626`)
  • Bug in pd.concat and .append with the same timezone get reset to UTC (:issue:`7795`)
  • Bug in Series and DataFrame .append raises AmbiguousTimeError if data contains datetime near DST boundary (:issue:`13626`)
  • Bug in DataFrame.to_csv() in which float values were being quoted even though quotations were specified for non-numeric values only (:issue:`12922`, :issue:`13259`)
  • Bug in DataFrame.describe() raising ValueError with only boolean columns (:issue:`13898`)
  • Bug in MultiIndex slicing where extra elements were returned when level is non-unique (:issue:`12896`)
  • Bug in .str.replace does not raise TypeError for invalid replacement (:issue:`13438`)
  • Bug in MultiIndex.from_arrays which didn’t check for input array lengths matching (:issue:`13599`)
  • Bug in cartesian_product and MultiIndex.from_product which may raise with empty input arrays (:issue:`12258`)
  • Bug in pd.read_csv() which may cause a segfault or corruption when iterating in large chunks over a stream/file under rare circumstances (:issue:`13703`)
  • Bug in pd.read_csv() which caused errors to be raised when a dictionary containing scalars is passed in for na_values (:issue:`12224`)
  • Bug in pd.read_csv() which caused BOM files to be incorrectly parsed by not ignoring the BOM (:issue:`4793`)
  • Bug in pd.read_csv() with engine='python' which raised errors when a numpy array was passed in for usecols (:issue:`12546`)
  • Bug in pd.read_csv() where the index columns were being incorrectly parsed when parsed as dates with a thousands parameter (:issue:`14066`)
  • Bug in pd.read_csv() with engine='python' in which NaN values weren’t being detected after data was converted to numeric values (:issue:`13314`)
  • Bug in pd.read_csv() in which the nrows argument was not properly validated for both engines (:issue:`10476`)
  • Bug in pd.read_csv() with engine='python' in which infinities of mixed-case forms were not being interpreted properly (:issue:`13274`)
  • Bug in pd.read_csv() with engine='python' in which trailing NaN values were not being parsed (:issue:`13320`)
  • Bug in pd.read_csv() with engine='python' when reading from a tempfile.TemporaryFile on Windows with Python 3 (:issue:`13398`)
  • Bug in pd.read_csv() that prevents usecols kwarg from accepting single-byte unicode strings (:issue:`13219`)
  • Bug in pd.read_csv() that prevents usecols from being an empty set (:issue:`13402`)
  • Bug in pd.read_csv() in the C engine where the NULL character was not being parsed as NULL (:issue:`14012`)
  • Bug in pd.read_csv() with engine='c' in which NULL quotechar was not accepted even though quoting was specified as None (:issue:`13411`)
  • Bug in pd.read_csv() with engine='c' in which fields were not properly cast to float when quoting was specified as non-numeric (:issue:`13411`)
  • Bug in pd.read_csv() in Python 2.x with non-UTF8 encoded, multi-character separated data (:issue:`3404`)
  • Bug in pd.read_csv(), where aliases for utf-xx (e.g. UTF-xx, UTF_xx, utf_xx) raised UnicodeDecodeError (:issue:`13549`)
  • Bug in pd.read_csv, pd.read_table, pd.read_fwf, pd.read_stata and pd.read_sas where files were opened by parsers but not closed if both chunksize and iterator were None. (:issue:`13940`)
  • Bug in StataReader, StataWriter, XportReader and SAS7BDATReader where a file was not properly closed when an error was raised. (:issue:`13940`)
  • Bug in pd.pivot_table() where margins_name is ignored when aggfunc is a list (:issue:`13354`)
  • Bug in pd.Series.str.zfill, center, ljust, rjust, and pad when passing non-integers, did not raise TypeError (:issue:`13598`)
  • Bug in checking for any null objects in a TimedeltaIndex, which always returned True (:issue:`13603`)
  • Bug in Series arithmetic raises TypeError if it contains datetime-like as object dtype (:issue:`13043`)
  • Bug Series.isnull() and Series.notnull() ignore Period('NaT') (:issue:`13737`)
  • Bug Series.fillna() and Series.dropna() don’t affect to Period('NaT') (:issue:`13737`
  • Bug in .fillna(value=np.nan) incorrectly raises KeyError on a category dtyped Series (:issue:`14021`)
  • Bug in extension dtype creation where the created types were not is/identical (:issue:`13285`)
  • Bug in .resample(..) where incorrect warnings were triggered by IPython introspection (:issue:`13618`)
  • Bug in NaT - Period raises AttributeError (:issue:`13071`)
  • Bug in Series comparison may output incorrect result if rhs contains NaT (:issue:`9005`)
  • Bug in Series and Index comparison may output incorrect result if it contains NaT with object dtype (:issue:`13592`)
  • Bug in Period addition raises TypeError if Period is on right hand side (:issue:`13069`)
  • Bug in Peirod and Series or Index comparison raises TypeError (:issue:`13200`)
  • Bug in pd.set_eng_float_format() that would prevent NaN and Inf from formatting (:issue:`11981`)
  • Bug in .unstack with Categorical dtype resets .ordered to True (:issue:`13249`)
  • Clean some compile time warnings in datetime parsing (:issue:`13607`)
  • Bug in factorize raises AmbiguousTimeError if data contains datetime near DST boundary (:issue:`13750`)
  • Bug in .set_index raises AmbiguousTimeError if new index contains DST boundary and multi levels (:issue:`12920`)
  • Bug in .shift raises AmbiguousTimeError if data contains datetime near DST boundary (:issue:`13926`)
  • Bug in pd.read_hdf() returns incorrect result when a DataFrame with a categorical column and a query which doesn’t match any values (:issue:`13792`)
  • Bug in .iloc when indexing with a non lex-sorted MultiIndex (:issue:`13797`)
  • Bug in .loc when indexing with date strings in a reverse sorted DatetimeIndex (:issue:`14316`)
  • Bug in Series comparison operators when dealing with zero dim NumPy arrays (:issue:`13006`)
  • Bug in .combine_first may return incorrect dtype (:issue:`7630`, :issue:`10567`)
  • Bug in groupby where apply returns different result depending on whether first result is None or not (:issue:`12824`)
  • Bug in groupby(..).nth() where the group key is included inconsistently if called after .head()/.tail() (:issue:`12839`)
  • Bug in .to_html, .to_latex and .to_string silently ignore custom datetime formatter passed through the formatters key word (:issue:`10690`)
  • Bug in DataFrame.iterrows(), not yielding a Series subclasse if defined (:issue:`13977`)
  • Bug in pd.to_numeric when errors='coerce' and input contains non-hashable objects (:issue:`13324`)
  • Bug in invalid Timedelta arithmetic and comparison may raise ValueError rather than TypeError (:issue:`13624`)
  • Bug in invalid datetime parsing in to_datetime and DatetimeIndex may raise TypeError rather than ValueError (:issue:`11169`, :issue:`11287`)
  • Bug in Index created with tz-aware Timestamp and mismatched tz option incorrectly coerces timezone (:issue:`13692`)
  • Bug in DatetimeIndex with nanosecond frequency does not include timestamp specified with end (:issue:`13672`)
  • Bug in `Series` when setting a slice with a `np.timedelta64` (:issue:`14155`)
  • Bug in Index raises OutOfBoundsDatetime if datetime exceeds datetime64[ns] bounds, rather than coercing to object dtype (:issue:`13663`)
  • Bug in Index may ignore specified datetime64 or timedelta64 passed as dtype (:issue:`13981`)
  • Bug in RangeIndex can be created without no arguments rather than raises TypeError (:issue:`13793`)
  • Bug in .value_counts() raises OutOfBoundsDatetime if data exceeds datetime64[ns] bounds (:issue:`13663`)
  • Bug in DatetimeIndex may raise OutOfBoundsDatetime if input np.datetime64 has other unit than ns (:issue:`9114`)
  • Bug in Series creation with np.datetime64 which has other unit than ns as object dtype results in incorrect values (:issue:`13876`)
  • Bug in resample with timedelta data where data was casted to float (:issue:`13119`).
  • Bug in pd.isnull() pd.notnull() raise TypeError if input datetime-like has other unit than ns (:issue:`13389`)
  • Bug in pd.merge() may raise TypeError if input datetime-like has other unit than ns (:issue:`13389`)
  • Bug in HDFStore/read_hdf() discarded DatetimeIndex.name if tz was set (:issue:`13884`)
  • Bug in Categorical.remove_unused_categories() changes .codes dtype to platform int (:issue:`13261`)
  • Bug in groupby with as_index=False returns all NaN’s when grouping on multiple columns including a categorical one (:issue:`13204`)
  • Bug in df.groupby(...)[...] where getitem with Int64Index raised an error (:issue:`13731`)
  • Bug in the CSS classes assigned to DataFrame.style for index names. Previously they were assigned "col_heading level<n> col<c>" where n was the number of levels + 1. Now they are assigned "index_name level<n>", where n is the correct level for that MultiIndex.
  • Bug where pd.read_gbq() could throw ImportError: No module named discovery as a result of a naming conflict with another python package called apiclient (:issue:`13454`)
  • Bug in Index.union returns an incorrect result with a named empty index (:issue:`13432`)
  • Bugs in Index.difference and DataFrame.join raise in Python3 when using mixed-integer indexes (:issue:`13432`, :issue:`12814`)
  • Bug in subtract tz-aware datetime.datetime from tz-aware datetime64 series (:issue:`14088`)
  • Bug in .to_excel() when DataFrame contains a MultiIndex which contains a label with a NaN value (:issue:`13511`)
  • Bug in invalid frequency offset string like “D1”, “-2-3H” may not raise ValueError (:issue:`13930`)
  • Bug in concat and groupby for hierarchical frames with RangeIndex levels (:issue:`13542`).
  • Bug in Series.str.contains() for Series containing only NaN values of object dtype (:issue:`14171`)
  • Bug in agg() function on groupby dataframe changes dtype of datetime64[ns] column to float64 (:issue:`12821`)
  • Bug in using NumPy ufunc with PeriodIndex to add or subtract integer raise IncompatibleFrequency. Note that using standard operator like + or - is recommended, because standard operators use more efficient path (:issue:`13980`)
  • Bug in operations on NaT returning float instead of datetime64[ns] (:issue:`12941`)
  • Bug in Series flexible arithmetic methods (like .add()) raises ValueError when axis=None (:issue:`13894`)
  • Bug in DataFrame.to_csv() with MultiIndex columns in which a stray empty line was added (:issue:`6618`)
  • Bug in DatetimeIndex, TimedeltaIndex and PeriodIndex.equals() may return True when input isn’t Index but contains the same values (:issue:`13107`)
  • Bug in assignment against datetime with timezone may not work if it contains datetime near DST boundary (:issue:`14146`)
  • Bug in pd.eval() and HDFStore query truncating long float literals with python 2 (:issue:`14241`)
  • Bug in Index raises KeyError displaying incorrect column when column is not in the df and columns contains duplicate values (:issue:`13822`)
  • Bug in Period and PeriodIndex creating wrong dates when frequency has combined offset aliases (:issue:`13874`)
  • Bug in .to_string() when called with an integer line_width and index=False raises an UnboundLocalError exception because idx referenced before assignment.
  • Bug in eval() where the resolvers argument would not accept a list (:issue:`14095`)
  • Bugs in stack, get_dummies, make_axis_dummies which don’t preserve categorical dtypes in (multi)indexes (:issue:`13854`)
  • PeriodIndex can now accept list and array which contains pd.NaT (:issue:`13430`)
  • Bug in df.groupby where .median() returns arbitrary values if grouped dataframe contains empty bins (:issue:`13629`)
  • Bug in Index.copy() where name parameter was ignored (:issue:`14302`)

安装

对多数用户来说，安装pandas最轻松的方式是安装 Anaconda ，其中包含了pandas。 Anaconda 是一个跨平台 Python 发行版，包含 conda 包和环境管理器，以及许多用于数据分析，数据科学和科学计算的软件包。 对绝大多数用户来说这是推荐的安装方式。

提供了通过源码、 PyPI 、及许多Linux发行版进行安装的说明。 同时也提供了 development 版本 。

支持的Python版本

官方 Python 2.7、3.4 及3.5

安装pandas

立即试用，无需安装！

在最轻松的情况下试用 pandas ，无需关心安装。

Wakari 是一个云端提供 IPython Notebook 的免费服务。

几分钟内就可以简单的创建一个帐号，然后在浏览器内通过 IPython Notebook 使用pandas。

通过Anaconda安装

对经验不足的用户来说，安装pandas以及接下来的`NumPy <http://www.numpy.org/>`__、`SciPy <http://www.scipy.org/>`__相关包是较为困难的事情。

最简单的方法是，不局限于安装pandas，而是包括Python及由最流行包（IPython, NumPy, Matplotlib, ...）组成的`SciPy <http://www.scipy.org/>`__集成包在内的`Anaconda <http://docs.continuum.io/anaconda/>`__。 这是一个用于数据分析及科学计算的跨平台（Linux, Mac OS X, Windows）Python发行版。

After running a simple installer, the user will have access to pandas and the rest of the SciPy stack without needing to install anything else, and without needing to wait for any software to be compiled. 运行一个简单的安装器，用户就可以使用pandas以及`SciPy <http://www.scipy.org/>`__集成包内其他内容，不必再安装任何东西，也不必等待任何软件编译。

Installation instructions for Anaconda can be found here. `Anaconda <http://docs.continuum.io/anaconda/>`__的安装说明可以在`此处 <http://docs.continuum.io/anaconda/install.html>`__找到。

A full list of the packages available as part of the Anaconda distribution can be found here. `Anaconda <http://docs.continuum.io/anaconda/>`__发行版中可用包的完整列表可以在`这里 <http://docs.continuum.io/anaconda/pkg-docs.html>`__找到。

An additional advantage of installing with Anaconda is that you don’t require admin rights to install it, it will install in the user’s home directory, and this also makes it trivial to delete Anaconda at a later date (just delete that folder). 随Anaconda还有无需管理员权限这个额外的好处，它安装在用户的home目录下，将来要删除它也很容易（删除该目录即可）。

Installing pandas with Miniconda 随Minicnoda一起安装pandas ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The previous section outlined how to get pandas installed as part of the Anaconda distribution. However this approach means you will install well over one hundred packages and involves downloading the installer which is a few hundred megabytes in size. 前面的小节概述了如何让pandas作为`Anaconda <http://docs.continuum.io/anaconda/>`__ 发行版的一部分共同安装。 然而这种方式意味着你要安装远多于100个包，并且需要下载100M出头的安装器。

If you want to have more control on which packages, or have a limited internet bandwidth, then installing pandas with Miniconda may be a better solution. 如果你想对这些包有更多的控制权，或者只有带宽有限的网络，那用`Miniconda <http://conda.pydata.org/miniconda.html>`__安装pandas可能是个更好的选择。

Conda is the package manager that the Anaconda distribution is built upon. It is a package manager that is both cross-platform and language agnostic (it can play a similar role to a pip and virtualenv combination). `Conda <http://conda.pydata.org/docs/>`__是`Anaconda <http://docs.continuum.io/anaconda/>`__发行版内置的包管理器。 这是一个跨平台且语言无关的包管理器。（扮演类似于pip及virtualenv二者结合的角色。）

Miniconda allows you to create a minimal self contained Python installation, and then use the Conda command to install additional packages. `Miniconda <http://conda.pydata.org/miniconda.html>`__允许你创建一个自包含的最小Python环境， 然后用`Conda <http://conda.pydata.org/docs/>`__命令安装额外的包。

First you will need Conda to be installed and downloading and running the Miniconda will do this for you. The installer can be found here 首先你需要将`Conda <http://conda.pydata.org/docs/>`__安装好，下载及安装`Miniconda <http://conda.pydata.org/miniconda.html>`__即可。 安装器可以在`这里找到<http://conda.pydata.org/miniconda.html>`__。

The next step is to create a new conda environment (these are analogous to a virtualenv but they also allow you to specify precisely which Python version to install also). Run the following commands from a terminal window:: 下一步是创建一个新的conda环境（跟virtualenv相似，并且允许你精确指定所使用的Python版本）。

conda create -n name_of_my_env python

This will create a minimal environment with only Python installed in it. To put your self inside this environment run:: 这会创建一个仅安装了Python的最小环境。 运行命令进入这个环境：

source activate name_of_my_env

On Windows the command is:: Windows下的命令是：

activate name_of_my_env

The final step required is to install pandas. This can be done with the following command:: 最后的步骤是安装pandas，用如下命令完成：

conda install pandas

To install a specific pandas version:: 安装指定版本的pandas：

conda install pandas=0.13.1

To install other packages, IPython for example:: 安装其他包，例如IPython：

conda install ipython

安装完整的`Anaconda <http://docs.continuum.io/anaconda/>`__发行版：

::
conda install anaconda

如果你需要pip中有而conda中无的包，很简单的安装pip，然后用pip安装这些包：

conda install pip
pip install django

从 PyPI 安装

pandas可以用pip从 PyPI 安装。

pip install pandas

很可能会需要安装一些依赖，包括NumPy，需要一个编译器去编译一些其依赖的代码，并且需要几分钟才能完成。

通过你的Linux发行版的包管理器安装。

如下表格的命令会从你的发行版安装基于Python 2的pandas。 你可能需要使用``python3-pandas``包来安装基于Python 3的pandas。

Distribution Status Download / Repository Link Install method
Debian stable official Debian repository sudo apt-get install python-pandas
Debian & Ubuntu unstable (latest packages) NeuroDebian sudo apt-get install python-pandas
Ubuntu stable official Ubuntu repository sudo apt-get install python-pandas
Ubuntu unstable (daily builds) PythonXY PPA; activate by: sudo add-apt-repository ppa:pythonxy/pythonxy-devel && sudo apt-get update sudo apt-get install python-pandas
OpenSuse stable OpenSuse Repository zypper in  python-pandas
Fedora stable official Fedora repository dnf install python-pandas
Centos/RHEL stable EPEL repository yum install python-pandas

从源码安装

查看:ref:帮助文档<contributing> 来获取完整的，从git源码安装的说明。更多内容，如果你想建立 pandas 开发环境，查看:ref:建立开发环境<contributing.dev_env>。

运行测试套件

当前，pandas配备了一套详尽的、代码覆盖率在98%左右的测试集。 如需在你的机器上运行测试去验证所有的东西都工作正常（以及所有软硬件依赖都已安装），需要确定你已经安装了`nose <http://readthedocs.org/docs/nose/en/latest/>`__，然后运行：

>>> import pandas as pd
>>> pd.test()
Running unit tests for pandas
pandas version 0.18.0
numpy version 1.10.2
pandas is installed in pandas
Python version 2.7.11 |Continuum Analytics, Inc.|
   (default, Dec  6 2015, 18:57:58) [GCC 4.2.1 (Apple Inc. build 5577)]
nose version 1.3.7
..................................................................S......
........S................................................................
.........................................................................

----------------------------------------------------------------------
Ran 9252 tests in 368.339s

OK (SKIP=117)

Dependencies 依赖 ————

  • numexpr: for accelerating certain numerical operations. numexpr uses multiple cores as well as smart chunking and caching to achieve large speedups. If installed, must be Version 2.1 or higher (excluding a buggy 2.4.4). Version 2.4.6 or higher is highly recommended.
  • numexpr：加速一些数学运算。 ``numexpr``利用多个核心，以及智能分块及缓存，来达到性能大幅提高的目的。 版本必须高于2.1（除了有bug的2.4.4），强烈推荐2.4.6及更高版本。
  • bottleneck: for accelerating certain types of nan evaluations. bottleneck uses specialized cython routines to achieve large speedups.
  • bottleneck：加速一些``nan``类型的求值。``bottleneck``使用专门的cython优化来提高性能。

注解

You are highly encouraged to install these libraries, as they provide large speedups, especially if working with large data sets. 强烈推荐你安装这些库，因为它们大幅提高了性能，尤其是工作在大数据集上的时候。

Optional Dependencies 可选的依赖 ~~~~~~~~~~~~~~~~~~~~~

  • Cython: Only necessary to build development version. Version 0.19.1 or higher.

  • Cython：只在构建开发版本时需要，0.19.1或更高版本。

  • SciPy: miscellaneous statistical functions

  • SciPy：各种各样的统计函数。

  • xarray: pandas like handling for > 2 dims, needed for converting Panels to xarray objects. Version 0.7.0 or higher is recommended.

  • PyTables: necessary for HDF5-based storage. Version 3.0.0 or higher required, Version 3.2.1 or higher highly recommended.

  • SQLAlchemy: for SQL database support. Version 0.8.1 or higher recommended. Besides SQLAlchemy, you also need a database specific driver. You can find an overview of supported drivers for each SQL dialect in the SQLAlchemy docs. Some common drivers are:

    • psycopg2: for PostgreSQL
    • pymysql: for MySQL.
    • SQLite: for SQLite, this is included in Python’s standard library by default.

  • matplotlib: for plotting

  • For Excel I/O:

    • xlrd/xlwt: Excel reading (xlrd) and writing (xlwt)
    • openpyxl: openpyxl version 1.6.1 or higher (but lower than 2.0.0), or version 2.2 or higher, for writing .xlsx files (xlrd >= 0.9.0)
    • XlsxWriter: Alternative Excel writer

  • Jinja2: Template engine for conditional HTML formatting.

  • boto: necessary for Amazon S3 access.

  • blosc: for msgpack compression using blosc

  • One of PyQt4, PySide, pygtk, xsel, or xclip: necessary to use read_clipboard(). Most package managers on Linux distributions will have xclip and/or xsel immediately available for installation.

  • Google’s `python-gflags <<https://github.com/google/python-gflags/>`__ , oauth2client , httplib2 and google-api-python-client : Needed for gbq

  • Backports.lzma: Only for Python 2, for writing to and/or reading from an xz compressed DataFrame in CSV; Python 3 support is built into the standard library.

  • One of the following combinations of libraries is needed to use the top-level read_html() function:

    警告

    • if you install BeautifulSoup4 you must install either lxml or html5lib or both. read_html() will not work with only BeautifulSoup4 installed.
    • 如果你需要安装`BeautifulSoup4`_，你必须先安装`lxml`_、`html5lib`_两者其中之一或者全部。
    • You are highly encouraged to read HTML reading gotchas. It explains issues surrounding the installation and usage of the above three libraries
    • 强烈建议你阅读:ref:HTML reading gotchas <html-gotchas>。这份文档解释了安装及使用上面三个库时会遇到的问题。
    • You may need to install an older version of BeautifulSoup4: Versions 4.2.1, 4.1.3 and 4.0.2 have been confirmed for 64 and 32-bit Ubuntu/Debian
    • 你也许需要安装一个版本较老的`BeautifulSoup4`_：在64及32位Ubuntu/Debian上已经确认的是4.2.1，4.1.3及4.0.2。
    • Additionally, if you’re using Anaconda you should definitely read the gotchas about HTML parsing libraries
    • 另外，如果你使用`Anaconda`_，你非常应该看看:ref:the gotchas about HTML parsing libraries <html-gotchas>。

    注解

    • if you’re on a system with apt-get you can do

      sudo apt-get build-dep python-lxml

      to get the necessary dependencies for installation of lxml. This will prevent further headaches down the line.

    • 如果你的操作系统中有``apt-get``，你可以

      sudo apt-get build-dep python-lxml

      获取`lxml`_需要的依赖。这样可以防止一些可能会遇到的编译问题。

注解

如果没有可选依赖，一些有用的特性可能无法工作。因此强烈推荐安装这些依赖。 一个打包好的发行版如 Anaconda ，或 Enthought Canopy <http://enthought.com/products/canopy> 也许值得考虑。

Contributing to pandas

Where to start?

All contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome.

If you are simply looking to start working with the pandas codebase, navigate to the GitHub “issues” tab and start looking through interesting issues. There are a number of issues listed under Docs and Difficulty Novice where you could start out.

Or maybe through using pandas you have an idea of your own or are looking for something in the documentation and thinking ‘this can be improved’...you can do something about it!

Feel free to ask questions on the mailing list or on Gitter.

Bug reports and enhancement requests

Bug reports are an important part of making pandas more stable. Having a complete bug report will allow others to reproduce the bug and provide insight into fixing. Because many versions of pandas are supported, knowing version information will also identify improvements made since previous versions. Trying the bug-producing code out on the master branch is often a worthwhile exercise to confirm the bug still exists. It is also worth searching existing bug reports and pull requests to see if the issue has already been reported and/or fixed.

Bug reports must:

  1. Include a short, self-contained Python snippet reproducing the problem. You can format the code nicely by using GitHub Flavored Markdown:

    ```python
>>> from pandas import DataFrame
>>> df = DataFrame(...)
...
```

  2. Include the full version string of pandas and its dependencies. In versions of pandas after 0.12 you can use a built in function:

    >>> from pandas.util.print_versions import show_versions
>>> show_versions()

    and in pandas 0.13.1 onwards:

    >>> pd.show_versions()

  3. Explain why the current behavior is wrong/not desired and what you expect instead.

The issue will then show up to the pandas community and be open to comments/ideas from others.

Working with the code

Now that you have an issue you want to fix, enhancement to add, or documentation to improve, you need to learn how to work with GitHub and the pandas code base.

Version control, Git, and GitHub

To the new user, working with Git is one of the more daunting aspects of contributing to pandas. It can very quickly become overwhelming, but sticking to the guidelines below will help keep the process straightforward and mostly trouble free. As always, if you are having difficulties please feel free to ask for help.

The code is hosted on GitHub. To contribute you will need to sign up for a free GitHub account. We use Git for version control to allow many people to work together on the project.

Some great resources for learning Git:

Getting started with Git

GitHub has instructions for installing git, setting up your SSH key, and configuring git. All these steps need to be completed before you can work seamlessly between your local repository and GitHub.

Forking

You will need your own fork to work on the code. Go to the pandas project page and hit the Fork button. You will want to clone your fork to your machine:

git clone git@github.com:your-user-name/pandas.git pandas-yourname
cd pandas-yourname
git remote add upstream git://github.com/pydata/pandas.git

This creates the directory pandas-yourname and connects your repository to the upstream (main project) pandas repository.

The testing suite will run automatically on Travis-CI once your pull request is submitted. However, if you wish to run the test suite on a branch prior to submitting the pull request, then Travis-CI needs to be hooked up to your GitHub repository. Instructions for doing so are here.

Creating a branch

You want your master branch to reflect only production-ready code, so create a feature branch for making your changes. For example:

git branch shiny-new-feature
git checkout shiny-new-feature

The above can be simplified to:

git checkout -b shiny-new-feature

This changes your working directory to the shiny-new-feature branch. Keep any changes in this branch specific to one bug or feature so it is clear what the branch brings to pandas. You can have many shiny-new-features and switch in between them using the git checkout command.

To update this branch, you need to retrieve the changes from the master branch:

git fetch upstream
git rebase upstream/master

This will replay your commits on top of the lastest pandas git master. If this leads to merge conflicts, you must resolve these before submitting your pull request. If you have uncommitted changes, you will need to stash them prior to updating. This will effectively store your changes and they can be reapplied after updating.

Creating a development environment

An easy way to create a pandas development environment is as follows.

Tell conda to create a new environment, named pandas_dev, or any other name you would like for this environment, by running:

conda create -n pandas_dev --file ci/requirements_dev.txt

For a python 3 environment:

conda create -n pandas_dev python=3 --file ci/requirements_dev.txt

警告

If you are on Windows, see here for a fully compliant Windows environment.

This will create the new environment, and not touch any of your existing environments, nor any existing python installation. It will install all of the basic dependencies of pandas, as well as the development and testing tools. If you would like to install other dependencies, you can install them as follows:

conda install -n pandas_dev -c pandas pytables scipy

To install all pandas dependencies you can do the following:

conda install -n pandas_dev -c pandas --file ci/requirements_all.txt

To work in this environment, Windows users should activate it as follows:

activate pandas_dev

Mac OSX / Linux users should use:

source activate pandas_dev

You will then see a confirmation message to indicate you are in the new development environment.

To view your environments:

conda info -e

To return to your home root environment in Windows:

deactivate

To return to your home root environment in OSX / Linux:

source deactivate

See the full conda docs here.

At this point you can easily do an in-place install, as detailed in the next section.

Creating a Windows development environment

To build on Windows, you need to have compilers installed to build the extensions. You will need to install the appropriate Visual Studio compilers, VS 2008 for Python 2.7, VS 2010 for 3.4, and VS 2015 for Python 3.5.

For Python 2.7, you can install the mingw compiler which will work equivalently to VS 2008:

conda install -n pandas_dev libpython

or use the Microsoft Visual Studio VC++ compiler for Python. Note that you have to check the x64 box to install the x64 extension building capability as this is not installed by default.

For Python 3.4, you can download and install the Windows 7.1 SDK. Read the references below as there may be various gotchas during the installation.

For Python 3.5, you can download and install the Visual Studio 2015 Community Edition.

Here are some references and blogs:

Making changes

Before making your code changes, it is often necessary to build the code that was just checked out. There are two primary methods of doing this.

  1. The best way to develop pandas is to build the C extensions in-place by running:

    python setup.py build_ext --inplace

    If you startup the Python interpreter in the pandas source directory you will call the built C extensions

  2. Another very common option is to do a develop install of pandas:

    python setup.py develop

    This makes a symbolic link that tells the Python interpreter to import pandas from your development directory. Thus, you can always be using the development version on your system without being inside the clone directory.

Contributing to the documentation

If you’re not the developer type, contributing to the documentation is still of huge value. You don’t even have to be an expert on pandas to do so! Something as simple as rewriting small passages for clarity as you reference the docs is a simple but effective way to contribute. The next person to read that passage will be in your debt!

In fact, there are sections of the docs that are worse off after being written by experts. If something in the docs doesn’t make sense to you, updating the relevant section after you figure it out is a simple way to ensure it will help the next person.

About the pandas documentation

The documentation is written in reStructuredText, which is almost like writing in plain English, and built using Sphinx. The Sphinx Documentation has an excellent introduction to reST. Review the Sphinx docs to perform more complex changes to the documentation as well.

Some other important things to know about the docs:

  • The pandas documentation consists of two parts: the docstrings in the code itself and the docs in this folder pandas/doc/.

    The docstrings provide a clear explanation of the usage of the individual functions, while the documentation in this folder consists of tutorial-like overviews per topic together with some other information (what’s new, installation, etc).

  • The docstrings follow the Numpy Docstring Standard, which is used widely in the Scientific Python community. This standard specifies the format of the different sections of the docstring. See this document for a detailed explanation, or look at some of the existing functions to extend it in a similar manner.

  • The tutorials make heavy use of the ipython directive sphinx extension. This directive lets you put code in the documentation which will be run during the doc build. For example:

    .. ipython:: python

    x = 2
    x**3

    will be rendered as:

    In [1]: x = 2

In [2]: x**3
Out[2]: 8

    Almost all code examples in the docs are run (and the output saved) during the doc build. This approach means that code examples will always be up to date, but it does make the doc building a bit more complex.

注解

The .rst files are used to automatically generate Markdown and HTML versions of the docs. For this reason, please do not edit CONTRIBUTING.md directly, but instead make any changes to doc/source/contributing.rst. Then, to generate CONTRIBUTING.md, use pandoc with the following command:

pandoc doc/source/contributing.rst -t markdown_github > CONTRIBUTING.md

The utility script scripts/api_rst_coverage.py can be used to compare the list of methods documented in doc/source/api.rst (which is used to generate the API Reference page) and the actual public methods. This will identify methods documented in in doc/source/api.rst that are not actually class methods, and existing methods that are not documented in doc/source/api.rst.

How to build the pandas documentation

Requirements

First, you need to have a development environment to be able to build pandas (see the docs on creating a development environment above). Further, to build the docs, there are some extra requirements: you will need to have sphinx and ipython installed. numpydoc is used to parse the docstrings that follow the Numpy Docstring Standard (see above), but you don’t need to install this because a local copy of numpydoc is included in the pandas source code. nbconvert and nbformat are required to build the Jupyter notebooks included in the documentation.

If you have a conda environment named pandas_dev, you can install the extra requirements with:

conda install -n pandas_dev sphinx ipython nbconvert nbformat

Furthermore, it is recommended to have all optional dependencies. installed. This is not strictly necessary, but be aware that you will see some error messages when building the docs. This happens because all the code in the documentation is executed during the doc build, and so code examples using optional dependencies will generate errors. Run pd.show_versions() to get an overview of the installed version of all dependencies.

警告

You need to have sphinx version >= 1.3.2.

Building the documentation

So how do you build the docs? Navigate to your local pandas/doc/ directory in the console and run:

python make.py html

Then you can find the HTML output in the folder pandas/doc/build/html/.

The first time you build the docs, it will take quite a while because it has to run all the code examples and build all the generated docstring pages. In subsequent evocations, sphinx will try to only build the pages that have been modified.

If you want to do a full clean build, do:

python make.py clean
python make.py build

Starting with pandas 0.13.1 you can tell make.py to compile only a single section of the docs, greatly reducing the turn-around time for checking your changes. You will be prompted to delete .rst files that aren’t required. This is okay because the prior versions of these files can be checked out from git. However, you must make sure not to commit the file deletions to your Git repository!

#omit autosummary and API section
python make.py clean
python make.py --no-api

# compile the docs with only a single
# section, that which is in indexing.rst
python make.py clean
python make.py --single indexing

For comparison, a full documentation build may take 10 minutes, a -no-api build may take 3 minutes and a single section may take 15 seconds. Subsequent builds, which only process portions you have changed, will be faster. Open the following file in a web browser to see the full documentation you just built:

pandas/docs/build/html/index.html

And you’ll have the satisfaction of seeing your new and improved documentation!

Building master branch documentation

When pull requests are merged into the pandas master branch, the main parts of the documentation are also built by Travis-CI. These docs are then hosted here.

Contributing to the code base

Code standards

pandas uses the PEP8 standard. There are several tools to ensure you abide by this standard. Here are some of the more common PEP8 issues:

  • we restrict line-length to 80 characters to promote readability
  • passing arguments should have spaces after commas, e.g. foo(arg1, arg2, kw1='bar')

The Travis-CI will run flake8 tool and report any stylistic errors in your code. Generating any warnings will cause the build to fail; thus these are part of the requirements for submitting code to pandas.

It is helpful before submitting code to run this yourself on the diff:

git diff master | flake8 --diff

Furthermore, we’ve written a tool to check that your commits are PEP8 great, pip install pep8radius. Look at PEP8 fixes in your branch vs master with:

pep8radius master --diff

and make these changes with:

pep8radius master --diff --in-place

Additional standards are outlined on the code style wiki page.

Please try to maintain backward compatibility. pandas has lots of users with lots of existing code, so don’t break it if at all possible. If you think breakage is required, clearly state why as part of the pull request. Also, be careful when changing method signatures and add deprecation warnings where needed.

Test-driven development/code writing

pandas is serious about testing and strongly encourages contributors to embrace test-driven development (TDD). This development process “relies on the repetition of a very short development cycle: first the developer writes an (initially failing) automated test case that defines a desired improvement or new function, then produces the minimum amount of code to pass that test.” So, before actually writing any code, you should write your tests. Often the test can be taken from the original GitHub issue. However, it is always worth considering additional use cases and writing corresponding tests.

Adding tests is one of the most common requests after code is pushed to pandas. Therefore, it is worth getting in the habit of writing tests ahead of time so this is never an issue.

Like many packages, pandas uses the Nose testing system and the convenient extensions in numpy.testing.

Writing tests

All tests should go into the tests subdirectory of the specific package. This folder contains many current examples of tests, and we suggest looking to these for inspiration. If your test requires working with files or network connectivity, there is more information on the testing page of the wiki.

The pandas.util.testing module has many special assert functions that make it easier to make statements about whether Series or DataFrame objects are equivalent. The easiest way to verify that your code is correct is to explicitly construct the result you expect, then compare the actual result to the expected correct result:

def test_pivot(self):
    data = {
        'index' : ['A', 'B', 'C', 'C', 'B', 'A'],
        'columns' : ['One', 'One', 'One', 'Two', 'Two', 'Two'],
        'values' : [1., 2., 3., 3., 2., 1.]
    }

    frame = DataFrame(data)
    pivoted = frame.pivot(index='index', columns='columns', values='values')

    expected = DataFrame({
        'One' : {'A' : 1., 'B' : 2., 'C' : 3.},
        'Two' : {'A' : 1., 'B' : 2., 'C' : 3.}
    })

    assert_frame_equal(pivoted, expected)
Running the test suite

The tests can then be run directly inside your Git clone (without having to install pandas) by typing:

nosetests pandas

The tests suite is exhaustive and takes around 20 minutes to run. Often it is worth running only a subset of tests first around your changes before running the entire suite. This is done using one of the following constructs:

  nosetests pandas/tests/[test-module].py
  nosetests pandas/tests/[test-module].py:[TestClass]
  nosetests pandas/tests/[test-module].py:[TestClass].[test_method]

.. versionadded:: 0.18.0

Furthermore one can run

pd.test()

with an imported pandas to run tests similarly.

Running the performance test suite

Performance matters and it is worth considering whether your code has introduced performance regressions. pandas is in the process of migrating to asv benchmarks to enable easy monitoring of the performance of critical pandas operations. These benchmarks are all found in the pandas/asv_bench directory. asv supports both python2 and python3.

注解

The asv benchmark suite was translated from the previous framework, vbench, so many stylistic issues are likely a result of automated transformation of the code.

To use all features of asv, you will need either conda or virtualenv. For more details please check the asv installation webpage.

To install asv:

pip install git+https://github.com/spacetelescope/asv

If you need to run a benchmark, change your directory to asv_bench/ and run:

asv continuous -f 1.1 upstream/master HEAD

You can replace HEAD with the name of the branch you are working on, and report benchmarks that changed by more than 10%. The command uses conda by default for creating the benchmark environments. If you want to use virtualenv instead, write:

asv continuous -f 1.1 -E virtualenv upstream/master HEAD

The -E virtualenv option should be added to all asv commands that run benchmarks. The default value is defined in asv.conf.json.

Running the full test suite can take up to one hour and use up to 3GB of RAM. Usually it is sufficient to paste only a subset of the results into the pull request to show that the committed changes do not cause unexpected performance regressions. You can run specific benchmarks using the -b flag, which takes a regular expression. For example, this will only run tests from a pandas/asv_bench/benchmarks/groupby.py file:

asv continuous -f 1.1 upstream/master HEAD -b ^groupby

If you want to only run a specific group of tests from a file, you can do it using . as a separator. For example:

asv continuous -f 1.1 upstream/master HEAD -b groupby.groupby_agg_builtins

will only run the groupby_agg_builtins benchmark defined in groupby.py.

You can also run the benchmark suite using the version of pandas already installed in your current Python environment. This can be useful if you do not have virtualenv or conda, or are using the setup.py develop approach discussed above; for the in-place build you need to set PYTHONPATH, e.g. PYTHONPATH="$PWD/.." asv [remaining arguments]. You can run benchmarks using an existing Python environment by:

asv run -e -E existing

or, to use a specific Python interpreter,:

asv run -e -E existing:python3.5

This will display stderr from the benchmarks, and use your local python that comes from your $PATH.

Information on how to write a benchmark and how to use asv can be found in the asv documentation.

Running Google BigQuery Integration Tests

You will need to create a Google BigQuery private key in JSON format in order to run Google BigQuery integration tests on your local machine and on Travis-CI. The first step is to create a service account.

Integration tests for pandas.io.gbq are skipped in pull requests because the credentials that are required for running Google BigQuery integration tests are encrypted on Travis-CI and are only accessible from the pydata/pandas repository. The credentials won’t be available on forks of pandas. Here are the steps to run gbq integration tests on a forked repository:

  1. First, complete all the steps in the Encrypting Files Prerequisites section.

  2. Sign into Travis using your GitHub account.

  3. Enable your forked repository of pandas for testing in Travis.

  4. Run the following command from terminal where the current working directory is the ci folder:

    ./travis_encrypt_gbq.sh <gbq-json-credentials-file> <gbq-project-id>

  5. Create a new branch from the branch used in your pull request. Commit the encrypted file called travis_gbq.json.enc as well as the file travis_gbq_config.txt, in an otherwise empty commit. DO NOT commit the *.json file which contains your unencrypted private key.

  6. Your branch should be tested automatically once it is pushed. You can check the status by visiting your Travis branches page which exists at the following location: https://travis-ci.org/your-user-name/pandas/branches . Click on a build job for your branch. Expand the following line in the build log: ci/print_skipped.py /tmp/nosetests.xml . Search for the term test_gbq and confirm that gbq integration tests are not skipped.

Running the vbench performance test suite (phasing out)

Historically, pandas used vbench library to enable easy monitoring of the performance of critical pandas operations. These benchmarks are all found in the pandas/vb_suite directory. vbench currently only works on python2.

To install vbench:

pip install git+https://github.com/pydata/vbench

Vbench also requires sqlalchemy, gitpython, and psutil, which can all be installed using pip. If you need to run a benchmark, change your directory to the pandas root and run:

./test_perf.sh -b master -t HEAD

This will check out the master revision and run the suite on both master and your commit. Running the full test suite can take up to one hour and use up to 3GB of RAM. Usually it is sufficient to paste a subset of the results into the Pull Request to show that the committed changes do not cause unexpected performance regressions.

You can run specific benchmarks using the -r flag, which takes a regular expression.

See the performance testing wiki for information on how to write a benchmark.

Documenting your code

Changes should be reflected in the release notes located in doc/source/whatsnew/vx.y.z.txt. This file contains an ongoing change log for each release. Add an entry to this file to document your fix, enhancement or (unavoidable) breaking change. Make sure to include the GitHub issue number when adding your entry (using `` :issue:`1234` `` where 1234 is the issue/pull request number).

If your code is an enhancement, it is most likely necessary to add usage examples to the existing documentation. This can be done following the section regarding documentation above. Further, to let users know when this feature was added, the versionadded directive is used. The sphinx syntax for that is:

.. versionadded:: 0.17.0

This will put the text New in version 0.17.0 wherever you put the sphinx directive. This should also be put in the docstring when adding a new function or method (example) or a new keyword argument (example).

Contributing your changes to pandas

Committing your code

Keep style fixes to a separate commit to make your pull request more readable.

Once you’ve made changes, you can see them by typing:

git status

If you have created a new file, it is not being tracked by git. Add it by typing:

git add path/to/file-to-be-added.py

Doing ‘git status’ again should give something like:

# On branch shiny-new-feature
#
#       modified:   /relative/path/to/file-you-added.py
#

Finally, commit your changes to your local repository with an explanatory message. Pandas uses a convention for commit message prefixes and layout. Here are some common prefixes along with general guidelines for when to use them:

  • ENH: Enhancement, new functionality
  • BUG: Bug fix
  • DOC: Additions/updates to documentation
  • TST: Additions/updates to tests
  • BLD: Updates to the build process/scripts
  • PERF: Performance improvement
  • CLN: Code cleanup

The following defines how a commit message should be structured. Please reference the relevant GitHub issues in your commit message using GH1234 or #1234. Either style is fine, but the former is generally preferred:

  • a subject line with < 80 chars.
  • One blank line.
  • Optionally, a commit message body.

Now you can commit your changes in your local repository:

git commit -m

Combining commits

If you have multiple commits, you may want to combine them into one commit, often referred to as “squashing” or “rebasing”. This is a common request by package maintainers when submitting a pull request as it maintains a more compact commit history. To rebase your commits:

git rebase -i HEAD~#

Where # is the number of commits you want to combine. Then you can pick the relevant commit message and discard others.

To squash to the master branch do:

git rebase -i master

Use the s option on a commit to squash, meaning to keep the commit messages, or f to fixup, meaning to merge the commit messages.

Then you will need to push the branch (see below) forcefully to replace the current commits with the new ones:

git push origin shiny-new-feature -f

Pushing your changes

When you want your changes to appear publicly on your GitHub page, push your forked feature branch’s commits:

git push origin shiny-new-feature

Here origin is the default name given to your remote repository on GitHub. You can see the remote repositories:

git remote -v

If you added the upstream repository as described above you will see something like:

origin  git@github.com:yourname/pandas.git (fetch)
origin  git@github.com:yourname/pandas.git (push)
upstream        git://github.com/pydata/pandas.git (fetch)
upstream        git://github.com/pydata/pandas.git (push)

Now your code is on GitHub, but it is not yet a part of the pandas project. For that to happen, a pull request needs to be submitted on GitHub.

Review your code

When you’re ready to ask for a code review, file a pull request. Before you do, once again make sure that you have followed all the guidelines outlined in this document regarding code style, tests, performance tests, and documentation. You should also double check your branch changes against the branch it was based on:

  1. Navigate to your repository on GitHub – https://github.com/your-user-name/pandas
  2. Click on Branches
  3. Click on the Compare button for your feature branch
  4. Select the base and compare branches, if necessary. This will be master and shiny-new-feature, respectively.

Finally, make the pull request

If everything looks good, you are ready to make a pull request. A pull request is how code from a local repository becomes available to the GitHub community and can be looked at and eventually merged into the master version. This pull request and its associated changes will eventually be committed to the master branch and available in the next release. To submit a pull request:

  1. Navigate to your repository on GitHub
  2. Click on the Pull Request button
  3. You can then click on Commits and Files Changed to make sure everything looks okay one last time
  4. Write a description of your changes in the Preview Discussion tab
  5. Click Send Pull Request.

This request then goes to the repository maintainers, and they will review the code. If you need to make more changes, you can make them in your branch, push them to GitHub, and the pull request will be automatically updated. Pushing them to GitHub again is done by:

git push -f origin shiny-new-feature

This will automatically update your pull request with the latest code and restart the Travis-CI tests.

If your pull request is related to the pandas.io.gbq module, please see the section on Running Google BigQuery Integration Tests to configure a Google BigQuery service account for your pull request on Travis-CI.

Delete your merged branch (optional)

Once your feature branch is accepted into upstream, you’ll probably want to get rid of the branch. First, merge upstream master into your branch so git knows it is safe to delete your branch:

git fetch upstream
git checkout master
git merge upstream/master

Then you can just do:

git branch -d shiny-new-feature

Make sure you use a lower-case -d, or else git won’t warn you if your feature branch has not actually been merged.

The branch will still exist on GitHub, so to delete it there do:

git push origin --delete shiny-new-feature

常见问题 (FAQ)

DataFrame 的内存使用

从 0.15.0 版本开始，你可以使用 dataframe 的 info 方法来查看 dataframe （包括索引）的内存使用。 这是一个可选属性，你可以通过设置 display.memory_usage (查看 options) 来指定调用 df.info() 方法的时候是否显示内存使用量。

举个例子，以下的 dataframe 的内存使用量在调用 df.info() 方法的时候被显示出来。

加号标志表示实际的内存使用量会更高一些，因为 pandas 无法计算类型为 dtype=object 的列中的内存使用。

0.17.1 新版功能.

输入 memory_usage='deep' 将会启用一个更精确的内存使用报告，它将会计算内部包括对象的全部使用量。 考虑到它的性能占用，使用这个参数前请三思。

默认的显示选项是 True ，但可以在调用 df.info() 时，通过 memory_usage 进行修改。

每一列的内存使用可以调用 memory_usage 方法。它将会翻译一个 Series 包括每一列的列名和以 bytes 表示的内存用量。 对于以上的 dataframe，每一列的内存使用和总体 dataframe 的内存使用都可以通过 memory_usage 方法显示出来：

默认情况下，dataframe 的索引的内存用量也会显示在返回的 Series 中，如不想显示它，可以使用 index=False 参数。

info 方法显示的内存使用量会利用 memory_usage 方法来确定 dataframe 的内存使用量，同时也会格式化输出可读的单位（二进制表示，比如 1KB = 1024 bytes）。

看更多 Categorical Memory Usage。

字节顺序问题

你可以能偶尔的需要处理一些不同字节顺序的机器生成的数据。为了解决这个问题，你需要在将数据转为 Series/DataFrame/Panel 之前使用一些方法，将数据转换基本的 NumPy 数组为你本地系统的字节顺序。 比如下面的几个方法：

查看 `NumPy 字节顺序<http://docs.scipy.org/doc/numpy/user/basics.byteswapping.html>`__ 获取更多信息。

在Qt 程序中可视化数据

目前在 pandas 中没有这样的支持，你可以使用第三方包 pandas-qt 获取这个功能。