Pandas DataFrames support label-based indexing with `.loc` and integer-position-based indexing with `.iloc`. For example, `df.iloc[3]` selects the row at integer position 3, while `df.iloc[3:5, 0:2]` selects rows at positions 3 and 4 and the first two columns. `df.loc[dates[0]]` selects a labeled row, and `df.loc[:, ["A", "B"]]` selects the labeled columns A and B. Boolean conditions can filter data: `df[df["A"] > 0]` returns rows whose value in column A is positive. The related `.at` and `.iat` accessors also provide direct data access. Assignment, such as `df["A"] = 0`, modifies column A in the DataFrame; this should not be described as meaning that every data access is pass-by-reference.

Pandas DataFrame Indexing Accessors and Boolean Filtering

A Pandas DataFrame can be subset by column names, row labels, integer positions, or Boolean conditions. Use `df["column_name"]` to select one column. For explicit two-dimensional selection, `.loc` uses labels and `.iloc` uses integer positions; for example, `df.iloc[3:5, 0:2]` selects rows at positions 3 and 4 and the first two columns. A label slice such as `df["20130102":"20130104"]` selects rows by index labels, while a condition such as `df[df["A"] > 0]` filters rows according to column values.

Google

University of Michigan - Ann Arbor

Pandas provides several tools for tidying data, each covered in its own child node: selecting data (indexing/slicing DataFrames), merging DataFrames with `pandas.concat()`, joining DataFrames on a key with `pandas.merge()` (SQL-style joins), reshaping data (stacking, unstacking, pivoting, melting), handling missing values (`dropna`, `fillna`), and converting between date-time representations.

Tools for Tidying Data

Missing data is important to handle early on in the data processing stage because it can easily affect regressions and other types of analysis at later stages of the pipeline. Pandas provides a variety of tools for dealing with missing values. 

Missing values will typically be stored as a ```np.nan``` and will appear in the printed data frames as ```NaN```. To drop all rows with ```NaN```, you can use the ```df.dropna(how="any")```. This will remove all rows that have at least 1 ```NaN```. 

You can also fill all NA values with the ```df.fillna(value = 5)``` and it will set all ```NaN``` entries to 5 in the data frame. You can also perform these operations on subsets of the data frame for more specific manipulation. 


Missing Data Handling

pandas.concat() combines multiple Series or DataFrame objects along an axis (rows by default), reassembling pieces without joining on a key. For example, splitting a DataFrame into slices and concatenating them reconstructs the original data using each piece's original index: ```python pieces = [df[:3], df[3:7], df[7:]] pd.concat(pieces) ``` This differs from key-based joins performed with pandas.merge() (see "Joining in Pandas").

Merging in Pandas

Other than merging, we can also perform join operations similar to SQL join function calls. 

```python
pd.merge(left, right, on="key")
```

This would join the left and right dat frames based on the variable named "key". 

For more specifics on merging details, such as inner or outer joins, please consult the specific documentation, found here :https://pandas.pydata.org/docs/user_guide/merging.html#merging-join. 

Joining in Pandas

Data doesn't always come in the format that is very useful or helpful for performing analysis and sometimes can be re-organized to make things easier to view. Reshaping can change the relative position of certain information to help draw emphasis between different pieces of data. Consider the following example:
```python
                     A         B
first second                    
bar   one     0.721555 -0.706771
      two    -1.039575  0.271860
baz   one    -0.424972  0.567020
      two     0.276232 -1.087401
```
Using the ```.stack()``` method the data can be modified to look like this
```python
first  second   
bar    one     A   -0.727965
               B   -0.589346
       two     A    0.339969
               B   -0.693205
baz    one     A   -0.339355
               B    0.593616
       two     A    0.884345
               B    1.591431
```
You can also include an argument when using the stack method to specify which columns you want to stack. The same logic and usage applies when using the inverse function ```.unstack()```.

Both ```.stack()``` and ```.unstack()``` will return a sorted copy of the dataframe. 

Reshaping Data

The Pandas library integrates seamlessly with Python's built-in `datetime` module, which is imported via `import datetime`. This integration provides tools to convert strings into datetime objects, enabling date comparisons and time-based sorting essential for time-series analysis. Pandas also facilitates time zone conversions for existing datetime objects; for example, converting a dataset from UTC+5 to UTC+8.

Learn Before

Related

Learn After