Python Dataframe Cheat Sheet



Sheet

This PySpark SQL cheat sheet is your handy companion to Apache Spark DataFrames in Python and includes code samples. You'll probably already know about Apache Spark, the fast, general and open-source engine for big data processing; It has built-in modules for streaming, SQL, machine learning and graph processing.

  • Python cheatsheet

Operators¶

Command

Description

*

multiplication operation: 2*3 returns 6

**

power operation: 2**3 returns 8

@

matrix multiplication:

returns

Python Dataframe Cheat Sheet For Beginners

Data Types¶

Data Frame function take a data (the values), index (the name of the index column), colu mns (the name of the column) parame ters. Colu mns are series. Take two columns: note the double brackets axis=0 can be omitted, is the default value. Inpl ace =True will apply the result to the original dataframe. This PySpark SQL cheat sheet is your handy companion to Apache Spark DataFrames in Python and includes code samples. You'll probably already know about Apache Spark, the fast, general and open-source engine for big data processing; It has built-in modules for streaming, SQL, machine learning and graph processing.

Command

Description

l=[a1,a2,,an]

Constructs a list containing the objects (a1, a2,..., an). You can append to the list using l.append().The (ith) element of (l) can be accessed using l[i]

t=(a1,a2,,an)

Constructs a tuple containing the objects (a1, a2,..., an). The (ith) element of (t) can be accessed using t[i]

Built-In Functions¶

Command

Description

len(iterable)

len is a function that takes an iterable, such as a list, tuple or numpy array and returns the number of items in that object.For a numpy array, len returns the length of the outermost dimension

returns 5.

zip

Make an iterator that aggregates elements from each of the iterables.

returns [(1,4),(2,5),(3,6)]

Iterating¶

Command

Description

forainiterable:

For loop used to perform a sequence of commands (denoted using tabs) for each element in an iterable object such as a list, tuple, or numpy array.An example code is

prints [1,4,9]

Comparisons and Logical Operators¶

Command

Description

ifcondition:

Performs code if a condition is met (using tabs). For example

squares (x) if (x) is (5), otherwise cubes it.

Dataframe

User-Defined Functions¶

Command

Description

lambda

Used for create anonymous one line functions of the form:

The code after the lambda but before variables specifies the parameters. The code after the colon tells python what object to return.

def

The def command is used to create functions of more than one line:

The code immediately following def names the function, in this example g .The variables in the parenthesis are the parameters of the function. The remaining lines of the function are denoted by tab indents.The return statement specifies the object to be returned.

Numpy¶

Command

Description

np.array(object,dtype=None)

np.array constructs a numpy array from an object, such as a list or a list of lists.dtype allows you to specify the type of object the array is holding.You will generally note need to specify the dtype.Examples:

A[i1,i2,,in]

Access a the element in numpy array A in with index i1 in dimension 1, i2 in dimension 2, etc.Can use : to access a range of indices, where imin:imax represents all (i) such that (imin leq i < imax).Always returns an object of minimal dimension.For example,

A[:,2]

returns the 2nd column (counting from 0) of A as a 1 dimensional array and

A[0:2,:]

returns the 0th and 1st rows in a 2 dimensional array.

np.zeros(shape)

Constructs numpy array of shape shape. Here shape is an integer of sequence of integers. Such as 3, (1, 2), (2, 1), or (5, 5). Thus

np.zeros((5,5))

Constructs an (5times 5) array while

np.zeros(5,5)

will throw an error.

np.ones(shape)

Same as np.zeros but produces an array of ones

np.linspace(a,b,n)

Returns a numpy array with (n) linearly spaced points between (a) and (b). For example

np.linspace(1,2,10)

returns

np.eye(N)

Constructs the identity matrix of size (N). For example

np.eye(3)

returns the (3times 3) identity matrix:

[begin{split}left(begin{matrix}1&0&00&1&0 0&0&1end{matrix}right)end{split}]

np.diag(a)

np.diag has 2 uses. First if a is a 2 dimensional array then np.diag returns the principle diagonal of the matrix.Thus

np.diag([[1,3],[5,6]])

returns [1,6].

If (a) is a 1 dimensional array then np.diag constructs an array with $a$ as the principle diagonal. Thus,

np.diag([1,2])

returns

[begin{split}left(begin{matrix}1&00&2end{matrix}right)end{split}]

np.random.rand(d0,d1,,dn)

Constructs a numpy array of shape (d0,d1,,dn) filled with random numbers drawn from a uniform distribution between :math`(0, 1)`.For example, np.random.rand(2,3) returns

np.random.randn(d0,d1,,dn)

Same as np.random.rand(d0,d1,,dn) except that it draws from the standard normal distribution (mathcal N(0, 1))rather than the uniform distribution.

A.T

Reverses the dimensions of an array (transpose).For example,if (x = left(begin{matrix} 1& 23&4end{matrix}right)) then x.T returns (left(begin{matrix} 1& 32&4end{matrix}right))

np.hstack(tuple)

Take a sequence of arrays and stack them horizontally to make a single array. For example

returns [1,2,3,2,3,4] while

returns (left( begin{matrix} 1&22&3 3&4 end{matrix}right))

np.vstack(tuple)

Like np.hstack. Takes a sequence of arrays and stack them vertically to make a single array. For example

returns

np.amax(a,axis=None)

By default np.amax(a) finds the maximum of all elements in the array (a).Can specify maximization along a particular dimension with axis.If

a=np.array([[2,1],[3,4]])#createsa2dimarray

then

np.amax(a,axis=0)#maximizationalongrow(dim0)

returns array([3,4]) and

np.amax(a,axis=1)#maximizationalongcolumn(dim1)

returns array([2,4])

np.amin(a,axis=None)

Same as np.amax except returns minimum element.

np.argmax(a,axis=None)

Performs similar function to np.amax except returns index of maximal element.By default gives index of flattened array, otherwise can use axis to specify dimension.From the example for np.amax

returns array([1,1]) and

returns array([0,1])

np.argmin(a,axis=None)

Same as np.argmax except finds minimal index.

np.dot(a,b) or a.dot(b)

Returns an array equal to the dot product of (a) and (b).For this operation to work the innermost dimension of (a) must be equal to the outermost dimension of (b).If (a) is a ((3, 2)) array and (b) is a ((2)) array then np.dot(a,b) is valid.If (b) is a ((1, 2)) array then the operation will return an error.

numpy.linalg¶

Command

Description

np.linalg.inv(A)

For a 2-dimensional array (A). np.linalg.inv returns the inverse of (A).For example, for a ((2, 2)) array (A)

returns

np.linalg.eig(A)

Returns a 1-dimensional array with all the eigenvalues of $A$ as well as a 2-dimensional array with the eigenvectors as columns.For example,

eigvals,eigvecs=np.linalg.eig(A)

returns the eigenvalues in eigvals and the eigenvectors in eigvecs.eigvecs[:,i] is the eigenvector of (A) with eigenvalue of eigval[i].

np.linalg.solve(A,b)

Constructs array (x) such that A.dot(x) is equal to (b). Theoretically should give the same answer as

but numerically more stable.

Pandas¶

Command

Description

pd.Series()

Constructs a Pandas Series Object from some specified data and/or index

pd.DataFrame()

Constructs a Pandas DataFrame object from some specified data and/or index, column names etc.

or alternatively,

Plotting¶

Command

Description

plt.plot(x,y,s=None)

The plot command is included in matplotlib.pyplot.The plot command is used to plot (x) versus (y) where (x) and (y) are iterables of the same length.By default the plot command draws a line, using the (s) argument you can specify type of line and color.For example ‘-‘, ‘- -‘, ‘:’, ‘o’, ‘x’, and ‘-o’ reprent line, dashed line, dotted line, circles, x’s, and circle with line through it respectively.Color can be changed by appending ‘b’, ‘k’, ‘g’ or ‘r’, to get a blue, black, green or red plot respectively.For example,

plots the cosine function on the domain (0, 10) with a green line with circles at the points (x, v)

Are you looking for examples of using Python for data analysis? This article is for you. We will show you how to accomplish the most common data analysis tasks with Python, from the features of Python itself to using modules like Pandas to a simple machine learning example with TensorFlow. Let’s dive in.

A Note About Python Versions

All examples in this cheat sheet use Python 3. We recommend using the latest stable version of Python, for example, Python 3.8. You can check which version you have installed on your machine by running the following command in the system shell:

Sometimes, a development machine will have Python 2 and Python 3 installed side by side. Having two Python versions available is common on macOS. If that is the case for you, you can use the python3 command to run Python 3 even if Python 2 is the default in your environment:
If you don’t have Python 3 installed yet, visit the Python Downloads page for instructions on installing it.

Python


Launch a Python interpreter by running the python3 command in your shell:

Libraries and Imports

The easiest way to install Python modules that are needed for data analysis is to use pip. Installing NumPy and Pandas takes only a few seconds:
Once you’ve installed the modules, use the import statement to make the modules available in your program:

Getting Help With Python Data Analysis Functions

If you get stuck, the built-in Python docs are a great place to check for tips and ways to solve the problem. The Python help() function displays the help article for a method or a class:
The help function uses the system text pagination program, also known as the pager, to display the documentation. Many systems use less as the default text pager, just in case you aren’t familiar with the Vi shortcuts here are the basics:

  • j and k navigate up and down line by line.
  • / searches for content in a documentation page.
    • After pressing / type in the search query, press Enter to go to the first occurrence.
    • Press n and N to go forward and back through the search results.
  • Ctrl+d and Ctrl+u move the cursor one page down and one page up, respectively.

Another useful place to check out for help articles is the online documentation for Python data analysis modules like Pandas and NumPy. For example, the Pandas user guides cover all the Pandas functionality with explanations and examples.

Basic language features

A quick tour through the Python basics:


There are many more useful string methods in Python, find out more about them in the Python string docs.

Working with data sources

Pandas provides a number of easy-to-use data import methods, including CSV and TSV import, copying from the system clipboard, and reading and writing JSON files. This is sufficient for most Python data analysis tasks:
Find all other Pandas data import functions in the Pandas docs.

Python Dataframe Cheat Sheet Pdf

Working with Pandas Data Frames

Pandas data frames are a great way to explore, clean, tweak, and filter your data sets while doing data analysis in Python. This section covers a few of the things you can do with your Pandas data frames.

Exploring data

Here are a few functions that allow you to easily know more about the data set you are working on:

Statistical operations

All standard statistical operations like minimums, maximums, and custom quantiles are present in Pandas:

Cleaning the Data

It is quite common to have not-a-number (NaN) values in your data set. To be able to operate on a data set with statistical methods, you’ll first need to clean up the data. The fillna and dropna Pandas functions are a convenient way to replace the NaN values with something more representative for your data set, for example, a zero, or to remove the rows with NaN values from the data frame.


Filtering and sorting

Here are some basic commands for filtering and sorting the data in your data frames.

Sheet

Machine Learning

While machine learning algorithms can be incredibly complex, Python’s popular modules make creating a machine learning program straightforward. Below is an example of a simple ML algorithm that uses Python and its data analysis and machine learning modules, namely NumPy, TensorFlow, Keras, and SciKit-Learn.

In this program, we generate a sample data set with pizza diameters and their respective prices, train the model on this data set, and then use the model to predict the price of a pizza of a diameter that we choose.

Once the model is set up we can use it to predict a result:

Summary

In this article, we’ve taken a look at the basics of using Python for data analysis. For more details on the functionality available in Pandas, visit the Pandas user guides. For more powerful math with NumPy (it can be used together with Pandas), check out the NumPy getting started guide.


To learn more about Python for data analysis, enroll in our Data Analysis Nanodegree program today.