This PySpark SQL cheat sheet is your handy companion to Apache Spark DataFrames in Python and includes code samples. You'll probably already know about Apache Spark, the fast, general and open-source engine for big data processing; It has built-in modules for streaming, SQL, machine learning and graph processing.
- Python cheatsheet
Operators¶
Command | Description |
---|---|
| multiplication operation: |
| power operation: |
| matrix multiplication: returns |
Python Dataframe Cheat Sheet For Beginners
Data Types¶
Data Frame function take a data (the values), index (the name of the index column), colu mns (the name of the column) parame ters. Colu mns are series. Take two columns: note the double brackets axis=0 can be omitted, is the default value. Inpl ace =True will apply the result to the original dataframe. This PySpark SQL cheat sheet is your handy companion to Apache Spark DataFrames in Python and includes code samples. You'll probably already know about Apache Spark, the fast, general and open-source engine for big data processing; It has built-in modules for streaming, SQL, machine learning and graph processing.
Command | Description |
---|---|
| Constructs a list containing the objects (a1, a2,..., an). You can append to the list using |
| Constructs a tuple containing the objects (a1, a2,..., an). The (ith) element of (t) can be accessed using |
Built-In Functions¶
Command | Description |
---|---|
|
returns |
| Make an iterator that aggregates elements from each of the iterables. returns |
Iterating¶
Command | Description |
---|---|
| For loop used to perform a sequence of commands (denoted using tabs) for each element in an iterable object such as a list, tuple, or numpy array.An example code is prints |
Comparisons and Logical Operators¶
Command | Description |
---|---|
| Performs code if a condition is met (using tabs). For example squares (x) if (x) is (5), otherwise cubes it. |
User-Defined Functions¶
Command | Description |
---|---|
| Used for create anonymous one line functions of the form: The code after the lambda but before variables specifies the parameters. The code after the colon tells python what object to return. |
| The def command is used to create functions of more than one line: The code immediately following |
Numpy¶
Command | Description |
---|---|
|
|
| Access a the element in numpy array A in with index i1 in dimension 1, i2 in dimension 2, etc.Can use
returns the 2nd column (counting from 0) of A as a 1 dimensional array and
returns the 0th and 1st rows in a 2 dimensional array. |
| Constructs numpy array of shape shape. Here shape is an integer of sequence of integers. Such as 3, (1, 2), (2, 1), or (5, 5). Thus
Constructs an (5times 5) array while
will throw an error. |
| Same as |
| Returns a numpy array with (n) linearly spaced points between (a) and (b). For example
returns |
| Constructs the identity matrix of size (N). For example
returns the (3times 3) identity matrix: [begin{split}left(begin{matrix}1&0&00&1&0 0&0&1end{matrix}right)end{split}] |
|
returns If (a) is a 1 dimensional array then
returns [begin{split}left(begin{matrix}1&00&2end{matrix}right)end{split}] |
| Constructs a numpy array of shape |
| Same as |
| Reverses the dimensions of an array (transpose).For example,if (x = left(begin{matrix} 1& 23&4end{matrix}right)) then |
| Take a sequence of arrays and stack them horizontally to make a single array. For example returns returns (left( begin{matrix} 1&22&3 3&4 end{matrix}right)) |
| Like returns |
| By default
then
returns
returns |
| Same as |
| Performs similar function to np.amax except returns index of maximal element.By default gives index of flattened array, otherwise can use axis to specify dimension.From the example for np.amax returns returns |
| Same as |
| Returns an array equal to the dot product of (a) and (b).For this operation to work the innermost dimension of (a) must be equal to the outermost dimension of (b).If (a) is a ((3, 2)) array and (b) is a ((2)) array then |
numpy.linalg¶
Command | Description |
---|---|
| For a 2-dimensional array (A). returns |
| Returns a 1-dimensional array with all the eigenvalues of $A$ as well as a 2-dimensional array with the eigenvectors as columns.For example,
returns the eigenvalues in |
| Constructs array (x) such that but numerically more stable. |
Pandas¶
Command | Description |
---|---|
pd.Series() | Constructs a Pandas Series Object from some specified data and/or index |
pd.DataFrame() | Constructs a Pandas DataFrame object from some specified data and/or index, column names etc. or alternatively, |
Plotting¶
Command | Description |
---|---|
| The plot command is included in plots the cosine function on the domain (0, 10) with a green line with circles at the points (x, v) |
Are you looking for examples of using Python for data analysis? This article is for you. We will show you how to accomplish the most common data analysis tasks with Python, from the features of Python itself to using modules like Pandas to a simple machine learning example with TensorFlow. Let’s dive in.
A Note About Python Versions
All examples in this cheat sheet use Python 3. We recommend using the latest stable version of Python, for example, Python 3.8. You can check which version you have installed on your machine by running the following command in the system shell:
Sometimes, a development machine will have Python 2 and Python 3 installed side by side. Having two Python versions available is common on macOS. If that is the case for you, you can use the python3 command to run Python 3 even if Python 2 is the default in your environment:
If you don’t have Python 3 installed yet, visit the Python Downloads page for instructions on installing it.
Launch a Python interpreter by running the python3 command in your shell:
Libraries and Imports
The easiest way to install Python modules that are needed for data analysis is to use pip. Installing NumPy and Pandas takes only a few seconds:
Once you’ve installed the modules, use the import statement to make the modules available in your program:
Getting Help With Python Data Analysis Functions
If you get stuck, the built-in Python docs are a great place to check for tips and ways to solve the problem. The Python help() function displays the help article for a method or a class:
The help function uses the system text pagination program, also known as the pager, to display the documentation. Many systems use less as the default text pager, just in case you aren’t familiar with the Vi shortcuts here are the basics:
- j and k navigate up and down line by line.
- / searches for content in a documentation page.
- After pressing / type in the search query, press Enter to go to the first occurrence.
- Press n and N to go forward and back through the search results.
- Ctrl+d and Ctrl+u move the cursor one page down and one page up, respectively.
Another useful place to check out for help articles is the online documentation for Python data analysis modules like Pandas and NumPy. For example, the Pandas user guides cover all the Pandas functionality with explanations and examples.
Basic language features
A quick tour through the Python basics:
There are many more useful string methods in Python, find out more about them in the Python string docs.
Working with data sources
Pandas provides a number of easy-to-use data import methods, including CSV and TSV import, copying from the system clipboard, and reading and writing JSON files. This is sufficient for most Python data analysis tasks:
Find all other Pandas data import functions in the Pandas docs.
Python Dataframe Cheat Sheet Pdf
Working with Pandas Data Frames
Pandas data frames are a great way to explore, clean, tweak, and filter your data sets while doing data analysis in Python. This section covers a few of the things you can do with your Pandas data frames.
Exploring data
Here are a few functions that allow you to easily know more about the data set you are working on:
Statistical operations
All standard statistical operations like minimums, maximums, and custom quantiles are present in Pandas:
Cleaning the Data
It is quite common to have not-a-number (NaN) values in your data set. To be able to operate on a data set with statistical methods, you’ll first need to clean up the data. The fillna and dropna Pandas functions are a convenient way to replace the NaN values with something more representative for your data set, for example, a zero, or to remove the rows with NaN values from the data frame.
Filtering and sorting
Here are some basic commands for filtering and sorting the data in your data frames.
Machine Learning
While machine learning algorithms can be incredibly complex, Python’s popular modules make creating a machine learning program straightforward. Below is an example of a simple ML algorithm that uses Python and its data analysis and machine learning modules, namely NumPy, TensorFlow, Keras, and SciKit-Learn.
In this program, we generate a sample data set with pizza diameters and their respective prices, train the model on this data set, and then use the model to predict the price of a pizza of a diameter that we choose.
Once the model is set up we can use it to predict a result:
Summary
In this article, we’ve taken a look at the basics of using Python for data analysis. For more details on the functionality available in Pandas, visit the Pandas user guides. For more powerful math with NumPy (it can be used together with Pandas), check out the NumPy getting started guide.
To learn more about Python for data analysis, enroll in our Data Analysis Nanodegree program today.