STAT 19000: Project 5 — Spring 2022

First, read both datasets into variables named stations and obs. Secondly, take a look at the head of both dataframes. You will notice, the station_id in the obs dataframe appears to correlate with the id column in the stations dataframe. This is a fairly common occurence when data has been normalized for a database. For our current project we will work with a single dataset.

pandas has a merge method that can be used to join two dataframes based on a common column. Here the id column from the stations dataframe matches the station_id column in the obs dataframe. Here is the explanation on merge.

Once merged, you will notice in the new dataframe, dat. That the id column from the obs dataframe is now labeled id_x, and the id column from the stations dataframe is now labeled id_y.

Use the pandas drop method to remove the id_y column. Use the pandas rename method to rename id_x to id, and name to station_name.

Great! We have cleaned up our dataframe so it is easier to work with, while learning a variety of useful pandas methods.

When looking at the new dataset, you may have noticed a mix of letters and numbers in the id column. Below are a few samples of the contents of that column.

The use of numbers and letters in this column are a variation on ksuid — a K-sortable globally unique id. The reason they are beneficial is that Ksuids are sortable by time and unique identifers (there is a minimal chance that any two id’s would be the same). If you are interested you can read more here.

Next, write a function called get_datetime that accepts a ksuid (as a string) and returns the datetime.

The following code should result in the following output.

To verify that the ordering claim is true, (for example,the sorting of ksuids resulted in obervations are in chronological order). We must first, use the sample method to get 10 random id values from the dat dataframe.

Secondly sort the values, then loop through the sorted list of values, and use your get_datetime function to print the datetime.

Can you confirm that sorting the ksuids automatically sorts the observations by datetime?

In this dataset we are given latitude and longitude values in degrees. We want to convert the degrees to radians. Write a function called degrees_to_radians that accepts a latitude or longitude value in degrees, and returns the same value in radians.

The formula to do this is.

$degrees*arctan2(0, -1)/180$

To test out your function you can use:

Write a function that accepts two pandas Series containing a latitude and longitude value. Also needs to be able to return the distance between two points in Kilometers. Call this function get_distance.

You can do this by using the Haversine formula.

$2*r*arcsin(\sqrt{sin^2(\frac{\phi_2 - \phi_1}{2}) + cos(\phi_1)*cos(\phi_2)*sin^2(\frac{\lambda_2 - \lambda_1}{2})})$

Where:

Test your function on the 2 rows with the following id values.

Great! Make sure to note these solutions for future use…​

Next, write a function called plot_stations. plot_stations should accept a dataset as an argument and produce a plot with the station locations plotted on a map.

For consistancy we will use plotly to produce the plot. This stackoverflow post will show some samples. For further understanding here is the explanation for the function.

We want to be careful we don’t plot the same point over and over. To avoid that we want to make sure we reduce the dataset (inside the function), this will plot each pair of latitude and longitude values only once.

Set hover_name to "station_id" so that hovering over a point will displays the station id.

Set scope to "usa" to reduce the map to the USA. Be sure to zoom in on the map so you can see the the stations within Indiana!