How to Easily Convert Pandas DataFrames to XArray DataSets

Pandas DataFrames serve us well in many applications, but for data with multiple coordinates or many attributes XArray may be a better fit. In this week’s video we explore how to convert a DataFrame to a DataSet using upper air balloon data as an example.

Pandas

Pandas has been a standard since its release in 2008 for dealing with tabular data. We’ve made it work for earth science datasets by either working with sets of DataFrames or complex multi-index DataFrames. While these solutions have solved the problem, a true multi-dimensional data structure has been missing from the Python ecosystem.

XArray

Xarray has proven itself to be a great tool for dealing with highly dimensional data. In planetary and atmospheric science we’ve battled this problem for many years. XArray is based on the idea of dimensions and coordinates. We have another video comparing those if you need a refresher.

Conversion of DataFrames to DataSets

While the conversion from tabular to N-dimensional structures can’t be totally automated, there is a to_xarray method available on DataFrames that is a good start. In this video we use the Siphon library to retrieve an upper air weather balloon sounding. We then convert that data to an XArray DataSet, assign attributes, clean up the data, and get a really useful structure in the end.

If you find these videos and tutorials useful, be sure to thank our friends at Unidata for all of the excellent work they do for the Python and earth science communities. Be sure to checkout the training services we offer in Python and a range of other topics.