Computer simulations of the Earth’s climate and weather generate huge amounts of data. These data are often persisted on HPC systems or in the cloud across multiple data assets of a variety of formats (netCDF, zarr, etc…). Finding, investigating, loading these data assets into compute-ready data containers costs time and effort. The data user needs to know what data sets are available, the attributes describing each data set, before loading a specific data set and analyzing it. In this notebook, we demonstrate the integration of data discovery tools such as intake and intake-esm (an intake plugin) with data stored in cloud optimized formats (zarr). We highlight (1) how these tools provide transparent access to local and remote catalogs and data, (2) the API for exploring arbitrary metadata associated with data, loading data sets into data array containers. We also showcase the Pangeo catalog, an open source project to enumerate and organize cloud optimized climate data stored across a variety of providers, and a place where several intake-esm collections are now publicly available. We use one of these public collections as an example to show how an end user would explore and interact with the data, and conclude with a short overview of the catalog’s online presence.

With observational data becoming widely available, researchers struggle to find information enabling assessment for its reliable use. A small first-step toward enabling data quality assessment of observational data is to associate the data with the sensor used to make the observations and to have the sensor description machine-harvestable. In the latest additions to the X-DOMES (Cross-Domain Observational Metadata for Enviromental Sensing) toolset, we have created targeted editors for creating SensorML documents to describe sensor models. The team has adjusted its delivery to enable integration of the X-DOMES content with the GEOCODES (JSON-LD/schema.org) EarthCube project. At our poster-session, we will highlight the new changes and capabilities and demonstrate the use of new X-DOMES tools.

Web 2.0 data delivery and visualization services have improved Earth system science workflows, yet scientists and researchers working with these applications require customized features that are not available on an application running on the browser. Tailoring Argovis’s data throughput so that users can gather data for their myriad tasks requires us to expose the underworkings of our Application Programming Interface (API). We provide a set of functions in a Jupyter notebook for users to retrieve Argo float profiles, platforms, metadata, spatial-temporal selections, and gridded products (including weather events) stored on Argovis. Charts and simple calculations made by the output of these functions provide users the means to write their python scripts. We have bundled the required libraries into a Docker container so that users do not need to install python libraries manually. All software dependencies are installed in the Docker container and run the notebooks within the docker environment. Instructions on how to build and run the container are included. We encourage users to improve, and expand these routines, and even extend them to other languages such as R, Matlab, or Julia, and share their work with us and the community. We welcome community feedback on these tutorial notebooks and are happy to support community-developed software on our platform.

As the size of geoscience datasets grows, scientists are eager to move away from a download-based workflow, where data files are downloaded a local computer for analysis, towards a more cloud-native workflow, where data is loaded on demand over the network. On-demand data loading offers several advantages, including increased reproducibility, provenance tracking, and, potentially, scalability using distributed cloud computing. In this notebook, we demonstrate how to load data on-demand using three different remote data access protocols: - OPeNDAP, the most common, well-established protocol - NetCDF over HTTP, enabled by the h5py library - Zarr over HTTP, a new format optimized for cloud object storage (e.g. Amazon S3) We then conduct a simple benchmarking exercise to explore the throughput and scalability of each service. We use Dask to parallelize reads from each access protocol and calculate the throughput as a function of number of parallel reads. One conclusion is that Zarr over HTTP, coupled with cloud object storage, shows favorable scaling up to hundreds of parallel processes. Finally, we compare the throughput of Zarr over HTTP on a few different clouds, including Google Cloud Storage, Jetstream Cloud, Wasabi Cloud, and Open Storage Network.

The Earthcube Geosemantics Framework (https://ecgs.ncsa.illinois.edu/) developed a prototype of a decentralized framework that combines the Linked Data and RESTful web services to annotate, connect, integrate, and reason about integration of geoscience resources. The framework allows the semantic enrichment of web resources and semantic mediation among heterogeneous geoscience resources, such as models and data. This notebook provides examples on how the Semantic Annotation Service can be used to manage linked controlled vocabularies using JSON Linked Data (JSON-LD), including how to query the built-in RDF graphs for existing linked standard vocabularies based on the Community Surface Dynamics Modeling System (CSDMS), Observations Data Model (ODM2) and Unidata udunits2 vocabularies, how to query build-in crosswalks between CSDMS and ODM2 vocabularies using SKOS, and how to add new linked vocabularies to the service. JSON-LD based definitions pro- vided by these endpoints will be used to annotate sample data available within the IML Critical Zone Observatory data repository using the Clowder Web Service API (https://data.imlczo.org/). By supporting JSON-LD, the Semantic Annotation Service and the Clowder framework provide examples on how portable and semantically defined metadata can be used to better annotate data across repositories and services.

The Scientific Paper of the Future (SPF) concept, initiated by the EarthCube OntoSoft Funded Project, encourages scientists to publish not only peer-reviewed journal articles, but also all associated data, software (data processing scripts), and computational workflows, in order to enable full science reproducibility. While the SPF concept was originally aimed at geoscientists, it can also be applied to interdisciplinary projects such as between ecology, economics, and maritime shipping. Multi-region input-output (MRIO) analysis is a method from economics for analyzing economic interdependencies between different regional entities. Entities can be countries, regions within a country, or groups of countries. MRIO can also be used to analyze other types of interdependencies, such as the environmental impact of one region’s activities on another. For this project, we use MRIO to analyze the global spread of marine non-indigenous species via cargo ships. Over 90% of global trade occurs by maritime shipping. Along with intended cargo, ships provide a means for marine organisms to move to locations beyond their natural ranges, mainly via hull fouling or in ballast tanks. These species can have harmful ecological and economic impacts at their destinations. By using MRIO to follow the imports and exports of commodities between countries, we can deduce the magnitude of seaborne trade connections based on physical volume of commodity traded, and therefore the magnitude and geographic distribution of marine biosecurity risk. MRIO model construction involved incorporating a diversity of data types from ecology, economics, and shipping, and has turned out to be a surprisingly complex endeavor. My poster will demonstrate the principles of an SPF by providing a diagram of the computational workflow involved in the model’s construction, including an explanation for each dataset incorporated into the model’s input parameters and each piece of software written to process the data and assemble and run the model.

The Magnetics Information Consortium (MagIC), hosted at http://earthref.org/MagIC is a database that serves as a Findable, Accessible, Interoperable, Reusable (FAIR) archive for paleomagnetic and rock magnetic data. It has a flexible, comprehensive data model that can accomodate most kinds of paleomagnetic data. The **PmagPy** software package is a cross-platform and open-source set of tools written in Python for the analysis of paleomagnetic data that serves as one interface to MagIC, accommodating various levels of user expertise. It is available through github.com/PmagPy. Because PmagPy requires installation of Python, several non-standard Python modules, and the PmagPy software package, there is a speed bump for many practitioners on beginning to use the software. In order to make the software and MagIC more accessible to the broad spectrum of scientists interested in paleo and rock magnetism, we have prepared a set of Jupyter notebooks, hosted on [jupyterhub.earthref.org](https://jupyterhub.earthref.org) which serve a set of purposes. 1) There is a complete course in Python for Earth Scientists, 2) a set of notebooks that introduce PmagPy (pulling the software package from the github repository) and illustrate how it can be used to create data products and figures for typical papers, and 3) show how to prepare data from the laboratory to upload into the MagIC database. The latter will satisfy expectations from NSF for data archiving and for example the AGU publication data archiving requirements.

An NSF EarthCube-funded project supported a field-based workshop designed to evaluate and refine the sedimentology/stratigraphy portion of the StraboSpot digital data management system. Eleven academics attended the workshop, representing a spectrum of career levels and specialties. The participants teach classes in sedimentology and conduct sedimentary research, but had not used any previous digital mobile apps in the field. The field component focused on learning the basic functionality of the StraboSpot app as a method of collecting digital data in the field. On the first day, teams of 2-3 participants measured a stratigraphic section in a highly visited locality of the well-studied Book Cliffs of central Utah. Teams saw how the vocabulary and spot functionality worked to collect sedimentary field data and to generate stratigraphic columns. The second day was spent measuring a more complex mixed carbonate-clastic sequence in the San Rafael Swell (Utah). Half of the third day was spent in discussion on major issues with workflow/vocabulary and getting feedback on how to simplify and streamline descriptive data collection functions (stratal attributes), and reviewing the more challenging interpretation functions (processes, depositional environments, and architecture). A major discussion point was how best to handle data collection and stratigraphic plotting of ‘interbedded’ intervals. As a result of the workshop, we streamlined workflow options and refined portions of the vocabulary. This field testing followed up on two previous workshops that solicited expert advice to develop the program categories and basic vocabulary for the sedimentary community. Overall, workshop participants were enthusiastic about the potential of digital data systems, and the ability to link annotated photographs and sketches to georeferenced localities. All participants indicated they were inclined to use StraboSpot in both teaching and research, particularly with versatile and customizable options.

The EarthCube BALTO (Brokered Alignment of Long-Tail Observations) project is aimed at developing new cyberinfrastructures that enables brokered access to diverse geoscience datasets. Towards achieving this BALTO objective, we developed a plug-in for the community extensible NSF open-source code ASPECT (Advanced Solver for Problems in Earth’s Convection) that permits ASPECT to read data from the BALTO server (OPeNDAP’s Hyrax open-source data server) over the web. We present a use-case of the BALTO-ASPECT client, which accesses lithospheric structures from the BALTO server to constrain a 3-D lithospheric modulated convection (LMC) modeling and melt generation beneath the Rungwe Volcanic Province (RVP) and the Malawi Rift. We test the hypothesis that at least part of the melt feeding the RVP is generated from LMC. In the model, we assume a rigid lithosphere, while for the asthenosphere we use non-Newtonian, temperature-, pressure- and porosity-dependent creep laws of peridotite. We find that a significant percentage of decompression melt from LMC occurs at a maximum depth of ~200 km beneath the axis of the Malawi Rift, consistent with the location and maximum depth of imaged low velocity zones. At shallower depths (~100 km), the melting region is focused beneath the RVP where there is rapid (~3 cm/yr) upwelling. Our results suggest that asthenospheric upwelling due to LMC is the main source of melt beneath the RVP and might also entrain the plume head materials with reported high 3He/4He values. We, therefore, propose that part of the melt beneath the northern Malawi Rift feeding the RVP can be generated by LMC without necessitating plumes impinging the base of the lithosphere at present. This use-case demonstrates the capability of the BALTO-ASPECT client to accelerate research by brokering input data from the BALTO server for modeling LMC and melt generation.

To incentivize the participation and contribution to the growth of an earth-science-based cyberinfrastructure, analytical environments need to be developed that allow automatic analysis and classification of data from connected data repositories. The purpose of this study is to investigate a machine learning technique for automatically detecting shear-sense-indicating clasts (i.e., sigma or delta clasts and mica fish) in photomicrographs, and finding their shear sense (i.e., sinistral (CCW) or dextral (CW) shearing). Previous work employed transfer learning, a technique in which a pre-trained Convolutional Neural Network (CNN) was repurposed, and artificially augmented image datasets to distinguish between CCW and CW shearing. Preprocessing images by denoising, a process in which noise at different scales is removed while preserving edges of an image, improved classification accuracy. However, upon randomizing the denoising parameters, the CNN model didn’t converge due to severe lack of data. While the efforts for acquiring more labeled data is ongoing, this work compensated for it by implementing a pre-processing “detection” system that automatically crops images to regions of image containing the clasts. This is done by utilizing YOLOv3, a CNN based image detection system that outputs a bounding box around an object of interest. YOLOv3 was trained using 93 photomicrographs containing bounding boxes of 344 shear-sense-indicating clasts. The retrained detector was tested on two sets: set A with 10 photomicrographs containing clasts and set B with 100 photomicrographs not containing clasts. All but one of the clasts in set A were correctly detected with an average confidence score of 96.6%. On set B, 72% of images correctly did not indicate presence of clasts. On the remaining images, where clasts were incorrectly identified, an average confidence score of 78.3% was observed. By utilizing a threshold on the confidence scores, the system could be made more accurate. Future work involves utilizing the bounding boxes output by the detection system to refine and improve the CNN model for classifying shear sense of clasts in photomicrographs.

Magnetics Information Consortium (MagIC), hosted at http://earthref.org/MagIC, is a database that serves as a Findable, Accessible, Interoperable, Reusable (FAIR) archive for paleomagnetic and rock magnetic data. It has a flexible, comprehensive data model that can accomodate most kinds of paleomagnetic data.The PmagPy software package is a cross-platform and open-source set of tools written in Python for the analysis of paleomagnetic data that serves as one interface to MagIC, accommodating various levels of user expertise. It is available through github.com/PmagPy. Because PmagPy requires installation of Python and the software package, there is a speed bump for many practitioners on beginning to use the software. In order to make the software and MagIC more accessible to the broad spectrum of scientists interested in paleo and rock magnetism, we have prepared a set of Jupyter notebooks, hosted on jupyterhub.earthref.org which serve a set of purposes. 1) There is a complete course in Python for Earth Scientists, and 2) a set of notebooks that introduce PmagPy (importing the software package from the github repository). These notebooks illustrate how to conduct statistical analyses, synthesize create data and create data visualizations of the type that are typically included in papers in the field. The notebooks also demonstrate how to prepare data from the laboratory for the MagIC database. This pathway gives additional tools to researchers so that they can satisfy data archiving requirements from NSF and publishers such as AGU.

As more analysis-ready datasets are provided on the cloud, we need to consider how researchers access data. To maximize performance and minimize costs, we move the analysis to the data. This notebook demonstrates a Pangeo deployment connected to multiple Dask Gateways to enable analysis, regardless of where the data is stored. Public clouds are partitioned into regions, a geographic location with a cluster of data centers. A dataset like the National Water Model Short-Range Forecast is provided in a single region of some cloud provider (e.g. AWS’s us-east-1). To analyze that dataset efficiently, we do the analysis in the same region as the dataset. That’s especially true for very large datasets. Making local “dark replicas” of the datasets is slow and expensive. In this notebook we demonstrate a few open source tools to compute “close” to cloud data. We use Intake as a data catalog, to discover the datasets we have available and load them as an xarray Dataset. With xarray, we’re able to write the necessary transformations, filtering, and reductions that compose our analysis. To process the large amounts of data in parallel, we use Dask. Behind the scenes, we’ve configured this Pangeo deployment with multiple Dask Gateways, which provide a secure, multi-tenant server for managing Dask clusters. Each Gateway is provisioned with the necessary permissions to access the data. By placing compute (the Dask workers) in the same region as the dataset, we achieve the highest performance: these worker machines are physically close to the machines storing the data and have the highest bandwidth. We minimize cost by avoiding egress costs: fees charged to the data provider when data leaves a cloud region.

Many ocean and climate models output ocean variables (like velocity, temperature, oxygen concentration etc.) in depth space. Property transport in the ocean generally follows isopycnals, but isopycnals often move up and down in depth space. A small difference in the vertical location of isopycnals between experiments can cause a large apparent difference in ocean properties when the experiments are compared in depth space. As a result, it is often useful to compare ocean variables in density space. This work compares two algorithms for plotting ocean properties in density coordinates, one written in FORTRAN with a python wrapper (xlayers), and one written in python (xarrayutils). Both algorithms conserve total salt content in the vertical, and both algorithms are easily parallelizable to enable plotting large datasets in density coordinates. We apply these algorithms to plot salinity in density space in some of the CMIP-6 models. In general, areas with net precipitation today experience increasing precipitation in higher greenhouse-gas scenarios, and areas with net evaporation today experience a further reduction in net precipitation in higher greenhouse-gas scenarios. By plotting salinity in density space, we visualize how changes in evaporation and precipitation at the surface propagate along isopycnals to influence salinity concentrations in the ocean interior.

ASPECT (Advanced Solver for Problems in Earth’s ConvecTion) is an analysis tool that simulates convection in the Earth’s mantle and other planets. The BALTO project has extended the ASPECT software so it can read data used to perform the simulations from the BALTO brokering server. These additions to the ASPECT codebase allow data to be remotely accessed and then processed as if the data were stored on the user’s local computer. The additions to ASPECT can be split into two distinct sections: a URL reader and a netCDF reader/translator. The URL reader uses the Data Access Protocol (DAP) to access remote data from supported web servers. Data values are transferred from the BALTO broker and converted within the URL reader plugin to match the format that is expected by the rest of the ASPECT code. Similarly, the netCDF plugin reads data stored using netCDF from the BALTO broker and transforms these data into the sph file format required by ASPECT to perform global mantle convection. The netCDF plugin can use either local or remote data. Once the NetCDF data are read, the plugin combines and formats the required variables (longitude, latitude, seismic velocity, and depth). These newly formatted values are then converted into the sph internal representation to be used as spherical harmonic data by ASPECT. The conversion and processing of data all takes place within the ASPECT program. Both plugins have been integrated to allow the user to lookup remote data in a seamless fashion and broaden the types of data that can be requested by the user.

The CF (Climate and Forecast) Conventions are a community-developed metadata standard for storing and describing Earth system science data in the netCDF binary data format. Numerous existing FOSS (Free and Open Source Software) and commercial software tools can explore, analyze, and visualize data that is encoded using the CF Conventions. The CF community holds annual workshops to develop, refine, and review enhancements to the CF Conventions and to manage the CF governance and processes. The EarthCube netCDF-CF project worked with the CF community on the development of extensions to netCDF-CF. Several of these have been accepted into the CF Conventions. Work on these extensions involved broad participation by members of the existing netCDF-CF community as well as members of science domains not traditionally represented in the netCDF-CF community. This presentation will provide an update of recent work and an overview of CF plans and future activities.