Customer Challenge:

NOAA Fisheries’ Northwest Fisheries Science Center (NWFSC) produces world class research in many areas and generates large numbers of datasets. A significant part of the challenge in providing data engineering and data management services to the NWFSC is managing the quantity of datasets generated annually. Unlike many other, much publicized engineering tasks within NOAA dealing with Big Data and pipelines designed for enormous throughput, the bulk of the datasets from the NWFSC are small. The challenge lies in maintaining the integrity, accuracy, metadata, updates to, access to, and lifecycle documentation for many, many data sets at once. 

Ensuring that the data that the NWFSC generates is accurately stored, securely backed up, and easily accessible to those who created it and those that want to make use of it is a primary concern for the Science Center.

ERT Solution and Services:

Through a contract with the Scientific Data Management group (SDM) at the NWFSC, ERT has aided in establishing processes and tools to assist or automate those processes, and created a robust framework for the future of scientific data at the NWFSC. Those tools have ranged from dedicated databases with web applications (Oracle and APEX) to python notebooks (Jupyter) and everything in between. ERT’s Developers have proven adept at matching the right tools (scale, need, stability, accessibility, user comfort) for the right job. SDM specializes in bespoke software development for the NWFSC scientific community and beyond.

Data Lifecycle Management: 

Specializing in Data Management, SDM with ERT have developed end-to-end management recommendations for the NWFSC that are in line with government best practices and implementable for the datasets developed at the Science Center. From the planning stages through metadata and data review and publication through InPort and Noaa.Data.Gov, to web-service enabled data hosting and retrieval, to archival submission, SDM can manage the entire lifecycle of data.

ERT developers specialize in bringing legacy systems into the modern, secure world, and maintaining the integrity of data that is moved along with the systems. SDM and ERT have worked on large systems serving the West Coast Region Trawl Rationalization Program, small utilities for automating data processing for individual programs at the Science Center, and everything in between. ERT Data Engineers make data compatible, accessible, and easily useable by a wide range of customers.