Satellite Earth observations (EOs) provide a wealth of vital information about the status of our planet. As satellite missions—and the data they collect—multiply, these observations are becoming increasingly important in spurring scientific advances and guiding decisionmaking across numerous fields, such as ecosystem monitoring, agriculture, forestry, hazards assessment, and climate change analysis. To ensure the widespread and effective application of satellite EO data by users and policymakers, it is crucial to validate these data rigorously to build confidence and trust in them.
Lack of standardization hampers our ability to monitor Earth system processes effectively, despite the ever-increasing amount of satellite Earth observation data at our disposal.
Ideally, satellite EO data should be accompanied by a quantitative and standardized assessment of their quality and suitability for intended applications. This criterion is often unmet, however, because accuracy standards and methodologies used for assessment vary greatly across EO missions. The lack of standardization hampers our ability to monitor Earth system processes effectively, despite the ever-increasing amount of satellite EO data at our disposal.
These issues have spurred many discussions in the research community recently, including at the 2023 Workshop on Land Product Validation and Evolution (LPVE). More than 100 scientists from around the world gathered in Frascati, Italy, to discuss how to standardize and assess the quality and suitability of satellite land data products and thereby build trust and confidence in EO data. Participants at the workshop, which was hosted by the European Space Agency (ESA) under the auspices of the Committee on Earth Observation Satellites Working Group on Calibration and Validation (CEOS WGCV), produce and validate satellite-derived terrestrial essential climate variables (ECVs). These variables include land surface temperature, soil moisture, land cover, fire, albedo, aboveground biomass, and others.
What’s Behind the Validation Problem?
“Validation” in this context refers to the process of independently assessing the quality of satellite-derived EO data [Justice et al., 2000]. In practice, this involves using independent ground-based observations to measure the same biophysical variable—over the same area and duration—inferred from spaceborne sensors. The ground observations serve as reference for the satellite observations, meaning they represent the best estimate of the quantity being analyzed. The statistical agreement between the satellite and reference values yields an uncertainty estimate for the satellite EO data, which establishes the level of confidence in the data needed for informing downstream applications and policymaking.
Although the validation process may seem straightforward, its practical implementation reveals several scientific and technical challenges.
Although the validation process may seem straightforward, its practical implementation reveals several scientific and technical challenges. The process may also be subject to assumptions and user decisions that introduce subjectivity [Loew et al., 2017].
The first fundamental challenge concerns the spatial representativeness of reference data, typically comprising discrete, point-like ground measurements, and their comparability to the satellite EO data, which is integrated over a much larger footprint. This is a crucial issue for measuring terrestrial variables, owing to the inherent spatial heterogeneity and temporal variability of land surface properties. Ideal reference data provide a meaningful estimate of an investigated variable at spatial and temporal scales comparable to the scales of the satellite measurements being validated.
Standard methods for enhancing the comparability of ground and satellite EO data are described in Figure 1. Despite best efforts, inherent disparities in the spatial and temporal scales of satellite and ground measurements give rise to collocation mismatch errors—meaning reference and satellite measurements aren’t perfectly comparable—which can be a significant source of uncertainty in a validation analysis.
The second key challenge is the reliability of reference measurements. Reference data are usually gathered by various means, including field data collection, drone and aircraft campaigns, and permanent instrumentation that’s often installed on towers distributed across networked sites. The underlying assumption is that such ground measurements, collected in closer proximity to the measurable quantity, are more accurate and better characterized than corresponding satellite EOs. This assumption often holds true, but not always.
In many instances, ground-based devices, such as low-cost off-the-shelf portable sensors (e.g., thermal or imaging cameras), may be much less accurate and more loosely characterized than highly sophisticated satellite sensors. Moreover, ground data often record indirect measurements of biophysical variables, such as soil moisture or leaf area index, and methods used to infer the desired variable from the raw observations introduce additional uncertainty. More broadly, field data used for validation are typically collected for various other scientific purposes, rather than specifically for satellite validation, meaning there is significant diversity in the sensors and practices used.
With the objective of addressing these limitations, the Land Product Validation (LPV) subgroup of the CEOS WGCV has for many years been developing and promoting standardized protocols and a common framework for validating and comparing satellite-derived terrestrial ECVs (Figure 2). Likewise, ESA has advocated the concept of Fiducial Reference Measurements. This concept offers a generic template and guidelines for how reference data should be collected in the field and provided with traceable information about their quality, including a full characterization of uncertainty [Niro et al., 2021].
Doing Data Collection Right
The availability of suitable reference data for satellite EO validation was a main topic at the recent LPVE workshop. Current validation data strongly rely on existing ecosystem monitoring networks, mainly the National Ecological Observatory Network in the United States, the Integrated Carbon Observation System in Europe, and the Terrestrial Ecosystem Research Network in Australia.
These continental-scale networks were developed to understand how ecosystems and their components respond to environmental changes. To this purpose, they collect data on many terrestrial properties—such as carbon and nutrient stocks, belowground and aboveground biomass, and plant and soil physical and chemical properties—using tower- and field-based measurements over many distributed sites selected to represent a wide range of ecoclimatic conditions.
Relying solely on existing infrastructure is insufficient because of persistent gaps in the availability of data needed to validate certain types of satellite Earth observation measurements.
Although they produce high-quality data, these networks were not originally designed for satellite validation purposes, and their data collection protocols should be adapted to better match EO validation requirements. Promising preliminary results of efforts to address this limitation were discussed at the workshop, as were remaining challenges, which mostly relate to ensuring spatial representativeness at the scale of satellite pixels and providing thorough uncertainty estimates for ground-based measurements.
However, relying solely on existing infrastructure is insufficient because of persistent gaps in the availability of data needed to validate certain types of satellite EO measurements. Workshop participants shared recent advances that address some of these gaps. For example, a new network of ground-based hyperspectral radiometers will provide reference data for validating satellite-observed surface reflectance, a key quantity used to derive a wide range of terrestrial biophysical variables [Goyens et al., 2021].
Substantial geographic gaps in available reference data also remain due to data scarcity, especially across Africa, Asia, and South America. Progress has been made in the past decade to fill these long-lasting gaps, although numerous constraints, such as limited funding and political barriers, persist. Overcoming these constraints will require greater international collaboration, coordinated by the CEOS WGCV.
Workshop participants also discussed the readiness of current capabilities to support validation needs of future satellite EO missions. Conversations focused mainly on missions designed to provide high spatial resolution (pixel scales of 50 meters) surface temperature data and those equipped with hyperspectral sensors covering the full spectral range, which will provide detailed data on land surface temperature, evapotranspiration, and soil and plant properties.
Historically, validation data and best practices developed by the CEOS LPV subgroup have largely focused on multispectral, coarse-resolution spaceborne sensors. Hence, a review of existing validation practices is needed to ensure they are also relevant to higher spatial and spectral resolution data and to consider what additional EO data could be obtained by fully exploiting the enhanced capabilities of future missions.
The spatial representativeness of reference data was another focal point during the meeting, with discussions covering notable progress in characterizing and quantifying errors arising from spatial mismatches. However, applying current upscaling methodologies to coarse-resolution satellite EO data for certain biophysical variables (e.g., leaf area index) results in uncertainties that fail to meet the rigorous requirements for climate modeling, such as those outlined by the Global Climate Observing System.
Workshop participants focused on minimizing uncertainties in the upscaling process by, for example, directly comparing satellite observations at pixel scale to ground-based data over appropriately selected and spatially uniform sites (i.e., the point-to-pixel approach in Figure 1), as currently recommended within the CEOS LPV best practices protocol for albedo validation. In parallel, discussions between the satellite EO community and the climate modeling community are needed to evaluate uncertainty requirements set for satellite EO data for climate-specific needs versus those of other applications (e.g., agriculture, hazards monitoring).
Accessibility and Usability, Now and in the Future
Ideally, validation data collection should be made publicly available following the FAIR guiding principles for the management and stewardship of scientific data.
The lack of a centralized repository for reference data from field campaigns and networks is a major gap that hinders data discoverability. Ideally, validation data collection should be made publicly available following the FAIR (findability, accessibility, interoperability, reusability) guiding principles for the management and stewardship of scientific data [Wilkinson et al., 2016].
Implementing FAIR principles requires harmonizing multisource data in terms of format, metadata, and content. A notable example of a centralized repository for reference and field data exists at the Oak Ridge National Laboratory Distributed Active Archive Center (ORNL DAAC), one of several FAIR-compliant NASA DAACs. The ORNL DAAC manages and distributes scientific data, primarily those collected by NASA-funded research projects, and provides guidance for data submission and archiving, including on best practices and quality review.
Even with ORNL and other DAACs, a unified access point for the entire validation community is still lacking, and the discoverability and interoperability of campaign data originating from a range of other funding sources and research initiatives are ongoing concerns.
Despite clear advances in satellite EO validation since the last LPVE workshop in 2018, many hurdles remain. It’s not surprising that funding for sustained validation measurements remains the main concern. Data collection field campaigns are costly and labor intensive but valuable for in-depth validation at local scales. Permanent networks and sites with dedicated instrumentation are also expensive, yet they are essential for ensuring temporal and geographical coverage. So their long-term financial sustainability must be secured to meet the validation needs of current and future satellite EO missions.
The main takeaway from the workshop is that we must improve communication about and broaden awareness of the importance and benefits of rigorous satellite EO data validation for the wider end user community, including policymakers and the public. This validation is key for satellite EO data to be used with confidence for a range of scientific and societal applications, from food security to climate mitigation and adaptation.
Acknowledgments
The authors thank all participants of the LPVE 2023 workshop for their precious scientific contributions and insightful comments. The authors express their gratitude to the chairs of the LPVE workshop, most especially Philippe Goryl (ESA), Valentina Boccia (ESA), Steffen Dransfeld (ESA), Marin Tudoroiu (ESA), Eric Vermote (NASA), Cody Anderson (U.S. Geological Survey), Fernando Camacho (EOLab, University of Valencia), Roberto Colombo (University of Milano-Bicocca), and Jadu Dash (University of Southampton), for their support throughout the sessions and for helping to summarize the highlights and recommendations gathered during the meeting.
References
Goyens, C., P. De Vis, and S. E. Hunt (2021), Automated generation of hyperspectral fiducial reference measurements of water and land surface reflectance for the Hypernets networks, in IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 7,920–7,923, Inst. of Electr. and Electron. Eng., New York, https://doi.org/10.1109/IGARSS47720.2021.9553738.
Justice, C., et al. (2000), Developments in the “validation” of satellite sensor products for the study of the land surface, Int. J. Remote Sens., 21(17), 3,383–3,390, https://doi.org/10.1080/014311600750020000.
Loew, A., et al. (2017), Validation practices for satellite‐based Earth observation data across communities, Rev. Geophys., 55(3), 779–817, https://doi.org/10.1002/2017RG000562.
Niro, F., et al. (2021), European Space Agency (ESA) calibration/validation strategy for optical land-imaging satellites and pathway towards interoperability, Remote Sens., 13(15), 3003, https://doi.org/10.3390/rs13153003.
Wilkinson, M. D., et al. (2016), The FAIR Guiding Principles for scientific data management and stewardship, Sci. Data, 3, 160018, https://doi.org/10.1038/sdata.2016.18.
Author Information
Fabrizio Niro (fabrizio.niro@ext.esa.int), Serco for European Space Agency, European Space Research Institute (ESA/ESRIN), Directorate of Earth Observation Programmes, Frascati, Italy; Michael Cosh, Hydrology and Remote Sensing Laboratory, U.S. Department of Agriculture, Beltsville, Md.; and Jaime Nickeson, Science Systems and Applications Inc., Science and Exploration Directorate, NASA Goddard Space Flight Center, Greenbelt, Md.