The Above Surface Networks (ASN) are an extension of Synoptic with the goal of collecting and sharing surface-based remote sensor observations. Surface-based remote sensors such as sodars, radiometers, and ceilometers are deployed frequently by researchers, public-, and private-sector organizations. The data from those sensors are rarely distributed widely. The ASN aim to change that.
Data Collection
We collect data via a number of techniques, including LDM, direct instrument connection, and FTP transfers. All data are ultimately converted into either compressed ASCII files or some form of hierarchical data format (NetCDF, HDF5, etc) for persistent storage.
We are constantly seeking additional suppliers of surface-based meteorolgical profiles, particularly those which report varaibles which can be used in NWP. If you are interested in participating in the ASN, please contact Synoptic for more information.
Storage
As a form of backup, raw data are stored in a highly-compressed format on a remote server. These files are not accessible to the data system, and can only be used to rebuild the primary data files.
Within the system data are stored in compressed high-speed indexed data files which allow quick reading of profiles from any sensor. The entire ASN station archive is stored in the same format, which means that requesting data from 3 hours ago, and 3 years ago requires the same (small) amount of effort.
Quality Control
A major goal of the ASN system is to provide streamlined QC for a variety of remote sensors, enabling streamlied use of the data in data assimilation and research processes. QC does not alter observed data, but makes note of observations which may contain unreliable data.
Potentially available data checks include:
- Range Checks
- Persistence Checks
Additional tests regarding the physicality of an observed profile or even spatial tests for certain variables in certain areas may be included in the future.
Sharing
The ASN is designed to facilitate collection and sharing of above-surface data from around the globe. We have three primary methods for sharing the data we collect. At this time, all data shared with the ASN is considered public.
Data Downloads
Coming Soon. We will offer the ability to download a relatively unlimited amount of data from any station in JSON format. Possibly others in the future.
MADIS
Any data we receive, which is not already provided to MADIS, will be passed to them, after going through range and persistence QC. Only variables which can be utilized in data assimilation (temperature, wind, moisture) will be sent at this time. This means primarily information from wind profilers and microwave radiometers will be sent.
Visual Tools
To assist researchers, station owners, and data users, the entire archive of data can be visually explored using the visualization interface. This interface combines a fast, on-the-fly tiled image creator, with an API which can (under limited circumstances), be used by external researchers. The main focus of the visualization is to increase the amount of information that can be extracted from the instrument while minimizing the effort to do so.
Continuous usability and functionality improvements are being made to this interface.
Data Decoding
Commercial remote sensors frequently use proprietary or non-standard data encoding formats in their raw data. Because of this, part of the ASN project is to develop and maintain a comprehensive repository of python programs for decoding data files from individual remote sensors. Ultimately this library will be made open-source, for use by the wider community.
You can see and learn about this library, called Pyodec, here http://pyodec.github.io.
We encourage operators of data to contribute their own decoding software (even if they are not in the python programming language), which produce simple array or object structures from encoded data files. The ASN system processes data in a file-based manner, meaning that a decoder is called on a complete data file, and not on individual observations. Once we open-source this project, we will solicit contributions from the community to improve and extend our decoding capabilities.