Skip to content

.details.md

Latest commit

 

History

History
130 lines (88 loc) · 3.51 KB

.details.md

File metadata and controls

130 lines (88 loc) · 3.51 KB
Raw

What it does

For CHP-1

  • Digest raw data
    • Signal from CHP-1
    • Tracker "async"
    • CHP-1 internal temperature from thermistor
  • Bad data ranges flagging
    • From manual set execution ranges
    • From acquisition signal physical limits
  • Converts signal to radiation
    • Computes temperature correction when possible
  • Plots
    • Overview of Clean/Dirty signal
    • Daily signal with and without dark
    • Overview of Direct radiation measurements
    • Daily Direct radiation measurements

For CM-21

  • Digest raw data
    • Signal from CHP-1
  • Bad data ranges flagging
    • From manual set execution ranges
    • From acquisition signal physical limits
  • Converts signal to radiation
  • Plots
    • Overview of Clean/Dirty signal
    • Daily signal with and without dark

For CHP-1 tracker

  • Digest async and step files for later analysis

Other processes

  • Quality Check of radiation data (QCRad)
    • Flags data using mainly the algorithm of C. N. Long and Y. Shi (2006)
  • Clear sky identification (CSid)
    • Flags data as affected by clouds or not with the algorithm of M. J. Reno and C. W. Hansen (2016)
  • Investigate long-term trends (Work in progress)
  • Creates TSI data used in analysis
  • Imports atmospheric pressure data from proxies
  • Keeps an md5sum of all input files to check for bit rot and other data corruption.

For EPPLEY-IR

  • Digest raw data
    • Signal from EPPLEY-IR

For Inclined CM-21

  • Digest raw data
    • Signal from an Inclined CHP-1

For TOT

  • Parse Global radiation data prepared with an external and independent process
    • Read *.TOT files
    • Do some plots and comparisons

Tools

  • inspect_days_duckdb.R interactive plot of some data in the duckdb
  • inspect_days_DB.R interactive plot of some data in the DB
  • inspect_days_Lap.R interactive plot of some data from source files
  • inspect_days_Lap_sirena.R interactive plot of some data from source files

TODO

  • Process more instruments
  • Import libRadtran data
  • Improve CSid algorithm
  • Import other references

Details

Development and Design

Some aspects on the implementation of this project.

  • We use a duckdb database for all measurements and additional data.
  • There are some files with extra meta data for the data in the database and the analysis performed.
  • We use features of the duckdb and arrow library, and also dplyr and data.table for data manipulations.
  • The analysis should be able to be performed with under 8Gb of RAM, but this is not assured.
  • There is a trade-of with the disk usage/wearing, especially when starting from scratch.
  • New data should be easy to be added on daily base on all levels.
  • New process and analysis should be easy to added for all data.
  • This is intended as a framework for all broadband instruments data analysis and processing.

Documentation and usage

There is no centralized documentation for the project. Although you can refer to:

  • Readme.md or other markdown files for a relevant overview
  • Summary notes on the start of each script
  • Comments inside each script
  • Compiled reports from each script
  • Follow the sequence of the scripts in 'execution' folder

Reproducibility

  • We use the renv R package to keep track of the projects dependencies.
  • Maybe will use a nix-shell environment as a more robust and portable method.

Journal

Contains parts of the logbook of the instruments maintenance and other notes.