ACS Preprocessing

This repository turns raw inline ac-s spectrophotometer recordings into quality-controlled, temperature-, salinity-, and scattering-corrected particulate absorption and attenuation spectra.

The workflow is organized as a sequence of Jupyter notebooks. It was configured for expedition PS113, but the processing parameters and input paths can be adapted to another expedition in config_PS113.ini.

Processing workflow

Run the notebooks in numerical order:

Step	Notebook	Purpose	Main output
0	acs_inline_mod_0.ipynb	Correct invalid 60 minute/second values embedded in filenames and optionally remove undersized files.	Renamed files; selected small files may be deleted after confirmation.
1	acs_inline_mod_1.ipynb	Read Compass .dat files, reconstruct UTC timestamps, extract absorption (a) and attenuation (c), and interpolate c onto the absorption wavelength grid.	*_extracted.txt and wavelength.txt
2	acs_inline_mod_2.ipynb	Correct the discontinuity between the ac-s detector wavelength ranges by shifting the spectra at configured wavelength pairs.	*_extracted_shifted.txt
3	acs_inline_mod_3.ipynb	Detect and mask spectral spikes using configurable slope, modified z-score, maximum-value, and relative-standard-deviation checks.	*_extracted_shifted_despiked.txt and diagnostic figures
4	acs_inline_mod_4.ipynb	Merge files, median-bin the spectra, interpolate underway temperature/salinity data, and apply instrument-specific temperature and salinity corrections.	acs_<freq>_merged.txt and acs_<freq>_merged_tscorr.txt
5	acs_inline_mod_5.ipynb	Estimate particulate absorption (ap) and attenuation (cp) from scheduled filtered-water periods; apply residual-temperature and scattering corrections; perform final quality checks; and calculate absorption line height at 676 nm.	acs_<freq>_merged_tscorr_p_TSalScatCorr.txt and acs_<freq>_merged_tscorr_p_lineheight676.txt
6	acs_inline_mod_6.ipynb	Optionally linearly interpolate the final absorption spectra onto a chosen wavelength grid.	*_interp.csv and an interpolation figure

Required input data

The measurement and calibration files are not included in this repository. Place them in the directory given by dir_acs in the configuration file:

• Raw Compass .dat files. Their names must contain a timestamp in the form YYYYMMDDHHMMSS, normally after the first underscore.
• An underway temperature/salinity CSV with no header and the columns datetime, temperature, salinity.
• The instrument .dev file containing its factory calibration temperature (tcal).
• The instrument-specific temperature and salinity coefficient spreadsheet used by modules 4 and 5.

Module 1 creates wavelength.txt; later modules use it to interpret each headerless spectral file. Those files are structured as:

datetime, a(wavelength 1), ..., a(wavelength N), c(wavelength 1), ..., c(wavelength N)

Environment

Create and activate the supplied Conda environment:

conda env create -f environment.yml
conda activate acs-preproc

If the fully pinned environment cannot be solved on your platform, create the environment from the shorter Conda requirements file instead:

conda create --name acs-preproc -c conda-forge --file requirements.txt
conda activate acs-preproc

JupyterLab itself is not listed in these environment files. Install or launch it from your preferred environment, then select the acs-preproc Python kernel when executing the notebooks.

Configuration

Edit config_PS113.ini before processing:

Section	Controls
mod_0	Minimum file size used by the optional cleanup step.
mod_1	Data directory, source timezone offset relative to UTC, and overwrite behavior.
mod_2	Consecutive wavelengths spanning each detector discontinuity and overwrite behavior.
mod_3	Enabled spike tests and their slope, z-score, maximum-value, and RSD thresholds.
mod_4	Temperature/salinity CSV, device file, coefficient spreadsheet, input suffix, and binning frequency.
mod_5	Hourly filtered-water interval, stability thresholds, attenuation/absorption equality option, and scattering-correction choice.

Paths under mod_4 may be relative to dir_acs, because the notebooks change into that directory before opening the measurement and calibration files. The suffix value selects the files merged by module 4; for the full workflow, it should match the final suffix produced by module 3 (the supplied value despiked does this).

Run the notebooks from the repository root so that the configuration and helper modules can be found. Modules 0 and 6 currently refer to config_PS133.ini, and Datetime_formating.ipynb and gap_ratio.py refer to config_PS143.ini; change those config.read(...) calls to config_PS113.ini or to your own configuration filename before using them. Module 5 also contains a one-off diagnostic cell that reads an absolute /isibhv/... path; skip or edit that cell when running the notebook elsewhere.

Outputs and logs

Processing products are written into dir_acs. Most are headerless comma-separated text files, and optional plots are saved below dir_acs/figures/.

Modules 1–6 initialize log.log in the directory from which the notebook server was launched, and the main processing modules write status messages there. Each module opens that file in write mode, so running the next notebook replaces the previous module's log. Progress bars are also displayed while long-running and multiprocessing steps execute.

Utilities

• rsd_median.py calculates a median-based relative standard deviation used during despiking and filtered-water stability checks.
• spectra_min.py estimates low absorption and attenuation values across a collection of extracted spectra.
• gap_ratio.py plots adjacent-band ratios to help locate a detector discontinuity; edit its configuration filename and example input before running it.
• expedition_shift_wavelength.txt records detector-gap wavelength pairs used for several expeditions and instruments.
• Datetime_formating.ipynb contains examples for converting underway temperature/salinity exports into the three-column CSV expected by module 4; its paths are expedition-specific and must be edited.

License

This project is licensed under the GNU General Public License v3.0. See LICENSE for the full terms.