% Proposed definition of the SPIF raw instrument data format
\documentclass[12pt,a4paper]{article}
%\usepackage[none]{FAAM_StdDoc}
\usepackage[T1]{fontenc}
\usepackage{helvet,mathptmx,graphics}
\usepackage[pdftex,colorlinks=true]{hyperref}
\usepackage{ifthen,calc}
\usepackage[usenames,dvipsnames]{color}
\usepackage{fancyhdr}
\usepackage{graphics}
\usepackage{booktabs,longtable}

% Set longtable margins to textwidth
\setlength\LTleft{0pt}
\setlength\LTright{0pt}
\setlength{\LTcapwidth}{\textwidth}

% Limit the depth of the table of contents to subsections
%\setcounter{tocdepth}{2}

% Set manually fixed text width
\newlength{\texttablecolwidth}
\setlength{\texttablecolwidth}{11cm}


% Define list for variables for netCDF files
\newcommand{\varlabel}[4][None]%
    {\footnotesize\textit{#2}~\texttt{#3}#4\ifthenelse{\equal{#1}{None}}%
                                           {\rule{0pt}{1ex}}%
                                           {~=~#1;}\normalsize}

% Define a max width of varitem. If varitem is greater than varitemwidth
% then put item text on next line.
\newlength{\varitemwidth}
\setlength{\varitemwidth}{3.5cm}
\newlength{\vartextwidth}
\setlength{\vartextwidth}{0.5\textwidth}

\newlength{\Mylen}
\newcommand{\varlistlabel}[1]%
    {\settowidth{\Mylen}{#1}%
     \ifthenelse{\lengthtest{\Mylen>\labelwidth}}%
                % if label > labelwidth
                {\parbox[b]{\labelwidth}%
                {\makebox[0pt][l]{#1}
                 \mbox{}\\}}%
                % else if label < labelwidth
                {#1}%
     \hfil\relax}

\newenvironment{varlist}[4][None]%
% Three args are value [none], type, name, (dim)
% (dim) is a bit of a cludge. If none then give {}
    {\vspace{2ex}
     \noindent
     \varlabel[#1]{#2}{#3}{#4}
     \begin{list}{}{%
        \footnotesize%
        \renewcommand{\makelabel}[1]{\varlistlabel{\quad \texttt{#3:##1}~=}}%
        % Vertical seperations
        \setlength{\parsep}{0.5ex}%
        \setlength{\partopsep}{0.5ex}%
        \setlength{\topsep}{1.5ex}%
        \setlength{\itemsep}{0.25ex}%
        % Horizontal seperations
        \setlength{\itemindent}{0cm}%
        \setlength{\listparindent}{0cm}%
        \setlength{\labelwidth}{\varitemwidth}%
        \setlength{\labelsep}{0em}%
        \setlength{\leftmargin}{\labelwidth+\labelsep}%
        \setlength{\rightmargin}{0cm}}}%
    {\normalsize\end{list}}



\begin{document}

\title{SPIF--Single Particle Image Format}
\author{The SPIF Working Group}
\date{March 2021\\v0.86}
%\docURL{Link to webpage when written}
%\docrelated{FAAM000002}

\maketitle
\begin{abstract}
This document defines the Single Particle Image Format (SPIF) file structure and and variables. The SPIF file is a NetCDF file and a universal file format for single particle image probe data.
\end{abstract}
\tableofcontents
%\pagebreak


\section{Motivation}
A variety of file formats are used by particle imaging instrument manufacturers. The formats are dictated by hardware and bandwidth limitations, and so may not have a convenient or conventional structure. However, after data has been acquired, and as hardware improves, the imposed limitations vanish. We propose a new file format for these imaging probes. The new format will specify a single structure which all imaging probe data formats may be mapped onto, leading to improved accessibility for users and refinement of data processing routines. We aim for this format to be adopted by instrument manufacturers. The format is also suitable for representing historical datasets and is suitable for long term archiving of raw data. File converters for some of the most commonly used probes are provided.


\section{Description}
The SPIF file format uses the NetCDF4 format. NetCDF4 is a structured binary file format capable of containing large datasets and has automatic compression utilities. NetCDF4 is widely supported on a variety of platforms and environments.
\par
Datasets will be contained in groups within a SPIF file. Only data from a single instrument (or instrument channel) may be stored in a given group although SPIF files can contain data from multiple instruments through the use of multiple groups.
Metadata will be provided for each instrument to provide information on data sources etc.

\section{SPIF File Definition}
\label{sec:FileDefinition}

\begin{description}
\item[root] The root of the file contains global attributes.
\item[instrument group] Each instrument data included within the file has a separate group.
\begin{description}
\item[root] Instrument metadata
\item[core] The SPIF-Core group contains the data necessary for subsequent analysis
\item[aux] The SPIF-Aux group contains additional data which is generated by a given instrument, but is not essential for data processing. This data is included to maintain integrity of the original dataset, making SPIF a suitable format for long term archiving.
\item[lvlX] Processed data may be included as a subgroup of either \texttt{core} or \texttt{aux} or as individual variables within the same group but with an appropriate \texttt{level} attribute.
\end{description}
\end{description}

\pagebreak
\subsection{SPIF Root}

The root of the SPIF file has all of the global data and attributes relating to the production of the file.
\\[2em]
\varlabel[``SPIF--Single Particle Image Format'']{char}{title}{}%\marginpar{\scriptsize$\leftarrow$I'm actually not keen on this name, though the acronym is cool, as it limits the scope of the format.}
\\
\varlabel[Institution that produced the file]{char}{institution}{}\\
\varlabel[Conventions being used -- SPIF-1.0, if following this document]{char}{version}{}\\
\varlabel[Link to web, paper, document reference describing datasets, project, etc]{char}{references}{}\\
\varlabel[creation/modification datetime, version number]{char}{history}{}\\
\varlabel[datetime, comment, changelog(?)]{char}{comment}{}\\
\\
\varlabel[Name of project]{char}{project}{}\\
\varlabel[Reference date of all datasets]{char}{start\_date}{}\\
\varlabel[Conventions used in file; e.g., SPIF-0.86]{char}{conventions}{}
%\marginpar{\scriptsize$\leftarrow$Will all the datasets have a common start date and project? If notmove these to instrumet group roots.}\\

\subsection{Instrument Group}

The root of the instrument group contains information pertaining to the individual instrument. It may also include information about the probe software module that was used to create it, this may be different to the version of the overall processing code.
\par
Some probes will have multiple channels, for example the 2DS has two orthogonal arms, with each detecting particles independently, essentially behaving as independent instruments. Since the primary coordinate applies only to particle sequences from a single instrument, for a probe that is made up of multiple instruments (e.g., 2DS, 3V-CPI, etc) where the particle detection will not be coincident, data is split into different instrument groups. These groups should be named using the core instrument name, followed by the channel name, separated by a dash. So for a 2DS instrument, the Instrument Groups would be named '2DS-H' and '2DS-V'.
\par
Metadata that should be defined in the Instrument group include:
\\
\\
\varlabel[Short name of instrument]{char}{instrument\_name}{}\\
\varlabel[Full descriptive name of instrument]{char}{instrument\_long\_name}{}\\
\varlabel[Instrument channel (if applicable)]{char}{instrument\_channel}{}\\
\varlabel[Institution operating instrument]{char}{institution}{}\\
\varlabel[Link to web, paper, document reference describing instrument]{char}{reference}{}\\
\varlabel[Serial number or instrument identifier]{char}{instrument\_serial\_number}{}\\
\varlabel[Manufacturer of instrument]{char}{manufacturer}{}\\
\varlabel[Instrument firmware version]{char}{instrument\_firmware}{}\\
\varlabel[Instrument software version]{char}{instrument\_software}{}\\
\varlabel[Name or description of platform instrument is mounted on]{char}{platform}{}\\
\varlabel[Raw data filename(s) used to generate the current instrument dataset]{array of char}{raw\_filenames}{}\\
\varlabel[Name and version of code used to generate SPIF output]{char}{spif\_processor}{}\\
\varlabel[Any further notes about instrument, platform, location, orientation, etc]{char}{comment}{}\\
\par
Although envisaged to be used exclusively with optical array probes, there is no reason that this format could not be used for any instrument type. Therefore there may not be suitable universal descriptive attributes for all instrument types. It may thus be desirable that these attributes are placed in a sub-group although this may add a layer of complexity for little gain. Below are variables which may be used for an optical array probe.

\vbox{
\begin{varlist}{int}{pixels}{()}
\item[long\_name] Numbers of pixels in \texttt{instrument};
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{resolution}{(scalar or 2)}
\item[long\_name] Physical resolution of array pixels \texttt{instrument};
\item[units] ``micrometer'';
\item[ancillary\_variables] \texttt{instrument:resolution\_err};
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{resolution\_err}{(scalar or 2)}
\item[long\_name] Uncertainty of physical resolution of array pixels \texttt{instrument};
\item[units] \texttt{instrument:resolution};
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{arm\_separation}{}
\item[long\_name] Physical distance between probe arms;
\item[units] ``millimeter'';
\item[ancillary\_variables] \texttt{instrument:arm\_separation\_err};
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{arm\_separation\_err}{}
\item[long\_name] Uncertainty of physical distance between probe arms;
\item[units] \texttt{instrument:arm\_separation};
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{boolean}{antishatter\_tips}{}
\item[long\_name] Use of antishatter-, or Korolev-, tips on probe arms;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{int}{bpp}{}
\item[long\_name] Bits per pixel used in image data;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{int}{wavelength}{}
\item[long\_name] Laser wavelength of instrument;
\item[units] ``nm'';
\item[comment] Any relevant comments;
\end{varlist}
}



\subsection{Instrument Core}

The Instrument Core group contains data necessary for analysis and processing of the raw data. There is a single coordinate for all variables in this group. This means that some of the arrays are larger than would otherwise be the case, but it also means that all the arrays are the same length which makes for more efficient storage, chunking, and retrieval. The coordinate is $images$, the length of \texttt{image\_ns} defines the length of the coordinate dimension, where the arrival time of the $n$th image is given by \texttt{image\_ns}($n$) + \texttt{image\_sec}($n$).
\par
There are two additional dimensions defined for the image data -- $slices$ and $array$. The $slices$ dimension corresponds to the image dimension in the direction of flight, and is set as the maximum number of slices encountered (or allowed) by the instrument. The $array$ dimension corresponds to the image dimension along the diode array, and is set as the number of diodes of the instrument.
\par
The instrument core data group consists of:

\vbox{
\begin{varlist}{float}{image\_ns}{($images$)}
\item[long\_name] ``image arrival time in nanoseconds'';
\item[units] ns since \texttt{image\_sec};
\item[ancillary\_variables] \texttt{image\_sec};
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{int}{image\_sec}{($images$)}
\item[standard\_name]``time'';
\item[long\_name] ``image arrival time in seconds'';
\item[timezone] ``UTC'';
\item[units] ``seconds since \texttt{start\_date} 00:00:00 +0000'';
\item[strftime\_format] ``\%F \%T \%z'';
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{int}{image\_len}{($images$)}
\item[long\_name] ``image event length in number of slices'';
\item[units] ``number of slices'';
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{uint8}{image}{($images$, $slices$, $array$)}
\item[long\_name] ``Image array'';
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{int}{buffer\_index}{($images$)}
\item[long\_name] ``index of data buffer from which image was extracted'';
\item[units] ``buffer number'';
\item[comment] Buffer information can be found in the Instrument Aux section;
\end{varlist}
}

\vbox{
\begin{varlist}{boolean}{overload}{($images$)}
\item[long\_name] ``Probe particle overload status'';
\item[comment] Any relevant comments;
\end{varlist}
}

In some cases, image buffers may contain pieces of information about each image in addition to the time, length, and image, as covered above. For example, DMT monoscale instruments include an image counter and depth of field (DOF) flag for each image; likewise, DMT greyscale instruments include an image counter and TAS for each image. In these cases where the additional information encoded in the data stream apply to single images, these parameters may be included in the Instrument Core section as additional parameters.




\subsection{Instrument Aux}

The Instrument Aux group contains auxiliary data relevant to a given instrument. This data is included to maintain integrity of the original dataset, making SPIF a suitable format for long term archiving. This group has its own time coordinate; this accommodates 1~Hz one dimensional data that may be transmitted in parallel to the two dimensional image data.

In this group, there are three parameters with defined names covering the aux timestamp and true airspeed (TAS). It is important to include TAS as a defined name, since this parameter is required to calculate sample volume for use in L2 derived microphysical parameters such as number concentration, liquid water content, etc.

\vbox{
\begin{varlist}{int}{time}{($time$)}
\item[standard\_name] ``time'';
\item[long\_name] ``aux time in seconds'';
\item[timezone] ``UTC'';
\item[units] ``seconds since \texttt{start\_date} 00:00:00 +0000'';
\item[strftime\_format] ``\%F \%T \%Z'';
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{int}{TAS\_original}{($time$)}
\item[long\_name] ``TAS used during original data acquisition'';
\item[units] ``m/s'';
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{int}{TAS\_corrected}{($time$)}
\item[long\_name] ``Corrected TAS'';
\item[units] ``m/s'';
\item[comment] Any relevant comments;
\end{varlist}
}

Note there are two TAS parameters defined. $TAS\_original$ represents the TAS used during original acquisition of the data. If this TAS is accurate, and no corrections are required, this is the only TAS that needs to be supplied, and will be used in all subsequent calculations. However, if post-flight corrections were required to the TAS data, these should be input to the $TAS\_corrected$ parameter. If present, $TAS\_corrected$ will be used in all subsequent calculations. The ratio of corrected to original TAS values will also be used to recalculate particle diameters, to account for any stretching or contraction of images that occurred due to incorrect probe sample rates.

If desired, other parameters can be added to the Aux group, and may include:

\begin{itemize}
\item Housekeeping data
\item Buffer time stamps
\item Image counters
\item Data acquisition timing words
\item Temperature
\item Altitude
\item Standard PADS aggregated data
\end{itemize}

\textbf{TODO: Add description of 'data discovery' terms.}

\subsection{Level 0 Processed Data}

Following extraction of image data into SPIF format, images can be analyzed to extract information about the particles they contain. At the most basic level, parameters of interest describe geometric and physical measurements of the identified particles. Thus, the Level 0 data contains basic information about identified particles such as:

\begin{itemize}
\item Diameters (more discussion on this below)
\item Area
\item Perimeter
\item Bounding box within image
\item Orientation
\item Right edge pixel count
\item Left edge pixel count
\item Center-in (boolean)
\item All-in (boolean)
\end{itemize}

Interpretation of particle diameter presents a challenge, as there are currently several definitions of particle diameter in use by the community, and a standard definition likely isn't reasonable, since different diameters are useful depending on the measurement scenario. Thus, to make SPIF useful to the broader community, it should include a wide set of diameters in use by the community.

An additional consideration for the inclusion of various particle diameters is how these diameters are named. Throughout the literature, varying names have been given to essentially identical diameters. In the diameter definitions here, an attempt will be made to standardize the names, while referencing other names used for a given diameter definition.

Note that the Level 0 particles will be sized using number of pixels -- conversion to sizing in microns takes place in Level 1. To differentiate measurements made in the different coordinate systems, sizing measurements in reference to number of pixels will have the prefix $N$, whereas measurements with units of microns will have the prefix $D$.

\begin{itemize}
\item [$N_p$] Max diameter in the photodiode-array dimension. Equivalent to $N_y$ (Korelev et al. 2000, Leroy 2016) and $L_5$ (Lawson 2011).
\item[$N_t$] Max diameter in the time dimension. Equivalent to $N_x$ (Korolev et al. 2000, Leroy 2016) and $L_1$ (Lawson 2011).
\item[$N_{eq}$] Diameter of circle with area equivalent to particle area.
\item[$N_s$] Diameter of minimum enclosing circle. Equivalent to $N_{max}$ (Heymsfield et al. 2013).
\item[$N_h$] Hypotenuse of triangle formed by $N_p$ and $N_t$.
\item[$N_m$] Mean of $N_p$ and $N_t$.
\item[$N_{slice\_count}$] Diameter in slice with maximum number of shaded pixels. Equivalent to $L_2$ (Lawson 2011).
\item[$N_{slice\_diff}$] Diameter in slice with greatest pixel separation. Equivalent to $L_4$ (Lawson 2011).
\item[$N_{reconst}$] Reconstructed circle diameter for center-in particles.
\item[$N_{hole}$] Max hole diameter as defined in Korolev 2007.
\end{itemize}

Each of the parameters discussed above applies to individual particles. For most OAPs, there can be multiple particles in a single image. Given this n-to-one relationship, the Level 0 particle data will require use of a new dimension corresponding to the number of particles detected, which is likely to be different than the number of images captured. The $particles$ dimension thus covers all parameters described in this section. With the additional dimension, there is a need for supplemental parameters which describe the relationship of detected particles to their original image, both in terms of a reference to the additional image, as well as a more exact temporal location, based on the particle's location in the image frame.

\begin{itemize}
\item[image\_index] Reference to source image index.
\item[particle\_sec] Particle arrival time in seconds.
\item[particle\_ns] Particle arrival time in nanoseconds.
\end{itemize}

With these factors in mind, the Level 0 data should be implemented as described below:

\vbox{
\begin{varlist}{int}{image\_index}{($particles$)}
\item[long\_name] ``reference to index of image containing current particle'';
\item[units] ``image number'';
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{particle\_ns}{($particles$)}
\item[long\_name] ``particle arrival time in nanoseconds'';
\item[units] ns since \texttt{particle\_sec};
\item[ancillary\_variables] \texttt{particle\_sec};
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{int}{particle\_sec}{($particles$)}
\item[standard\_name] ``time'';
\item[long\_name] ``image arrival time in seconds'';
\item[timezone] ``UTC'';
\item[units] ``seconds since \texttt{start\_date} 00:00:00 +0000'';
\item[strftime\_format] ``\%F \%T \%z'';
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{N\_p}{($particles$)}
\item[long\_name] ``Max diameter in the photodiode-array dimension'';
\item[equivalent\_names] ``N\_y, L5''
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{N\_t}{($particles$)}
\item[long\_name] ``Max diameter in the time dimension'';
\item[equivalent\_names] ``N\_x, L1''
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{N\_eq}{($particles$)}
\item[long\_name] ``Diameter of circle with area equivalent to particle area'';
\item[equivalent\_names]
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{N\_s}{($particles$)}
\item[long\_name] ``Diameter of minimum enclosing circle'';
\item[equivalent\_names] ``N\_max''
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{N\_h}{($particles$)}
\item[long\_name] ``Hypotenuse of triangle formed by N\_p and N\_t'';
\item[equivalent\_names]
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{N\_m}{($particles$)}
\item[long\_name] ``Mean of N\_p and N\_t'';
\item[equivalent\_names]
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{N\_slice\_count}{($particles$)}
\item[long\_name] ``Diameter of slice with maximum number of shaded pixels'';
\item[equivalent\_names] ``L2''
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{N\_slice\_diff}{($particles$)}
\item[long\_name] ``Diameter of slice with greatest pixel separation'';
\item[equivalent\_names] ``L4''
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{N\_reconst}{($particles$)}
\item[long\_name] ``Reconstructed circle diameter for center-in particles'';
\item[equivalent\_names]
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{N\_hole}{($particles$)}
\item[long\_name] ``Max hole diameter'';
\item[equivalent\_names]
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}


\vbox{
\begin{varlist}{float}{area}{($particles$)}
\item[long\_name] ``Number of shaded pixels'';
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{perimeter}{($particles$)}
\item[long\_name] ``Number of pixels around perimeter of particle'';
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{series of int}{bbox}{($particles$, $coords$)}
\item[long\_name] ``Bounding box of particle'';
\item[units] p0, t0, pN, tN pixel coordinates;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{l\_edge\_count}{($particles$)}
\item[long\_name] ``Number of shaded pixels along left (0th) photodiode boundary for current particle'';
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{r\_edge\_count}{($particles$)}
\item[long\_name] ``Number of shaded pixels along right (Nth) photodiode boundary for current particle'';
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{boolean}{center\_in}{($particles$)}
\item[long\_name] ``Flag indicating if particle is center-in'';
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{boolean}{all\_in}{($particles$)}
\item[long\_name] ``Flag indicating if particle is all-in'';
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{ellipse\_maj\_axis}{($particles$)}
\item[long\_name] ``The length of the major axis of the ellipse fit to particle'';
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{ellipse\_min\_axis}{($particles$)}
\item[long\_name] ``The length of the minor axis of the ellipse fit to particle'';
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{ellipse\_angle}{($particles$)}
\item[long\_name] ``Orientation of particle fit to ellipse'';
\item[units] radians;
\item[comment] ``Given in radians with range from -pi/2 to pi/2. Reference is parallel to photodiode array.'';
\end{varlist}
}

\vbox{
\begin{varlist}{float}{fit\_maj\_axis}{($particles$)}
\item[long\_name] ``The length of particle along the least-squares fit line orientation'';
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{fit\_min\_axis}{($particles$)}
\item[long\_name] ``The length of particle perpendicular to the least-squares fit line orientation'';
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{fit\_angle}{($particles$)}
\item[long\_name] ``Orientation of particle fit to line using least-squares technique'';
\item[units] radians;
\item[comment] ``Given in radians with range from -pi/2 to pi/2. Reference is parallel to photodiode array.'';
\end{varlist}
}

\vbox{
\begin{varlist}{float}{fit\_r2}{($particles$)}
\item[long\_name] ``R squared value from least-squares line fit'';
\item[units] unitless;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{F}{($particles$)}
\item[long\_name] ``Shape factor of particle.'';
\item[units] unitless;
\item[comment] ``As defined in Holroyd 1986.'';
\end{varlist}
}

\vbox{
\begin{varlist}{float}{S}{($particles$)}
\item[long\_name] ``Ratio of fully shaded slices to total slices'';
\item[units] unitless;
\item[comment] ``As defined in Holroyd 1986.'';
\end{varlist}
}

\subsection{Level 1 Processed Data}

Whereas Level 0 data presents particle information as simply properties of an image, Level 1 contains particle properties linked to physical, real-world quantities. In Level 1, there are two primary categories of data:

\begin{enumerate}
\item Particle properties scaled to physical dimensions ($\mu m$, etc.) using the resolution of the instrument
\item Parameters classifying particles into habits or other categories
\end{enumerate}

When generating scaled particle properties, care must be taken to correct for improper scaling in the image time direction due to inconsistencies between the probe sampling rate and the speed of the aircraft. These inconsistencies can happen for various reasons -- the most common reasons include exceeding TAS limits of the probe; having incorrect or constant airspeed inputs supplied to the probe; or problems with local pitot measurements due to icing, blockages, or other reasons. In these cases, the image scaling along the time direction can be corrected by scaling it by the correct TAS divided by the original TAS.

Scaled particle properties should include the following parameters:

\vbox{
\begin{varlist}{int}{image\_index}{($particles$)}
\item[long\_name] ``reference to index of image containing current particle'';
\item[units] ``image number'';
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{D\_p}{($particles$)}
\item[long\_name] ``Max diameter in the photodiode-array dimension'';
\item[equivalent\_names] ``D\_y, L5''
\item[units] microns;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{D\_t}{($particles$)}
\item[long\_name] ``Max diameter in the time dimension'';
\item[equivalent\_names] ``D\_x, L1''
\item[units] microns;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{D\_eq}{($particles$)}
\item[long\_name] ``Diameter of circle with area equivalent to particle area'';
\item[equivalent\_names]
\item[units] microns;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{D\_s}{($particles$)}
\item[long\_name] ``Diameter of minimum enclosing circle'';
\item[equivalent\_names] ``D\_max''
\item[units] microns;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{D\_h}{($particles$)}
\item[long\_name] ``Hypotenuse of triangle formed by D\_p and D\_t'';
\item[equivalent\_names]
\item[units] microns;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{D\_m}{($particles$)}
\item[long\_name] ``Mean of D\_p and D\_t'';
\item[equivalent\_names]
\item[units] microns;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{D\_slice\_count}{($particles$)}
\item[long\_name] ``Diameter of slice with maximum number of shaded pixels'';
\item[equivalent\_names] ``L2''
\item[units] microns;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{D\_slice\_diff}{($particles$)}
\item[long\_name] ``Diameter of slice with greatest pixel separation'';
\item[equivalent\_names] ``L4''
\item[units] microns;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{D\_reconst}{($particles$)}
\item[long\_name] ``Reconstructed circle diameter for center-in particles'';
\item[equivalent\_names]
\item[units] microns;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{D\_korolev}{($particles$)}
\item[long\_name] ``Korolev-corrected diameter'';
\item[equivalent\_names]
\item[refererence] ``Korolev, 2007''
\item[units] microns;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{N\_korolev}{($particles$)}
\item[long\_name] ``Korolev-corrected diameter'';
\item[equivalent\_names]
\item[refererence] ``Korolev, 2007''
\item[units] pixels;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{ellipse\_maj\_axis}{($particles$)}
\item[long\_name] ``The length of the major axis of the ellipse fit to particle'';
\item[units] microns;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{ellipse\_min\_axis}{($particles$)}
\item[long\_name] ``The length of the minor axis of the ellipse fit to particle'';
\item[units] microns;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{fit\_maj\_axis}{($particles$)}
\item[long\_name] ``The length of particle along the least-squares fit line orientation'';
\item[units] microns;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{fit\_min\_axis}{($particles$)}
\item[long\_name] ``The length of particle perpendicular to the least-squares fit line orientation'';
\item[units] microns;
\item[comment] Any relevant comments;
\end{varlist}
}

\vbox{
\begin{varlist}{float}{area}{($particles$)}
\item[long\_name] ``Area of shaded pixels'';
\item[units] microns$^2$;
\item[comment] Any relevant comments;
\end{varlist}
}

The second component of Level 1 data contains parameters used to classify particles. Examples of such classification parameters could include results from the Holroyd habit classification technique, or the Korolev habit classification technique. There are no set requirements regarding which classification methods are included, but any included parameters should follow the structure outlined below.

\vbox{
\begin{varlist}{float}{habit}{($particles$)}
\item[long\_name] ``Name of habit classification technique'';
\item[key] parameter1: char1, parameter2: char2, etc;
\item[comment] Any relevant comments;
\end{varlist}
}

Where the $key$ attribute uses keyword: value pairs to specify the classification types and their corresponding character code for the given parameter.

In cases where classifications have auxiliary parameters necessary for classification (e.g., the $R$ parameter in the Korolev classification scheme), these should also be included in this section. These parameters should be named in such a way where they can be easily identified, and should include a long description and units parameter, as appropriate.


\section{Level 2 Processed Data}

Level 2 Processed Data contains binned histograms and calculated bulk microphysical parameters calculated using the Level 1 per-particle data using a fixed time base. These parameters are either one-dimensional (bulk parameters) or two-dimensional (binned histograms), where a reference to time is the principle dimension in both cases. Thus, all parameters in Level 2 use $Time$ as their primary dimension, calculated using a fixed time base, and encompassing all particle detection timestamps in the file. In the case of binned data, the second dimension is $Bins$.






%\subsection{Level \textit{n} processed data}
%
%It may be desirable to include some processed data along with the unadulterated raw image data. These variables should include an \texttt{level} attribute to indicate the appropriate processing level applied;
%\begin{description}
%\item[level 0] Raw, or pixel, units\marginpar{\scriptsize$\leftarrow$Should all unadulterated raw data, eg \texttt{particle\_image} include a \texttt{level} of 00?}
%\item[level 1] Geophysical units
%\item[level 2] Bulk properties in geophysical units?
%\end{description}
%\par
%The variables could be included in a subgroup of the Instrument core or aux groups. Where it is placed will depend on the appropriate coordinate time. For example, level 0 data could be placed within the core group as these variables will use the same nanosecond and seconds time coordinates as the raw single particle data. Level 2 data such as total concentration and median volume diameter will use the same time coordinates as probe air speed data so would probably be put in a subgroup of the aux group.
%\par
%For monoscale image probes level zero processed data may include:
%\begin{itemize}
%\item Single particle properties
%\item Area
%\item Filled Area
%\item Perimeter
%\item Position on Array
%\item Num Edge0 Pixels On
%\item Num Edge1 Pixels On
%\item Dmax, Dx, Dy etc etc
%\item Number of particles in image event
%\item Corrections to dead pixels?
%\end{itemize}
%%
%%
\section{SPIF Code Definitions}

\textit{Will complete this section when code is more final.}
%Code that produces the SPIF files is written in python~3 using the h5py module to interface with the hdf5 library. The structural layout of the code is shown in figure~\ref{fig:SoftwareStructure}.
%\begin{figure}
%\begin{center}
%\resizebox{0.5\linewidth}{!}{\includegraphics{images/SPIF_program_structure}}
%\caption{Functional diagram of possible software structure. Using self-contained layers and algorithm functions should more easily allow code additions and modification. See text for more details.\label{fig:SoftwareStructure}}
%\end{center}
%\end{figure}
%The code is organised in different layers, the lowest level being the raw data translator shown in figure~\ref{fig:SoftwareStructure} in green. These code functions read in the raw data from the probe and put it into variables that are easily digested by the hdf5 read/write layer, down in pink.
%\par
%The h5py module accepts python dictionaries and numpy variables and iterables. Eventually this section will describe the details of the expected outputs of the data translators however at this stage it should be reasonably obvious from the hdf5 structures given in section~\ref{sec:FileDefinition}. Each hdf5 group will be a dictionary, with each variable being a subdictionary with the attributes of these variables being either native python or numpy variables types.

\subsection{Data Chunking}

It is quite possible for a translator to read in an entire raw data file for a single probe, parse that into the appropriate (and possibly large) python variables, and pass the resultant dictionary to the NetCDF read/writer module for writing into the SPIF file. This would result in on-the-fly chunking and filtering by the NetCDF library. It is also possible that the raw data files are too numerous or large to be handled all together and so subsets of the data need to be streamed to the read/write module piecemeal. This will also occur if the SPIF file is being written in realtime. This will mean that chunking and filtering may have to be set manually. Thus there may be information passed from the translator to the read/write module that is not stored in the SPIF file but is used to construct the SPIF file.



% Changelog
% V1.0 First release
% V1.1 Added L0 parameters. Renamed Particle references in Core to Image.


\label{LastPage}
\end{document}
