optmat.datasets#

Curated datasets: literature values with their provenance and conditions.

A source is one paper (or one compilation, theory table or database); its record, data/sources/<source>.toml, holds the citation, where the values are in the paper library, how they were measured or computed, and one entry per dataset: a set of values of one quantity for one material. A dataset’s values are either extracted from a library table into data/values/<family>/<material>/<source>__<dataset>.csv (the family is the quantity’s; optmat.vocabulary) by that source’s script under tools/extract/, or, when they exist only in the text, listed in the record with their page.

The files keep the values as printed. Everything that turns them into physics happens here, once:

units and printed scale factors are applied;
wavelengths (and wavenumbers) in standard air become vacuum wavelengths;
corrections of printed values (errata) are applied, each checked against the value it corrects;
quantities that determine the polarizability (a refractivity at a stated state, a polarizability, a molar refractivity) are converted to the per-particle polarizability through Lorentz-Lorenz with the number density the source meant.

Quantities that constrain dispersion only indirectly (phase-matching density ratios, group delays, oscillator strengths, Cauchy moments) are loaded and kept, but have no polarizability of their own; the fitting layer gives them forward models.

Module Attributes

`DATA_DIRECTORY`	The curated data shipped with the package.
`VALUES_DIRECTORY`	The extracted data files, `<family>/<material>/<source>__<dataset>.csv`.
`NOT_STATED`	The abscissa cell of a wavelength the source does not give.

Functions

`data_file`(quantity, file)	Return where a dataset's file lives.
`datasets`([material, quantity])	Return the curated datasets, optionally for one material or one quantity.
`load_source`(path, *[, defer])	Load one source record and its datasets.
`parse_state`(spec, where)	Parse a dataset's state.
`sources`()	Return every curated source, by id.

Classes

`Dataset`(source, id, material, quantity, ...)	Values of one quantity for one material from one source, as printed.
`Erratum`(dataset, row, column, printed, ...)	A correction of one printed value, with its justification.
`Source`(id, citation, doi, library, identity, ...)	One source of values: its citation, provenance and datasets.
`Uncertainty`([relative, absolute, column, ...])	How the uncertainty of a dataset's values is stated.

optmat.datasets.DATA_DIRECTORY: Final = PosixPath('/home/docs/checkouts/optmat/checkouts/7/src/optmat/data')#: The curated data shipped with the package.

class optmat.datasets.Dataset(source, id, material, quantity, unit, scale, abscissa, abscissa_unit, wavelength_medium, state, density, uncertainty, use, reproduced_from, table, notes, parameters, excluded_rows, rows, abscissae, values, sigmas, columns=<factory>, column_units=<factory>, digitised='')[source]#

Bases: object

Values of one quantity for one material from one source, as printed.

Attributes:

sourcestr: The source id.
idstr: The dataset id, unique within its source.
materialstr: The material’s id (optmat.materials).
quantitystr: What the values are: “refractivity” (n - 1), “polarizability”, “molar_refractivity”, “refractivity_per_density” ((n - 1) / N), or a quantity with no polarizability of its own (e.g. “phase_match_density_ratio”, “group_delay_dispersion”, “oscillator_strength”, “cauchy_moment”).
unitstr: The unit of the values after scale: “1” for dimensionless, a polarizability unit (“au”, “angstrom3”, “cm3”, “SI”), “cm3/mol”, “cm3” (for a refractivity per density) or a quantity-specific unit.
scalefloat: The printed values times scale are in unit.
abscissastr: What the x column is (ABSCISSAE): “wavelength” (a wavelength, wavenumber or photon energy; “inf” is the static limit and “not stated” a wavelength the source does not give), “order”, “temperature”, “pressure”, “density”, “intensity”, “duration”, “field”, “label” or “none”.
abscissa_unitstr: The unit of the x column: for a wavelength “m”, “nm”, “um”, “angstrom”, a wavenumber (“cm-1”, “um-1”) or a photon energy (“eV”, “hartree”, “rydberg”); “” for a label or none.
wavelength_mediumstr: “vacuum”, or “standard_air” (15 degrees C, 101 325 Pa, Edlen 1966) for wavelengths and wavenumbers measured in air.
stateState or None: The state the values refer to or are reduced to; None for per-particle quantities.
densitystr: How the source related the values to its state: “real” (real-gas density), “ideal” (ideal-gas density) or “unknown”.
uncertaintyUncertainty: How the uncertainty is stated.
usestr: “primary” (enters fits), “reproduced” (a copy of another source’s values; never fitted beside the original), “check” (compared, not fitted) or “excluded”.
reproduced_fromtuple of str: Source ids (or citations, for sources outside the library) of reproduced values.
tablestr: The library table id, or “text” with the page in notes, or, for values read from a figure, the library figure id.
notesstr: Anything else a reader needs.
parametersdict: Quantity-specific metadata (the reference material of a ratio, the harmonic of a phase-matching measurement, the length a group delay refers to, …); see the data-format page of the manual. beyond_resonance = true marks values on the far side of a resonance, where n - 1 may be negative.
excluded_rowsdict of str to str: Rows of the file left out of the dataset, each with the reason (the paper’s own exclusions, points marked extrapolated, …).
rowstuple of str: Row ids.
abscissaetuple of str: The x column, as printed (after errata).
valuestuple of str: Value column, as printed (after errata).
sigmastuple of str: Per-point uncertainty column, as printed, or empty strings.
columnsdict of str to tuple of str: Extra per-row columns, as printed (after errata): x_upper, the upper edge of an interval, in the abscissa’s unit; or a per-row condition in the unit of column_units.
column_unitsdict of str to str: The unit of each extra column (x_upper’s is the abscissa’s).
digitisedstr: “” for values printed as numbers; “vector” for values read from a figure’s vector drawing in the PDF (the coordinates of its markers and lines, the axes calibrated from the drawing’s own ticks); “raster” for values read from a figure’s image. Figure data are used only where the source prints no number; they state their digitising uncertainty, and raster data never enter a fit.

abscissa: str#

abscissa_unit: str#

abscissa_values()[source]#

Return the abscissa as numbers in SI (a wavelength as printed).

Returns:

numpy.ndarray: A temperature in K, a pressure in Pa, a density in m^-3, an intensity in W/m^2, a field in V/m, an order as printed; a wavelength, wavenumber or energy in its printed unit (use vacuum_wavelength() for the vacuum wavelength).

Raises:

DataError: If the abscissa is a label or none.

abscissae: tuple[str, ...]#

angular_frequency()[source]#

Return the angular frequencies 2 pi c / lambda_vac.

Returns:

numpy.ndarray: Angular frequency of each value, rad/s; 0 for a static value.

column_units: Mapping[str, str]#

column_values(name)[source]#

Return an extra column as numbers, as printed.

Parameters:

namestr: The column (x_upper, or a per-row condition).

Returns:

numpy.ndarray: The column, in its unit (x_upper in the abscissa’s).

Raises:

DataError: If the dataset has no such column.

columns: Mapping[str, tuple[str, ...]]#

density: Literal['real', 'ideal', 'unknown']#

digitised: str = ''#

excluded_rows: Mapping[str, str]#

id: str#

property key: str#

The dataset’s global id.

Returns:

str: source/dataset.

material: str#

notes: str#

parameters: Mapping[str, object]#

quantity: str#

reproduced_from: tuple[str, ...]#

rows: tuple[str, ...]#

scale: float#

scaled_values()[source]#

Return the values times scale, in unit.

Returns:

numpy.ndarray: The values.

sigmas: tuple[str, ...]#

source: str#

standard_uncertainty()[source]#

Return each value’s stated uncertainty as one standard deviation.

Per-point, relative and absolute statements are combined in quadrature.

Returns:

numpy.ndarray: One-sigma uncertainty in unit; NaN where none is stated.

state: State | None#

table: str#

uncertainty: Uncertainty#

unit: str#

use: str#

vacuum_wavelength()[source]#

Return the vacuum wavelengths.

Returns:

numpy.ndarray: Vacuum wavelength of each value, m; inf for a static value, NaN where the source does not state it.

Raises:

DataError: If the abscissa is not a wavelength.

values: tuple[str, ...]#

wavelength_medium: str#

class optmat.datasets.Erratum(dataset, row, column, printed, corrected, reason, evidence)[source]#

Bases: object

A correction of one printed value, with its justification.

Attributes:

datasetstr: The dataset id within the source.
rowstr: The row id in the data file; “” for a correction of the whole dataset.
columnstr: “x” (the abscissa), “value”, “sigma” or an extra column of the file; or, with row = "", “scale” (a misprinted column factor).
printedstr: The value as printed (and as extracted or recorded); checked on loading.
correctedstr: The corrected value, in the printed unit.
reasonstr: Why the printed value is wrong.
evidencestr: How the correction was established.

column: str#

corrected: str#

dataset: str#

evidence: str#

printed: str#

reason: str#

row: str#

optmat.datasets.NOT_STATED: Final = 'not stated'#: The abscissa cell of a wavelength the source does not give.

class optmat.datasets.Source(id, citation, doi, library, identity, kind, technique, notes, entries, errata, formulas=())[source]#

Bases: object

One source of values: its citation, provenance and datasets.

A dataset is built, and its data file read, when it is first asked for (datasets builds them all, dataset() one), so the record’s citation, errata and formulas cost no data file.

Attributes:

idstr: The source id, the record’s file stem.
citationstr: The citation as it should be printed.
doistr: The DOI, or “” if the work has none.
librarystr: The paper-library directory, or “” if the work is not in the library.
identitystr: The library’s bibliographic status of the record (“VALIDATED”, …).
kindstr: “measurement”, “theory”, “compilation”, “fit” or “database”.
techniquestr: How the values were obtained.
notesstr: Anything else a reader needs.
entriestuple of dict: The record’s dataset entries, as recorded ([[datasets]]).
erratatuple of Erratum: The corrections applied.
formulastuple of dict: The source’s published formulas, as recorded ([[formulas]]); the ones with a form are built into models by optmat.gas.models.formulas.

citation: str#

dataset(dataset_id)[source]#

Return one dataset, reading only its data file.

Parameters:

dataset_idstr: The dataset’s id within the source.

Returns:

Dataset: The dataset, errata applied; the same object on every call.

Raises:

KeyError: If the source has no such dataset.
DataError: If its entry or data file is malformed.

property datasets: tuple[Dataset, ...][source]#

The datasets, errata applied, in the record’s order.

Returns:

tuple of Dataset: Every dataset of the source.

doi: str#

entries: tuple[Mapping[str, object], ...]#

errata: tuple[Erratum, ...]#

formulas: tuple[Mapping[str, object], ...] = ()#

id: str#

identity: str#

kind: str#

library: str#

notes: str#

technique: str#

class optmat.datasets.Uncertainty(relative=None, absolute=None, column=False, level='not stated', systematic_relative=None, relative_rows=<factory>)[source]#

Bases: object

How the uncertainty of a dataset’s values is stated.

Attributes:

relativefloat or None: Per-point relative uncertainty, at level.
absolutefloat or None: Per-point absolute uncertainty in the printed unit (before scale), at level.
columnbool: True if the data file carries a per-point sigma column in the printed unit.
levelstr: The confidence level as stated (“1 sigma”, “probable error”, “90 % confidence”, “95 % confidence”, “99 % confidence”, “2 sigma”, “3 sigma”, “rms”, “bound”), or “not stated”.
systematic_relativefloat or None: A stated relative systematic uncertainty common to every point (a scale error), as one standard deviation; the prior of the dataset’s scale in a fit.
relative_rowsdict of str to float: Per-row relative uncertainties that replace relative for those rows (a source that states a larger error near a resonance, say).

absolute: float | None = None#

column: bool = False#

level: str = 'not stated'#

relative: float | None = None#

relative_rows: Mapping[str, float]#

property stated: bool#

Whether any per-point uncertainty is given.

Returns:

bool: True if a column, a relative or an absolute uncertainty is stated.

systematic_relative: float | None = None#

property to_standard: float#

The factor from the stated level to one standard deviation.

Returns:

float: 1 when the level is one sigma or not stated.

optmat.datasets.VALUES_DIRECTORY: Final = PosixPath('/home/docs/checkouts/optmat/checkouts/7/src/optmat/data/values')#: The extracted data files, <family>/<material>/<source>__<dataset>.csv.

optmat.datasets.data_file(quantity, file)[source]#

Return where a dataset’s file lives.

Parameters:

quantitystr: The dataset’s quantity, whose family is the directory.
filestr: The record’s file, <material>/<source>__<dataset>.csv.

Returns:

pathlib.Path: data/values/<family>/<file>.

Raises:

DataError: If the quantity is unknown.

optmat.datasets.datasets(material=None, *, quantity=None)[source]#

Return the curated datasets, optionally for one material or one quantity.

Parameters:

materialstr, optional: Only this material: its id or an alias (a gas species such as "He", or "SiO2").
quantitystr, optional: Only this quantity (e.g. “refractivity”).

Returns:

tuple of Dataset: The matching datasets, in source order.

Examples

>>> all(d.material == "He" for d in datasets("He"))
True

optmat.datasets.load_source(path, *, defer=False)[source]#

Load one source record and its datasets.

Parameters:

pathpathlib.Path: The record, data/sources/<source>.toml.
deferbool, optional: Build each dataset, and read its data file, only when it is first asked for, as the catalogue does; a malformed dataset then raises there. By default every dataset is built and checked here.

Returns:

Source: The source, with its datasets read (unless deferred) and errata applied.

Raises:

DataError: If the record or a data file is malformed or inconsistent.

optmat.datasets.parse_state(spec, where)[source]#

Parse a dataset’s state.

Parameters:

specobject: A state name, a table with a pressure and a temperature, or None.
wherestr: The dataset, for error messages.

Returns:

State or None: The state, or None if none is given.

Raises:

DataError: If the state is unknown or incomplete.

optmat.datasets.sources()[source]#

Return every curated source, by id.

Returns:

dict of str to Source: The sources, loaded once per process.

Examples

>>> "huber_1974" in sources()
True