Data containers

Data containers#

pytransit.utils.io provides containers that hold observations together with their metadata. They exist because real datasets are not a single array: a light curve comes with a passband, an instrument, an exposure time, a sector, and a supersampling rate, and keeping those in parallel lists next to the data is how bookkeeping errors happen.

There are two pairs, one for photometry and one for radial velocities:

Class	Holds
`LCData`	One light curve: times, fluxes, errors, covariates, and metadata.
`LCDataGroup`	An ordered collection of LCData objects.
`RVData`	One radial velocity dataset: times, velocities, errors, and covariates.
`RVDataGroup`	An ordered collection of RVData objects.

from pytransit.utils.io import LCData, LCDataGroup, RVData, RVDataGroup

A single light curve#

from pytransit.utils.io import LCData

lc = LCData(time, flux, error=error,
            passband='TESS', instrument='TESS',
            sector=14, exptime=0.0014, nsamples=1)

Only time and flux are required. Everything else has a default, and the white noise estimate defaults to the point-to-point estimate \(\mathrm{std}(\Delta F)/\sqrt{2}\).

Note

exptime is in days, consistent with the transit models.

The arrays are stored without copying when they already are float64 ndarrays, so the caller and the LCData object may share memory.

LCData also carries a small toolkit for the routine cleaning steps:

lc.running_median(width=15)             # running median of the flux
lc.outlier_mask(nsigma=3.0, width=15)   # which points a clip would remove
lc.remove_outliers(nsigma=3.0)          # clip them, in place; returns the count
lc.add_time_covariates(order=2)         # add normalised time and its powers as covariates
lc.linear_model()                       # least-squares fit of the flux to the covariates

Outlier clipping is done against a running median rather than a global mean, so it removes points that deviate from the local trend and leaves the transit itself alone.

Groups#

A group is an ordered collection. The insertion order is the light curve order, which is what defines the light curve indices a model will use, so nothing is ever sorted implicitly.

Groups are built by addition or from a sequence:

lcs = lc1 + lc2 + lc3
lcs = LCDataGroup([lc1, lc2, lc3])
lcs = sum(light_curves)                 # a list of LCData objects

Nested groups are flattened, and adding the same LCData instance twice raises a ValueError.

A group exposes every per-light-curve quantity as a bulk property, which is what makes it easy to feed a transit model:

lcs.times          # list of time arrays
lcs.fluxes         # list of flux arrays
lcs.errors         # list of error arrays
lcs.pbids          # passband index per light curve
lcs.exptimes       # exposure time per light curve
lcs.nsamples       # supersampling rate per light curve
lcs.passband_names # the unique passband names, in pbid order
lcs.lcslices       # slices splitting a concatenated array back into light curves

These map directly onto the arguments of set_data, described in Data setup.

Selecting and sorting#

select returns a new group of the datasets matching all the given criteria. Tuple-valued attributes such as passband and pids match by membership, everything else by equality, and a criterion may be a single value or a sequence of accepted values.

lcs.select(passband='TESS')
lcs.select(instrument='MuSCAT2', sector=[14, 15])
lcs.select(pids=0)                      # light curves with planet 0 transiting

sorted_by returns a new group in a different order, keyed by 'time' (the default), by any attribute name, or by a callable.

lcs.sorted_by()                 # by first observation time
lcs.sorted_by('passband')
lcs.sorted_by(lambda d: d.npt)

Groups also support indexing, slicing, boolean masks, and iteration.

Important

select, sorted_by, and slicing return new groups, but the LCData objects inside them are shared by reference, not copied. Modifying a light curve through one group modifies it in every group holding it.

Inspecting and cleaning#

plot draws the light curves in a grid of panels sharing their y limits, so transit depths can be compared by eye. It is the entry point to the cleaning workflow.

lcs.plot(ncols=5, show_median=True, nsigma=3.0)
lcs.plot(passbands='TESS', show_linear_model=True)

The panels can be filtered by passband, instrument, sector, or planet id, the running median can be overlaid with its n-sigma clipping limits, and the variability explained by the covariates can be drawn on top.

Bad light curves are removed in two steps: mark, then remove.

lcs.plot()                        # eyeball the light curves
lcs.mark_for_removal([2, 7, 9])   # mark the bad ones by their panel index
lcs.plot()                        # the marked ones now have a gray background
lcs.remove_marked()

Marking only sets a flag. The marked light curves stay in the group and keep appearing in every bulk property until remove_marked is called, so nothing a model sees changes until you commit. The index shown in the upper left corner of each panel is exactly the index mark_for_removal takes.

Warning

Marks are shared, removals are not. The flag lives in the LCData objects, which groups share by reference, so marking through a select result marks the light curve in the original group too. remove_marked drops the light curves from that group only.

Remove before handing the group to a model: the removal renumbers every order-derived index, including pbids and lcslices.

Radial velocities#

RVData and RVDataGroup mirror the photometric containers, with two differences that follow from the data:

Times are in days, velocities in m/s.
The uncertainties are required. Unlike photometry, a radial velocity measurement essentially always comes with an uncertainty, so error is a positional argument rather than an optional one.

from pytransit.utils.io import RVData

rv = RVData(time, rv, error, instrument='HARPS')

The instrument name matters here: it identifies the dataset’s systemic velocity and jitter parameters, so it must be unique within a group.

API#

Light curves#

class pytransit.utils.io.lcdata.LCData(time, flux, error=None, covariates=None, passband='white', pids=None, noise=None, instrument='', sector=-1, segment=0, exptime=0.0, nsamples=1)[source]#

Bases: _Data

Data and metadata for a single light curve.

Parameters:

time – Mid-exposure times, converted to a 1D float64 array. Must all be finite.
flux – Fluxes, converted to a 1D float64 array of the same length as time. Normally normalised so that the out-of-transit level is close to unity.
error – Per-point flux uncertainties, converted to a 1D float64 array of the same length as time. Defaults to None, in which case error falls back to the white noise estimate repeated.
covariates – Covariate matrix with shape (npt, ncov). A 1D array of length npt is treated as a single covariate, and None gives an empty (npt, 0) matrix.
passband – Passband name, or a sequence of names for a light curve combining several. Stored as a tuple of strings.
pids – Indices of the planets transiting in this light curve, for modelling multiplanet systems. An integer is accepted for a single planet. Stored as a tuple of ints. Defaults to None meaning “unspecified”, which is distinct from an empty sequence meaning “no planet transits here”.
noise – White noise estimate. Defaults to the point-to-point estimate nanstd(diff(flux)) / sqrt(2), the same estimator BaseLPF uses, or to NaN for a light curve with fewer than two points.
instrument – Instrument name.
sector – Sector, night, or similar observation id. Defaults to -1 for “unspecified”.
segment – Segment id within the sector.
exptime – Exposure time in days.
nsamples – Number of supersamples used to integrate the model over the exposure.

Notes

The arrays are stored without copying when they already are float64 ndarrays, so the caller and the object may share memory.

A light curve also carries a boolean marked flag, False at construction, that LCDataGroup.mark_for_removal sets and LCDataGroup.remove_marked acts on. It records the state of an interactive session rather than a property of the data, so it is not a constructor argument and does not affect anything a model sees.

property size#: Number of datapoints.

property npt#: Number of datapoints.

property ncov#: Number of covariates.

property error#

Per-point flux uncertainty.

The array given at construction, or the white noise estimate repeated if none was given. noise is estimated from the point-to-point flux scatter and is not derived from error, so the two are independent descriptions of the uncertainty.

property has_error#: True if an explicit uncertainty array was given.

property has_pids#: True if the transiting planets were specified.

running_median(width=15)[source]#

Running median of the flux, width points wide.

Parameters:: width – Width of the median filter in points. Must be a positive odd integer, and is clipped to the length of the light curve when it is longer.

outlier_mask(nsigma=3.0, width=15)[source]#

Boolean mask flagging the flux points further than nsigma from the running median.

The scatter is a robust MAD estimate of the residuals from the running median, so that a few strong outliers cannot inflate the very threshold meant to catch them. A light curve with no scatter at all flags nothing.

Parameters:

nsigma – Clipping threshold in units of the robust residual scatter.
width – Width of the median filter in points. Must be a positive odd integer.

remove_outliers(nsigma=3.0, width=15)[source]#

Remove the flux points further than nsigma from the running median, in place.

The times, fluxes, covariates, and uncertainties are all filtered together, and the noise estimate is refreshed unless it was given explicitly. Returns the number of points removed.

Parameters:

nsigma – Clipping threshold in units of the robust residual scatter.
width – Width of the median filter in points. Must be a positive odd integer.

add_time_covariates(order=1)[source]#

Add the normalised time and its powers as covariates, in place.

The time is mapped linearly onto -1 … 1 and its powers 1 … order are appended to the existing covariates, so that a linear-in-covariates baseline can absorb a polynomial trend in time. The powers start from one because the intercept is added by the baseline model itself, and a constant column would only make the design matrix singular.

Parameters:: order – Highest power of the normalised time to add. Must be a positive integer.

linear_model()[source]#

Least-squares linear model of the flux in terms of the covariates.

The covariate columns are standardised and an intercept is added, and the flux is fitted against them by linear least squares. The result is the part of the flux variability the covariates can explain, which is what LCDataGroup.plot overlays with show_linear_model. A light curve with no covariates gives its mean flux.

class pytransit.utils.io.lcdata.LCDataGroup(data=())[source]#

Bases: _DataGroup

A container of LCData objects.

The light curves keep their insertion order, which defines both the light curve index (lcid) an LPF will use and the order of passband_names. Nothing is sorted implicitly; use sorted_by if you need a different order.

Parameters:: data – A LCData, a sequence of them, or another LCDataGroup. Nested groups are flattened.

property fluxes#: List of 1D flux arrays.

property noises#: Array of white noise estimates.

property passbands#: List of per-light-curve passband name tuples.

property pids#

List of per-light-curve transiting-planet index tuples.

An entry is None for a light curve whose transiting planets were not specified.

property sectors#: Array of sector ids.

property segments#: Array of segment ids.

property exptimes#: Array of exposure times in days.

property nsamples#: Array of supersampling factors.

property has_pids#: True if every light curve names its transiting planets.

property n_planets#

Number of planets implied by the planet indices.

One past the largest index given by any light curve, or zero if none of them specify their planets.

property passband_names#

Unique passband names, ordered by first appearance.

Suitable as the passbands argument of a BaseLPF, which expects the passbands to be ordered from blue to red. The order follows insertion order rather than being sorted, so arrange the light curves accordingly.

property pbids#

Passband index per light curve, indexing into passband_names.

Raises:: ValueError – If any light curve has more than one passband. An LPF passband carries a single radius ratio and limb darkening pair, so a light curve combining several passbands has no single correct index.

property wnids#

Noise block index per light curve.

Light curves sharing an instrument, sector, and segment share a white noise parameter. With the default metadata every light curve falls into block 0, matching BaseLPF’s own default, and the blocks separate as soon as the metadata is filled in. Use _group_ids directly for a different grouping.

property piis#

Running index of each light curve within its instrument.

Together with ins, this gives LinearModelBaseline the per-light-curve tokens it uses to name its baseline parameters:

lpf.ins, lpf.piis = lcs.ins, lcs.piis

property lcslices#

Slices splitting a concatenated array back into per-light-curve arrays.

The slices follow the order of the light curves in the group, so an array created by concatenating any of the per-light-curve quantities can be split back into a list:

timea = concatenate(lcs.times)
times = [timea[sl] for sl in lcs.lcslices]

These are the slices BaseLPF stores as lcslices for the same data.

property marked#: Boolean array telling which light curves are marked for removal.

property n_marked#: Number of light curves marked for removal.

mark_for_removal(indices)[source]#

Mark the given light curves for removal, modifying the group in place.

Marking only sets a flag: the marked light curves stay in the group and keep appearing in every bulk property until remove_marked is called, so nothing a model sees changes until then. plot draws the marked light curves on a light gray background, so the workflow is to plot the group, mark the bad light curves by the index shown in the upper left corner of their panels, plot again to check, and then remove them.

Parameters:: indices – Index of a light curve, a sequence, set, or array of indices, a boolean mask with one entry per light curve, or a slice. Negative indices count from the end.

Notes

Marking is additive, so several calls accumulate, and unmark clears the marks again. The flag lives in the LCData objects themselves, which the groups share by reference, so marking a light curve through a select result or a slice marks it in the group it came from as well. The marked light curves can be selected with select(marked=True).

Examples

lcs.plot()                       # Eyeball the light curves
lcs.mark_for_removal([2, 7, 9])  # Mark the bad ones by their panel index
lcs.plot()                       # The marked ones now have a gray background
lcs.remove_marked()

unmark(indices=None)[source]#

Clear the removal marks of the given light curves, modifying the group in place.

Parameters:: indices – The light curves to unmark, in any of the forms mark_for_removal accepts. Defaults to None for all of them.

remove_marked()[source]#

Remove the light curves marked for removal from the group, in place.

The group shrinks and every index derived from the order, such as pbids, wnids, piis, and lcslices, is renumbered the next time it is asked for, which is why the removal should happen before the group is handed to a BaseLPF. Removing every light curve leaves a legal empty group.

Unlike marking, removal is not shared: the light curves are dropped from this group only, and any other group holding them keeps them, still marked, until its own remove_marked.

remove_outliers(nsigma=3.0, width=15)[source]#

Remove the outlying flux points from every light curve, in place.

Each light curve is clipped separately against its own running median, see LCData.remove_outliers. Returns the total number of points removed. The light curves themselves are always kept, even if one loses every point; use remove_marked to drop whole light curves.

Parameters:

nsigma – Clipping threshold in units of the robust residual scatter.
width – Width of the median filter in points. Must be a positive odd integer.

add_time_covariates(order=1)[source]#

Add the normalised time and its powers as covariates to every light curve, in place.

Each light curve’s time is mapped onto -1 … 1 separately, see LCData.add_time_covariates.

Parameters:: order – Highest power of the normalised time to add. Must be a positive integer.

property covariates#: List of 2D covariate matrices.

property errors#: List of 1D uncertainty arrays.

property has_covariates#: True if any dataset has at least one covariate.

property has_errors#: True if every dataset was given an explicit uncertainty array.

property ins#: Instrument names, under the name LinearModelBaseline looks for.

property instruments#: List of instrument names.

property ncovs#: Array of per-dataset covariate counts.

property npts#: Array of per-dataset datapoint counts.

plot(ncols=5, figsize=None, passbands=None, instruments=None, sectors=None, pids=None, annotate=True, show_index=True, show_xticks=True, show_median=False, median_width=15, nsigma=3.0, show_linear_model=False, median_kwargs=None, linear_model_kwargs=None, errorbars=False, xoffset=None, ylim=None, alpha=0.5, **kwargs)[source]#

Plot the light curves in a grid of subplots.

The panels share their y limits so the transit depths can be compared by eye, but not their x limits, since the light curves generally cover different times. Each panel’s time axis is offset by its own zero point by default, keeping the tick labels short. The light curves marked for removal are drawn on a light gray background, show_median overlays the running median with its n-sigma limits for spotting outlying points, and show_linear_model overlays the variability the covariates can explain.

Parameters:

ncols – Number of columns in the subplot grid, clipped to the number of light curves.
figsize – Figure size in inches. Defaults to a 13-inch-wide figure 2.5 inches per row tall.
passbands – Plot only the light curves matching these criteria. Each takes a single value or a sequence of accepted values, and None means no filtering. A pids criterion never selects a light curve with unspecified transiting planets.
instruments – Plot only the light curves matching these criteria. Each takes a single value or a sequence of accepted values, and None means no filtering. A pids criterion never selects a light curve with unspecified transiting planets.
sectors – Plot only the light curves matching these criteria. Each takes a single value or a sequence of accepted values, and None means no filtering. A pids criterion never selects a light curve with unspecified transiting planets.
pids – Plot only the light curves matching these criteria. Each takes a single value or a sequence of accepted values, and None means no filtering. A pids criterion never selects a light curve with unspecified transiting planets.
annotate – Show the instrument name and the passband in the upper right corner of each panel.
show_index – Show each light curve’s index in the group in the upper left corner of its panel. This is the index mark_for_removal takes, and it is the position in the group plot was called on even when the panels are filtered.
show_xticks – Draw the x axis ticks and labels. Switching them off packs more light curves onto the screen when eyeballing the data.
show_median – Overlay the running median of the flux and its n-sigma limits, for spotting the points remove_outliers would clip.
median_width – Width of the median filter in points. Must be a positive odd integer.
nsigma – Limits to draw around the running median, as a multiple of the robust MAD scatter of the residuals from it. Either a single number or a sequence of them, in which case one band is drawn per value.
show_linear_model – Overlay the least-squares linear model of the flux in terms of the covariates, showing how much of the variability the covariates can explain. Light curves without covariates are left alone, see LCData.linear_model.
median_kwargs – Line properties (c, lw, alpha, zorder, and anything else matplotlib.axes.Axes.plot takes) for the running median and the linear model. They override the defaults, which draw both lines on top of the flux points. The n-sigma bands take their colour from median_kwargs.
linear_model_kwargs – Line properties (c, lw, alpha, zorder, and anything else matplotlib.axes.Axes.plot takes) for the running median and the linear model. They override the defaults, which draw both lines on top of the flux points. The n-sigma bands take their colour from median_kwargs.
errorbars – Plot the flux uncertainties as error bars. The uncertainties fall back to the estimated point-to-point scatter for the light curves without explicit errors, see LCData.error and has_errors.
xoffset – Time subtracted from the plotted times. Defaults to None for a per-panel offset, and 0.0 plots the times as they are.
ylim – Y limits shared by all the panels. Defaults to None for automatic limits.
alpha – Opacity of the plotted flux.
**kwargs – Passed to matplotlib.axes.Axes.plot or matplotlib.axes.Axes.errorbar.

Returns:

matplotlib.figure.Figure

Raises:

ValueError – If the group is empty or nothing matches the given criteria.

Examples

lcs.plot(ncols=4, passbands=['g', 'r'], instruments='MuSCAT2')

select(**criteria)#

Return a new group with the datasets matching all the given criteria.

Each keyword names an attribute of the contained objects. A tuple-valued attribute such as passband or pids matches by membership, everything else by equality. A criterion can be a single value or a sequence of accepted values, in which case a dataset matches if it matches any of them, so pids=[0, 2] selects the datasets with a transiting planet 0 or 2, and an empty sequence selects nothing. A dataset with unspecified pids (None) is never selected by a pids criterion. Criteria given as None are ignored.

property size#: Number of datasets.

sorted_by(key='time')#

Return a new group ordered by key.

Parameters:: key – ‘time’, the name of any attribute of the contained objects, or a callable taking one of them and returning a sort key.

property times#: List of 1D time arrays.

property tmax#: Latest time in the group.

property tmin#: Earliest time in the group.

Radial velocities#

class pytransit.utils.io.rvdata.RVData(time, rv, error, covariates=None, instrument='')[source]#

Bases: _Data

Radial velocities and metadata for a single instrument or campaign.

Parameters:

time – Mid-measurement times in days, converted to a 1D float64 array. Must all be finite.
rv – Radial velocities in m/s, converted to a 1D float64 array of the same length as time.
error – Radial velocity uncertainties in m/s, a 1D float64 array of the same length as time. Required: unlike photometry, an RV measurement essentially always comes with an uncertainty, and RVLPF needs one per point.
covariates – Covariate matrix with shape (npt, ncov) holding activity indicators, airmass, or similar. A 1D array of length npt is treated as a single covariate, and None gives an empty (npt, 0) matrix.
instrument – Instrument name. RVLPF interpolates it into the names of the systemic velocity and jitter parameters, rv_shift_{instrument} and rv_err_{instrument}, so it must be unique within a group.

Notes

The arrays are stored without copying when they already are float64 ndarrays, so the caller and the object may share memory.

property size#: Number of datapoints.

property npt#: Number of datapoints.

property ncov#: Number of covariates.

property noise#

Point-to-point scatter estimate, nanstd(diff(rv)) / sqrt(2).

A read-only diagnostic. Unlike LCData this is never a constructor argument, since RV uncertainties are always given, and it is not used by any model: RVLPF fits a separate jitter term per instrument on top of error.

property has_error#

RV uncertainties are required.

Type:: Always True

class pytransit.utils.io.rvdata.RVDataGroup(data=())[source]#

Bases: _DataGroup

A container of RVData objects.

The datasets keep their insertion order, which is the order RVLPF uses when it builds its per-dataset systemic velocity and jitter parameters. Nothing is sorted implicitly; use sorted_by if you need a different order.

Parameters:: data – An RVData, a sequence of them, or another RVDataGroup. Nested groups are flattened.

property rvs#: List of 1D radial velocity arrays in m/s.

property noises#: Array of per-dataset point-to-point scatter estimates.

property rvis#

Instrument labels, one per dataset, ready for RVLPF.

Returned as a list rather than an array on purpose: RVLPF.__init__ tests if rvis:, which raises on an ndarray.

Raises:: ValueError – If two datasets share an instrument label. RVLPF names one systemic velocity and one jitter parameter after each label, so duplicates would collide.

property covariates#: List of 2D covariate matrices.

property errors#: List of 1D uncertainty arrays.

property has_covariates#: True if any dataset has at least one covariate.

property has_errors#: True if every dataset was given an explicit uncertainty array.

property ins#: Instrument names, under the name LinearModelBaseline looks for.

property instruments#: List of instrument names.

property ncovs#: Array of per-dataset covariate counts.

property npts#: Array of per-dataset datapoint counts.

property piis#: Running index of each dataset within its instrument.

select(**criteria)#

Return a new group with the datasets matching all the given criteria.

Each keyword names an attribute of the contained objects. A tuple-valued attribute such as passband or pids matches by membership, everything else by equality. A criterion can be a single value or a sequence of accepted values, in which case a dataset matches if it matches any of them, so pids=[0, 2] selects the datasets with a transiting planet 0 or 2, and an empty sequence selects nothing. A dataset with unspecified pids (None) is never selected by a pids criterion. Criteria given as None are ignored.

property size#: Number of datasets.

sorted_by(key='time')#

Return a new group ordered by key.

Parameters:: key – ‘time’, the name of any attribute of the contained objects, or a callable taking one of them and returning a sort key.

property times#: List of 1D time arrays.

property tmax#: Latest time in the group.

property tmin#: Earliest time in the group.

property wnids#: Noise block index per dataset, grouping the datasets by instrument.

Data containers

Contents

Data containers#

A single light curve#

Groups#

Selecting and sorting#

Inspecting and cleaning#

Radial velocities#

API#

Light curves#

Radial velocities#