LINCC Frameworks Software Tools & Areas of Focus

Category: Solar System
Brief Description: An image processing library for moving object detection implemented with GPUs. Based on a Maximum Likelihood detection algorithm for moving astronomical objects.
Github Page: https://github.com/dirac-institute/kbmod
Distribution Page: https://github.com/dirac-institute/kbmod
Documentation: https://epyc.astro.washington.edu/~kbmod/

Category: Solar System
Brief Description: A package for interfacing with IMCCE’s SkyBoT VO tool (Berthier et. al. 2006) and JPL’s SSD (Solar System Dynamics) API service to check asteroid linkages for previously known objects. Designed to validate linkage tools, KOFFI takes (x, y) coordinates within a FITS file or (RA, dec) coordinates for a list of potential detections and compares them with results from those APIs.
Github Page: https://github.com/lincc-frameworks/koffi
Distribution Page: https://pypi.org/project/koffi/
Documentation: https://github.com/lincc-frameworks/koffi

Category: Data Storage, Spatial Data Analysis
Brief Description: A HATS catalog is a partitioning of objects on a sphere with efficient storage of large data sets in Parquet files. It was designed for storing data from large astronomy surveys, but can be used for other use cases where you have large data with some spherical properties. This library provides rudimentary access, with more analytic capabilities in LSDB.
Github Page: https://github.com/astronomy-commons/hats
Distribution Page: https://pypi.org/project/hats/
Documentation: https://hats.readthedocs.io/en/latest/

Category: Data Storage, Spatial Data Analysis
Brief Description: Utility for ingesting large survey data into HATS structure. Once created, the catalog can be read and analyzed with LSDB.
Github Page: https://github.com/astronomy-commons/hats-import
Distribution Page: https://pypi.org/project/hats-import/
Documentation: https://hats-import.readthedocs.io/en/latest/

Category: Machine Learning
Brief Description: An efficient tool to hunt for rare and anomalous sources in large astronomical imaging surveys. Not tied to a specific anomaly detection algorithm/model or a specific class of rare/anomalous objects, but rather intended to support any algorithm that the user may want to apply on imaging data.
Github Page: https://github.com/lincc-frameworks/hyrax
Distribution Page: https://github.com/lincc-frameworks/hyrax
Documentation: https://github.com/lincc-frameworks/hyrax

Category: Spatial Data Analysis
Brief Description: LSDB is a python framework that enables simple, fast spatial analysis of extremely large astronomical catalogs (e.g. querying and crossmatching ~10⁹ sources). It aims to address large-scale data processing challenges, in particular those brought up by LSST.
Github Page: https://github.com/astronomy-commons/lsdb
Distribution Page: https://pypi.org/project/lsdb/
Documentation: https://docs.lsdb.io/en/stable/

Category: Data Storage, Core Package
Brief Description: Nested-pandas and nested-dask are extension packages that wrap the Pandas and Dask APIs respectively and empower efficient analysis of nested associated datasets.

Github Pages: 

• https://github.com/lincc-frameworks/nested-pandas
• https://github.com/lincc-frameworks/nested-dask

Distribution Pages: 

• https://pypi.org/project/nested-pandas/
• https://pypi.org/project/nested-dask/

Documentation: 

• https://nested-pandas.readthedocs.io/en/latest/
• https://nested-dask.readthedocs.io/en/latest/

Category: Time Domain Simulation
Brief Description: A package for large-scale, time-domain forward-modeling of astronomical light curve data. Simulations incorporate realistic effects, including survey cadence, dust extinction, and instrument noise models. LightCurveLynx is designed to enable user extensibility, such as adding new source models, effects, and instruments, while ensuring scalability.
Github Page: https://github.com/lincc-frameworks/lightcurvelynx
Distribution Page: https://pypi.org/project/lightcurvelynx/
Documentation: https://lightcurvelynx.readthedocs.io/en/latest/

Category: Development Utility
Brief Description: This project template codifies LINCC-Framework’s best practices for python code organization, testing, documentation, and automation. It is meant to help new python projects get started quickly, letting the user focus on writing code. The template takes care of the minutiae of directory structures, tool configurations, and automated testing until the user is ready to take over.
Github Page: https://github.com/lincc-frameworks/python-project-template
Distribution Page: https://github.com/lincc-frameworks/python-project-template
Documentation: https://lincc-ppt.readthedocs.io/en/latest/

Category: Development Utility
Brief Description: LF Workflow Dash is your easy solution for monitoring and managing GitHub Actions workflows. Track workflows across any number of repositories, check status and other relevant metrics, and quickly modify via yaml, all in one customizable dashboard.
Github Page: https://github.com/lincc-frameworks/lf-workflow-dash
Distribution Page: https://github.com/lincc-frameworks/lf-workflow-dash
Documentation: https://github.com/lincc-frameworks/lf-workflow-dash

Category: Development Utility
Brief Description: Simple script to re-format the output of ASV (air-speed-velocity) benchmarking outputs for use in github messages.
Github Page: https://github.com/lincc-frameworks/asv-formatter
Distribution Page: https://pypi.org/project/lf-asv-formatter/
Documentation: https://github.com/lincc-frameworks/asv-formatter

Category: Development Utility
Brief Description: A lightweight package for annotating and extracting citable portions of scientific code from Python modules. The purpose of this package is to make it easier for code authors to track what citations they need for their code.
Github Page: https://github.com/lincc-frameworks/citation-compass
Distribution Page: https://pypi.org/project/citation-compass/
Documentation: https://github.com/lincc-frameworks/citation-compass/blob/main/docs/notebooks/intro_notebook.ipynb

Over its lifetime, Rubin LSST will compile an unprecedented catalog of tens of billions of astrophysical objects. This wealth of data will enable astronomers to answer a range of scientific and statistical questions about our universe, such as: 

• Understanding structure by analyzing the distribution of objects
• Modeling the changes of variable sources over time
• Allowing prompt localization of electromagnetic counterparts to gravitational wave sources
• Providing interpretable and information-rich cross-correlations against multi-wavelength datasets, including the Cosmic Microwave Background

Supporting these science questions requires key functionality in an analysis framework, including the ability to: 

• Store and manipulate catalog data at scale
• Perform distributed computation over this data
• Use spatial structure within searches and statistical computation
• Interoperate with data from other surveys
• Access these catalogs without having to download them directly

The LINCC Frameworks team is developing LSDB, an infrastructure to facilitate the analysis of large survey data based on efficient spatial partitioning. Driven by the initial use cases of catalog cross-matching and distributed time series analysis, the team is developing an end-to-end technology stack. The work includes coordinating with the broader astronomy community on developing standardized data formats, enabling cloud or HPC-based analysis, and the development of a full suite of software tools.

Rubin is uniquely capable of monitoring the sky at unprecedented depths, detecting and characterizing the time-variability of tens of billions of astrophysical objects. Fast and efficient access to astronomical light curves (LCs) will enable the discovery of the most energetic events in the Universe and provide the first systematic characterization of variability within the Milky Way. Scientific discoveries we expect from this:

• Identification of the pre-explosion outbursts from supernova progenitors, which are poorly understood and which hold the key to understanding the chemical enrichment of the intergalactic medium
• Understanding the evolution and death of stars by detecting the very rarest and the most distant transients that represent the most extreme ways stars die
• Probing the properties of dark matter by mapping the distribution of mass in the Milky Way using the kinematics of Cepheids and RR Lyrae

Supporting these science questions requires key functionality:

• A datastore for lightcurves that scales to the size of LSST data sets (10^10 sources)
• Algorithms to automatically search and analyze light curves, including measures of multi-band periods, detection of outbursts, generation of classification features, identification of changes in the state of a source, and measures of distance between lightcurves accounting for phase and period variation
• A framework that scales user-defined algorithms to the size of LSST data
• Trained neural network architectures for classifying sparsely sampled time series data (including trusted training samples)
• A fast and scalable catalog cross-matching engine to match sources from the LSST with existing data sets to provide panchromatic information for detected sources

The LINCC Frameworks team is developing multiple projects to provide support for scalable and automated time series analysis at Rubin data scales. The nested-pandas and nested-dask libraries add support for embedded time series data in Pandas and Dask settings, providing an intuitive user interface for this type of data. Interoperability of these libraries with LSDB provides further scalability and functionality, such as catalog cross-matching.

Asteroids and comets are the remnants of the Solar System’s early assembly. Their history of accretion, collisions, and perturbation by existing and vanished giant planets is preserved in their orbital elements and size distributions. Rubin has the potential to discover the nearest (Near Earth Objects; NEOs) and the most distant (Trans Neptunian Objects; TNOs) asteroid populations, mapping the Solar System in unprecedented detail. The discoveries we expect from this framework include:

• Detection and impact probabilities for ~80% of NEOs with sizes over 140m (the impact of which would cause devastation on a regional scale)
• The discovery of interstellar objects as they pass through the Solar System
• A 10-fold increase in the number of known TNOs, which can elucidate the evolution and origin of the Solar System
• The ability to find objects that originate from the inner Oort Cloud and potentially additional planets beyond 100 au (astronomical units)

The LINCC Frameworks team is currently working to scale the KBMOD algorithm – a shift-and-stack search that finds objects that may not be bright enough to detect in a single image. The goal is to enable the efficient detection of faint objects at Rubin’s data scales. Key technical challenges include: 

• Scaling this approach to Rubin’s massive data volume
• Improving sensitivity and accuracy

A key element in turning 2D images from LSST into a 3D view is determining distances to the astronomical objects studied. Scientifically, this 2D to 3D conversion is essential to map the expansion history of the Universe and the growth of cosmological structure as a function of time, which is the key to understanding dark energy, galaxy formation/evolution, and the physical processes that drive transient and variable phenomena. Photometric redshifts, or photo-zs, use the observed colors of the galaxies (and potentially other information) to produce an estimate of the distance to an object, which can be critically important information to understand the physics that may have given rise to it. 

The development of photo-z estimation methods is an active field of research that gives rise to a number of software needs for application to Rubin data. The openly developed Redshift Assessment Infrastructure Layers (RAIL) software library was initiated by the Dark Energy Science Collaboration (DESC) to establish a unified framework for comprehensive photo-z development, validation, and optimization within the scope of cosmological analysis. LINCC Frameworks is working in collaboration or coordination with DESC as appropriate and with members of the other LSST Science Collaborations to extend RAIL to other extragalactic use cases beyond cosmology, including: 

• The addition of new photo-z performance metrics;
• Working with Rubin Observatory on the compatibility of RAIL with the Rubin Science Platform to enable key commissioning activities
• Extending the probabilistic representations of photo-z uncertainty, i.e., probability density functions, p(z)
• Improving uncertainty estimation for individual and ensemble photo-z distributions
• Generally investing effort into optimizations that improve the robustness of the software at the LSST scale

Throughout their development work, the Frameworks team watches for common problems and develops cross-project infrastructure that will be beneficial for a wide range of use cases. An example of this is the reusable Python project template (based on the Copier package) that reduces the overhead of setting up new astronomy projects. This template has been used throughout the Framework’s projects to provide baseline productionisation capabilities such as continuous integration testing, style checking, and integration with readthedocs. For more information, see the Tech Talk.