Overview#
Purpose#
The “timeseries refinery” is a powerful tool to build complete time series information systems.
It provides:
a versioned time series (compact) storage system with a simple yet powerful API (direct in python + REST)
an edition/supervision workflow for experts
computed versioned time series using a simple yet expressive dedicated formula language (think about Excel formulas, but simpler and specialised for time series) with a simple API
a cache system for the computed series
a central repository of time series (stored or computed)
rich metadata support and querying/filtering over various kinds of metadata
support for timezone aware and naive time series
a data-mesh distributed architecture allowing to connect to other refineries (for collaborative work) or other time series silos
a task manager and a mini-framework to handle regular ingestion of time series from external or internal sources, and also model runs
a tool to detect/watch the series lagging behind (because of failing sources)
a powerful low-code declarative dashboard system (specially targeted at energy comodity markets)
a rich web user interface allowing exploration of the catalog, visualisation and edition of series, formulas, caches, tasks, watchers and dashboards
It can be used as a building block for machine learning, model optimization and validation, both for inputs and outputs.
Core objects#
At the centre sits the series: a named, versioned sequence of timestamped values. Everything else builds on it. A formula computes a new series from existing ones; a group bundles related series together; a basket is a saved query that resolves to a set of series; a folder places a series in a tree for browsing. Around these, supervision records manual corrections, the formula cache materializes computed series for speed, tasks and scrapers feed data in on a schedule, and dashboards present it.
One distinction underlies the whole system: the value date of a point (when it happened) versus its insertion date (when it was recorded). Keeping both is what gives the Refinery its time-travel and backtesting abilities.
- series#
- primary series#
A named, versioned sequence of timestamped values, stored directly in the database. Every update is kept as a revision, so the series can be read as it stood at any past insertion date. See Working with series in Python.
- formula#
A computed series defined by a lisp-like expression over other series. Formulas are read-only and evaluated on demand; they inherit their history from their inputs. See Formulas (computed series) and the Formula Language Reference.
- group#
An ensemble of related series sharing a common structure — typically the members or scenarios of a forecast. See Groups.
- basket#
A saved search query. It resolves, on demand, to the set of series (or groups) that currently match it. See Search Query Language Reference.
- folder#
A position in a path tree used to organise series for browsing. A series belongs to at most one folder. See Folders.
- supervision#
The mechanism for applying manual corrections on top of an automated feed while preserving the original data. A series is accordingly
unsupervised,supervisedorhandcrafted. See Supervision.- formula cache#
Materialized formula results, refreshed by a policy, so that expensive formulas are served quickly instead of recomputed on every read. See Formula cache.
- task#
A background job run by the scheduler: scraping, cache refresh, CSV import, replication… See Task Manager Operations.
- scraper#
A task that fetches data from an external source on a schedule and updates the corresponding series.
- dashboard#
A saved arrangement of figures and balances used to view series together. See Dashboards.
- figure#
A single plot or table within a dashboard.
- balance#
An accounting-style aggregation that sums the signed contributions of several series (for example production minus consumption). See Dashboards.
- value date#
The timestamp of a data point — when the observation or forecast applies.
- insertion date#
- revision date#
When a value was recorded in the store. Each update creates a new revision identified by its insertion date; reading a series as of an insertion date is how the Refinery time-travels.
- source#
Where a series comes from:
local(this instance) or the name of a remote Refinery it is replicated from.- metadata#
Key/value information attached to a series. User metadata is free-form and versioned; internal metadata is maintained by the system (value type, timezone-awareness, supervision status, source…).
- tzaware#
- tznaive#
Whether a series’ timestamps carry a timezone (
tzaware) or not (tznaive).
Usages#
The time series repository can be used from the user interface. It has also been prominently designed with a data analyst friendly Python level API (in addition to the standard REST API).
The refinery can be used:
as a standalone tool
as a framework to build your own applications on top
as a library that you can embed in your (Python) applications
It provides several extension points through a plugin system. The API can be extended, the formula system can receive new custom operators/functions.
It has been designed to cooperate nicely within any existing IT infrastructure addressing time series data needs.
Its primary purposes and design goals are:
to give maximum autonomy to non-programmers working with time series data (analysts, data scientists), without having to beg the IT department
to give very user-friendly APIs to manipulate data and computations
to provide an extensive audit trail from the acquired external data to the final dashboard product (it makes it easy to understand the provenance of all the data)
to foster good data governance and collaboration over the time series repository
all that while providing an efficient storage system and horizontal scalability
License#
The Timeseries Refinery - Timeseries management tool
Copyright (C) 2024 Pythonian
This library is an open source software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.