The Timeseries Refinery — full documentation Source: https://refinery.docs.pythonian.fr/ =============================================================================== # https://refinery.docs.pythonian.fr/index.html =============================================================================== The Timeseries Refinery *********************** The Timeseries Refinery is a platform to store, transform and monitor time series data. It keeps a full revision history of every series — enabling time-travel and backtesting — centralizes data from internal silos and external APIs, and exposes a shared, curated catalog to its users, from Python, the browser, Excel or an MCP server. At a glance, the Refinery lets you: * store **versioned** time series with full history (time-travel, backtesting); * **centralize** data from internal silos and external APIs, queried on the fly or scraped on schedule; * **automate** data processes (ETL, predictions, computations, sanity checks…); * **aggregate** and **transform** series with formulas; * build a common **catalog** — a shared, curated data referential. [image] Content ^^^^^^^ * Getting Started * Installation * The different ways to install (pypi, docker, ansible) * Installing on Linux, Macos, Windows * Upgrading and migrating * Installing older versions, migrating from older versions * Overview * Purpose * Core objects * Usages * License * Working with the Refinery * Basics * General setup reminder * How to create / update / read "primary" time series * How to Exists / Rename / Delete time series * Going further: playing with revisions * Adding and fetching metadata * Using formulas : create a formula * Similarities and differences with primary series * Using the formula editor * Seeing all the formulas * Create a time series information system * Pipelines : starting with injecting data * Using the task manager * Writing and using scrapers for primary series * Building formulas on top to get curated series * Feeding a model with series, getting the output in the system, with the task manager * To the next level * Doing your own app using the Refinery * Data acquisition: a structured path * Performance aspects: stored time series and formulas * Formulas: when to use a cache/materialized view * How to cache formulas * Tasks system: how to organize and schedule tasks * Data architecture at scale: putting the Data Mesh to work * Core Components * Working with series in Python * Table of Contents * Creating a series * Updating a series * Working with versions * Point and version erasure * Point erasure with NaN * Version erasure with strip * Retrieving history * Working with metadata * Replacing a series entirely * Checking series existence * Renaming a series * Deleting a series * Finding series * Getting series information * Working with logs * Staircase operations * Basic staircase * Block staircase * Other use cases * Time Series Operations API * Working with series in the browser * Table of Contents * Creating a series * Importing from a CSV * Exploring history * Working with metadata * Renaming a series * Deleting a series * Finding series * Getting series information * Working with logs * Viewing series * Supervision * The supervision workflow * Making manual corrections * Viewing data provenance * Understanding supervision status * The web editor * Selecting and navigating * Editing values * Editing several series at once * Decimal formatting * Supervision API Reference * Formulas (computed series) * Table of Contents * Introduction * Creating Your First Formula * Formula Language Basics * Updating a Formula * Deleting a Formula * The formula editor * Using Formula Series * Formula-Specific API Methods * Formula Dependencies * Formula Insertion Dates * Formula Evaluation Context * Additional Formula Methods * Renaming Series and Formula Propagation * Deleting Series Referenced in Formulas * Performance and Caching * Advanced: Creating Custom Operators * Declaring a new operator * More Transformation Examples * Autotrophic series operator * Formula Language Reference * Formula API Reference * Formula Language Reference * Table of Contents * Getting Started * Your First Formula * Basic Syntax Rules * Language Basics * Data Types * Series Access and Options * Basic Series Access * Series Options * Options Operator * Arithmetic and Mathematical Operations * Scalar Operations * Series-to-Series Operations * Mathematical Functions * Trigonometric Functions * Temporal Operations * Resampling (Downsampling) * Upsampling (Increasing Frequency) * Time Shifting * Rolling Operations * Cumulative Operations * Integration * Block Staircase (Future Slicing) * Aggregation and Statistics * Row-wise Operations * Day-of-Year Aggregation (Folded Series) * Practical Aggregation Examples * Data Processing and Filtering * Conditional and Comparison Operations * Clipping and Bounds * Forward Curves * Priority and Layering * Date-based Slicing * Date-based Filtering with Cron Rules * Revision Date Control * Date and Time Functions * Current Date and Time * Date Creation * Date Arithmetic * Month Boundary Functions * Timezone Operations * Holiday Calendar Series * Special Series Generation * Advanced Series Discovery * Dynamic Series Selection * Basket-based Operations * Formula cache * When to use it * How it works * Configuring a policy * Cache operations * Programmatic access * Groups * Table of Contents * Introduction and Stored Groups * Group Formulas * Group Formula Operators * Creating and Using Group Formulas * Group Formula Metadata * Formula-Specific Methods * Formula Bindings: Creating Groups from Series Formulas * Common Use Cases * Group API Reference * Primary Group Operations * Formula Group Operations * Folders * The folders page * Managing folders and series * Setting a folder from tsinfo * Finding series by folder * Programmatic access * Search Query Language Reference * Table of Contents * Quick Start * Your First Search * More find API options * Finding Groups * Saving Searches with Baskets * The basket editor * Query Syntax * Structure * Basic Syntax Rules * Basic Search Operators * Name-based Search: "by.name" * Metadata Key Search: "by.metakey" * Metadata Value Search: "by.metavalue" * Everything Operator: "by.everything" * Timezone-Aware Filter: "by.tzaware" * Metadata Value Search: "by.metaitem" * Internal Metadata Search: "by.internal-metaitem" * Internal Metadata Key Search: "by.internal-metakey" * Basket Search: "by.basket" * Supervision status shortcuts * Value Comparison Search * Logical Operators * AND Operator: "by.and" * OR Operator: "by.or" * NOT Operator: "by.not" * Formula-Specific Operators * Formula Detection: "by.formula" * Formula Content Search: "by.formulacontents" * Tree and Cache Operators * Tree Path Filters * Cache Policy Filters * Task Manager Operations * Introduction * Overview * Installation * Quick start * Time Series Monitoring * Overview * Adding Series to Monitor * Understanding the Status Dashboard * Status Indicators * Monitoring Features * Series Information * Formula Dependencies * Namespaces and Organization * Removing Series * Excel addin * What * Install Xlwings client * Uninstall Xlwings client * Base use * Configuration options * Prefix name options * Layout * Revisiting the past * "Not A Number" handling * Resampling * Common pitfalls * Testing * Dashboards * The dashboard admin page * Managing dashboards * Viewing a dashboard * Managing figures * Figure Types * Timeseries * Table * Scatter * Daily Profile * Candlestick * Balance * Group * Waterfall * History heatmap * History curve * Model statistics * Balances * Programmatic access * Access and deployment * The MCP server * What it exposes * Connecting a client * Authentication and roles * Built-in guidance * Available tools * The HTTP API * Interactive reference * Speaking to it directly * Securing the Refinery * Setting up basic auth * Working with OpenID Connect * Setting up KeyCloak clients * Confidential client * Public client * Setting up the Refinery configuration files * Associating roles to users * Using the command line * Using the settings section * Settings * Horizons * Timezones * Users * Operations Guide * Configuration Reference * Configuration File Locations and Discovery * Required Sections * [dburi] Section - Database Connections * [storage] Section - Series Storage Backends * Optional Sections * [sources] Section - Data Mesh Configuration * [auth] Section - Authentication for External Sources * [server-auth] Section - Server Authentication Configuration * [dashboard] Section - External Dashboard URL Configuration * Examples and Use Cases * Basic Single Instance Setup * Multi-Database Development Setup * Data Mesh Configuration with Client Authentication * Enterprise Server Authentication Configuration * Best Practices * Naming Conventions * Data Governance * Formula Development * Dual Storage Operations * Configuration * Moving series between backends * Operational notes * Series replication * replicate_series_from_refinery * replicate_from_basket * Formula Debugging Techniques * Table of Contents * Common Semantic Issues * Progressive Debugging Strategy * Using the Formula Editor * Inspect Intermediate Results with python * Data Alignment Problems * Series with Different Frequencies * Different Date Ranges * Date Range Slicing for Optimization * Missing Data Propagation * Understanding Holes propagation * Date and Time Semantic Errors * TZ-Aware vs TZ-Naive: A Fundamental Distinction * Time Shifting for Business Period Alignment * Formula Dependencies and Recursion * Deep Formula Performance * Understanding the Reactive System * Troubleshooting * Installation Issues * Runtime Issues * Web Interface Issues * Performance Issues * Migration Issues * Getting Help * API documentation * Base Series Operations * tsa.get * "mainsource.get()" * tsa.update * "mainsource.update()" * tsa.multi_update * "mainsource.multi_update()" * tsa.replace * "mainsource.replace()" * tsa.exists * "mainsource.exists()" * tsa.source * "mainsource.source()" * tsa.delete * "mainsource.delete()" * tsa.rename * "mainsource.rename()" * tsa.type * "mainsource.type()" * tsa.interval * "mainsource.interval()" * tsa.inferred_freq * "mainsource.inferred_freq()" * tsa.insertion_dates * "mainsource.insertion_dates()" * tsa.history * "mainsource.history()" * tsa.staircase * "mainsource.staircase()" * tsa.block_staircase * "mainsource.block_staircase()" * tsa.strip * "mainsource.strip()" * tsa.log * "mainsource.log()" * tsa.metadata * "mainsource.metadata()" * tsa.internal_metadata * "mainsource.internal_metadata()" * tsa.replace_metadata * "mainsource.replace_metadata()" * tsa.update_metadata * "mainsource.update_metadata()" * tsa.old_metadata * "mainsource.old_metadata()" * tsa.list_metadata_keys * "mainsource.list_metadata_keys()" * tsa.list_internal_metadata_keys * "mainsource.list_internal_metadata_keys()" * tsa.tree * "mainsource.tree()" * tsa.path_series * "mainsource.path_series()" * tsa.series_path * "mainsource.series_path()" * tsa.set_series_path * "mainsource.set_series_path()" * tsa.delete_path * "mainsource.delete_path()" * tsa.rename_path * "mainsource.rename_path()" * tsa.info * "mainsource.info()" * tsa.sources * "mainsource.sources()" * Search and Baskets * tsa.find * "mainsource.find()" * tsa.catalog * "mainsource.catalog()" * tsa.register_basket * "mainsource.register_basket()" * tsa.basket * "mainsource.basket()" * tsa.basket_definition * "mainsource.basket_definition()" * tsa.list_baskets * "mainsource.list_baskets()" * tsa.rename_basket * "mainsource.rename_basket()" * tsa.delete_basket * "mainsource.delete_basket()" * Groups * tsa.group_exists * "mainsource.group_exists()" * tsa.group_get * "mainsource.group_get()" * tsa.group_update * "mainsource.group_update()" * tsa.group_replace * "mainsource.group_replace()" * tsa.group_delete * "mainsource.group_delete()" * tsa.group_rename * "mainsource.group_rename()" * tsa.group_type * "mainsource.group_type()" * tsa.group_source * "mainsource.group_source()" * tsa.group_interval * "mainsource.group_interval()" * tsa.group_inferred_freq * "mainsource.group_inferred_freq()" * tsa.group_insertion_dates * "mainsource.group_insertion_dates()" * tsa.group_history * "mainsource.group_history()" * tsa.group_log * "mainsource.group_log()" * tsa.group_metadata * "mainsource.group_metadata()" * tsa.group_internal_metadata * "mainsource.group_internal_metadata()" * tsa.group_old_metadata * "mainsource.group_old_metadata()" * tsa.update_group_metadata * "mainsource.update_group_metadata()" * tsa.replace_group_metadata * "mainsource.replace_group_metadata()" * tsa.group_find * "mainsource.group_find()" * tsa.group_catalog * "mainsource.group_catalog()" * Supervision * tsa.edited * "mainsource.edited()" * tsa.supervision_status * "mainsource.supervision_status()" * Formulas * tsa.register_formula * "mainsource.register_formula()" * tsa.formula * "mainsource.formula()" * tsa.oldformulas * "mainsource.oldformulas()" * tsa.eval_formula * "mainsource.eval_formula()" * tsa.formula_depth * "mainsource.formula_depth()" * tsa.formula_components * "mainsource.formula_components()" * tsa.depends * "mainsource.depends()" * tsa.register_formula_bindings * "mainsource.register_formula_bindings()" * tsa.bindings_for * "mainsource.bindings_for()" * tsa.register_group_formula * "mainsource.register_group_formula()" * tsa.group_formula * "mainsource.group_formula()" * tsa.group_formula_depth * "mainsource.group_formula_depth()" * tsa.group_eval_formula * "mainsource.group_eval_formula()" * tsa.group_depends * "mainsource.group_depends()" * Excel * tsa.values_markers_origins * "mainsource.values_markers_origins()" * Dashboards * tsa.dashboard.update_figure * "DashboardAPI.update_figure()" * tsa.dashboard.figures_catalog * "DashboardAPI.figures_catalog()" * tsa.dashboard.figure_exists * "DashboardAPI.figure_exists()" * tsa.dashboard.get_figure * "DashboardAPI.get_figure()" * tsa.dashboard.figure_dependents * "DashboardAPI.figure_dependents()" * tsa.dashboard.delete_figure * "DashboardAPI.delete_figure()" * tsa.dashboard.rename_figure * "DashboardAPI.rename_figure()" * tsa.dashboard.dashboards_catalog * "DashboardAPI.dashboards_catalog()" * tsa.dashboard.update_dashboard * "DashboardAPI.update_dashboard()" * tsa.dashboard.dashboard_exists * "DashboardAPI.dashboard_exists()" * tsa.dashboard.get_dashboard * "DashboardAPI.get_dashboard()" * tsa.dashboard.delete_dashboard * "DashboardAPI.delete_dashboard()" * tsa.dashboard.rename_dashboard * "DashboardAPI.rename_dashboard()" * tsa.dashboard.balance_list * "DashboardAPI.balance_list()" * tsa.dashboard.balance_formula * "DashboardAPI.balance_formula()" * tsa.dashboard.balance_exists * "DashboardAPI.balance_exists()" * tsa.dashboard.update_balance * "DashboardAPI.update_balance()" * tsa.dashboard.rename_balance * "DashboardAPI.rename_balance()" * tsa.dashboard.delete_balance * "DashboardAPI.delete_balance()" * Formula cache * tsa.new_cache_policy * "mainsource.new_cache_policy()" * tsa.edit_cache_policy * "mainsource.edit_cache_policy()" * tsa.delete_cache_policy * "mainsource.delete_cache_policy()" * tsa.set_cache_policy * "mainsource.set_cache_policy()" * tsa.unset_cache_policy * "mainsource.unset_cache_policy()" * tsa.cache_free_series * "mainsource.cache_free_series()" * tsa.cache_policies * "mainsource.cache_policies()" * tsa.cache_policy_series * "mainsource.cache_policy_series()" * tsa.cache_series_policy * "mainsource.cache_series_policy()" * tsa.has_cache * "mainsource.has_cache()" * tsa.delete_cache * "mainsource.delete_cache()" * tsa.refresh_series_policy_now * "mainsource.refresh_series_policy_now()" * Tasks * tsa.task.schedule * "TaskAPI.schedule()" * tsa.task.prepare * "TaskAPI.prepare()" * tsa.task.unprepare * "TaskAPI.unprepare()" * tsa.task.tasks * "TaskAPI.tasks()" * tsa.task.prepared * "TaskAPI.prepared()" * Administration * tsa.set_user_role * "mainsource.set_user_role()" * tsa.user_roles * "mainsource.user_roles()" * tsa.change_backend * "mainsource.change_backend()" =============================================================================== # https://refinery.docs.pythonian.fr/getting_started/index.html =============================================================================== Getting Started *************** Contents: ^^^^^^^^^ * Installation * The different ways to install (pypi, docker, ansible) * Installing on Linux, Macos, Windows * Upgrading and migrating * Installing older versions, migrating from older versions * Overview * Purpose * Core objects * Usages * License * Working with the Refinery * Basics * Create a time series information system * To the next level =============================================================================== # https://refinery.docs.pythonian.fr/getting_started/installation.html =============================================================================== Installation ************ The different ways to install (pypi, docker, ansible) ===================================================== The installation can be realised by three means: * directly from pypi (the public one, where one finds a stable - if old - version of the refinery, or the private ones at *pypi.pythonian.fr* or *pypi-pro.pythonian.fr*, where we make the latest fixes and major versions available to subscribers) * from docker images (available to subscribers) * via an Ansible-based deployment system that automates installation, configuration and updates of refinery instances (available to subscribers) To get a fresh installation of the Refinery on your Python environment, just type: pip install tshistory_refinery For an upgrade, create a new virtual environment and proceed like with a new installation. Installing on Linux, Macos, Windows =================================== Once pip-installed, you need also a working Postgres database. On Linux, getting postgres can be made using https://www.postgresql.org/download/linux/. On MacOs, doing a: brew install postgresql For Windows: get it from https://www.postgresql.org/download/windows/. Once installed and configured, you can create first a postgresql database: $ createdb my_time_series Then initialize the database schema: $ tsh init-db postgresql:///my_time_series --no-dry-run Last item: we need a configuration file "tshistory.cfg" in the home directory, containing: [dburi] refinery = postgresql:///my_time_series From this point you can start a local server and start using the refinery. tsh webstart Note: The *tsh webstart* command is suitable for a local development environement. For production purposes you probably want to use a wsgi container such as *gunicorn* or *waitress*.The *app* object used by wsgi containers can be found at *tshistory_refinery.wsgi:app*. Upgrading and migrating ======================= Pythonian publishes a major release every year. Also a number of smaller bug fix releases typically happen. There are two aspects to the upgrade: * deploying the new code version * running the data migration (if there is one) For a pypi installed instance, upgrading is as simple as doing a: pip install tshistory_refinery --upgrade Note: **Upgrading to 1.0**The 1.0 release (refinery_pro 0.5.0) introduces the dual storage system and mandates a "[storage]" section in "tshistory.cfg". After deploying the new code and *before* running the migration, add: [storage] .path = /path/to/storage Your existing series stay on the PostgreSQL backend; the path hosts the new filesystem (fs1) backend. See the configuration reference for the details. After having done that, you should run the migration, like this: tsh migrate The migrate command accepts a *--no-interactive* parameter to run it without confirmation or question. It can run like this: (base) aurelien@trantor:~$ tsh migrate refinery --no-interactive Running migrations for tshistory. Versions: from 0.20.0 to 0.20.0 Nothing to migrate for `tshistory`, skipping. Running migrations for tshistory-supervision. Versions: from 0.13.0 to 0.13.0 Nothing to migrate for `tshistory-supervision`, skipping. Running migrations for tshistory-formula. Versions: from 0.16.0 to 0.16.2 Migration plan for `tshistory-formula`: ['0.16.1'] Running migrations for tshistory-refinery. Versions: from 0.9.0 to 0.9.1 Migration plan for `tshistory-refinery`: ['0.9.1'] Running migrations for tswatch. Versions: from 0.1.0 to 0.3.0 Migration plan for `tswatch`: ['0.3.0', '0.2.0'] Running migrations for refinery-pro. Versions: from 0.1.0 to 0.1.0 Nothing to migrate for `refinery-pro`, skipping. Warning: You should have a dev or preprod environment to test your migration against. Always make sure you have tested the migration on this environment, with a fresh production database dump, to make sure it will work smoothly. The migrations are intensely tested before shipping but one is never too careful. Installing older versions, migrating from older versions ======================================================== We will herein document installation and migration starting from the *0.8.3* release, which has been available on PyPi since september 2023. If one wants to install this old release today (this is written in february 2025), this will fail unless: * the Python version is 3.9 (pandas 1.5 is still available in binary form for this version, for later versions, a compilation from source will happen, which often ends badly) * some dependencies, direct and indirect, must be corrected (as they have evolved to newer, incompatible versions) You probably want to create a *requirements.txt* file and put this inside: numpy==1.26.4 werkzeug==2.3.8 tshistory_refinery==0.8.3 tshistory==0.19.4 You then need to do: $ pip install -r requirements.txt Let's assume you want to upgrade from this to newer versions. We will describe the steps to go from 0.8.3 to 0.9.0, then 0.9.1, then 0.9.2, and lastly to the "pro" version. Warning: It is possible in theory to jump several versions at once. However due to bugs in the earlier versions of the migration system (up to 0.9.1) it is advised to perform it step-wise. So to go from *0.8.3* to *0.9.0* one should: * adjust the *pip install* stanza with a *-i https://:@pypi.pythonian.fr/simple* * also populate a requirements.txt with the following: tshistory_refinery==0.9.0 tshistory==0.20.0 tsview==0.19.0 tshistory_supervision==0.13.0 tshistory_formula==0.16.1 tshistory_xl==0.8.0 rework==0.17.0 rework_ui==0.16.0 Lastly: $ pip install -r requirements.txt The *refinery.cfg* configuration file is now deprecated and you should make sure you have a properly formed *tshistory.cfg* file instead. For instance, if you had a *refinery.cfg* with the following: [db] uri = postgresql:///myrefinery You will want this *tshistory.cfg*: [dburi] refinery = postgresql:///myrefinery For *0.9.0* to *0.9.1*: tshistory_refinery==0.9.1 tshistory==0.20.1 tsview==0.20.1 tshistory_supervision==0.13.1 tshistory_formula==0.16.2 tshistory_xl==0.8.0 rework==0.17.2 rework_ui==0.16.0 For *0.9.1* to *0.9.2*: $ pip install tshistory_refinery==0.9.2 --upgrade To get the *Pro* version, from *0.9.2*: $ pip install refinery_pro==0.2.0 $ dashboard init-db $ tsh migrate --no-interactive =============================================================================== # https://refinery.docs.pythonian.fr/getting_started/package_overview.html =============================================================================== Overview ******** Purpose ======= The "timeseries refinery" is a powerful tool to build complete time series information systems. It provides: * a **versioned** time series (compact) storage system with a simple yet powerful API (direct in python + REST) * an edition/supervision workflow for experts * **computed** versioned time series using a simple yet expressive dedicated **formula language** (think about Excel formulas, but simpler and specialised for time series) with a simple API * a cache system for the computed series * a **central repository** of time series (stored or computed) * rich **metadata** support and **querying/filtering** over various kinds of metadata * support for **timezone aware** and naive time series * a **data-mesh** distributed architecture allowing to connect to other refineries (for **collaborative work**) or other time series silos * a task manager and a mini-framework to handle regular **ingestion** of time series from external or internal sources, and also **model runs** * a tool to **detect/watch** the series lagging behind (because of failing sources) * a powerful **low-code declarative dashboard** system (specially targeted at energy comodity markets) * a **rich web user interface** allowing exploration of the catalog, visualisation and edition of series, formulas, caches, tasks, watchers and dashboards It can be used as a building block for machine learning, model optimization and validation, both for inputs and outputs. Core objects ============ At the centre sits the **series**: a named, versioned sequence of timestamped values. Everything else builds on it. A **formula** computes a new series from existing ones; a **group** bundles related series together; a **basket** is a saved query that resolves to a set of series; a **folder** places a series in a tree for browsing. Around these, **supervision** records manual corrections, the **formula cache** materializes computed series for speed, **tasks** and **scrapers** feed data in on a schedule, and **dashboards** present it. One distinction underlies the whole system: the **value date** of a point (when it happened) versus its **insertion date** (when it was recorded). Keeping both is what gives the Refinery its time-travel and backtesting abilities. series primary series A named, versioned sequence of timestamped values, stored directly in the database. Every update is kept as a revision, so the series can be read as it stood at any past insertion date. See Working with series in Python. formula A computed series defined by a lisp-like expression over other series. Formulas are read-only and evaluated on demand; they inherit their history from their inputs. See Formulas (computed series) and the Formula Language Reference. group An ensemble of related series sharing a common structure — typically the members or scenarios of a forecast. See Groups. basket A saved search query. It resolves, on demand, to the set of series (or groups) that currently match it. See Search Query Language Reference. folder A position in a path tree used to organise series for browsing. A series belongs to at most one folder. See Folders. supervision The mechanism for applying manual corrections on top of an automated feed while preserving the original data. A series is accordingly "unsupervised", "supervised" or "handcrafted". See Supervision. formula cache Materialized formula results, refreshed by a policy, so that expensive formulas are served quickly instead of recomputed on every read. See Formula cache. task A background job run by the scheduler: scraping, cache refresh, CSV import, replication… See Task Manager Operations. scraper A task that fetches data from an external source on a schedule and updates the corresponding series. dashboard A saved arrangement of figures and balances used to view series together. See Dashboards. figure A single plot or table within a dashboard. balance An accounting-style aggregation that sums the signed contributions of several series (for example production minus consumption). See Dashboards. value date The timestamp of a data point — when the observation or forecast applies. insertion date revision date When a value was recorded in the store. Each update creates a new revision identified by its insertion date; reading a series *as of* an insertion date is how the Refinery time-travels. source Where a series comes from: "local" (this instance) or the name of a remote Refinery it is replicated from. metadata Key/value information attached to a series. *User* metadata is free-form and versioned; *internal* metadata is maintained by the system (value type, timezone-awareness, supervision status, source…). tzaware tznaive Whether a series' timestamps carry a timezone ("tzaware") or not ("tznaive"). Usages ====== The time series repository can be used from the user interface. It has also been prominently designed with a data analyst friendly Python level API (in addition to the standard REST API). The refinery can be used: * as a standalone tool * as a framework to build your own applications on top * as a library that you can embed in your (Python) applications It provides several extension points through a plugin system. The API can be extended, the formula system can receive new custom operators/functions. It has been designed to cooperate nicely within any existing IT infrastructure addressing time series data needs. Its primary purposes and design goals are: * to give maximum **autonomy** to non-programmers working with time series data (analysts, data scientists), without having to beg the IT department * to give very **user-friendly APIs** to manipulate data and computations * to provide an extensive **audit trail** from the acquired external data to the final dashboard product (it makes it easy to understand the provenance of all the data) * to foster good **data governance** and **collaboration** over the time series repository * all that while providing an **efficient storage** system and **horizontal scalability** License ======= The Timeseries Refinery - Timeseries management tool Copyright (C) 2024 Pythonian This library is an open source software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. To know more on LGPL license, click here. =============================================================================== # https://refinery.docs.pythonian.fr/getting_started/tutorials/index.html =============================================================================== Working with the Refinery ************************* In this large section, we will gradually introduce all the important notions needed to exploit the Refinery up to its full potential. We start with an exposition of the basic notions, API and UI usage. Once familiarized with the core notions, we will show how to build a simple Time Series Information System. Only then will we expose the most advanced aspects, including performance management and the distributed system aspects. Contents: ^^^^^^^^^ * Basics * Create a time series information system * To the next level =============================================================================== # https://refinery.docs.pythonian.fr/getting_started/tutorials/beginner.html =============================================================================== Basics ****** General setup reminder ====================== Setup your machine ------------------ In this chapter, we assume you have setup a Refinery instance on some server. Let's install the client on you local machine: $ pip install tshistory_refinery And save the config file "tshistory.cfg" in the home directory, containing: [dburi] refinery = https://refinery..pythonian.fr/api If the instance is behind some security protection, you have to fill the *[auth]* section. [auth] refinery.uri = https://refinery..pythonian.fr/api refinery.login = refinery.password = That's for http basic auth. For a more complicated security setup, see the security chapter. The naming convention : an essential step ----------------------------------------- First and foremost, users of the refinery need to gather and define together the naming convention. This is an important aspect of the **data governance**. This convention concerns all the series (primary and formulas) available in the data referential. Every person of the team needs to know it very well. They will define new data following this convention and find quickly and efficiently the data they need. Here are some tips: * define the separator of your keywords. Let's choose a dot for our example "." * gather all the types of keywords you could need. Let's assume we work with fruits transactions, here is our list: * type of aliment * type of fruit * country of origin * country of selling * type of transaction (sell/buy) * unit (currency/weight) * farmer * buyer * granularity * choose the most intuitive order of keywords: **aliment.fruit.transa ction.farmer.buyer.from_country.to_country.unit.granularity** example : *fruit.banana.buy.misterfield.superfrenchshop.civ.fr.eur/t.d* Note: Some of these aspects probably belong to the **metadata** of the series. However a good name goes a long way when it comes to making a data referential easily usable for everyone. So encoding some metadata both in metadata proper and in the title definitely makes sense. Initialize the timeseries client -------------------------------- from tshistory.api import timeseries tsa = timeseries() How to create / update / read "primary" time series =================================================== Create a series --------------- import pandas as pd ts = pd.Series( data=[1, 2, 3], index=pd.date_range(start="2024-11-01", freq="D", periods=3) ) seriesname = "fruit.banana.buy.misterfield.superfrenchshop.civ.fr.eur/t.d" tsa.update(seriesname, ts, author="fruit@lover") Get your series --------------- >>> ts = tsa.get(seriesname) >>> ts 2024-11-01 1.0 2024-11-02 2.0 2024-11-03 3.0 Name: fruit.banana.buy.misterfield.superfrenchshop.civ.fr.eur/t.d, dtype: float64 Update your series ------------------ Overwrite existing values and add one for the 4th of November: ts = pd.Series( data=[5, 6, 7, 10], index=pd.date_range(start="2024-11-01", freq="D", periods=4) ) seriesname = "fruit.banana.buy.misterfield.superfrenchshop.civ.fr.eur/t.d" tsa.update(seriesname, ts, author="fruit@lover") Get the new version of the timeseries: >>> ts = tsa.get(seriesname) >>> ts 2024-11-01 5.0 2024-11-02 6.0 2024-11-03 7.0 2024-11-04 10.0 Freq: D, Name: fruit.banana.buy.misterfield.superfrenchshop.civ.fr.eur/t.d, dtype: float64 Note: The update is as important as get.Let's explain its semantics: * its input series is used to patch the existing series state * existing points with unchanged values will just be dropped * new points will be added * modified points will be updated * points with nan values will erase existing points * if there is no difference between the inserted points and the ones in base, no revision is created * it returns a series of the points that have been either added, update or erased * if the series doesn't already exist, it is created How to Exists / Rename / Delete time series =========================================== >>> tsa.exists(seriesname) True >>> tsa.rename(seriesname, 'new.name') >>> tsa.exists(seriesname) False >>> tsa.delete('new.name') >>> tsa.exists('new.name') False Going further: playing with revisions ===================================== Now let's assume that we work with a forecast: # create a forecast series insertion_dates = pd.date_range( start=pd.Timestamp('2024-11-01T09:00+00:00'), end=pd.Timestamp('2024-11-04T09:00+00:00'), freq='D' ) ts = pd.Series( [1, 2, 3], index = pd.date_range(start='2024-11-01', periods=3, freq='D') ) for idx, insertion_date in enumerate(insertion_dates): tsa.update( 'my_morning_forecast', ts+idx, 'sensei', insertion_date=insertion_date ) Let's use the get operator. It will give us the last version of each timestamp. >>> tsa.get("my_morning_forecast") 2024-11-01 1.0 2024-11-02 2.0 2024-11-03 2.0 2024-11-04 2.0 2024-11-05 3.0 2024-11-06 4.0 Name: my_morning_forecast, dtype: float64 Now, let's investigate our forecast versions. >>> tsa.history("my_morning_forecast") { Timestamp('2024-11-01 09:00:00+0000', tz='UTC'): 2024-11-01 1.0 2024-11-02 2.0 2024-11-03 3.0 Name: my_morning_forecast, dtype: float64, Timestamp('2024-11-02 09:00:00+0000', tz='UTC'): 2024-11-01 1.0 2024-11-02 2.0 2024-11-03 3.0 2024-11-04 4.0 Name: my_morning_forecast, dtype: float64, Timestamp('2024-11-03 09:00:00+0000', tz='UTC'): 2024-11-01 1.0 2024-11-02 2.0 2024-11-03 2.0 2024-11-04 3.0 2024-11-05 4.0 Name: my_morning_forecast, dtype: float64, Timestamp('2024-11-04 09:00:00+0000', tz='UTC'): 2024-11-01 1.0 2024-11-02 2.0 2024-11-03 2.0 2024-11-04 2.0 2024-11-05 3.0 2024-11-06 4.0 Name: my_morning_forecast, dtype: float64 } If we want to see only what changed from one version to the next: >>> tsa.history("my_morning_forecast", diffmode=True) { Timestamp('2024-11-01 09:00:00+0000', tz='UTC'): 2024-11-01 1.0 2024-11-02 2.0 2024-11-03 3.0 Name: my_morning_forecast, dtype: float64, Timestamp('2024-11-02 09:00:00+0000', tz='UTC'): 2024-11-04 4.0 Name: my_morning_forecast, dtype: float64, Timestamp('2024-11-03 09:00:00+0000', tz='UTC'): 2024-11-03 2.0 2024-11-04 3.0 2024-11-05 4.0 Name: my_morning_forecast, dtype: float64, Timestamp('2024-11-04 09:00:00+0000', tz='UTC'): 2024-11-04 2.0 2024-11-05 3.0 2024-11-06 4.0 Name: my_morning_forecast, dtype: float64 } Let's assume we want the forecast of the 3rd of Nov at 12am, after the morning run: >>> tsa.get("my_morning_forecast", revision_date=pd.Timestamp("2024-11-03T12:00:00+00:00")) 2024-11-01 1.0 2024-11-02 2.0 2024-11-03 2.0 2024-11-04 3.0 2024-11-05 4.0 Name: my_morning_forecast, dtype: float64 Adding and fetching metadata ============================ tsa.update_metadata( "my_morning_forecast", { 'fruit': 'orange', 'buyer': 'superfrenchshop', 'seasonal': 0, 'calibration': '2024-10-01' } ) >>> tsa.metadata("my_morning_forecast") { 'buyer': 'superfrenchshop', 'fruit': 'orange', 'seasonal': 0, 'calibration': '2024-10-01' } Note: Metadata is a very important tool to structure the time series referential. We insisted on the naming convention in the beginning and indeed good names should be good keys and embed a number of metadata aspects of the series. But it is not always enough.Metadata, at the Python API level, will be managed with flat Python dictionaries. Using formulas : create a formula ================================= A low code formula system has been implemented into the Refinery to transform your raw data into enriched indicators. Time travel capabilities are enabled in all levels of aggregation. tsa.register_formula( 'my_simple_formula', '(resample (series "my_morning_forecast") "h" #:method "interpolate" )' ) With this formula, we resample our daily series "my_morning_forecast" to an hourly series. To get the result of this formula, you can use the *get* method: >>> tsa.get("my_simple_formula") 2024-11-01 00:00:00 1.000000 2024-11-01 01:00:00 1.041667 2024-11-01 02:00:00 1.083333 2024-11-01 03:00:00 1.125000 2024-11-01 04:00:00 1.166667 ... 2024-11-05 20:00:00 3.833333 2024-11-05 21:00:00 3.875000 2024-11-05 22:00:00 3.916667 2024-11-05 23:00:00 3.958333 2024-11-06 00:00:00 4.000000 Freq: h, Name: my_simple_formula, Length: 121, dtype: float64 Let's add other series to create more complex formulas: ts = pd.Series( data=3, index=pd.date_range(start="2024-10-31", freq="h", periods=24) ) seriesname = "my_series_obs" tsa.update(seriesname, ts, author="fruit@lover") Now, let's create a formula with the priority operator. We can see that a formula can work with primary and/or formulas in a transparent way. tsa.register_formula( 'my_combined_formula', '(priority (series "my_series_obs") (series "my_simple_formula") )' ) Note: The priority operator combines its input series as layers. For each timestamp in the union of all series time stamps, the value comes from the first series that provides a value. >>> tsa.get("my_combined_formula") 2024-10-31 00:00:00 3.000000 2024-10-31 01:00:00 3.000000 2024-10-31 02:00:00 3.000000 2024-10-31 03:00:00 3.000000 2024-10-31 04:00:00 3.000000 ... 2024-11-05 20:00:00 3.833333 2024-11-05 21:00:00 3.875000 2024-11-05 22:00:00 3.916667 2024-11-05 23:00:00 3.958333 2024-11-06 00:00:00 4.000000 Name: my_combined_formula, Length: 145, dtype: float64 Note: See a more detailed description of the formula system in the following chapter : Formulas (computed series) Similarities and differences with primary series ================================================ You can easily determine if a series is a primary or a formula by using the *type* method : >>> tsa.type("my_simple_formula") 'formula' >>> tsa.type("my_series_obs") 'primary' Another way to identify the type of a series is to look at the internal metadata : >>> tsa.internal_metadata("my_simple_formula") { 'formula': '(resample (series "my_morning_forecast") "h" #:method "interpolate")', 'tzaware': False, 'index_type': 'datetime64[ns]', 'value_type': 'float64', 'contenthash': 'fd1ca7df8b9a543c4a232ac9125e89ab0cef493a', 'index_dtype': '>> tsa.internal_metadata("my_series_obs") { 'tzaware': False, 'tablename': 'my_series_obs', 'index_type': 'datetime64[ns]', 'value_type': 'float64', 'index_dtype': ' --namespace calibration Here are now the necessary python functions. Necessary imports: import json import pandas as pd from sklearn.linear_model import LinearRegression from sklearn.metrics import r2_score from tshistory.api import timeseries from dbcache.api import vkvstore Calibration per se: def calibrate(tsa): # input x = tsa.get('meteo.area.fr.2t.k.era5.obs.h') # output y = tsa.get('power.load.fr.mwh.entsoe.obs.h') x.name = 'x' y.name = 'y' common = pd.concat([x, y], axis=1, join='inner') predictor = LinearRegression(n_jobs=-1) predictor.fit(X=common[['x']], y=common[['y']]) slope = predictor.coef_[0][0] intercept = predictor.intercept_[0] # we want to store some indicator of fitness: r2 = r2_score( common[['y']], predictor.predict(common[['x']]) ) # we prepare the serialization of the calibration parameters # the pickle method should be avoided for archive purpose results = { 'slope': slope, 'intercept': intercept, 'r2': r2, } return results Save the calibration parameters: def save_coefficients(tsa, results): store = vkvstore(tsa.uri, 'calibration') # vkvstore use string as keys and bstring as values json.dumps(results) store.set( 'linear-calibration-key', json.dumps(results).encode('utf-8') ) Note: Note the “calibration” string in the vkvstore api which references the string used as namespace in the dbcache init-db command. Now for the forecasting, we need to load the saved parameters: def load_coefficients(tsa): store = vkvstore(tsa.uri, 'calibration') stored = store.get('linear-calibration-key') return json.loads(stored.decode('utf-8')) And finally read the input data, apply the model on it and store the results as a timeseries: def forecast(tsa, coefficients): x = tsa.get('meteo.area.fr.2t.k.era5.obs.h') predicted = x * coefficients['slope'] + coefficients['intercept'] name = 'power.load.fr.mwh.linear.fcst.h' tsa.update( name, predicted, 'linear-model' ) We can now assemble these operations in two distincts tasks: @task(inputs=(), domain='models') def linear_calibration(task): with task.capturelogs(std=True): tsa = timeseries() results = calibrate(tsa) save_coefficients(tsa, results) @task(inputs=(), domain='models') def linear_forecast(task): with task.capturelogs(std=True): tsa = timeseries() coefficients = load_coefficients(tsa) forecast(tsa, coefficients) This tasks can then be scheduled, accordingly to your calibration/forecasting rules. =============================================================================== # https://refinery.docs.pythonian.fr/getting_started/tutorials/advanced.html =============================================================================== To the next level ***************** In this chapter we will explore aspects of the system that go beyond the simplest steps taken to build a relatively simple time series information system. We will address: * performance issues regarding stored and computed series * working in depth with the task manager to build robust pipelines * how to efficiently organize the acquisition of a vast referential of raw series * good data architecture Doing your own app using the Refinery ===================================== The Refinery works well as a stand-alone tool, until you need to do any of the following: * data acquisition with dedicated tasks * new api points * new formula operators The first item is by far the most common. We propose to start with a skeleton app that we will call "myapp". It structure is as follows (standard Python package app): myapp ├── myapp │   └── __init__.py └── setup.py The *setup.py* should be minimal and may contain this: from setuptools import setup setup(name='myapp', version='0.1.0', author='MyCorp', author_email='myself@mycorp.fr', description='Provide scrapers and tasks for my refinery', packages=['myapp'], install_requires=[ 'tshistory_refinery' ], ) Data acquisition: a structured path =================================== In the chapter Pipelines : starting with injecting data we described an effective yet rudimentary strategy to implement data acquisition. The Refinery provides a tool to enable a better organization of this. We will organize the scraping into two modules: * *scrap_collection.py* * *scrap_config.py* The project application structure should be like this: myapp/ ├── myapp │   ├── __init__.py │   ├── scrap_collection.py │   └── scrap_config.py └── setup.py The first one, *scrap_collection*, will host all our scraping functions. The second one, *scrap_config*, will provide a global scraping configuration for the application and of course use the scraping functions defined in the first one. Let's examine *scrap_collection.py*, restarting from the minimalistic scraper we wrote for the Pipelines : starting with injecting data. import pandas as pd from entsoe import EntsoePandasClient from entsoe.exceptions import NoMatchingDataError def get_res_generation_forecast_entsoe(zone, fromdate, todate): metadata = { 'source': 'entsoe', 'country': zone, 'category': 'res generation', 'series-type': 'forecast' } try: df = EntsoePandasClient(api_key='apikey').query_wind_and_solar_forecast( zone, start=fromdate.tz_localize('Europe/Paris'), end=todate.tz_localize('Europe/Paris'), ) except NoMatchingDataError: return pd.DataFrame(), metadata return df, metadata Here are the differences with the previous one: * there is a *zone* parameter rather than a hard-coded 'FR' zone * we have a pair of *fromdate* and *todate* parameters * we don't insert any data, we just return what we got (a time series and a dict to hold the metadata) Let's try to do something out of this immediately, this time we will populate the *scrap_config* module: from functools import partial from tshistory_refinery.scrap import ( Scrap, Scrapers ) from myapp.scrap_collection import get_res_generation_forecast_entsoe SCRAPERS = Scrapers( Scrap( names={ 'power.prod.solar.fr.mwh.entsoe.fcst.h': 'Solar', 'power.prod.windonshore.fr.mwh.entsoe.fcst.h': 'Wind Onshore' }, func=partial(get_res_generation_forecast_entsoe, 'FR'), schedrule='0 5 * * * *', fromdate='(shifted (today) #:days -1)', todate='(shifted (today) #:days 7)', initialdate='(date "2015-1-1")' ), Scrap( names={ 'power.prod.solar.be.mwh.entsoe.fcst.h': 'Solar', 'power.prod.windoffshore.be.mwh.entsoe.fcst.h': 'Wind Offshore', 'power.prod.windonshore.be.mwh.entsoe.fcst.h': 'Wind Onshore' }, func=partial(get_res_generation_forecast_entsoe, 'BE'), schedrule='0 5 * * * *', fromdate='(shifted (today) #:days -1)', todate='(shifted (today) #:days 7)', initialdate='(date "2015-1-1")' ) ) Let's analyze this step by step, as there is a lot to unpack. As you can see at first glance, we can re-use our scraping functions for several uses and generate not just a pair of time series. At the top-level, we define this big global variable named SCRAPERS, which will indeed contain the scrapers hold by the *Scrapers* object. The *Scrapers* object will collect as many individual scrap configurations as needed (as coma separated objects). These are to be defined with the *Scrap* object. Let's look at the first one. names={ 'power.prod.solar.fr.mwh.entsoe.fcst.h': 'Solar', 'power.prod.windonshore.fr.mwh.entsoe.fcst.h': 'Wind Onshore' } The names in this case is a mapping from the final series name we want to the series names the scraping function will acquire (which is determined by the data provider, obtained through documentation ... or looking at what comes at the end of their api). A scraping function can get one or many series. func=partial(get_res_generation_forecast_entsoe, 'FR'), This *func* parameter takes the scraping function. It is called with *fromdate* and *todate* as *keyword* arguments — the parameter names are load-bearing — and only those defined by the scrap declaration are passed: funcname(fromdate: pd.Timestamp, todate: pd.Timestamp) The values are timezone-aware Pandas timestamps. Our concrete scrap function in fact wants a first *zone* parameter, which is why, to be compliant, we give *func* a partial application of the scrap function with a concrete value ('FR'). Hence we can re-use this technique for as many zones as upstream knows, and in our example with 'BE'. Next: the scheduling rule. schedrule='0 5 * * * *', This declares a cron rule with support for seconds. In this case we want it to run every hour of every day at minute 5 and second 0. Then come the parameters for the horizon/time windows. fromdate='(shifted (today) #:days -1)', todate='(shifted (today) #:days 7)', initialdate='(date "2015-1-1")' *initialdate* specifies the first time stamp wanted for the series, *fromdate* and *todate* specify the time window for a given time series update. Note: *initialdate* will be used once, when the series we want does not yet exist. The initial series history will be built from there. The three timestamps are defined using a mini expression language comparable to that of the formula language. The tree primitives are: * *today* -> returns a timestamp as of today * *date * -> builds a timestamp from a string * *shifted * -> shifts a date by a given delta Note: The for *shifted* is using a keyword notation. The available keywords are *#:hours*, *#:days*, *#:weeks*, *#:months* In our case, since we are fetching a 7 days forecast, it is entirely reasonable to fetch data from yesterday to the next 7 days. The "Scrap" object accepts a few more optional parameters: * "target_granularities" — a list of granularity suffixes (".h", ".d", ".m", ...). For a base series named "...raw", the framework automatically registers a resampled **formula** at each granularity (e.g. "prod.h" from "prod.raw"), so consumers can read the base or any coarser aggregate. * "resample_method" — the aggregation method used by those derived formulas (default ""mean""). * "fill" / "fill_limit" — a filling policy applied to the base series inside the derived formulas ("fill_limit" requires "fill"). * "precious" — when "True", protects the series: the destructive "fetch_history" backfill is refused. * "history" — when "True", the scraping function returns a full history (several revisions) instead of a single current series. For example, to scrape a raw series and automatically expose hourly, daily and monthly aggregates: Scrap( names={ 'power.prod.hydro.fr.raw': 'actual_generation_fr', }, func=partial(get_hydro_generation, 'FR'), schedrule='0 5 * * * *', fromdate='(shifted (today) #:days -1)', todate='(shifted (today) #:days 1)', initialdate='(date "2015-1-1")', target_granularities=['.h', '.d', '.m'], resample_method='mean', fill='ffill', fill_limit=4, ) This feeds the base "power.prod.hydro.fr.raw" series and registers three resampled formulas — "power.prod.hydro.fr.h", ".d" and ".m" — each a forward-filled mean of the base at the matching granularity. Well, having explained all of that, are we done yet ? Almost ! The last step will be about the final setup: we need the task system to be made aware of what we have done, and we will do it through a command line operation of the *myapp* application. This entails having a *cli* (for "command line interface") module in *myapp*, whose tree now looks like that: myapp/ ├── myapp │   ├── cli.py │   ├── __init__.py │   ├── scrap_collection.py │   └── scrap_config.py └── setup.py Let's write the *cli.py* module contents: import click from rework import api from tshistory.api import timeseries from tshistory_refinery import cache from myapp.scrap_config import SCRAPERS @click.group() def myapp(): pass @myapp.command(name='setup-scrapers') def setup_scrapers(): tsa = timeseries() engine = tsa.engine with cache.suspended_policies(engine): for scraper in SCRAPERS.scrapers.values(): scraper.prepare_task(engine) @myapp.command(name='first-imports') def first_imports(): tsa = timeseries() engine = tsa.engine for scr in SCRAPERS.scrapers.values(): if scr.initialdate is not None: inputdata = scr.build_inputs('fetch_history') inputdata['reset'] = 0 api.schedule( engine, 'fetch_history', inputdata=inputdata, domain='scrapers' ) Pfew ! Ah, the ultimate step: inform the *setup.py* of the existence of command line operations. We add a keyword *entry_point* to the *setup* function inside, like this, after the *install_requires* keyword: entry_points={ 'console_scripts': [ 'myapp=myapp.cli:myapp' ] } At installation time, this creates the *myapp* command line tool. You can launch the initial import (to fetch the history from *initialdate*) with: $ myapp first-imports And then schedule the regular updates like that: $ myapp setup-scrapers Now we are really set. Performance aspects: stored time series and formulas ==================================================== Managing a time series information system entails understanding a number of performance aspects. These are related to two main metrics: * data volumetry * amount of computation done On data, the standard metric in the industry is the number of points. This is too imprecise however to understand read and write performance aspects, and also the storage density aspect of the Refinery. The time series storage logical model is built on a tree of blocks. We basically store: * version entities, each pointing to an initial block * a tree of blocks of compressed points - up to 150 points can be lumped together into a single block This model is unremarkable when it comes to store a lot of versions with few points. The worst case would be one version with one point. However it shines when it comes to handling large amounts of points with a reasonnable amount of versions; for instance, meteo forecasts which come three to four times per day with 15 days worth of hourly forecasts. Density storage in these circumstances is excellent. On computation (for the formulas), the effort is directly proportional to: * the amount of data involved, * the complexity of involved operations. A simple *resample* formula is cheap and can be computed on the fly with a reasonably small overhead. But in a developped time series information system, formulas of formulas exist and it is possible to have a very deep level of formulas made of formulas until one hits the ground of stored time series, hence increasing latency, sometimes very noticeably. When the performance of complex formulas becomes an issue, it is possible to set up a regularly refreshed materialized view, or *cache*, which will make operational use of the complex formulas manageable again. This aspect is addressed in the next chapter. Formulas: when to use a cache/materialized view =============================================== There exist two main scenarios leading to use a formula cache: * formulas with autotrophic operators * very complex (computation intensive) formulas The first case relates to custom operators made to act as a proxy for another time series repository. In the Refinery we have two builtin operators that are autotrophic: *series* and *constant*. These operators do not transform data, they are a source of data. Writing an autotrophic operator is a viable option if (and only if): * the remote source provides an API that can be adapted for the needs of the Refinery, * the latency of the remote source remains reasonnably low. Even in the best case, such an operator can have some nasty performance characteristics from the point of view of the remote source: a new load at arbitrary times that interferes with the load as initially designed. Caching formulas made with such operators can help smooth out the load and establish good relationships with the upstream source. The case of computationally intensive formulas is simple: we need to have access to a freshly computed version of the formula. We will explain how to do this in the next chapter. How to cache formulas ===================== Theory ------ Caching a formula is not a matter of pushing a button and get a working cache ! That would be too easy ... The very nature of versioned time series forces us to think about how to do it properly. Let's have a look at a series from the point of view of the two time axis: [image] The blue lines represent new points along the value date axis. We have new blue lines for each version. On this graph, we should realize that we are seeing elements of a forecast, for many points go into the future. Understanding the relationship between the *value dates* (horizontal axis) and *insertion dates* (vertical axis), and their regularities, is key to set up a working cache policy. So let's define immediately the notion of *cache policy*. It is characterized by: * an *initial revision date* (or insertion date) * a *revision rule* for the advancement of the revision dates * a time window associated with the current revision date (*look_before* / *look_after*) Note: We use the terms *revision date* and *insertion date* as pure synonyms. If we apply these notions to the time series seen above, we can propose a cache policy like the following: [image] The optimal cache policy will exploit our knowledge of the time series temporality aspects. So with the *initial revision date* we put an arbitrary point in the past before which the time series formula has in effect no materialized counterpart. This is because old enough versions get more rarely accessed as time moves on. There has to be a compromise between time and space. Revisions before the *initial revision date* will be computed on the fly. The *revision rule* should match the expected refresh frequency of the time series. Since a formula's insertion dates is naturally the union of all the underlying components, it makes sense to think about this expected refresh frequency: we want to avoid getting spurious revisions, and we need to have all the relevant (business wise) ones. Lastly, the time window allows querying the system with limited bounds, so as to minimize the load when building the cache. For instance, using the full horizon all the time could have nasty performance implications. It must be configured so as to: * permit catching possible updates to the *observed* domain, * guarantee the lossless ingestion of the *predictive* part of the curve. Now let's observe how the presence of a formula cache affects the *reading* of such a time series: [image] Note: There is another aspect not mentioned in the above drawing: when the time delta between *now* and the latest cache revision date is two "ticks" behind according to the revision date rule (that is: potentially stale), we do actually use the formula to get the result. Practice -------- The Web UI provides a whole section dedicated to the maintenance of cache policies. [image] The list at the top shows the existing cache policies. We can create or edit a policy, leading us to the following policy creation form: [image] You need to give it a clear name. A cache policy will be associated with as many formulas as you want, which makes sense provided they have the same common time and data patterns, as explained in the previous chapter. The fields *initial revision date*, *look before* and *look after* all use the same Lisp-like mini language. Here's the list of available functions: * today -> produces a timestamp as of *now* * shifted -> takes keywords *seconds*, *minutes*, *hours*, *days*, *weeks*, *months* and *years* to shift the input timestamp by the required time delta * date -> turns the date string input into an actual timestamp * monthstart / monthend / yearstart / yearend -> project the input timestamp on the corresponding period boundary The *look before* and *look after* will generally use a *now* variable rather than *today*. When the cache is being updated, this variable will be valued with the corresponding revision date. Examples: (today) (shifted (today) #:weeks -1) (date "2024-1-1") (shifted now #:days 15) The *revision date rule* and *schedule rule* must be described according to the cron syntax. When in doubt, you should assign the same value to *schedule rule* and to *revision date rule*. It sometimes makes sense to schedule the actual cache update using a different rule than the revision date rule (to smooth out the strain on server resources for instance). When the cache policy has been created, it can be of course *edited* again, *activated* (otherwise nothing happens) or *deleted*. [image] The last step is managing the series associated with the policy. One gets there by clicking the policy name in the list. [image] Tasks system: how to organize and schedule tasks ================================================ In this chapter we will survey the basics of the tasks system, how to write and run tasks, and the best ways to organize them. The built-in tasks system allows to organize, schedule and run tasks for (non limitative list): * data acquisition * data monitoring * running models Note: The builtin tasks manager is not exclusive of any other tasks manager you want to use. Dask, Airflow or other such tools are fine in and by themselves and can certainly be used along with the Refinery if their workflows are more compelling for your specific use cases. Basics: operations, domains, inputs, tasks ------------------------------------------ Let's review the most important concepts. A *task* is the execution of an *operation*. An *operation* is a Python function that will run as a task. Here's an example: from rework.api import task @task def hello(task): with task.capturelogs(std=True): print('Hello') Let's examine what's there: * the operation is marked by the *task* decorator * it must be a one parameter function, whose parameter is a *task* object. * the *task* object provides a number of methods, such as *capturelogs* * the *capturelogs* method accepts an *std* parameter, which when True turns the output of the print function into logs; *capturelogs* records the logs from any logger within its context Such a task will also run in the "default" *domain*. A domain is basically a label allowing to group operations. It certainly makes sense to e.g. have separate domains such as *models* and *scrapers*. The Web UI permits filtering based on domains. from datetime import timedelta from rework.api import task import rework.io as rio from tshistory.api import timeseries from eflower.models import run_co2_forecast @task(domain='models', inputs=(rio.string('zone', required=True),), timeout=timedelta(hours=2)) def co2_model(task): tsa = timeseries() with task.capturelogs(std=True): run_co2_forecast(tsa, zone=task.input['zone']) This example introduces three other parameters of the "task" decorator: * "domain" (string) — a label for grouping operations and routing them to a specific worker pool, * "inputs" (tuple of "rework.io" entities) — typed input fields exposed in the Web UI "Launchers" tab, * "timeout" ("datetime.timedelta") — an upper bound on the task duration; tasks running longer are aborted by the monitor. The inputs come as an attribute of the "task" object — a dictionary. The next section catalogs the available input types. Input types reference --------------------- The "inputs" tuple of the "@task" decorator uses entities defined in "rework.io" (conventionally imported as "rio"). Each entity declares a named input field that the Web UI "Launchers" tab turns into an interactive form widget. All input types accept the same three common options: * "required" (bool, default "False") — the field must be filled at launch time, otherwise the task fails to start * "default" — value used when the field is left empty * "choices" (list) — restricts the allowed values to a fixed set, rendered as a dropdown in the UI The available types are listed below, each with a usage example. "rio.string" Python "str", rendered as a text field. The workhorse input. rio.string('zone', required=True) "rio.number" Python "int" or "float" (integer if the input has no decimal point, float otherwise), rendered as a numeric field. rio.number('horizon', default=7) "rio.boolean" Python "bool", rendered as a toggle. rio.boolean('with_calendar', default=False) "rio.file" Python "bytes", rendered as a file upload. Typically used with "required=True" for ingestion tasks. rio.file('csv', required=True) "rio.moment" Python "str" (a psyl expression), rendered as a text field whose value is evaluated to a datetime at task launch time. Accepts expressions like "(today)" or "(shifted (today) #:days 7)". rio.moment('fromdate', default='(today)') "rio.datetime" Python "str" (ISO 8601), rendered as a text field parsed as an absolute datetime. rio.datetime('asof', required=True) Example combining several types and options: from rework.api import task import rework.io as rio @task(inputs=( rio.string('zone', choices=['fr', 'be', 'de'], required=True), rio.moment('fromdate', default='(today)'), rio.number('horizon', default=7), rio.boolean('with_calendar', default=True), )) def my_forecast(task): ... Patterns for clean task code ---------------------------- Putting all the logic inside a task wrapper makes the code hard to test and to reuse outside of rework. The recommended pattern is to split the code: * Domain logic in a regular Python module ("utils.py", "models.py", "scrap.py"... whatever name fits) — pure functions of the form "func(tsa, …)", with no dependency on rework, callable from a notebook, a script or a unit test. * Thin "@task" wrappers in "tasks.py" that just plumb the rework inputs into a domain function call, capture logs, and let the function do the work. A typical task wrapper looks like this: from rework.api import task import rework.io as rio from tshistory.api import timeseries from myapp.models import run_forecast @task(inputs=( rio.string('zone', required=True), rio.number('horizon', default=7), )) def forecast(task): with task.capturelogs(std=True): inputs = task.input run_forecast( timeseries(), zone=inputs['zone'], horizon=inputs['horizon'], ) The same can be written more concisely by spreading "task.input" as keyword arguments: @task(inputs=( rio.string('zone', required=True), rio.number('horizon', default=7), )) def forecast(task): with task.capturelogs(std=True): run_forecast(timeseries(), **task.input) This works as long as the "rio" input names match the function parameter names. Note: Make sure the task function name differs from the imported domain function name, otherwise Python scoping turns the call into infinite recursion. Keeping the domain logic in a regular Python module also makes it easy to test in isolation. Test it before wrapping it in a "@task" decorator. Once your task is written and tested, register it with rework so workers can pick it up. Registering, running and scheduling tasks ----------------------------------------- Operations defined in a Python module are not magically known from the system: an explicit registration step has to be performed. Once registered, a task can be launched manually or programmed to run on a cron schedule. We'll go through both paths, then show how applications usually bundle them in a single "setup-tasks" command. Manual registration uses the "rework" CLI directly. Assuming the operations live in a "tasks.py" module: $ rework register-operations my_refinery myapp/tasks.py registered 2 new operation (0 already known) The reverse operation is available too: $ rework unregister-operation my_refinery helloworld preparing de-registration of: delete helloworld default /home/aurelien/tasks.py 10.211.55.4 really remove those? [y/N]: y delete helloworld default /home/aurelien/tasks.py 10.211.55.4 For tasks to actually execute, a monitor must run for their domain. The monitor is a long-lived process that maintains a pool of worker subprocesses: $ rework monitor my_refinery --maxruns 2 --maxworkers 4 --minworkers 1 --domain models --vacuum P4D Let's unpack this: * "minworkers" specifies the minimum amount of running workers in the pool, * "maxworkers" specifies the absolute maximum of workers, * "maxruns" indicates how many tasks a worker is allowed to run before being shut down (and eventually replaced), * "domain" specifies the domain, * "vacuum" defines the retention duration of tasks in the "done" state, using the iso 8601 durations notation. With the operation registered and a monitor running, tasks can be triggered from the Web UI "Launchers" tab, or programmatically with "rework.api.schedule". For real-world deployments, registration and cron scheduling are bundled into a single "setup-tasks" CLI command of the application, run once at deploy time. It calls "freeze_operations" (the Python equivalent of "rework register-operations") and "rework.api.prepare" to install both the operations and their schedules: import click from inireader import reader from sqlhelp.pgapi import pgdb as create_engine from rework.api import freeze_operations, prepare @click.command(name='setup-tasks') def setup_tasks(): from myapp import tasks # noqa: F401 -- registers the @task ops config = reader('rework.cfg') engine = create_engine(list(config['dburi'].values())[0]) recorded, known = freeze_operations(engine) prepare(engine, 'forecast', rule='0 8 * * *') print(f'recorded {len(recorded)} new operations, ' f'{len(known)} already known') The "import myapp.tasks" is what makes the "@task" decorators execute (and thus register the operations in the in-memory registry). "freeze_operations" writes them to the "rework.operation" table. "prepare" adds the cron rule to the "rework.sched" table. The rule uses the standard 5-field cron syntax: "minute hour day-of-month month day-of-week". A few common patterns: * "0 * * * *" — every hour, on the hour * "*/15 * * * *" — every 15 minutes * "0 8 * * *" — every day at 08:00 * "0 8 * * 1-5" — every weekday at 08:00 * "30 6 1 * *" — the 1st of each month at 06:30 For more complex rules, crontab.guru is a helpful interactive parser. "prepare" is idempotent for the same combination of operation name, domain, host, rule and inputs: calling it again updates the schedule in place rather than creating a duplicate. A more robust variant deletes the application's existing operations before re-freezing them, which makes the command safe to re-run after renaming a task or changing its inputs: with engine.begin() as cn: cn.execute( "delete from rework.operation where path like '%%myapp/tasks%%'" ) freeze_operations(engine) Domains and monitor strategy ---------------------------- The notion of *domains* permits to group operations together. There can be a semantic aspect to the grouping (let's not mix models and scrapers) and performance aspects, since each domain will be associated with a monitor, which has its own settings (configuration of the workers pool and tasks retention duration). Out of the box, the Refinery uses two domains: *timeseries* and *scrapers*. * *timeseries* is used to run the formula cache operations, * *scrapers* is used to run the scraping operations. You of course are encouraged to have your own domains. Reasonable proposals seen in the field are: * *models* to run your models, * *high-frequency* to run operations that really want to run often, like e.g. every minute (hence they will clog the tasks list and you want to be able to easily filter them out) Don't forget to run at least one monitor for each domain ! Note: It is possible to run several monitors for one given domain. Hypothetical scenario: your computing units are too small to run all operations of a given domain in parallel, so you build several ones, using a Python environment with the same dependency versions on several machines. Last point: if you have e.g. models but with incompatible requirements (typically in terms of versions), you can put them in different domains, managed by monitors living in different Python environments. Tasks: using the Web UI ----------------------- The task manager provides a convenient Web UI to provide the users a dependable view of the state of the system. The UI provides a number of tabbed views and a couple of direct views. Let's walk through them. [image] The tasks list is the first thing to be seen when we land on the tasks manager views. It provides a direct access to the tasks (status, inputs, domain, logs) and a few actions, in inverse chronological order (most recent at the top). By default, the complete list is not loaded. By scrolling downwards, one gets new lazily loaded items. The actions available on tasks are: * relaunch (can be useful when a task has been aborted for some reason) * delete (to unclutter the list) * abort (available only on running tasks) Note: For the filters (on the service, inputs and status columns) to work appropriately, having everything loaded at once is more convenient. The "Tasks" tab provides an action to force load all known tasks in the list. [image] Clicking on the curved arrow symbol will force load all tasks. By clicking on a task item (blue link on the operation name), one jumps to the task log (available if it is running or done). [image] For failed tasks, clicking on the *traceback* link in red opens the task back trace. [image] These two last views are important tools to understand what is going on with your tasks. The next tab shows the monitors status. As we have seen, monitors provides workers for a given domain. We can see their state (their status is updated using a dead-man switch scheme) and their active workers. [image] It is possible to ask for a shutdown (the worker will disappear after finishing its current task), or a kill (the worker is stopped and removed immediately). The "Launchers" tab provides a zone to manually launch a task, either through a "launch now" action (when there are no inputs) or through an "open form" dialog to allow inputing the relevant operations parameters before launching. [image] Note: For the tasks to be directly launchable from the Launchers tab, their declaration **must** contain an *inputs* declaration (even with an empty tuple if you need none). Only with this formalism can the UI know it can present actions, and which ones, to the end-user. The "Schedulers" tab lists all pre-scheduled tasks with their inputs and schedule rule, also allowing to do an immediate launch if needed. [image] The "Plan" tab shows what tasks (amongst those that are pre-scheduled) are going to fire, and at which moment, with their inputs, in the next few hours. [image] The final "Services" tab lists all the operations, notably with their host and file system path. [image] Data architecture at scale: putting the Data Mesh to work ========================================================= For a number of reasons, it may be a good idea to run not just one instance of the Refinery, but several interconnected in a data mesh, and also have some of them interconnected. Let's see some of the reasons: * team/topics diversity * data architecture choice Making teams work together -------------------------- Energy commodity companies sometimes have significant trading desks with a sizeable amount of analysts, data scientists and IT quant people working together. These people have different specialties, scopes and requirements. One useful way to address their specific needs is to provide a number of Refineries, each dedicated to a specific group. Let's imagine an hypothetical department where you have analysts working on oil, coal, power and gas. The "power" people are interested into renewables, nuclear, hydro and fossil fuel plants, and "gas" people are interested in piped and lng gas, also in storages and terminals. You get the idea. All these people work on production and demand models, which entails having access to meteorological data (forecasts). Putting everyone's data into one big data lake, I mean a big Refinery instance would quickly turn it into a data swamp, I mean a terrible mess. What we can propose is that each theme gets its own Refinery instance. You would have power, gas, oil, meteo, etc. refineries. But then, the power and gas people (for instance) would need access to the meteo data, for their forecasting models need meteo forecasts as inputs. Also the management would like to have a trans-commodity view of their position and get something consolidated: we can give them a Refinery connected to all others (and very little, if any, data of its own). In this setup, each team can focus on its own needs, and delegate what they don't master to other teams. The important power team (of several analysts and data scientists) would rely on the meteo team to provide them with the relevant solar and wind forecasts. The gas people modelling the demand would also ask the meteorologists about temperature forecasts in some places of interest. So each teams gets its own Refinery, which means a local "data governance", naming convention, workflows, models, etc. and at the same time data is easily shared cross-teams. The Medallion Architecture -------------------------- This architecture was invented to address the needs of Data Lake users and avoid the Data Swamp syndrome which can afflict them. In the medallion architecture, we have layers of data. Each layer corresponds to a data state ("bronze" for raw data, "silver" holds the pre- processed raw data and "gold" contains completely curated data fit for consumption by everyone). This layered approach can be an inspiration for the organization of Refineries, with: * a "bronze" Refinery holding data as scraped from the upstream providers (in versioned time series form of course), including missing data, outliers, changing granularities discontinuities and other warts, * a "silver" Refinery holding mostly formulas based on the "bronze" level, where we do proper resampling, basic outliers elimination, bogus data overrides, basic aggregations and data assemblies (using e.g. the *add* and *priority* formula operators) - this layer knows "bronze" which its use as a secondary, read-only source * a "gold" Refinery used as inputs for the models and also storing their outputs, providing data for dashboard and any other decision support tools - this layer uses the "silver" curated time series Note: In a Data Lake, we have a hodge-podge of raw (json, xml) data, non relational and relational (with normalized and dernormalized relations) data bases, and so on; this complexity cries for the application of a doctrine such as the Medallion Architecture. On the other hand, Refineries are light-weight and provide structure from the start, but still the layering idea has some merit and could be put to work for instances with either a lot of time series at each levels, or a lot of stake holders at each level, to provide more overall clarity and better data governance. =============================================================================== # https://refinery.docs.pythonian.fr/core_components/index.html =============================================================================== Core Components *************** Contents: ^^^^^^^^^ * Working with series in Python * Working with series in the browser * Supervision * Formulas (computed series) * Formula Language Reference * Formula cache * Groups * Folders * Search Query Language Reference * Task Manager Operations * Time Series Monitoring * Excel addin * Testing * Dashboards * The MCP server * The HTTP API * Securing the Refinery * Settings =============================================================================== # https://refinery.docs.pythonian.fr/core_components/usage.html =============================================================================== Working with series in Python ***************************** Table of Contents ================= * Creating a series * Updating a series * Working with versions * Point and version erasure * Retrieving history * Working with metadata * Replacing a series entirely * Checking series existence * Renaming a series * Deleting a series * Finding series * Getting series information * Working with logs * Staircase operations * Time Series Operations API Creating a series ================= Here’s a simple example: >>> import pandas as pd >>> from tshistory.api import timeseries >>> >>> tsa = timeseries('postgresql://me:password@localhost/mydb') >>> >>> series = pd.Series([1, 2, 3], ... pd.date_range(start=pd.Timestamp(2017, 1, 1), ... freq='D', periods=3)) # db insertion >>> tsa.update('my_series', series, 'babar@pythonian.fr') ... 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 3.0 Freq: D, Name: my_series, dtype: float64 # note how our integers got turned into floats # (there are no provisions to handle integer series as of today) # retrieval >>> tsa.get('my_series') ... 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 3.0 Name: my_series, dtype: float64 Note that we generally adopt the convention to name the time series api object "tsa". Updating a series ================= The "update" method is the fundamental operation for time series management, designed for incrementally updating series as new data arrives over time. This is good. Now, let's insert more: >>> series = pd.Series([2, 7, 8, 9], ... pd.date_range(start=pd.Timestamp(2017, 1, 2), ... freq='D', periods=4)) # db insertion >>> tsa.update('my_series', series, 'babar@pythonian.fr') ... 2017-01-03 7.0 2017-01-04 8.0 2017-01-05 9.0 Name: my_series, dtype: float64 You get back the *new information* you put inside and this is why the *2* doesn't appear (it was already put there in the first step). >>> tsa.get('my_series') ... 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 7.0 2017-01-04 8.0 2017-01-05 9.0 Name: my_series, dtype: float64 It is important to note that the third value was *replaced*, and the two last values were just *appended*. As noted the point at "2017-1-2" wasn't a new information so it was just ignored. Working with versions ===================== The "insertion_dates" method is one of the three fundamental API points (along with "get" and "update"). Every update creates a new version of the series, and this method returns when each version was created: >>> tsa.insertion_dates('my_series') [pd.Timestamp('2018-09-26 17:10:36.988920+02:00'), pd.Timestamp('2018-09-26 17:12:54.508252+02:00')] >>> # get insertions within a date range >>> tsa.insertion_dates('my_series', ... from_insertion_date=pd.Timestamp('2018-09-26 17:11:00+02:00')) [pd.Timestamp('2018-09-26 17:12:54.508252+02:00')] These timestamps identify the versions of your series and are what you use with "get" to retrieve any past state. Point and version erasure ========================= Point erasure with NaN ---------------------- You can erase specific points in a series by updating with NaN values: >>> # erase the point at 2017-01-02 >>> erasure = pd.Series([np.nan], index=[pd.Timestamp('2017-01-02')]) >>> tsa.update('my_series', erasure, 'cleanup@example.com', keepnans=True) >>> # by default, erased points are not shown >>> tsa.get('my_series') 2017-01-01 1.0 2017-01-03 7.0 2017-01-04 8.0 2017-01-05 9.0 Name: my_series, dtype: float64 >>> # use keepnans=True to see erased points >>> tsa.get('my_series', keepnans=True) 2017-01-01 1.0 2017-01-02 NaN 2017-01-03 7.0 2017-01-04 8.0 2017-01-05 9.0 Name: my_series, dtype: float64 Version erasure with strip -------------------------- **WARNING: This is a DESTRUCTIVE operation that should only be used as a LAST RESORT.** The "strip" method permanently removes all versions after a given insertion date: >>> # check existing versions >>> tsa.insertion_dates('my_series') [pd.Timestamp('2018-09-26 17:10:36.988920+02:00'), pd.Timestamp('2018-09-26 17:12:54.508252+02:00'), pd.Timestamp('2018-09-26 17:15:00.000000+02:00')] >>> # DANGER: permanently remove versions after 17:12 >>> tsa.strip('my_series', pd.Timestamp('2018-09-26 17:12:00+02:00')) >>> # versions are gone forever >>> tsa.insertion_dates('my_series') [pd.Timestamp('2018-09-26 17:10:36.988920+02:00')] This operation cannot be undone. Use with extreme caution. Retrieving history ================== We can access the whole history (or parts of it) in one call: >>> history = tsa.history('my_series') ... >>> >>> for idate, series in history.items(): # it's a dict ... print('insertion date:', idate) ... print(series) ... insertion date: 2018-09-26 17:10:36.988920+02:00 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 3.0 Name: my_series, dtype: float64 insertion date: 2018-09-26 17:12:54.508252+02:00 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 7.0 2017-01-04 8.0 2017-01-05 9.0 Name: my_series, dtype: float64 Note how this shows the full serie state for each insertion date. Also the insertion date is timzeone aware. Specific versions of a series can be retrieved individually using the "get" method with the "revision_date" parameter (using timestamps obtained from "insertion_dates"): >>> tsa.get('my_series', revision_date=pd.Timestamp('2018-09-26 17:11+02:00')) ... 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 3.0 Name: my_series, dtype: float64 >>> >>> tsa.get('my_series', revision_date=pd.Timestamp('2018-09-26 17:14+02:00')) ... 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 7.0 2017-01-04 8.0 2017-01-05 9.0 Name: my_series, dtype: float64 It is possible to retrieve only the differences between successive insertions: >>> diffs = tsa.history('my_series', diffmode=True) ... >>> for idate, series in diffs.items(): ... print('insertion date:', idate) ... print(series) ... insertion date: 2018-09-26 17:10:36.988920+02:00 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 3.0 Name: my_series, dtype: float64 insertion date: 2018-09-26 17:12:54.508252+02:00 2017-01-03 7.0 2017-01-04 8.0 2017-01-05 9.0 Name: my_series, dtype: float64 Working with metadata ===================== Series can have metadata attached to help document and organize them: >>> tsa.update_metadata('temperature_sensor', { ... 'unit': 'celsius', ... 'location': 'building_a', ... 'sensor_type': 'PT100', ... 'frequency': 'hourly' ... }) >>> tsa.metadata('temperature_sensor') {'unit': 'celsius', 'location': 'building_a', 'sensor_type': 'PT100', 'frequency': 'hourly'} >>> # update metadata (merges with existing) >>> tsa.update_metadata('temperature_sensor', {'calibrated': '2023-01-15'}) >>> tsa.metadata('temperature_sensor') {'unit': 'celsius', 'location': 'building_a', 'sensor_type': 'PT100', 'frequency': 'hourly', 'calibrated': '2023-01-15'} >>> # replace all metadata >>> tsa.replace_metadata('temperature_sensor', {'unit': 'fahrenheit', 'status': 'active'}) >>> tsa.metadata('temperature_sensor') {'unit': 'fahrenheit', 'status': 'active'} >>> # view metadata history >>> tsa.old_metadata('temperature_sensor') [(pd.Timestamp('2023-01-01 10:00:00+00:00'), {'unit': 'celsius', 'location': 'building_a', 'sensor_type': 'PT100', 'frequency': 'hourly'}), (pd.Timestamp('2023-01-02 11:00:00+00:00'), {'unit': 'celsius', 'location': 'building_a', 'sensor_type': 'PT100', 'frequency': 'hourly', 'calibrated': '2023-01-15'}), (pd.Timestamp('2023-01-03 09:00:00+00:00'), {'unit': 'fahrenheit', 'status': 'active'})] Beyond managing metadata for individual series, you can also discover what metadata keys are used across all series in your refinery instance: >>> # list all metadata keys in use >>> tsa.list_metadata_keys() ['calibrated', 'frequency', 'location', 'sensor_type', 'status', 'unit'] Replacing a series entirely =========================== In specific circumstances, you may need to completely replace a series - for example, when you are working with forecast data and you need only the last updated forecast. The "replace" method provides this capability: >>> # create initial series >>> series = pd.Series([10, 20, 30], ... pd.date_range(start=pd.Timestamp(2025, 1, 2), ... freq='D', periods=3)) >>> tsa.update('stock_levels_forecast', series, 'operator@example.com', insertion_date=pd.Timestamp(2025,1,1)) >>> # later, replace the entire series with a new forecast >>> new_series = pd.Series([70, 50, 60], ... pd.date_range(start=pd.Timestamp(2025, 1, 3), ... freq='D', periods=3)) >>> tsa.replace('stock_levels_forecast', new_series, 'admin@example.com', insertion_date=pd.Timestamp(2025,1,2)) >>> tsa.get('stock_levels_forecast') 2025-01-03 70.0 2025-01-04 50.0 2025-01-05 60.0 Freq: D, Name: stock_levels_forecast, dtype: float64 The "replace" method completely overwrites the series with new data, removing any points not present in the new series. Note: It's important to note that "replace" preserves the complete version history. The replace operation appears as a new insertion date in the series history: >>> tsa.insertion_dates('stock_levels_forecast') [pd.Timestamp('2025-01-01 00:00:00+0000', tz='UTC'), # original update pd.Timestamp('2025-01-02 00:00:00+0000', tz='UTC')] # replace operation >>> # history shows both the original and replaced versions >>> history = tsa.history('stock_levels_forecast') >>> for idate, series in history.items(): ... print(f'insertion date: {idate}') ... print(series) ... insertion date: 2025-01-01 00:00:00+00:00 2025-01-02 10.0 2025-01-03 20.0 2025-01-04 30.0 Name: stock_levels_forecast, dtype: float64 insertion date: 2025-01-02 00:00:00+00:00 2025-01-03 70.0 2025-01-04 50.0 2025-01-05 60.0 Name: stock_levels_forecast, dtype: float64 This means you can always retrieve previous states of the series before the replace operation using "revision_date". Checking series existence ========================= To check if a series exists: >>> tsa.exists('my_series') True >>> tsa.exists('non_existent') False Renaming a series ================= To rename a series: >>> tsa.rename('old_name', 'new_name') >>> tsa.exists('old_name') False >>> tsa.exists('new_name') True Deleting a series ================= To remove a series from the database: >>> tsa.delete('my_series') >>> tsa.get('my_series') # returns None Finding series ============== To find series in the database: >>> # find all series >>> tsa.find() ['my_series', 'temperature_fr', 'calculated_avg', 'temperature_paris'] >>> # find with metadata >>> results = tsa.find('(by.name "temperature")', meta=True) >>> results ['temperature_fr', 'temperature_paris'] >>> # access directly the metadata of found series >>> results[0].meta {'unit': 'celsius', 'location': 'fr'} See the Search Query Language Reference documentation for comprehensive query capabilities. The older "catalog()" method is still available but returns everything at once from all sources in a slightly cumbersome structure: >>> tsa.catalog() {('refinery', 'tsh'): [('my_series', 'primary'), ('temperature_fr', 'primary'), ('temperature_paris', 'primary')], ('meteo', 'tsh'): [('calculated_avg', 'formula')]} The keys are (instance name, namespace) pairs, one per source. The "find()" API is generally preferred for its flexibility. Getting series information ========================== To get detailed information about a series: >>> tsa.type('my_series') 'primary' >>> tsa.interval('my_series') (pd.Timestamp('2025-01-01'), pd.Timestamp('2025-01-05')) >>> tsa.source('my_series') 'local' >>> # get inferred frequency >>> tsa.inferred_freq('my_series') 'D' # daily frequency >>> # get various informations with internal_metadata. >>> # tzawareness, value type (float, string) and supervision_status >>> tsa.internal_metadata('my_series') {'left': '2025-01-01T00:00:00', 'right': '2025-01-05T00:00:00', 'tzaware': False, 'tablename': 'my_series', 'index_type': 'datetime64[ns]', 'value_type': 'float64', 'index_dtype': '>> tsa.log('my_series', limit=5) [{'date': pd.Timestamp('2018-09-26 17:12:54.508252+02:00'), 'author': 'babar@pythonian.fr', 'meta': {}, 'rev': 2}, {'date': pd.Timestamp('2018-09-26 17:10:36.988920+02:00'), 'author': 'babar@pythonian.fr', 'meta': {}, 'rev': 1}] The most recent revision comes first; "limit" hence returns the latest revisions. Staircase operations ==================== The staircase operations are specialized methods for forecast backtesting and time-consistent analysis. They reconstruct series as they were available at specific lead times, which is essential for evaluating forecast accuracy without look-ahead bias. Basic staircase --------------- A staircase series can be defined as a series whose values originate from successive revisions with a fixed time span between revision date and value date. Let us take an example assuming a series called "daily_series" has been created with insertions given by the following table (row indices are value dates, column indices are insertion dates): +---------------------------+---------------------------+---------------------------+---------------------------+ | | 2020-01-01 00:00+00 | 2020-01-02 00:00+00 | 2020-01-03 00:00+00 | |===========================|===========================|===========================|===========================| | 2020-01-01 | 1.1 | | | +---------------------------+---------------------------+---------------------------+---------------------------+ | 2020-01-02 | 2.1 | 2.2 | | +---------------------------+---------------------------+---------------------------+---------------------------+ | 2020-01-03 | 3.1 | 3.2 | 3.3 | +---------------------------+---------------------------+---------------------------+---------------------------+ | 2020-01-04 | | 4.2 | 4.3 | +---------------------------+---------------------------+---------------------------+---------------------------+ | 2020-01-05 | | | 5.3 | +---------------------------+---------------------------+---------------------------+---------------------------+ Supposing this series is a forecast published on a daily basis, the day-ahead forecast series — values for which the time span between revision date and value date is 1 day or more — can be reconstructed as follows: >>> tsa.staircase('daily_series', ... from_value_date=pd.Timestamp('2020-01-01'), ... to_value_date=pd.Timestamp('2020-01-07'), ... delta=pd.Timedelta(days=1)) ... 2020-01-02 2.1 2020-01-03 3.2 2020-01-04 4.3 2020-01-05 5.3 Name: daily_series, dtype: float64 The name "staircase" refers to the way values are picked from the history: +---------------------------+---------------------------+---------------------------+---------------------------+ | | 2020-01-01 00:00+00 | 2020-01-02 00:00+00 | 2020-01-03 00:00+00 | |===========================|===========================|===========================|===========================| | 2020-01-01 | | | | +---------------------------+---------------------------+---------------------------+---------------------------+ | 2020-01-02 | **2.1** | | | +---------------------------+---------------------------+---------------------------+---------------------------+ | 2020-01-03 | | **3.2** | | +---------------------------+---------------------------+---------------------------+---------------------------+ | 2020-01-04 | | | **4.3** | +---------------------------+---------------------------+---------------------------+---------------------------+ | 2020-01-05 | | | **5.3** | +---------------------------+---------------------------+---------------------------+---------------------------+ Block staircase --------------- The "block_staircase" method is a more sophisticated and controllable version. It rebuilds a series from successive blocks of history, each linked to a distinct revision date. This is useful for analyzing how forecasts evolve over time with a consistent publication schedule. A common use case is daily revisions at 10am with a 24-hour maturity offset, where each day's values come from the revision published the day before: >>> bsc = tsa.block_staircase( ... name='forecast_series', ... from_value_date=pd.Timestamp('2020-01-03', tz='utc'), ... to_value_date=pd.Timestamp('2020-01-05', tz='utc'), ... revision_freq={'days': 1}, ... revision_time={'hour': 10}, ... revision_tz='UTC', ... maturity_offset={'hours': 24}, ... maturity_time={'hour': 4} ... ) For a more detailed example, consider the series "hourly_series" with the following insertions: +----------------------+----------------------+----------------------+----------------------+----------------------+ | | 2020-01-01 06:00+00 | 2020-01-01 14:00+00 | 2020-01-02 06:00+00 | 2020-01-02 14:00+00 | |======================|======================|======================|======================|======================| | 2020-01-01 00:00+00 | 1.1 | 1.2 | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2020-01-01 08:00+00 | 2.1 | 2.2 | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2020-01-01 16:00+00 | 3.1 | 3.2 | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2020-01-02 00:00+00 | 4.1 | 4.2 | 4.3 | 4.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2020-01-02 08:00+00 | 5.1 | 5.2 | 5.3 | 5.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2020-01-02 16:00+00 | 6.1 | 6.2 | 6.3 | 6.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2020-01-03 00:00+00 | 7.1 | 7.2 | 7.3 | 7.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2020-01-03 08:00+00 | 8.1 | 8.2 | 8.3 | 8.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2020-01-03 16:00+00 | 9.1 | 9.2 | 9.3 | 9.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2020-01-04 00:00+00 | | | 10.3 | 10.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2020-01-04 08:00+00 | | | 11.3 | 11.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2020-01-04 16:00+00 | | | 12.3 | 12.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ The day-ahead forecast with revisions at 9am can be computed as follows: >>> tsa.block_staircase('hourly_series', ... from_value_date=pd.Timestamp('2020-01-01', tz='utc'), ... to_value_date=pd.Timestamp('2020-01-05', tz='utc'), ... revision_freq={'days': 1}, ... revision_time={'hour': 9}, ... revision_tz='utc', ... maturity_offset={'days': 1}, ... maturity_time={'hour': 0}) ... 2020-01-02 00:00:00+00:00 4.1 2020-01-02 08:00:00+00:00 5.1 2020-01-02 16:00:00+00:00 6.1 2020-01-03 00:00:00+00:00 7.3 2020-01-03 08:00:00+00:00 8.3 2020-01-03 16:00:00+00:00 9.3 2020-01-04 00:00:00+00:00 10.4 2020-01-04 08:00:00+00:00 11.4 2020-01-04 16:00:00+00:00 12.4 Name: hourly_series, dtype: float64 With "revision_time={'hour': 9}", values are picked from the 6am insertions (except for 2020-01-04 where the latest available revision is 2020-01-02 14:00). Using "revision_time={'hour': 20}" would instead select values from the 2pm insertions only. The arguments of "block_staircase" should be used as follows: * "from_value_date" and "to_value_date": time range on which values are retrieved * "revision_freq": revision frequency, as a dict with integer values and keys taken from "['years', 'months', 'weeks', 'bdays', 'days', 'hours', 'minutes', 'seconds']" * "revision_time": revision time, as a dict with integer values and keys taken from "['year', 'month', 'day', 'weekday', 'hour', 'minute', 'second']", used for revision date initialisation; successive revision dates are then obtained by adding "revision_freq" * "revision_tz": time zone in which revision date and time are expressed * "maturity_offset": time span between each revision date and the start of its block of values, as a dict with integer values and keys taken from "['years', 'months', 'weeks', 'bdays', 'days', 'hours', 'minutes', 'seconds']"; if not specified the revision date is the block start date * "maturity_time": start time of each block, as a dict with integer values and keys taken from "['year', 'month', 'day', 'hour', 'minute', 'second']"; the block start date is obtained by adding "maturity_offset" to the revision date then applying "maturity_time"; if not specified the block start date is the revision date shifted by "maturity_offset" Other use cases --------------- The "block_staircase" method covers multiple use cases, such as week- ahead revisions or revision by business day. Week-ahead staircase ~~~~~~~~~~~~~~~~~~~~ Consider a series named "weekly_series" with the following insertions: +----------------------+----------------------+----------------------+----------------------+----------------------+ | | 2021-01-05 (Tue) | 2021-01-07 (Thu) | 2021-01-12 (Tue) | 2021-01-14 (Thu) | |======================|======================|======================|======================|======================| | 2021-01-11 (Mon) | 1.1 | 1.2 | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-12 (Tue) | 2.1 | 2.2 | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-13 (Wed) | 3.1 | 3.2 | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-14 (Thu) | 4.1 | 4.2 | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-15 (Fri) | 5.1 | 5.2 | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-16 (Sat) | 6.1 | 6.2 | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-17 (Sun) | 7.1 | 7.2 | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-18 (Mon) | 8.1 | 8.2 | 8.3 | 8.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-19 (Tue) | 9.1 | 9.2 | 9.3 | 9.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-20 (Wed) | 10.1 | 10.2 | 10.3 | 10.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-21 (Thu) | | | 11.3 | 11.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-22 (Fri) | | | 12.3 | 12.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-23 (Sat) | | | 13.3 | 13.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-24 (Sun) | | | 14.3 | 14.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-25 (Mon) | | | 15.3 | 15.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-26 (Tue) | | | 16.3 | 16.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ | 2021-01-27 (Wed) | | | 17.3 | 17.4 | +----------------------+----------------------+----------------------+----------------------+----------------------+ The week-ahead staircase with weekly revision on Friday can be retrieved as follows: >>> tsa.block_staircase('weekly_series', ... from_value_date=pd.Timestamp('2021-01-10'), ... to_value_date=pd.Timestamp('2021-01-30'), ... revision_freq={'days': 7}, ... revision_time={'weekday': 4}, ... revision_tz='utc', ... maturity_offset={'days': 3}, ... maturity_time={'hour': 0}) ... 2021-01-11 1.2 2021-01-12 2.2 2021-01-13 3.2 2021-01-14 4.2 2021-01-15 5.2 2021-01-16 6.2 2021-01-17 7.2 2021-01-18 8.4 2021-01-19 9.4 2021-01-20 10.4 2021-01-21 11.4 2021-01-22 12.4 2021-01-23 13.4 2021-01-24 14.4 2021-01-25 15.4 2021-01-26 16.4 2021-01-27 17.4 Name: weekly_series, dtype: float64 It is also possible to retrieve a month-ahead staircase by using "revision_freq={'months': 1}" and, for example, "revision_time={'day': 15}" to perform monthly revision every 15th of the month. Revision by business day ~~~~~~~~~~~~~~~~~~~~~~~~ The "block_staircase" method also supports business day frequency. Consider a series named "business_day_series" with the following insertions: +----------------+----------------+----------------+----------------+----------------+----------------+----------------+ | | 2021-01-13 | 2021-01-14 | 2021-01-15 | 2021-01-16 | 2021-01-17 | 2021-01-18 | | | (Wed) | (Thu) | (Fri) | (Sat) | (Sun) | (Mon) | |================|================|================|================|================|================|================| | 2021-01-13 | 3.1 | | | | | | | (Wed) | | | | | | | +----------------+----------------+----------------+----------------+----------------+----------------+----------------+ | 2021-01-14 | 4.1 | 4.2 | | | | | | (Thu) | | | | | | | +----------------+----------------+----------------+----------------+----------------+----------------+----------------+ | 2021-01-15 | 5.1 | 5.2 | 5.3 | | | | | (Fri) | | | | | | | +----------------+----------------+----------------+----------------+----------------+----------------+----------------+ | 2021-01-16 | 6.1 | 6.2 | 6.3 | 6.4 | | | | (Sat) | | | | | | | +----------------+----------------+----------------+----------------+----------------+----------------+----------------+ | 2021-01-17 | | 7.2 | 7.3 | 7.4 | 7.5 | | | (Sun) | | | | | | | +----------------+----------------+----------------+----------------+----------------+----------------+----------------+ | 2021-01-18 | | | 8.3 | 8.4 | 8.5 | 9.6 | | (Mon) | | | | | | | +----------------+----------------+----------------+----------------+----------------+----------------+----------------+ | 2021-01-19 | | | | 9.4 | 9.5 | 11.6 | | (Tue) | | | | | | | +----------------+----------------+----------------+----------------+----------------+----------------+----------------+ | 2021-01-20 | | | | | 10.5 | 12.6 | | (Wed) | | | | | | | +----------------+----------------+----------------+----------------+----------------+----------------+----------------+ | 2021-01-21 | | | | | | 13.6 | | (Thu) | | | | | | | +----------------+----------------+----------------+----------------+----------------+----------------+----------------+ A business-day-ahead staircase series with daily business-day revision: >>> tsa.block_staircase('business_day_series', ... from_value_date=pd.Timestamp('2021-01-13'), ... to_value_date=pd.Timestamp('2021-01-21'), ... revision_freq={'bdays': 1}, ... revision_tz='utc', ... maturity_offset={'bdays': 1}) ... 2021-01-14 4.1 2021-01-15 5.2 2021-01-16 6.2 2021-01-17 7.2 2021-01-18 8.3 2021-01-19 11.6 2021-01-20 12.6 2021-01-21 13.6 Name: business_day_series, dtype: float64 Time Series Operations API ========================== The time series API provides comprehensive methods for managing time series data: class mainsource(*args, **kwargs) API façade for the main source (talks directly to the storage) The api documentation is carried by this object. The http client provides exactly the same methods. update(name, updatets, author, metadata=None, insertion_date=None, keepnans=False, **kw) Update a series named by with the input pandas series. This creates a new version of the series. Only the _changes_ between the last version and the provided series are part of the new version. A series made of the changed points is returned. If there was no change, an empty series is returned and no new version is created. New points are added, changed points are changed, points with NaN are dropped if *keepnans* is False (by default) or _erased_ if True. The *author* is mandatory. The *metadata* dictionary allows to associate any metadata with the new series revision. It is possible to force an *insertion_date*, which can only be higher than the previous *insertion_date*. >>> import pandas as pd >>> from tshistory.api import timeseries >>> >>> tsa = timeseries('postgres://me:password@localhost/mydb') >>> >>> series = pd.Series([1, 2, 3], ... pd.date_range(start=pd.Timestamp(2017, 1, 1), ... freq='D', periods=3)) # db insertion >>> tsa.update('my_series', series, 'babar@pythonian.fr') ... 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 3.0 Freq: D, Name: my_series, dtype: float64 Parameters: * **name** (*str*) * **updatets** (*Series*) * **author** (*str*) * **metadata** (*dict** | **None*) * **insertion_date** (*datetime** | **None*) * **keepnans** (*bool** | **None*) Return type: *Series* | None replace(name, replacets, author, metadata=None, insertion_date=None, **kw) Replace a series named by with the input pandas series. This creates a new version of the series. The series is completely replaced with the provided values. The *author* is mandatory. The *metadata* dictionary allows to associate any metadata with the new series revision. It is possible to force an *insertion_date*, which can only be higher than the previous *insertion_date*. Parameters: * **name** (*str*) * **replacets** (*Series*) * **author** (*str*) * **metadata** (*dict** | **None*) * **insertion_date** (*datetime** | **None*) Return type: *Series* | None exists(name) Checks the existence of a series with a given name. Parameters: **name** (*str*) Return type: bool source(name) Provide the source name of a series. When coming from the main source, it returns 'local'. Parameters: **name** (*str*) Return type: str | None get(name, revision_date=None, from_value_date=None, to_value_date=None, inferred_freq=False, keepnans=False, **kw) Get a series by name. By default one gets the latest version. By specifying *revision_date* one can get the closest version matching the given date. The *from_value_date* and *to_value_date* parameters permit to specify a narrower date range (by default all points are provided). If the series does not exists, a None is returned. >>> tsa.get('my_series') ... 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 3.0 Name: my_series, dtype: float64 Parameters: * **name** (*str*) * **revision_date** (*datetime** | **None*) * **from_value_date** (*datetime** | **None*) * **to_value_date** (*datetime** | **None*) * **inferred_freq** (*bool*) * **keepnans** (*bool*) Return type: *Series* | None insertion_dates(name, from_insertion_date=None, to_insertion_date=None, from_value_date=None, to_value_date=None, limit=None, **kw) Get the list of all insertion dates (as pandas timestamps). Parameters: * **name** (*str*) * **from_insertion_date** (*datetime** | **None*) * **to_insertion_date** (*datetime** | **None*) * **from_value_date** (*datetime** | **None*) * **to_value_date** (*datetime** | **None*) * **limit** (*int** | **None*) history(name, from_insertion_date=None, to_insertion_date=None, from_value_date=None, to_value_date=None, diffmode=False, keepnans=False, **kw) Get all versions of a series in the form of a dict from insertion dates to series version. It is possible to restrict the versions range by specifying *from_insertion_date* and *to_insertion_date*. It is possible to restrict the values range by specifying *from_value_date* and *to_value_date*. If *diffmode* is set to True, we don't get full series values between two consecutive insertion date but only the difference series (with new points, updated points and deleted points). This is typically more costly to compute but can be much more compact, and it encodes the same information as with *diffmode* set to False. >>> history = tsa.history('my_series') ... >>> >>> for idate, series in history.items(): # it's a dict ... print('insertion date:', idate) ... print(series) ... insertion date: 2018-09-26 17:10:36.988920+02:00 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 3.0 Name: my_series, dtype: float64 insertion date: 2018-09-26 17:12:54.508252+02:00 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 7.0 2017-01-04 8.0 2017-01-05 9.0 Name: my_series, dtype: float64 Parameters: * **name** (*str*) * **from_insertion_date** (*datetime** | **None*) * **to_insertion_date** (*datetime** | **None*) * **from_value_date** (*datetime** | **None*) * **to_value_date** (*datetime** | **None*) * **diffmode** (*bool*) * **keepnans** (*bool*) Return type: *Dict*[*datetime*, *Series*] | None staircase(name, delta, from_value_date=None, to_value_date=None, revision_date=None, keepnans=False) Compute a series whose value dates are the most recent constrained to be *delta* time after the insertion dates of the series. This kind of query typically makes sense for forecast series where the relationship between insertion date and value date is sound. Parameters: * **name** (*str*) * **delta** (*timedelta*) * **from_value_date** (*datetime** | **None*) * **to_value_date** (*datetime** | **None*) * **revision_date** (*datetime** | **None*) * **keepnans** (*bool*) Return type: *Series* | None block_staircase(name, from_value_date=None, to_value_date=None, revision_freq=None, revision_time=None, revision_tz='UTC', maturity_offset=None, maturity_time=None, revision_date=None, keepnans=False) Staircase a series by block This is a more sophisticated and controllable version of the *staircase* method. Computes a series rebuilt from successive blocks of history, each linked to a distinct revision date. The revision dates are taken at regular time intervals determined by *revision_freq*, *revision_time* and *revision_tz*. The time lag between revision dates and value dates of each block is determined by *maturity_offset* and *maturity_time*. name: str unique identifier of the series from_value_date: pandas.Timestamp from which values are retrieved to_value_date: pandas.Timestamp to which values are retrieved revision_freq: dict giving revision frequency, of which keys must be taken from ['years', 'months', 'weeks', 'bdays', 'days', 'hours', 'minutes', 'seconds'] and values as integers. Default is daily frequency, i.e. {'days': 1} revision_time: dict giving revision time, of which keys should be taken from ['year', 'month', 'day', 'weekday', 'hour', 'minute', 'second'] and values must be integers. It is only used for revision date initialisation. The next revision dates are then obtained by successively adding *revision_freq*. Default is {'hour': 0} revision_tz: str giving time zone in which revision date and time are expressed. Default is 'UTC' maturity_offset: dict giving time lag between each revision date and start time of related block values. Its keys must be taken from ['years', 'months', 'weeks', 'bdays', 'days', 'hours', 'minutes', 'seconds'] and values as integers. Default is {}, i.e. the revision date is the block start date maturity_time: dict fixing start time of each block, of which keys should be taken from ['year', 'month', 'day', 'weekday', 'hour', 'minute', 'second'] and values must be integers. The start date of each block is thus obtained by adding *maturity_offset* to revision date and then applying *maturity_time*. Default is {}, i.e. block start date is just the revision date shifted by *maturity_offset* Parameters: * **from_value_date** (*datetime** | **None*) * **to_value_date** (*datetime** | **None*) * **revision_freq** (*Dict**[**str**, **int**] **| **None*) * **revision_time** (*Dict**[**str**, **int**] **| **None*) * **revision_tz** (*str*) * **maturity_offset** (*Dict**[**str**, **int**] **| **None*) * **maturity_time** (*Dict**[**str**, **int**] **| **None*) * **revision_date** (*datetime** | **None*) * **keepnans** (*bool*) catalog(allsources=True) Produces a catalog of all series in the form of a mapping from source to a list of (name, kind) pair. By default it provides the series from all sources. If *allsources* is False, only the main source is listed. Parameters: **allsources** (*bool*) Return type: *Dict*[*Tuple*[str, str], *List*[*Tuple*[str, str]]] find(query=None, limit=None, meta=False, sources=[], _source='local') Return a list of series descriptors matching the query. A series descriptor is a string-like object (exhibiting the series name) with additional attributes. If *meta* has been set to True, the .meta (for normal metadata) and .imeta (for internal metadata) fields will be populated (non None). Lastly, the .source and .kind attributes provides the series source and kind. Here is an example: tsa.find( '(by.and ' ' (by.tzaware)' ' (by.name "power capacity") ' ' (by.metakey "plant")' ' (by.not (by.or ' ' (by.metaitem "plant_type" "oil")' ' (by.metaitem "plant_type" "coal")))' ' (by.metaitem "unit" "mwh")' ' (by.metaitem "country" "fr"))' ) This builds a query for timezone aware series about french power plants (in mwh) which are not of the coal or oil fuel type. The following filters can be used from the search module: * by.tzaware: no parameter, yields time zone aware series names * by.name : takes a space separated string of word, yields series names containing the substrings (in order) * by.metakey : takes a string, strictly matches all series having this metadata key * by.metaitems : takes a string (key) and an str (or numerical) value and yields all series strictly matching this metadata item * by.and: takes a variable number of filters as above to combine them * by.or: takes a variable number of filters as above to combine them * by.not: produce the negation of a filter Also inequalities on metadata values can be used: * <, <=, >, >=, =: take a string key, a value (str or num) As in *(<= "max_capacity" 900)* Parameters: * **query** (*str** | **None*) * **limit** (*int** | **None*) * **meta** (*int** | **None*) * **sources** (*List**[**str**]*) * **_source** (*str** | **None*) Return type: *List*[*ts*] interval(name) Return a pandas interval object which provides the smallest and highest value date of a series. Parameters: **name** (*str*) Return type: *Interval* inferred_freq(name, revision_date=None, from_value_date=None, to_value_date=None) Return a tuple of timedelta, float (between 0 and 1). The timedelta represents the period (or 'freq' in pandas parlance) and the number the quality of the period, which may vary because of the irregularity of the series. Parameters: * **name** (*str*) * **revision_date** (*Timestamp** | **None*) * **from_value_date** (*Timestamp** | **None*) * **to_value_date** (*Timestamp** | **None*) Return type: *Tuple*[*Timedelta*, float] | None metadata(name) Return a series metadata dictionary. Parameters: **name** (*str*) Return type: *Dict*[str, *Any*] | None internal_metadata(name) Return a series internal metadata dictionary. Parameters: **name** (*str*) Return type: *Dict*[str, *Any*] replace_metadata(name, metadata, user='no-user') Replace a series metadata with a dictionary from strings to anything json-serializable. Parameters: * **name** (*str*) * **metadata** (*dict*) * **user** (*str*) Return type: None update_metadata(name, metadata, user='no-user') Update a series metadata with a dictionary from strings to anything json-serializable. Parameters: * **name** (*str*) * **metadata** (*dict*) * **user** (*str*) Return type: None old_metadata(name) Get a list of the the older versions of the metadata. Parameters: **name** (*str*) Return type: *List*[*Tuple*[*Timestamp*, dict, str]] list_metadata_keys() List available metadata keys Return type: *List*[str] type(name) Return the type of a series, for instance 'primary' or 'formula'. Parameters: **name** (*str*) Return type: str log(name, limit=None, fromdate=None, todate=None) Return a list of revisions for a given series, in reverse chronological order, with filters. Revisions are dicts of: * rev: revision id (int) * author: author name * date: timestamp of the revision * meta: the revision metadata Parameters: * **name** (*str*) * **limit** (*int** | **None*) * **fromdate** (*Timestamp** | **None*) * **todate** (*Timestamp** | **None*) Return type: *List*[*Dict*[str, *Any*]] rename(name, newname, propagate=True, **_kw) Rename a series. The target name must be available. Parameters: * **name** (*str*) * **newname** (*str*) * **propagate** (*bool*) Return type: None delete(name, force=False) Override to add dependency checking. Delete a series. By default, refuses to delete if the series is referenced by formulas. Use force=True to override this safety check. Parameters: * **name** (*str*) * **force** (*bool*) Return type: None strip(name, insertion_date) Remove revisions after a specific insertion date. This is an irreversible operation. Parameters: * **name** (*str*) * **insertion_date** (*datetime*) Return type: None rename_basket(name, newname, group=False) Rename a basket. Parameters: * **name** (*str*) * **newname** (*str*) * **group** (*bool*) =============================================================================== # https://refinery.docs.pythonian.fr/core_components/web_usage.html =============================================================================== Working with series in the browser ********************************** This page covers the same basic operations as Working with series in Python, but from the web interface. Editing the values of an existing series is done in the series editor (see Supervision). Table of Contents ================= * Creating a series * Importing from a CSV * Exploring history * Working with metadata * Renaming a series * Deleting a series * Finding series * Getting series information * Working with logs * Viewing series Creating a series ================= A new series can be created from the **Create** entry of the left menu, which opens the series editor in creation mode. You fill in: * **Name**: must be unique (validated in real time) * **From / To**: initial time bounds * **Timezone**: chosen from a drop-down list * **Frequency**: a number and a Pandas alias ("MS" for monthly, "D" for daily, "h" for hourly, etc.) * **Data type**: Float or String * **Value**: optional initial fill value Clicking Preview generates the timestamp grid and displays a preview in the graph. The series can then be saved. [image] Importing from a CSV ==================== Several series can be loaded at once from a CSV file with the "ingest_from_csv" task. The CSV holds the dates in its first column and one series per remaining column, the column header giving the series name: date,temp.paris,temp.berlin 2026-01-01 00:00,3.2,1.1 2026-01-01 01:00,3.0,0.8 2026-01-01 02:00,2.7,0.5 Each column is applied with "update" — values are appended or patched, not replaced — empty cells are skipped, and duplicate dates keep the last row. [image] Launch the "ingest_from_csv" task from the rework task manager: upload the CSV and set the options. The parsing options are: * "sep" — column separator (default ","). * "decimal" — decimal separator (default "."). * "thousands" — thousands separator. * "encoding" — file encoding (default "utf-8"). * "na_values" — extra strings to read as missing values. * "dayfirst" — parse day-first dates (default "false"). * "date_format" — an explicit date format. * "timezone" — localise (or convert) the dates to this timezone. * "insertion_date" — record the update under this insertion date. * "author" — author of the update. Exploring history ================= From the **info page** of a series, you can explore its successive versions by activating the **history mode** selector, which replays the series across its revisions. A slider (the blue bar) and a *view all* option let you move through the versions. The selected version is drawn on the second graph, with its points coloured relative to the version's date: * **green** for past values, * **red** for future values. Hovering the second graph updates the third graph, which shows how the value at a single timestamp evolved across the revisions. [image] Working with metadata ===================== The **Metadata** tab of the info page shows the series' user metadata and lets you add or edit it. Metadata is versioned, and you can browse its earlier versions at the bottom of the page. [image] Renaming a series ================= The info page has a **Rename** button (blue, at the top right), available for local series only. Renaming asks for a confirmation and updates every formula that references the series to the new name. [image] Deleting a series ================= There are two ways to delete a series from the browser: the **Delete** page (accessible from the left menu) — to remove one or several series at once — and the **Delete** button on a series' **info page**. A series that is referenced by a formula cannot be deleted: the attempt stops with a message and the series is kept. Remove or repoint the referencing formulas first — you can see the Dependents of a series from the **Dependents** tab of the info page. [image] Finding series ============== The **Catalog** lists all series and groups in a flat mode. A filter panel on the left narrows the list; the filters combine into a single search expression (the same query language as Search Query Language Reference): * **Kind** — primary series and/or formulas. * **Value type** — float and/or string series. * **Timezone awareness** — tzaware or naive series. * **Sources** — which source, when several are configured. * **Name** and **formula content** fragments. * **Metadata** key/value filters. Each result row shows the series **name** (a link to its info page) and a **kind** badge, and, on the right, badges for its value type (a "string" badge), supervision status, source, storage and timezone- awareness. Hovering the small *M* marker of a series that carries metadata opens a **popover** listing its metadata. For a local formula the row offers an **edit formula** link; for a plain series it shows its left and right value bounds. [image] Getting series information ========================== At the top of a series' info page, a row of **badges** next to the name summarises it at a glance: * **timezone awareness** — "tzaware" or "tznaive"; * **supervision status** — "unsupervised", "supervised" or "handcrafted"; * **source** — "local", or the name of the remote source it comes from; * **storage backend** — "pg" (PostgreSQL) or "fs1". [image] Working with logs ================= The info page's **Logs** tab lists the series' successive revisions, one row per revision: * **#** — the revision number; * **Author** — who recorded the revision; * **Date** — when it was recorded (its insertion date); * **Meta** — the metadata attached to that revision (for example the request window a scraper recorded). You type how many revisions you want to see in the *Logs displayed* field, then click *See more logs* to load them. [image] The **Logs** tab only exists for primary series. Formula series get a **Formula cache** tab instead, showing their cache state (see Formula cache), and — as soon as the formula has been edited at least once — a **Formula history** tab listing its successive definitions. Viewing series ============== **Quick View** plots a set of series and groups together on an interactive graph. You reach it from a folder's **View** link, from a search query, or from a basket. Add what to plot with the **Series**, **Basket** and **Group** buttons. Each loaded item then appears in a row below the graph, with its name (a link to its info page), a status marker, an axis toggle to put it on the **1st** or **2nd Y axis** — handy to mix series of different scales, such as prices and volumes — a **Select** action to highlight it, and a **Remove** button to drop it. The top bar controls the view: * the **timezone** and the value-date window — a *from* and a *to* date, or a named horizon (the horizons of Settings), with arrows to shift it; * a **Cache** toggle — untick it to force a live re-computation instead of serving cached values; * the number of **decimals** displayed; * a **permalink** to share the exact view, and an **Edition** button that opens the loaded series in the editor. The graph itself is interactive (zoom, pan, hover tooltips), and **Show legend** toggles the legend. A **.csv** button downloads the plotted data — the series and every member of the loaded groups — as a single date-indexed table. [image] =============================================================================== # https://refinery.docs.pythonian.fr/core_components/supervision.html =============================================================================== Supervision *********** The supervision mechanism enables data quality management when working with external data sources. As an analyst, you can correct erroneous values while the system maintains the original data and ensures that provider corrections automatically flow through. The supervision workflow ======================== Consider a typical scenario: you receive daily market data from a provider. Sometimes this data contains errors that need immediate correction for your reports. The supervision system allows you to: 1. Apply manual corrections to fix errors immediately 2. Continue receiving updates from the provider 3. Have provider corrections automatically replace your manual fixes 4. Maintain full audit trail of all changes Making manual corrections ========================= Manual corrections can be applied in three ways: >>> corrected_values = pd.Series([95.5, 96.0], ... index=[pd.Timestamp('2023-03-15'), ... pd.Timestamp('2023-03-16')]) >>> tsa.update('market-prices', corrected_values, 'analyst@corp.com', manual=True) The same manual corrections can be made through the web interface or the Excel client - both automatically use manual=True when you edit values. Your corrections appear immediately in the data. When the provider later sends corrected data through the normal update process, their fixes automatically replace your manual corrections. Viewing data provenance ======================= The ".edited" method shows you which values are original versus manually edited: >>> series, markers = tsa.edited('market-prices') The returned series contains the current values, while markers is True where data was manually edited and False for provider data. To retrieve the raw series — the automated feed before any manual correction — pass "upstream=True" to "get": >>> raw = tsa.get('market-prices', upstream=True) This returns the data as the source delivered it, ignoring every manual edit. Understanding supervision status ================================ Every series has a supervision status in its metadata: * "unsupervised": series updated only through normal updates (never with manual=True) * "handcrafted": series created and maintained entirely with manual updates (always with manual=True, whether from Python, Excel client, or web UI) * "supervised": series containing both provider updates and manual corrections This helps you quickly identify which series have manual interventions. The web editor ============== The Series Editor is the web interface for visualising and editing time series directly in the browser, accessible from the series catalog. [image] The editor is organised into three areas: * **Graph** (top): displays the series values over the selected time window, with Plotly controls for zoom, pan and reset. When edits are pending, modified values appear in orange alongside the originals. Overrides are also marked visually. * **Data table** (bottom left): shows date-value pairs for the visible window. * **Statistics panel** (right): shows real-time descriptive statistics for the visible window — First/Last insertion dates, Start/End value dates, Min, Max, Sum, Count, NaN count, Mean, Std, P25, P50, P75, and inferred frequency. The top bar controls the time window (from/to dates), the timezone, decimal precision, and provides a permalink and a series info link. Selecting and navigating ------------------------ A click selects a row and sets it as the anchor for subsequent keyboard or shift-click extensions. Shift+click extends the selection from the anchor to the clicked row. Click-and-drag selects all rows crossed. Escape deselects everything. [image] Keyboard navigation: * Up/Down arrows: move focus one row * Shift+Up/Down: extend the selection * Ctrl+Up/Down: jump to the start or end of the series * PageUp/PageDown: scroll one screen Editing values -------------- *Direct editing*: clicking on a value cell makes it editable. The original value remains visible, and the change can be cancelled before saving. *Copy and paste*: data copied from Excel or any spreadsheet can be pasted directly. The editor parses the content to extract date-value pairs, handling multiple date formats, and matches them to existing timestamps. The current selection can also be copied to the clipboard in TSV format for use in a spreadsheet. *Filling missing values*: the Fill NAs button identifies each contiguous group of missing values and fills them by linear interpolation between the last valid point before the gap and the first valid point after it. Fill All processes all visible gaps at once. *Linear correction*: an affine transformation can be applied to the selected values. The user enters a slope and an intercept; each selected value is transformed as "y' = slope × y + intercept". This is typically used to correct a sensor offset or an incorrect scale factor. [image] *Batch deletion*: pressing the Delete key removes the values from all selected cells in one operation, turning them into missing values. Editing several series at once ------------------------------ The editor is a spreadsheet that can hold several series side by side, each as a column aligned on a shared date axis. Only primary series are editable; formula series are read-only. Its main use is **correcting a formula at its source**. Open the formula in the editor from its *tsinfo* page with the **show values** link. A formula is computed and cannot be edited directly, but the **Expand Formula** button reveals the primary source series it is built from, each as an editable column — so you can correct the underlying data with the formula's own values in view, all in the same table. [image] Several series can also be loaded into the editor from three other places: * **A folder** (see Folders) — its **Edition** link opens every series it contains. * **A basket** — its **View** link opens its series in Quick view, from where they can be sent to the editor. * **Quick View** — a page that displays a set of series, with an **Edition** button that opens them all in the editor. With several columns, the tools above become two-dimensional: * **Rectangular selection** — the selection can span several series (columns) as well as several dates (rows). Shift+click, click-and- drag and Shift+arrow extend the rectangle across columns; clicking a column header selects a whole series. * **2D copy and paste** — the selected block is copied as a tab- separated grid (columns = series), so it drops straight into a spreadsheet; conversely a multi-column block copied from a spreadsheet is pasted back as a rectangle anchored at the focused cell, writing only into editable cells. * **Batch operations across series** — Delete, Fill All and the linear correction apply to every selected or edited cell, whatever series it belongs to. * **Saving** — a single save commits the pending edits to every series that was modified. Decimal formatting ------------------ The number of decimal places displayed can be adjusted with the +/- buttons in the top bar. Thousands are separated by spaces. These preferences are retained between sessions. Supervision API Reference ========================= class mainsource(*args, **kwargs) API façade for the main source (talks directly to the storage) The api documentation is carried by this object. The http client provides exactly the same methods. edited(name, revision_date=None, from_value_date=None, to_value_date=None, inferred_freq=False, keepnans=False) Returns the base series and a second boolean series whose entries indicate if an override has been made or not. Parameters: * **name** (*str*) * **revision_date** (*Timestamp** | **None*) * **from_value_date** (*Timestamp** | **None*) * **to_value_date** (*Timestamp** | **None*) * **inferred_freq** (*bool** | **None*) * **keepnans** (*bool*) Return type: *Tuple*[*Series*, *Series*] supervision_status(name) Returns the supervision status of a series. Possible values are *unsupervised*, *handcrafted* and *supervised*. Parameters: **name** (*str*) Return type: str =============================================================================== # https://refinery.docs.pythonian.fr/core_components/formula.html =============================================================================== Formulas (computed series) ************************** Table of Contents ================= * Introduction * Creating Your First Formula * The formula editor * Using Formula Series * Advanced: Creating Custom Operators * Formula Language Reference * Formula API Reference Introduction ============ Formulas are computed time series that derive their values dynamically from other series through calculations. Unlike primary series that store data directly in the database, formulas are evaluated on-demand using a powerful expression language. Key characteristics: * **Read-only**: Formulas cannot be updated directly with new data points. Their values are entirely determined by the formula expression and the underlying series data. * **Versioned**: Formulas automatically inherit version history from their constituent series. When you request a formula at a specific revision date, it computes using the series versions that existed at that time. * **Lazy evaluation**: Formula values are computed only when requested, not when underlying data changes. This ensures efficient resource usage. * **Composable**: Formulas can reference other formulas, allowing you to build complex calculations from simpler building blocks. * **Cacheable**: For performance-critical scenarios, formula results can be materialized into a cache that refreshes periodically. * **API transparency**: Formula and stored series are indistinguishable through the read API ("get", "history", "metadata", etc.). You access them identically - the system handles whether to fetch stored data or compute formula values. Only specific methods like "type()" or "formula()" reveal whether a series is stored or computed. Creating Your First Formula =========================== Let's start with a simple example that adds two series together. Note that the series referenced in the formula must already exist in the database: >>> from tshistory.api import timeseries >>> tsa = timeseries() >>> # register a formula that computes total energy >>> # both solar_production and wind_production must exist >>> tsa.register_formula( ... 'total_energy', ... '(add (series "solar_production") (series "wind_production"))' ... ) >>> # use it exactly like a stored series >>> total = tsa.get('total_energy') >>> print(total) 2024-01-01 150.5 2024-01-02 142.3 2024-01-03 163.7 Name: total_energy, dtype: float64 The formula is now permanently registered and will automatically compute whenever requested. Formula Language Basics ----------------------- Formulas use a Lisp-like syntax with these key elements: ;; operator and arguments in parentheses (add (series "a") (series "b")) ;; multiply by constant 1.2 (* 1.2 (series "price")) ;; reference a series (series "temperature_celsius") ;; forward fill missing values using optional parameter (series "noisy_data" #:fill "ffill") ;; daily average from hourly data (resample (series "hourly") "D" #:method "mean") ;; nested operations compute (revenue + other_income) / costs (div (add (series "revenue") (series "other_income")) (series "costs")) For the complete list of available operators, see Formula Language Reference. Updating a Formula ------------------ To modify an existing formula, use "register_formula" again with the same name: >>> # original formula >>> tsa.register_formula( ... 'total_energy', ... '(add (series "solar") (series "wind"))' ... ) >>> # update to include hydro >>> tsa.register_formula( ... 'total_energy', ... '(add (series "solar") (series "wind") (series "hydro"))' ... ) >>> # the formula now includes three components >>> print(tsa.formula('total_energy')) (add (series "solar") (series "wind") (series "hydro")) Deleting a Formula ------------------ Remove a formula when it's no longer needed: >>> tsa.delete('total_energy') >>> tsa.exists('total_energy') False The formula editor ================== Formulas can also be written and edited from the browser, with the **formula editor** — reached from the *Formula* menu (to create one) or from the *edit formula* links found across the interface (the catalog, baskets, ...) and a series' info page. It has a **Series** and a **Group** tab, depending on what the formula produces. You give the formula a **name** and edit its definition in two synchronised panes: * **Formula edition** — a text editor for the lisp expression, with syntax highlighting; * **Editor** — a structured, form-based view of the same expression, with a drop-down per operator and a field per argument, handy for discovering the available operators and their parameters. A live **plot** below previews the result as you edit, and a **series info** link opens the series' info page. **Create** saves a new formula and **Update** saves changes to an existing one; undo/redo and a copy/paste mode are also available. [image] Using Formula Series ==================== Formula series have some specific behaviors and methods that distinguish them from primary series. Formula-Specific API Methods ---------------------------- >>> # get the formula expression >>> expr = tsa.formula('total_energy') >>> print(expr) '(add (series "solar") (series "wind") (series "hydro"))' >>> # check if a series is a formula >>> tsa.type('total_energy') 'formula' When formulas reference other formulas, you can see the expanded expression. For example, if 'renewable' is defined as "(add (series "solar") (series "wind"))" and 'total' is defined as "(add (series "renewable") (series "hydro"))": >>> tsa.formula('total') '(add (series "renewable") (series "hydro"))' >>> tsa.formula('total', expanded=True) '(add (add (series "solar") (series "wind")) (series "hydro"))' Formula Dependencies -------------------- Understanding what a formula depends on is crucial for debugging and maintenance: >>> # direct dependencies of a formula >>> tsa.formula_components('total_energy') {'total_energy': ['solar', 'wind', 'hydro']} For nested formulas where 'total' uses 'renewable' which in turn uses 'solar' and 'wind', the expanded view shows the full dependency tree: >>> tsa.formula_components('total', expanded=True) {'total': [{'renewable': ['solar', 'wind']}, 'hydro']} Formula Insertion Dates ----------------------- Formulas have a unique behavior regarding insertion dates. They inherit the union of all insertion dates from their components. If solar was updated on January 1 and 3, and wind was updated on January 2 and 3, the formula shows all three dates: >>> tsa.insertion_dates('total_energy') [Timestamp('2024-01-01 09:00:00+0000', tz='UTC'), Timestamp('2024-01-02 09:00:00+0000', tz='UTC'), Timestamp('2024-01-03 09:00:00+0000', tz='UTC')] This means the formula's history contains a version for every change in any component series. Formula Evaluation Context -------------------------- Formulas can use special operators that access evaluation context. The "today()" operator is particularly useful for creating rolling windows: >>> # formula using today() operator >>> tsa.register_formula( ... 'last_30_days_avg', ... '(rolling (slice (series "temperature") ' ... ' #:fromdate (shifted (today) #:days -30)) 7)' ... ) When called normally, "today()" returns the current date: >>> current = tsa.get('last_30_days_avg') When called with a revision_date, "today()" becomes that date, allowing the formula to compute as if evaluated in the past: >>> historical = tsa.get('last_30_days_avg', ... revision_date='2023-06-01T00:00:00Z') In this case, the formula computes as if "today" was June 1, 2023, creating a 30-day window ending on that date. Additional Formula Methods -------------------------- Testing formulas without registering them is useful during development: >>> result = tsa.eval_formula('(add (series "a") (series "b"))') This evaluates the expression and returns the computed series without saving the formula. Check the depth of a formula: >>> tsa.formula_depth('complex_formula') 20 View historical formula definitions when a formula has been modified over time: >>> history = tsa.oldformulas('total_energy') >>> for formula, timestamp, author in history: ... print(f"{timestamp} by {author}: {formula}") 2024-01-01 10:00:00+00:00 by alice: (add (series "solar") (series "wind")) 2024-02-15 14:30:00+00:00 by bob: (add (series "solar") (series "wind") (series "hydro")) Renaming Series and Formula Propagation --------------------------------------- When renaming a series that is referenced in formulas, the system can automatically update all formulas to use the new name: >>> # rename with propagation (default) >>> tsa.rename('solar', 'solar_pv') This automatically rewrites all formulas that reference 'solar' to use 'solar_pv' instead. The system prevents renaming if the new name would conflict with existing references in formulas. >>> # rename without propagation >>> tsa.rename('solar', 'solar_pv', propagate=False) Without propagation, formulas referencing the old name will break. Use this only when you intend to update formulas manually or delete them. Deleting Series Referenced in Formulas -------------------------------------- The system refuses to delete a series that is still referenced by formulas or group formulas — the error names the referencing items: >>> tsa.delete('solar') ValueError: cannot delete `solar`: referenced by series total_energy The check can be overridden with "force=True"; the referencing formulas are then left in place and will fail at evaluation time: >>> tsa.delete('solar', force=True) >>> tsa.get('total_energy') ValueError: No such series `solar` To enumerate what depends on a series before deleting it, use "tsa.depends('solar', reverse=True)". Performance and Caching ----------------------- Complex formulas with deep nesting or expensive computations may benefit from caching. The cache system materializes formula results and refreshes them periodically, improving query performance. Cache Impact on Read Operations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When a formula has an active cache, the system intelligently decides whether to use cached or live data. **get() method behavior with cache**: The system automatically detects if the cache is stale by analyzing the regularity of cache insertion dates. If more than 2 expected update intervals have passed since the last cache update, it considers the cache stale. For stale cache, the system: * Retrieves the cached historical data * Computes fresh data for a window defined by the cache policy's "look_before" and "look_after" parameters * Patches the cached data with the fresh computation (fresh data overwrites cached for overlapping periods) * Returns only the date range you requested This ensures you get cached performance for historical data while still receiving fresh data for recent periods. Use "nocache=True" to bypass the cache entirely and force live computation. **insertion_dates() method behavior with cache**: Returns the cache's insertion dates, but transparently completes the list with uncached formula insertion dates for any periods before the cache was initialized. This provides a complete view of the formula's version history regardless of cache coverage. Use "nocache=True" to get the formula's actual insertion dates without any cache influence. The cache system is designed to be transparent - it automatically provides the best available data based on freshness and coverage. See Formulas: when to use a cache/materialized view for detailed guidance on cache configuration. Advanced: Creating Custom Operators =================================== This is a fundamental need. Operators are fixed python functions exposed through a lispy syntax. Applications need a variety of fancy operators. Declaring a new operator ------------------------ One just needs to decorate a python function with the "func" decorator: from tshistory_formula.registry import func @func('identity') def identity(series: pd.Series) -> pd.Series: """ Return the timeseries. Example: `(identity (series "my_series"))` """ return series The operator will be known to the outer world by the name given to "@func", not the python function name (which can be arbitrary). You *must* provide correct type annotations : the formula language is statically typed and the typechecker will refuse to work with an untyped operator. Note: The function's docstring is mandatory. It will appear in the formula operators documentation. For the tool to assimilate this custom operator, a new section has to be added to the *setup.py`* configuration. Configuration example below: from setuptools import find_packages, setup setup(name='custom', version='0.1.0', author='me', author_email='me@company.fr', description='Custom refinery applicative part', packages=find_packages(), install_requires=[ 'tshistory_refinery', ], entry_points={ 'forceimports': [ 'forceimports=custom.funcs:IMPORTCALLBACK' ] } ) This is enough to get a working *transformation* operator. However operators built to construct series rather than just transform pre- existing series are more complicated. More Transformation Examples ---------------------------- Here's another simple operator with parameters: @func('scale') def scale(series: pd.Series, factor: float) -> pd.Series: """ Perform a scalar multiplication to each value of the series. Example: `(scale (series "my_series") 3.14)` """ return series * factor Usage in formulas: (scale (series "temperature_celsius") 1.8) Autotrophic series operator --------------------------- We start with an example, a "proxy" operator that gets a series from an existing time series silo (on the fly) to be served as it came from your local installation. We would use it like this: "(proxy "a-name" #:parameter 42.3)" As we can see it can look like the "series" operator, though its signature might be more complicated (this will be entirely dependent on the way to enumerate series in the silo). Hence "proxy" must be understood as an alternative to "series" itself. Here is how the initial part would look: from tshistory_formula.registry import func, finder, metadata, history, insertion_dates @func('proxy', auto=True) def proxy(__interpreter__, __from_value_date__, __to_value_date__, __revision_date__, name: str, parameter=0): """ Yields data from silo database, taking the silo reference as "name" input. Example: `(proxy (name "abc_123456") #:parameter 0)` """ # we assume there is some python client available # for the tier timeseries silo return silo_client.get( parameter=parameter, fromdate=__from_value_date__, todate=__to_value_date__, revdate=__revision_date__ ) This is a possible implementation of the API *get* protocol. Ths dunder methods are a mandatory part of the signature. The other parameters (positional or keyword) are at your convenience and will be exposed to the formula users. We must also provide an helper for the formula system to detect the presence of this particular kind of operator in a formula (because it is not like other mere *transformation* operators). Let's have it: @finder('proxy') def proxy_finder(cn, tsh, tree): return { tree[1]: tree } Let us explain the parameters: * *cn* is a reference to the current database connection * *tsh* is a reference to the internal API implementation object (and you will need the *cn* object to use it) * *tree* is a representation of the formula restricted to the proxy operator use When implementing a proxy-like operator, one generally won't need the first two items. But here is an example of what the *tree* would look like: ['proxy, 'a-name', '#:parameter, 77] Yes, the half-quoted *'proxy* and *'#:parameters* are not typos. These are respectively a: * symbol (simimlar to a variable name in Python) * keyword (similar to a Python keyword) In the finder return dictionary, only the key of the dictionary is important: it should be globally unique and will be used to provide an (internal) alias for the provided series name. For instance, in our example, if *parameter* has an impact on the returned series identity, it should be part of the key. Like this: @finder('proxy') def proxy_finder(cn, tsh, tree): return { f'tree[1]-tree[2]': tree } We also have to map the *metadata*, *insertion_dates* and the *history* API methods. @metadata('proxy') def proxy_metadata(cn, tsh, tree): return { f'proxy:{tree[1]}-{tree[2]}': { 'tzaware': True, 'source': 'silo-proxy', 'index_type': 'datetime64[ns, UTC]', 'value_type': 'float64', 'index_dtype': '|M8[ns]', 'value_dtype': '>> tsa.formula_components('my-series') {'my-series': ['component-a', 'component-b']} >>> tsa.formula_components('my-series-2', expanded=True) {'my-series-2': [{'sub-component-1': ['component-a', 'component-b']}, 'component-b']} Parameters: * **name** (*str*) * **expanded** (*bool*) Return type: *Dict*[str, list] | None formula_depth(name) Compute the depth of a formula. The depth is the maximum number of formula series sub expressions that have to be traversed to get to the bottom. Parameters: **name** (*str*) eval_formula(formula, revision_date=None, from_value_date=None, to_value_date=None, tz=None, nocache=False) Execute a formula on the spot. tsa.eval_formula('(add (series "sales.fr") (series "sales.be"))') Parameters: * **formula** (*str*) * **revision_date** (*Timestamp** | **None*) * **from_value_date** (*Timestamp** | **None*) * **to_value_date** (*Timestamp** | **None*) * **nocache** (*bool*) Return type: *Series* class mainsource(*args, **kwargs) API façade for the main source (talks directly to the storage) The api documentation is carried by this object. The http client provides exactly the same methods. new_cache_policy(name, initial_revdate, look_before, look_after, revdate_rule, schedule_rule) Create a cache policy. Parameters: * **name** (*str*) * **initial_revdate** (*str*) * **look_before** (*str*) * **look_after** (*str*) * **revdate_rule** (*str*) * **schedule_rule** (*str*) Return type: None edit_cache_policy(name, initial_revdate, look_before, look_after, revdate_rule, schedule_rule) Modify an existing cache policy (by name). Parameters: * **name** (*str*) * **initial_revdate** (*str*) * **look_before** (*str*) * **look_after** (*str*) * **revdate_rule** (*str*) * **schedule_rule** (*str*) Return type: None delete_cache_policy(name) Delete a cache policy (by name). Parameters: **name** (*str*) Return type: None set_cache_policy(policyname, seriesnames) Associate series with a cache policy. Parameters: * **policyname** (*str*) * **seriesnames** (*List**[**str**]*) Return type: None unset_cache_policy(seriesnames) Dis-associate series from a cache policy. Parameters: **seriesnames** (*List**[**str**]*) Return type: None cache_free_series(allsources=True) List the series that are available for association with a cache policy. Parameters: **allsources** (*bool*) cache_policies() Return a list of cache policies names. cache_policy_series(policyname) Return the list of series associated with a cache policy. Parameters: **policyname** (*str*) has_cache(seriesname) Predicate to verify is a series formula has a cache. Parameters: **seriesname** (*str*) delete_cache(seriesname) Purge the cache of a formula. Parameters: **seriesname** (*str*) =============================================================================== # https://refinery.docs.pythonian.fr/core_components/formula_language.html =============================================================================== Formula Language Reference ************************** The Refinery formula language allows you to create computed time series using a powerful Lisp-like functional language. Formulas are evaluated dynamically and can reference other series, perform calculations, and apply transformations. Table of Contents ================= * Getting Started * Language Basics * Series Access and Options * Arithmetic and Mathematical Operations * Temporal Operations * Aggregation and Statistics * Data Processing and Filtering * Date and Time Functions * Advanced Series Discovery Getting Started =============== Your First Formula ------------------ # Register a simple formula that adds two series tsa.register_formula( 'total_sales', '(add (series "online_sales") (series "retail_sales"))' ) # Use the formula series like any other series total = tsa.get('total_sales') Basic Syntax Rules ------------------ Formulas are expressed in a Lisp-like syntax using operators, positional (mandatory) parameters and keyword (optional) parameters. The general form is: "( ... #: ... #: )" Key rules: 1. **Parentheses**: Every operation is wrapped in parentheses "(function arg1 arg2 ...)" 2. **Quotes**: String literals use double quotes ""series_name"" 3. **Keywords**: Optional parameters use "#:keyword value" syntax 4. **Nesting**: Functions can be nested "(add (series "a") (mul (series "b") 2))" 5. **Operator names**: Can contain dashes or arbitrary characters Language Basics =============== Data Types ---------- ;; Numbers (integers and floats) 42 3.14159 -100 ;; Strings (series names, dates, etc.) "daily_temperature" "2024-01-01" "Europe/Paris" ;; Booleans #t ;; true #f ;; false ;; Dates (created with date function) (date "2024-01-01") (date "2024-01-01" #:tz "America/New_York") ;; Series (result of series operations) (series "stock_price") Series Access and Options ========================= Basic Series Access ------------------- ;; Simple series reference (series "temperature") ;; Series from different sources (if configured) (series "remote_data") ;; Automatically resolves from available sources Series Options -------------- Series options control how data is handled when combining with other series: ;; Fill missing values (series "incomplete_data" #:fill "ffill") ;; Forward fill (series "sparse_data" #:fill "bfill") ;; Backward fill (series "default_zero" #:fill 0) ;; Fill with constant value (series "mixed_fill" #:fill "ffill, bfill") ;; Forward fill then backward fill ;; Preserve NaN values (useful for priority operations) (series "mask_data" #:keepnans #t) ;; Bound the filling: at most 3 consecutive NaNs get filled (series "gappy_data" #:fill "ffill" #:limit 3) ;; Add weights for aggregation operations (series "important_station" #:weight 2.0) (series "minor_station" #:weight 0.5) ;; Combine multiple options (series "weather_data" #:fill "ffill" #:weight 1.5 #:keepnans #f) Options Operator ---------------- ;; Apply options to any series expression (options (series "raw_data") #:fill "ffill" #:weight 2) ;; Apply options to computed series (options (add (series "a") (series "b")) #:weight 1.5) Arithmetic and Mathematical Operations ====================================== Scalar Operations ----------------- ;; Add constant to series (+ 100 (series "base_value")) ;; Add 100 to each point (+ 273.15 (series "celsius")) ;; Convert Celsius to Kelvin ;; Multiply by constant (* 365 (series "daily_value")) ;; Annualize daily values (* -1 (series "debt")) ;; Flip sign ;; Divide by constant (/ (series "total") 1000) ;; Convert to thousands (/ (series "percentage") 100) ;; Convert percentage to decimal ;; Power operations (** (series "growth_rate") 12) ;; Compound monthly to annual (** (series "distance") 2) ;; Square for area calculations Series-to-Series Operations --------------------------- ;; Addition (variable number of arguments) (add (series "q1") (series "q2") (series "q3") (series "q4")) (add (series "domestic") (series "international")) ;; Multiplication (mul (series "price") (series "quantity")) ;; Revenue calculation (mul (series "usd_amount") (series "usd_to_eur_rate")) ;; Currency conversion ;; Division (div (series "revenue") (series "costs")) ;; Profit margin (div (series "total_sales") (series "num_stores")) ;; Sales per store ;; Subtraction (takes exactly 2 series) (sub (series "revenue") (series "costs")) ;; Profit calculation (sub (series "target") (series "actual")) ;; Variance analysis Mathematical Functions ---------------------- ;; Absolute value (abs (series "temperature_diff")) ;; Remove negative signs (abs (sub (series "forecast") (series "actual"))) ;; Forecast error magnitude ;; Rounding (round (series "precise_values")) ;; Round to nearest integer (round (* 100 (series "percentage"))) ;; Round percentages **min** / **max** - Scalar min/max functions (min 5.5 2.1 8.9) ;; Returns 2.1 (max 10 20 30 40) ;; Returns 40 (max 3.14 2.71 1.41) ;; Returns 3.14 Trigonometric Functions ----------------------- All trigonometric functions work with degrees (not radians): ;; Basic trigonometric functions (trig.sin (series "angle_degrees")) ;; Sine (trig.cos (series "angle_degrees")) ;; Cosine (trig.tan (series "angle_degrees")) ;; Tangent ;; Inverse trigonometric functions (output in degrees) (trig.arcsin (series "ratio")) ;; Arcsine (trig.arccos (series "ratio")) ;; Arccosine (trig.arctan (series "slope")) ;; Arctangent ;; Two-argument arctangent (for vector angles) (trig.row-arctan2 (series "y_component") (series "x_component")) ;; Example: Convert wind components to direction (trig.row-arctan2 (series "wind_north") (series "wind_east")) Temporal Operations =================== Resampling (Downsampling) ------------------------- Resampling reduces the frequency of data by aggregating values: ;; Basic resampling with different aggregation methods (resample (series "hourly_temperature") "D") ;; Daily mean (default) (resample (series "minute_prices") "h" #:method "last") ;; Hourly last value (resample (series "tick_volume") "15T" #:method "sum") ;; 15-minute volume sum (resample (series "daily_rain") "M" #:method "sum") ;; Monthly rainfall total ;; Advanced resampling (resample (series "irregular_data") "W" #:method "median") ;; Weekly median (resample (series "high_freq") "Q" #:method "std") ;; Quarterly volatility (resample (series "sensors") "Y" #:method "count") ;; Annual data point count ;; Local-time bucket boundaries (default is UTC); tzaware output is converted back to UTC (resample (series "canadian_hourly") "D" #:tz "America/Toronto") ;; daily mean on local days ;; Common frequency codes: ;; min - minutes, h - hours, D - days, W - weeks ;; ME - month end, QE - quarter end, YE - year end ;; Business frequencies: B - business days, BME - business month end **freq** and **nfreq** - Frequency builders (freq "h") ;; Hourly frequency (freq "D") ;; Daily frequency (nfreq 15 "min") ;; 15-minute frequency (nfreq 3 "ME") ;; Quarterly frequency ;; Use with resample (resample (series "data") (freq "h")) ;; To hourly (resample (series "data") (nfreq 15 "min")) ;; To 15-minute (resample (series "data") (nfreq 2 "W")) ;; To bi-weekly Upsampling (Increasing Frequency) --------------------------------- Upsampling increases the frequency by interpolating or forward- filling: ;; Convert monthly data to daily (upsample (series "monthly_budget") #:freq "D" #:origin_freq "M") ;; Convert annual data to quarterly with interpolation (upsample (series "yearly_target") #:freq "Q" #:origin_freq "Y" #:method "interpolate") ;; Convert weekly data to daily (upsample (series "weekly_sales") #:freq "D" #:origin_freq "W") ;; Local-time boundaries (DST-correct), instead of UTC (upsample (series "yearly_target") #:freq "h" #:origin_freq "Y" #:tz "Europe/Paris") Time Shifting ------------- ;; Basic shifting (time-shifted (series "data") #:hours 6) ;; Shift forward 6 hours (time-shifted (series "data") #:days -1) ;; Shift back 1 day Three main use cases: 1. **Fix badly indexed series** - When timestamps are wrong by a consistent offset ;; Series was recorded in wrong timezone, shift by 5 hours (time-shifted (series "misindexed_data") #:hours 5) 2. **Business period alignment** - e.g., gas day (06:00 to 06:00) ;; Shift gas flow data so gas day aligns with calendar day (time-shifted (series "gas_flow") #:hours -6) ;; Now midnight values represent start of gas day 3. **"What if" scenarios** - Exploring hypothetical time shifts ;; What if maintenance was delayed by 2 days? (time-shifted (series "maintenance_schedule") #:days 2) Rolling Operations ------------------ Rolling operations compute statistics over a moving window: ;; Moving averages (rolling (series "stock_price") 20) ;; 20-period moving average (default: mean) (rolling (series "temperature") 7 #:method "mean") ;; 7-day average temperature (rolling (series "sales") 30 #:method "median") ;; 30-day median sales ;; Rolling aggregations (rolling (series "daily_volume") 5 #:method "sum") ;; 5-day volume sum (rolling (series "prices") 10 #:method "min") ;; 10-day minimum price (rolling (series "prices") 10 #:method "max") ;; 10-day maximum price ;; Rolling statistics (rolling (series "returns") 252 #:method "std") ;; Annual volatility (252 trading days) (rolling (series "earnings") 4 #:method "count") ;; 4-quarter data availability ;; Technical analysis examples (rolling (series "close") 50 #:method "mean") ;; 50-day moving average (rolling (series "high") 14 #:method "max") ;; 14-day highest high (rolling (series "low") 14 #:method "min") ;; 14-day lowest low Cumulative Operations --------------------- ;; Cumulative sum (cumsum (series "daily_sales")) ;; Running total of sales (cumsum (series "rainfall")) ;; Total rainfall to date ;; Cumulative product (cumprod (series "daily_returns")) ;; Cumulative returns (cumprod (+ 1 (series "growth_rates"))) ;; Compound growth Integration ----------- The integration operator combines a stock series with a flow series: ;; Basic integration (integration "stock-series-name" "flow-series-name") ;; Integration with fill (handles gaps in stock series) (integration "inventory-level" "daily-changes" #:fill #t) ;; Example: Tank level tracking (integration "tank-level-readings" "flow-meter-data") ;; Drop the stock readings coming after the last flow point (integration "tank-level-readings" "flow-meter-data" #:remove_stock_trail #t) This operator integrates the flow series from the last known stock value, useful for tracking cumulative quantities where you have occasional absolute readings (stock) and continuous change measurements (flow). Block Staircase (Future Slicing) -------------------------------- The block-staircase operator extracts a consistent "slice of the future" from historical forecast revisions: ;; Extract 24-hour-ahead forecasts from daily runs (block-staircase "temperature-forecast" #:revision_freq_days 1 ;; Daily forecast runs #:revision_time_hours 6 ;; Run at 6am #:maturity_offset_days 1) ;; Extract 1-day-ahead slice ;; Extract 2-hour-ahead predictions from hourly model runs (block-staircase "demand-forecast" #:revision_freq_hours 1 ;; Hourly updates #:maturity_offset_hours 2) ;; Extract 2-hour-ahead slice What it does: For each point in the output series, it finds the forecast that was made *maturity_offset* time ago. This reconstructs what you "thought would happen" at a consistent horizon. Use cases: - Analyze forecast accuracy at specific horizons ("How good are our day-ahead forecasts?") - Regulatory compliance ("What did we publish 24 hours in advance?") - Compare model performance across different lead times Aggregation and Statistics ========================== Row-wise Operations ------------------- These operations work across multiple series at each time point: ;; Row-wise mean (simple average) (row-mean (series "station_1") (series "station_2") (series "station_3")) ;; Weighted row-wise mean (row-mean (series "main_sensor" #:weight 3) (series "backup_sensor" #:weight 1)) ;; Row-wise min/max (row-min (series "bid_price_1") (series "bid_price_2") (series "bid_price_3")) (row-max (series "server_cpu_1") (series "server_cpu_2")) ;; Row-wise standard deviation (row-std (series "measurement_1") (series "measurement_2") (series "measurement_3")) ;; Row-wise median (row-median (series "station_1") (series "station_2") (series "station_3")) ;; Control NaN handling (row-mean (series "sensor_a") (series "sensor_b") #:skipna #f) ;; Include NaNs Day-of-Year Aggregation (Folded Series) --------------------------------------- The doy-agg operator creates statistical profiles by day-of-year, useful for "folded" visualizations where multiple years are overlaid on the same plot: ;; Create median profile from 5 years of data (doy-agg (series "daily_temperature") 5 #:method "median") ;; Mean profile with stricter data requirements (doy-agg (series "energy_consumption") 4 #:method "mean" #:valid_aggr_ratio 0.8) ;; Require 80% of years to have data ;; Handle leap years for financial data (doy-agg (series "daily_revenue") 3 #:leap_day_rule "ignore") ;; Always 365 days ;; depth is optional — omit it to use every available year (doy-agg (series "daily_temperature") #:method "mean") ;; Bound the output period with #:fromdate / #:todate (doy-agg (series "daily_temperature") 5 #:fromdate (date "2026-01-01") #:todate (date "2026-12-31")) This creates the statistical summary line (median, mean, etc.) that you would typically overlay on a folded plot showing individual years. The result has one value per day-of-year, perfect for: * Climate analysis (current year vs historical normal) * Seasonal pattern detection * Year-over-year comparisons on the same axis * Building baseline expectations Practical Aggregation Examples ------------------------------ ;; Average temperature across weather stations (row-mean (series "station_north" #:weight 1) (series "station_south" #:weight 1) (series "station_city" #:weight 2)) ;; City station has double weight ;; Portfolio value calculation '(add (mul (series "stock_a_price") (series "stock_a_shares")) (mul (series "stock_b_price") (series "stock_b_shares")) (mul (series "stock_c_price") (series "stock_c_shares")))' ;; Risk calculation (worst case scenario) '(row-min (series "optimistic_forecast") (series "pessimistic_forecast") (series "realistic_forecast"))' Data Processing and Filtering ============================= Conditional and Comparison Operations ------------------------------------- ;; Comparison operators (return 0/1 series by default) (> (series "temperature") 25) ;; Greater than (>= (series "score") 70) ;; Greater or equal (< (series "price") 100) ;; Less than (<= (series "volume") 1000) ;; Less or equal (== (series "status") 1) ;; Equal to (<> (series "value") 0) ;; Not equal to ;; With custom true/false values (> (series "temp") 25 #:true_value 100 #:false_value -100) (== (series "flag") 1 #:true_value 1 #:false_value -1) ;; Series-to-series comparisons (> (series "actual") (series "target")) Clipping and Bounds ------------------- ;; Remove negative values (clip (series "temperature_anomaly") #:min 0) ;; Cap maximum values (clip (series "cpu_usage") #:max 100) ;; Set both bounds (clip (series "normalized_score") #:min 0 #:max 1) ;; Replace out-of-bounds values instead of removing them (clip (series "sensor_reading") #:min 0 #:max 1000 #:replacemin #t #:replacemax #t) ;; The #:min and #:max bounds are constants: clip cannot use ;; another series (e.g. a rolling statistic) as a dynamic bound. Forward Curves -------------- Note: This operator is only accessible within the pro version. If interested, please visit: https://timeseries.pythonian.fr/#ouroffer. The "full-fwd-price" operator assembles a full forward price curve from sliding maturity series: for every day of the output range, the value comes from the series whose maturity has the shortest delivery period still covering that day (daily quotes win over weekly ones, weekly over monthly, and so on). (full-fwd-price (today) "standard" (label "D+1" (series "market.eex.d1.price")) (label "W+1" (series "market.eex.w1.price")) (label "M+1" (series "market.eex.m1.price"))) Each input series is tagged with its maturity code through "label". Accepted maturities: "D+1..3", "WkEnd+1", "W+1..3", "M+1..3", "Q+1..3", "S+1..3" (seasons) and "Cal+1..3" (years). The second argument names the calendar convention used to compute the maturity boundaries. Priority and Layering --------------------- The priority operator combines series by taking the first available value: ;; Basic data fallback chain '(priority (series "real_time_data") (series "delayed_data") (series "historical_estimate"))' ;; Sensor data with backup '(priority (series "primary_sensor") (series "secondary_sensor") (constant 20 (date "2020-01-01") (date "2030-01-01") "h" (today)))' ;; Financial data with different frequencies '(priority (series "high_freq_prices") ;; Prefer high-frequency data (series "daily_prices") ;; Fall back to daily (series "weekly_estimates"))' ;; Last resort ;; Combine actual and forecast data '(priority (series "actual_sales") ;; Use actual when available (series "forecasted_sales"))' ;; Fill gaps with forecast Date-based Slicing ------------------ ;; Slice from specific date (slice (series "historical_data") #:fromdate (date "2020-01-01")) ;; Slice to specific date (slice (series "forecast_data") #:todate (date "2025-12-31")) ;; Slice specific date range '(slice (series "training_data") #:fromdate (date "2018-01-01") #:todate (date "2022-12-31"))' ;; Dynamic slicing (last year of data) '(slice (series "recent_data") #:fromdate (shifted (today) #:days -365))' Date-based Filtering with Cron Rules ------------------------------------ Filter data based on calendar patterns: ;; Business hours only (9 AM to 5 PM, weekdays) '(date-filter (series "office_temperature") (cronrule #:hour "9-17" #:day_week "1-5"))' # Keep only the on-the-hour and half-hour points '(date-filter (series "quarter_hourly") (cronrule #:minute "0,30"))' ;; First day of each month '(date-filter (series "monthly_report_data") (cronrule #:day_month "1"))' ;; Quarterly data (first day of Q1, Q2, Q3, Q4) '(date-filter (series "quarterly_metrics") (cronrule #:month "1,4,7,10" #:day_month "1"))' ;; Weekend data only '(date-filter (series "leisure_activity") (cronrule #:day_week "0,6"))' # Sunday=0, Saturday=6 ;; Peak hours in specific timezone '(date-filter (series "energy_consumption") (cronrule #:hour "18-22") #:tzone "Europe/Paris")' ;; Summer months only (June, July, August) '(date-filter (series "cooling_costs") (cronrule #:month "6,7,8"))' ;; Intraday window to the minute — from_time inclusive, to_time exclusive '(date-filter (series "intraday_prices") (cronrule #:from_time "08:30" #:to_time "18:00"))' ;; Business hours on weekdays (handy for sub-hourly series) '(date-filter (series "sensor_15min") (cronrule #:from_time "08:00" #:to_time "18:00" #:day_week "1-5"))' Revision Date Control --------------------- The asof operator fetches a series at a specific revision date: ;; Get series as it was yesterday (asof (shifted (now) #:days -1) (series "volatile-series")) ;; Get series as of a specific date (asof (date "2024-01-01") (series "historical-data")) ;; Combine with other operations (add (asof (date "2023-12-31") (series "2023-values")) (series "2024-adjustments")) Date and Time Functions ======================= Current Date and Time --------------------- ;; Current timestamp (now) ;; Current UTC timestamp (now #:tz "America/New_York") ;; Current time in NY timezone (now #:tz nil) ;; Naive timestamp (no timezone) ;; Today at midnight (today) ;; Today at midnight UTC (today #:tz "Europe/London") ;; Today at midnight London time (today #:tz nil) ;; Today naive (no timezone) Date Creation ------------- ;; Create specific dates (date "2024-01-01") ;; January 1, 2024 UTC (date "2024-07-15" #:tz "Asia/Tokyo") ;; July 15, 2024 Tokyo time (date "2024-12-31") ;; December 31, 2024 UTC (date "2024-01-01" #:tz nil) ;; January 1, 2024 naive (no tz) ;; Date with time (date "2024-06-15T14:30:00") ;; June 15, 2024 at 2:30 PM UTC (date "2024-06-15T09:00:00" #:tz "Europe/Paris") ;; 9 AM Paris time ;; Note: #:tz nil yields a naive timestamp Date Arithmetic --------------- ;; Add/subtract time periods (shifted (date "2024-01-01") #:days 30) ;; 30 days later (shifted (date "2024-01-01") #:weeks -2) ;; 2 weeks earlier (shifted (date "2024-01-01") #:months 6 #:days 15) ;; 6 months 15 days later ;; Complex date arithmetic (shifted (today) #:years 1 #:months -3 #:days 5) ;; 1 year, minus 3 months, plus 5 days (shifted (now) #:hours -24 #:minutes -30) ;; 24.5 hours ago ;; Common patterns (shifted (today) #:days -7) ;; One week ago (shifted (today) #:days -365) ;; One year ago (approximately) (shifted (today) #:months 1) ;; One month from now Month Boundary Functions ------------------------ ;; First day of month (start-of-month (date "2024-05-15")) ;; Returns 2024-05-01 00:00:00 (start-of-month (today)) ;; First day of current month ;; Last day of month (end-of-month (date "2024-02-15")) ;; Returns 2024-02-29 00:00:00 (leap year) (end-of-month (today)) ;; Last day of current month ;; Slice a series on the current month (slice (series "sales") #:fromdate (start-of-month (today)) #:todate (end-of-month (today))) Timezone Operations ------------------- ;; Convert naive series to timezone-aware (tzaware (series "local_temperatures") "Europe/Berlin") ;; Convert timezone-aware to naive (naive (series "utc_timestamps") "America/Chicago") ;; Convert timestamp to different timezone (tzaware-stamp (date "2024-01-01T12:00:00") "Asia/Singapore") ;; Practical examples (tzaware (series "server_logs") "UTC") ;; Standardize to UTC (naive (series "market_data") "America/New_York") ;; Convert to NYSE local time Holiday Calendar Series ----------------------- Generate holiday indicator series (1.0 for holidays, 0.0 for regular days). Takes a 2-letter country code (ISO 3166-1 alpha-2): ;; French holidays (tzaware by default) (holidays "fr" (date "2025-01-01") (date "2025-12-31")) ;; US holidays as naive timestamps (naive (holidays "us" (date "2025-01-01") (date "2025-12-31")) "America/New_York") ;; German holidays for current year (holidays "de" (start-of-month (today)) (end-of-month (shifted (today) #:months 12))) Special Series Generation ========================= **constant** - Generate a constant-valued time series (constant 42.5 (date "2020-01-01") (date "2025-12-31") "D" (today)) ;; Creates daily series with value 42.5 from 2020 to 2025 ;; With timezone (constant 100 (date "2020-01-01" #:tz "Europe/Paris") (date "2025-12-31" #:tz "Europe/Paris") "h" (today #:tz "Europe/Paris")) Advanced Series Discovery ========================= Dynamic Series Selection ------------------------ Instead of hardcoding series names, you can dynamically find and load series: ;; Find and load all temperature series (findseries (by.metaitem "sensor_type" "temperature")) ;; Find series with fill options (findseries (by.name "daily") #:fill "ffill") ;; Get just the names (for inspection) (findnames (by.metaitem "region" "europe")) ;; Use in calculations (row-mean (findseries (by.metaitem "station_type" "weather"))) ;; Average all weather stations ;; Filter by timezone awareness (naive defaults to #f) (findseries (by.metaitem "type" "price")) ;; Only tzaware series (default: naive=#f) (findseries (by.metaitem "type" "price") #:naive #f) ;; Only tzaware series (explicit) (findseries (by.metaitem "type" "daily") #:naive #t) ;; Only naive series Basket-based Operations ----------------------- ;; Use predefined baskets (findseries (by.basket "energy_meters")) (add (findseries (by.basket "revenue_streams"))) ;; Combine with other operations (rolling (row-mean (findseries (by.basket "temperature_sensors"))) 7) =============================================================================== # https://refinery.docs.pythonian.fr/core_components/cache.html =============================================================================== Formula cache ************* A formula is normally evaluated every time it is read. For a heavy formula — a deep dependency tree, many fine-granularity (sub-quarter- hour) series pulled over the whole history, or costly operators — this can make reads slow (a sluggish *tsinfo*). The **formula cache** *materialises* the result of such a series: its values are pre- computed and stored, so a read serves them directly instead of re- evaluating the formula. When to use it ============== Cache a formula when it is slow to display, typically because of: * a deep dependency tree; * many fine-granularity series read over a long history; * costly operators ("resample", "priority", "block-staircase", or multi-site aggregates such as "findseries" + "add"). How it works ============ Three things are worth understanding. * **Refresh** — the cache is filled and kept up to date by a rework task, "refresh_formula_cache", run on each policy's schedule ("schedule_rule"); it walks the policy's series in dependency order and materialises new revision dates incrementally. Its on-demand counterpart, "refresh_formula_cache_now", is behind the *Refresh* button. Creating a policy and mapping series does not populate the cache — a refresh has to run. * **Freshness** — when a cached series is read, if its cache has not been refreshed for more than about twice its usual cadence, it is considered stale and recomputed live over the policy's window. In normal operation the cache is served directly, giving a fast read. * **Intermediate nodes speed up their parents** — when the engine evaluates a formula, it stops expanding the dependency tree as soon as it reaches a cached sub-series and reads the cached values directly. Caching an intermediate node therefore cuts out the computation of everything below it, for *every* formula that depends on it. Configuring a policy ==================== Caching is driven by a **policy**, which has five parameters: +----------------------+---------------------------------------------------------+ | parameter | role | |======================|=========================================================| | "initial_revdate" | earliest revision date to cache from | +----------------------+---------------------------------------------------------+ | "look_before" | lower bound of the materialised value window | +----------------------+---------------------------------------------------------+ | "look_after" | upper bound of the materialised value window | +----------------------+---------------------------------------------------------+ | "revdate_rule" | cron rule: which revision dates to snapshot | +----------------------+---------------------------------------------------------+ | "schedule_rule" | cron rule: when the refresh job runs | +----------------------+---------------------------------------------------------+ "initial_revdate", "look_before" and "look_after" are date expressions (a small lisp), evaluated at refresh time — for example "(date "2025-01-01")", "(today)", "(shifted now #:days -60)" or "(shifted now #:days 15)". The "now" variable is bound to the revision date being refreshed and is only available in "look_before" / "look_after". Do not use it in "initial_revdate": it passes the policy validation but makes the refresh task fail — use a fixed date or "(today)"-based expressions there. The two cron rules serve different purposes: "revdate_rule" selects which revision dates to snapshot (e.g. "0 * * * *" — every hour on the hour), while "schedule_rule" decides when the refresh task actually runs (e.g. "5 * * * *" — five past the hour). The small offset lets data pipelines finish before the snapshot is taken. [image] Warning: **The value window is the main pitfall.** "look_before" / "look_after" define the range of *value dates* the cache materialises (and that the live recompute uses). It must cover the whole history you display. A window that is too narrow (for instance "look_before = (today)", "look_after = (today)") caches almost nothing, so the cache brings no benefit. Use a wide range, for example "look_before = (date "2025-01-01")" (or "(shifted now #:days -180)") and "look_after = (shifted now #:days 15)". **Examples.** # Hourly cache, a week of revisions, six months of value history initial_revdate (shifted (today) #:days -7) look_before (shifted now #:days -180) look_after (shifted now #:days 15) revdate_rule 0 * * * * # snapshot every hour schedule_rule 5 * * * * # refresh at 5 past the hour # Once-a-day cache, fixed history start initial_revdate (date "2025-01-01") look_before (date "2025-01-01") look_after (shifted now #:days 15) revdate_rule 0 6 * * * # one snapshot a day, 06:00 schedule_rule 15 6 * * * # refresh at 06:15 Cache operations ================ From the cache management interface: * **Refresh** — triggers an immediate refresh (the "refresh_formula_cache_now" task) over the window. It does not purge anything and does not rebuild an emptied cache. * **Activate / Deactivate** — schedules or removes the periodic "refresh_formula_cache" task. A policy can only be edited while it is deactivated. * **Delete** — purges the cache of **all** the policy's series and removes the policy and its series mappings (everything has to be recreated). * **Free series** — moving a series to the free list purges that series' cache and unmaps it from the policy. A series' cache is emptied in three cases: **deleting its policy** (every series), **moving the series to the free list** (that series only), and **editing its formula** — a change to the formula invalidates its own cache and those of its dependents. Programmatic access =================== A policy can be created and populated from the API: tsa.new_cache_policy( 'daily', initial_revdate='(shifted (today) #:days -7)', look_before='(shifted now #:days -180)', look_after='(shifted now #:days 15)', revdate_rule='0 * * * *', schedule_rule='5 * * * *', ) tsa.set_cache_policy('daily', ['my.heavy.formula']) Related methods: "cache_policies()" and "cache_free_series()" list the policies and the formulas still available for caching; "cache_series_policy" and "has_cache" inspect a series; "delete_cache" clears a series' cache; and "refresh_series_policy_now" triggers an immediate refresh of a policy. See Formulas (computed series) for the full cache API reference. =============================================================================== # https://refinery.docs.pythonian.fr/core_components/groups.html =============================================================================== Groups ****** Table of Contents ================= * Introduction and Stored Groups * Group Formulas * Common Use Cases * Group API Reference Introduction and Stored Groups ============================== Groups in tshistory are collections of related time series that share the same time index and are managed as a unit. They are particularly useful for handling scenarized time series or multivariate time series data where multiple series need to be kept in sync. [image] They come in two flavors: primary groups (stored) and formula groups. Here's how creating a primary group looks like: import pandas as pd from tshistory.api import timeseries tsa = timeseries() group_data = pd.DataFrame({ 'low': [17.0, 21.6, 18.2], 'mid': [20.5, 22.1, 23.4], 'high': [21.3, 22.6, 24.9] }, index=pd.date_range('2025-01-01', periods=3, freq='M')) tsa.group_update('subsidiary1.revenues.fcst', group_data, 'operator') Note: Groups in tshistory have a **fixed schema** once created: you can't change the **columns** nor, like with time series, fundamental attributes such as tz-awareness.All "group_update" and "group_replace" operations must match the exact column structure of the original group.This constraint ensures: * **Data Integrity**: Prevents accidental schema changes that could break downstream consumers * **Version Consistency**: All historical versions of a group maintain the same structure Group Formulas ============== Group formulas enable powerful computed groups using the formula language. Like series formulas, they are evaluated on-demand and inherit versioning from their components. Group Formula Operators ----------------------- The formula language provides specialized operators for working with groups: **group** Retrieves a group from storage or formula, similar to the "series" operator: (group "subsidiary1.revenues.fcst") **group-add** Performs element-wise addition of multiple groups. All groups must have compatible indexes: ;; obtain low, mid and high revenue scenarios for a company with 3 subsidiaries (group-add (group "subsidiary1.revenues.fcst") (group "subsidiary2.revenues.fcst") (group "subsidiary3.revenues.fcst")) This operator aligns the groups by their time index and adds corresponding columns. **group-add-series** Adds a series to every column of a group. Useful for adjustments or calibrations: ;; convert temperatures from kelvin to celsius (group-add-series (group "temperatures_kelvin") (series "kelvin_to_celsius_offset")) The series is broadcast to all scenarios in the group. **label and group-from-series** Constructs a new group by binding multiple series together as named scenarios: ;; create scenarios from individual series (group-from-series (label "high" (series "forecast_high")) (label "mid" (series "forecast_mid")) (label "low" (series "forecast_low"))) Each "label" creates a named column in the resulting group. This is particularly useful for creating scenario-based groups from individual forecast series. **seasonal** Turns a single series into a group of day-of-year profiles — one column per year — for folded, year-over-year plots: ;; five years of daily demand, one column per year (seasonal (series "power.demand.d") 5) The "depth" argument sets how many past years to include (omit it to use every available year). It is the group counterpart of the "doy- agg" series operator, which instead collapses those years into a single profile. **subst and substitute-in-series-formula** Turns an existing *series* formula into a group, by substituting groups for some of the series it references. The last "subst" argument names a *family*: substitutions of the same family advance together on their scenarios (the groups must share their scenario names), while distinct families combine as a cartesian product. With a single family, each scenario of the result is the formula evaluated on the matching scenarios of the groups: (substitute-in-series-formula "energy-balance" (subst "production" "production-scenarios" "energy") (subst "consumption" "consumption-scenarios" "energy")) If "energy-balance" is defined as "(add (series "production") (series "consumption"))", the result has one scenario per scenario of the input groups, each being the sum of the corresponding columns. With two families, all the combinations are produced: (substitute-in-series-formula "energy-cost" (subst "demand" "demand-scenarios" "weather") (subst "price" "price-scenarios" "market")) If "energy-cost" is "(mul (series "demand") (series "price"))", "demand-scenarios" carries N weather scenarios and "price-scenarios" M market scenarios, the result has N×M scenarios — one per (weather, market) pair. Creating and Using Group Formulas --------------------------------- Register a group formula using the API: >>> tsa.register_group_formula( ... 'eu_production_mwh', ... '(group-add (group "france_production_mwh") ' ... ' (group "germany_production_mwh") ' ... ' (group "spain_production_mwh"))' ... ) Once registered, use it like any other group: >>> df = tsa.group_get('eu_production_mwh') >>> print(df.head()) low mid high 2025-01-01 00:00:00 48.5 52.2 55.8 2025-01-02 00:00:00 49.1 52.7 56.4 Group Formula Metadata ---------------------- Group formulas can have metadata like primary groups: >>> tsa.update_group_metadata('eu_production_mwh', { ... 'unit': 'mwh', ... 'frequency': 'daily', ... 'scenarios': 'energy production forecasts' ... }) Formula-Specific Methods ------------------------ Several methods are specific to group formulas: >>> # get the formula definition >>> formula = tsa.group_formula('eu_production_mwh') >>> print(formula) '(group-add (group "france_production_mwh") ...)' >>> # get expanded formula (resolving nested formulas) >>> expanded = tsa.group_formula('eu_production_mwh', expanded=True) >>> # test a formula without registering >>> result = tsa.group_eval_formula( ... '(group-add (group "test1") (group "test2"))' ... ) Formula Bindings: Creating Groups from Series Formulas ------------------------------------------------------ The bindings system is a powerful mechanism that transforms series formulas into group formulas by replacing selected series references with groups. This allows you to apply the same calculation logic across multiple scenarios simultaneously. **Core Concept** Given a series formula that combines multiple series, you can "bind" some of those series to groups. The formula then evaluates column-wise across the bound groups, producing a group as output. **The Family Concept** A "family" groups together series/groups that play equivalent roles in the formula. Key rules: * All groups within a family must have the same number of columns (scenarios) * The formula is evaluated column-by-column across families * Column 1 of each group in a family is used together, then column 2, etc. **Example: Weather Scenario Modeling** Consider a formula that combines temperature and wind data with adjustments: # original series formula tsa.register_formula( 'weather_index', '(add (mul (series "temp_base") 0.7) ' ' (mul (series "wind_base") 0.3) ' ' (series "seasonal_adjustment"))' ) Now create groups for different weather scenarios: # temperature scenarios (3 scenarios: cold, normal, warm) temp_scenarios = pd.DataFrame({ 'cold': [5, 6, 7], 'normal': [15, 16, 17], 'warm': [25, 26, 27] }, index=dates) tsa.group_replace('temp_scenarios', temp_scenarios, 'operator') # wind scenarios (must also have 3 scenarios to match) wind_scenarios = pd.DataFrame({ 'calm': [5, 5, 5], 'moderate': [15, 15, 15], 'strong': [30, 30, 30] }, index=dates) tsa.group_replace('wind_scenarios', wind_scenarios, 'operator') Bind the formula to create a group: # define the binding binding = pd.DataFrame([ ['temp_base', 'temp_scenarios', 'weather'], ['wind_base', 'wind_scenarios', 'weather'], # seasonal_adjustment remains a regular series ], columns=['series', 'group', 'family']) # register the bound group tsa.register_formula_bindings( 'weather_index_scenarios', # new group name 'weather_index', # source formula binding ) Result: >>> result = tsa.group_get('weather_index_scenarios') >>> print(result.columns) ['scenario_1', 'scenario_2', 'scenario_3'] # each column computed as: # scenario_1: temp_scenarios['cold'] * 0.7 + wind_scenarios['calm'] * 0.3 + seasonal_adjustment # scenario_2: temp_scenarios['normal'] * 0.7 + wind_scenarios['moderate'] * 0.3 + seasonal_adjustment # scenario_3: temp_scenarios['warm'] * 0.7 + wind_scenarios['strong'] * 0.3 + seasonal_adjustment **Multiple Families Example** Families are useful when you want different binding strategies for different parts of the formula: # formula with different types of inputs tsa.register_formula( 'complex_calc', '(add (series "regional_data") ' ' (mul (series "global_factor") (series "local_factor")))' ) # binding with two families binding = pd.DataFrame([ ['regional_data', 'regions_group', 'regions'], # 5 regions ['local_factor', 'local_scenarios', 'scenarios'], # 3 scenarios # global_factor remains unbound (same for all combinations) ], columns=['series', 'group', 'family']) This would create a group with 15 columns (5 regions × 3 scenarios), exploring all combinations. **Key Points** * Unbound series in the formula remain as series (broadcast to all columns) * All groups in the same family must have identical column counts * The binding creates a "bound" type group that dynamically evaluates the formula * Use "bindings_for(name)" to retrieve the binding configuration for a group Common Use Cases ================ Groups are ideal for: * **Ensemble Forecasts**: Scenarized stochastic weather scenarios They may (depending on various factors) be interesting with: * **Financial Data**: OHLC (Open, High, Low, Close) price data * **IoT Sensors**: Multiple sensor readings from the same device * **Economic Indicators**: Related economic metrics that should be kept in sync In many cases, it will be more convenient to handle data acquisition as individual time series, and then create a group from them (using the *group-from-series* formulaic operator). Group API Reference =================== Primary Group Operations ------------------------ class mainsource(*args, **kwargs) API façade for the main source (talks directly to the storage) The api documentation is carried by this object. The http client provides exactly the same methods. group_exists(name) Checks the existence of a group with a given name. Parameters: **name** (*str*) Return type: bool group_type(name) Return the type of a group, for instance 'primary', 'formula' or 'bound' Parameters: **name** (*str*) Return type: str group_find(query=None, limit=None, meta=False, sources=[], _source='local') Return a list of group descriptors matching the query. A series descriptor is a string-like object (exhibiting the series name) with additional attributes. If *meta* has been set to True, the .meta (for normal metadata) and .imeta (for internal metadata) fields will be populated (non None). Lastly, the .source and .kind attributes provides the series source and kind. Here is an example: tsa.group_find( '(by.and ' ' (by.tzaware)' ' (by.name "power capacity") ' ' (by.metakey "plant")' ' (by.not (by.or ' ' (by.metaitem "plant_type" "oil")' ' (by.metaitem "plant_type" "coal")))' ' (by.metaitem "unit" "mwh")' ' (by.metaitem "country" "fr"))' ) The following filters can be used from the search module: * by.tzaware: no parameter, yields time zone aware series names * by.name : takes a space separated string of word, yields series names containing the substrings (in order) * by.metakey : takes a string, strictly matches all series having this metadata key * by.metaitems : takes a string (key) and an str (or numerical) value and yields all series strictly matching this metadata item * by.and: takes a variable number of filters as above to combine them * by.or: takes a variable number of filters as above to combine them * by.not: produce the negation of a filter Also inequalities on metadata values can be used: * <, <=, >, >=, =: take a string key, a value (str or num) As in *(<= "max_capacity" 900)* Parameters: * **query** (*str** | **None*) * **limit** (*int** | **None*) * **meta** (*int** | **None*) * **sources** (*List**[**str**]*) * **_source** (*str** | **None*) Return type: *List*[*ts*] group_rename(name, newname, propagate=True, **_kw) Rename a group. The target name must be available. Parameters: * **name** (*str*) * **newname** (*str*) * **propagate** (*bool*) Return type: None group_source(name) Provide the source name of a group. When coming from the main source, it returns 'local'. Parameters: **name** (*str*) Return type: str | None group_get(name, revision_date=None, from_value_date=None, to_value_date=None) Get a group by name. By default one gets the latest version. By specifying *revision_date* one can get the closest version matching the given date. The *from_value_date* and *to_value_date* parameters permit to specify a narrower date range (by default all points are provided). If the group does not exists, a None is returned. Parameters: * **name** (*str*) * **revision_date** (*Timestamp** | **None*) * **from_value_date** (*Timestamp** | **None*) Return type: *DataFrame* | None group_interval(name) Return the value date range of a group. Parameters: **name** (*str*) Return type: *Interval* group_inferred_freq(name, revision_date=None, from_value_date=None, to_value_date=None) Return the inferred frequency of a group. Parameters: * **name** (*str*) * **revision_date** (*Timestamp** | **None*) * **from_value_date** (*Timestamp** | **None*) * **to_value_date** (*Timestamp** | **None*) Return type: *Tuple*[*Timedelta*, float] | None group_insertion_dates(name, from_insertion_date=None, to_insertion_date=None) Get the list of all insertion dates for any given group Parameters: * **name** (*str*) * **from_insertion_date** (*Timestamp** | **None*) * **to_insertion_date** (*Timestamp** | **None*) Return type: *List*[*Timestamp*] group_history(name, from_value_date=None, to_value_date=None, from_insertion_date=None, to_insertion_date=None) Get all versions of a group in the form of a dict from insertion dates to dataframe. It is possible to restrict the versions range by specifying *from_insertion_date* and *to_insertion_date*. It is possible to restrict the values range by specifying *from_value_date* and *to_value_date*. Parameters: * **name** (*str*) * **from_value_date** (*Timestamp** | **None*) * **to_value_date** (*Timestamp** | **None*) * **from_insertion_date** (*Timestamp** | **None*) * **to_insertion_date** (*Timestamp** | **None*) Return type: *Dict*[*Timestamp*, *DataFrame*] group_replace(name, df, author, metadata=None, insertion_date=None) Replace a group named by with the input dataframe. This creates a new version of the group. The group is completely replaced with the provided values. The *author* is mandatory. The *metadata* dictionary allows to associate any metadata with the new group revision. It is possible to force an *insertion_date*, which can only be higher than the previous *insertion_date*. Parameters: * **name** (*str*) * **df** (*DataFrame*) * **author** (*str*) * **metadata** (*dict** | **None*) * **insertion_date** (*Timestamp** | **None*) Return type: None group_delete(name, force=False) Override to add dependency checking. Delete a group. By default, refuses to delete if the group is referenced by group formulas. Use force=True to override this safety check. Parameters: * **name** (*str*) * **force** (*bool*) Return type: None group_internal_metadata(name) Return a group internal metadata dictionary. Parameters: **name** (*str*) Return type: *Dict*[str, *Any*] | None group_metadata(name) Return a group metadata dictionary. Parameters: **name** (*str*) Return type: *Dict*[str, *Any*] | None group_old_metadata(name) Get a list of the the older versions of the metadata. Parameters: **name** (*str*) Return type: *List*[*Tuple*[*Timestamp*, dict, str]] update_group_metadata(name, meta, user='no-user') Update a group metadata with a dictionary from strings to anything json-serializable. Parameters: * **name** (*str*) * **meta** (*Dict**[**str**, **Any**]*) * **user** (*str*) Return type: None replace_group_metadata(name, meta, user='no-user') Replace a group metadata with a dictionary from strings to anything json-serializable. Parameters: * **name** (*str*) * **meta** (*Dict**[**str**, **Any**]*) * **user** (*str*) Return type: None group_catalog(allsources=True) Produces a catalog of all groups in the form of a mapping from source to a list of (name, kind) pair. Parameters: **allsources** (*bool*) Return type: *Dict*[*Tuple*[str, str], *List*[*Tuple*[str, str]]] Formula Group Operations ------------------------ class mainsource(*args, **kwargs) API façade for the main source (talks directly to the storage) The api documentation is carried by this object. The http client provides exactly the same methods. register_group_formula(name, formula) Define a group as a named formula. You can use any operator (including those working on series) provided the top-level expression is a group. Parameters: * **name** (*str*) * **formula** (*str*) Return type: None group_formula(name, display=True, expanded=False, remote=True, level=-1) Get the group formula associated with a name. Parameters: * **name** (*str*) * **display** (*bool*) * **expanded** (*bool*) * **remote** (*bool*) * **level** (*int*) Return type: str | None group_eval_formula(formula, revision_date=None, from_value_date=None, to_value_date=None, tz=None) Execute a group formula on the spot. tsa.group_eval_formula('(group-add (group "group1") (group "group2"))') Parameters: * **formula** (*str*) * **revision_date** (*Timestamp** | **None*) * **from_value_date** (*Timestamp** | **None*) * **to_value_date** (*Timestamp** | **None*) Return type: *Series* register_formula_bindings(groupname, formulaname, bindings) Define a group by association of an existing series formula and a *bindings* object. The designated series formula will be then interpreted as a group formula. And the bindings object provides mappings that tell which components of the formula are to be interpreted as groups. Given a formula named "form1": (add (series "foo") (series "bar") (series "quux")) ... where one wants to treat "foo" and "bar" as groups. The binding is expressed as a dataframe: binding = pd.DataFrame( [ ['foo', 'foo-group', 'group'], ['bar', 'bar-group', 'group'], ], columns=('series', 'group', 'family') ) The complete registration looks like: register_formula_bindings( 'groupname', 'form1', pd.DataFrame( [ ['foo', 'foo-group', 'group'], ['bar', 'bar-group', 'group'], ], columns=('series', 'group', 'family') )) Within a given family, all groups must have the same number of members (series) and the member roles are considered equivalent (e.g. meteorological scenarios). Parameters: * **groupname** (*str*) * **formulaname** (*str*) * **bindings** (*DataFrame*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/core_components/folders.html =============================================================================== Folders ******* Series can be organised into a hierarchy of **folders** — a tree that makes a large catalog easier to browse. A folder is identified by a dot-separated **path** (for example "production.france"), and a series belongs to at most one folder. Series that have not been placed in any folder are shown as **unclassified**. Folders do not need to be created ahead of time, and an empty folder is kept until you delete it. The folders page ================ Folders live under the "/folders" page. It shows the folder tree on the left; selecting a folder lists the series it contains. A special **Unclassified** entry gathers every series that has no folder yet. [image] Managing folders and series =========================== **Create a folder** Add a folder under the selected node. A folder can be created empty and stays until you delete it — you do not have to place a series in it first. **Move series (drag and drop)** Drag one or several selected series onto a folder to place them there. Moves are applied in batches, so large selections are handled in one go. **Cut, copy and paste** Select series, use *Cut* or *Copy*, then *Paste* them into the target folder. This is an alternative to drag and drop for reorganising series. **Rename a folder** Renaming a folder rewrites the whole subtree beneath it, so its sub-folders and their series move with it. **Delete a folder** Deleting a folder (after a confirmation) removes it and all of its sub-folders. The series they held are not deleted — they simply become *unclassified*. **Select and filter** Series can be multi-selected before a move or a cut/paste, and each folder's contents can be filtered in place to find a series quickly. **Unclassified series** The *Unclassified* entry lists series with no folder. Dropping a series there — or moving it out of every folder — removes its folder association. Setting a folder from tsinfo ============================ The folder of a series can also be set from its information page (*tsinfo*), on the **Folder** tab. It shows the current folder and offers a text field to type a new path ("folder1.folder2.folder3") and save it. The target folder does not need to exist beforehand; clearing the field removes the series from its folder. [image] Finding series by folder ======================== Folders are searchable with the query language (see Search Query Language Reference): (by.at-path "production.france") ;; series in this exact folder (by.at-path "production" #:children #t) ;; also series in sub-folders (by.without-path) ;; unclassified series Paths are dot-separated, with no leading slash. These predicates work anywhere the search language is accepted, baskets included. Note: The same query drives the "findseries" formula operator, so you can build a formula over every series in a folder without listing them. For instance "(add (findseries (by.at-path "production.france")))" sums all the series in "production.france"; add "#:children #t" to include its sub-folders. Programmatic access =================== Folders can also be managed from the API: from tshistory.api import timeseries tsa = timeseries() tsa.set_series_path('prod.plant.mw.actual.15min', 'production.france') tsa.series_path('prod.plant.mw.actual.15min') # 'production.france' tsa.path_series('production.france') # the series in that folder tsa.tree() # every folder path tsa.rename_path('production', 'prod') # move the whole subtree tsa.delete_path('prod.france') # remove the folder tsa.set_series_path('prod.plant.mw.actual.15min', None) # unclassify "set_series_path" places a series in a folder (creating the folder if needed) or, with "None", removes it from its folder. "rename_path" moves an entire subtree, carrying its series along. "delete_path" removes a folder and its sub-folders, and the series they held become unclassified — the series themselves are never deleted. =============================================================================== # https://refinery.docs.pythonian.fr/core_components/search_language.html =============================================================================== Search Query Language Reference ******************************* The Refinery search system uses a powerful query language to find time series and groups based on metadata, names, and other properties. This guide provides comprehensive documentation with extensive examples. Table of Contents ================= * Quick Start * Query Syntax * Basic Search Operators * Logical Operators * Formula-Specific Operators * Tree and Cache Operators Quick Start =========== Your First Search ----------------- # find all series containing "temperature" in their name results = tsa.find('(by.name "temperature")') assert results == ['room_temperature', 'outside_temperature', 'water_temperature'] # find series by metadata results = tsa.find('(by.metaitem "sensor_type" "humidity")', meta=True) assert results[0].meta == {'sensor_type': 'humidity', 'unit': 'percent', 'location': 'building_a'} # combine conditions results = tsa.find('(by.and (by.name "daily") (by.metaitem "region" "europe"))', limit=50) assert len(results) <= 50 Note: As the examples above show, results look like strings - which they are ! But they carry .meta and imeta attributes which, when queried with meta=True, get populated. Note: "find()" does not filter on timezone-awareness: naive and tzaware series alike are returned. To restrict a search, use "by.tzaware" or its negation: # find only timezone-naive series naive_series = tsa.find('(by.not (by.tzaware))') # combine with other conditions to find naive series naive_temp = tsa.find('(by.and (by.name "temperature") (by.not (by.tzaware)))') The *formula* operator "findseries" behaves differently: since series of both kinds cannot be mixed within a formula, it keeps only the tzaware ones by default (see its "#:naive" option). More find API options --------------------- The find method offers additional capabilities for more advanced use cases. You can retrieve all series using either "find()" with no parameters or the explicit "(by.everything)" operator. In federated setups with multiple data sources, the "sources" parameter lets you filter results by origin. The result objects also carry useful attributes: ".source" indicates where the data comes from (local or remote) and ".kind" shows whether it's a primary (stored) or formula (computed) series. # get all series - two equivalent ways allseries = tsa.find() # no parameters allseries = tsa.find('(by.everything)') # explicit # filter by source in federated setups localonly = tsa.find('(by.everything)', sources=['local']) assert all(s.source == 'local' for s in localonly) remote_series = tsa.find('(by.name "sales")', sources=['remote']) assert all(s.source == 'remote' for s in remote_series) # examine result attributes results = tsa.find('(by.name "temperature")') assert results[0].source in ('local', 'remote') assert results[0].kind in ('primary', 'formula') Finding Groups -------------- The same search capabilities are available for groups through the "group_find" method. It supports all the same operators and parameters as "find", but searches within the groups namespace instead of series. # find groups by name groups = tsa.group_find('(by.name "weather")') assert groups == ['weather_station_paris', 'weather_forecast_lyon'] Note: To create actual groups (DataFrames) from **series** search results, you have two approaches: # approach 1: using group-from-series formula operator tsa.register_group_formula( 'sensor_group', '''(group-from-series (label "sensor1" (series "temperature_sensor_1")) (label "sensor2" (series "temperature_sensor_2")) (label "sensor3" (series "temperature_sensor_3")))''' ) # approach 2: python code to build dataframe from search results results = tsa.find('(by.metaitem "type" "sensor")') df = pd.DataFrame({ name: tsa.get(name) for name in results }) Saving Searches with Baskets ---------------------------- Baskets allow you to save frequently used search queries for reuse. This is particularly useful for complex queries or when the same set of series needs to be referenced multiple times. Baskets work for both series and groups independently - they maintain separate namespaces controlled by the "group" parameter. # register a basket for series tsa.register_basket('energy_sensors', '(by.and (by.metaitem "type" "sensor") (by.metaitem "category" "energy"))') # register a basket for groups tsa.register_basket('weather_groups', '(by.name "weather")', group=True) # list all series baskets series_baskets = tsa.list_baskets() assert 'energy_sensors' in series_baskets # execute a basket query (supports same parameters as find) results = tsa.basket('energy_sensors', limit=10, meta=True) assert results[0].meta['type'] == 'sensor' # use the basket in formulas formula = '(add (findseries (by.basket "energy_sensors")))' The basket editor ----------------- Baskets can also be managed from the browser, on the **Baskets** page. A panel on the left lists the existing baskets, with a filter on the basket name. Selecting a basket shows it in one of two modes: * **View** — the series the basket currently yields (*This basket yields N series*), each row linking to its info page, with an *edit formula* link for formula series; * **Edit** — the basket's query, edited in the same dual editor as the formula editor (a text pane for the lisp expression and a synchronised structured one, see Formulas (computed series)). From there you can **Save** the changes, **Rename** the basket, **Save a copy under a new name**, or **Delete** it. **Create new basket** starts a new, empty one. [image] Query Syntax ============ Structure --------- ;; Basic structure ( ) ;; With multiple arguments ( ... ) ;; Optional parameters use the #:keyword notation ( #: ) ;; Nested queries (by.and (by.name "sales") (by.metaitem "region" "north")) Basic Syntax Rules ------------------ 1. **Parentheses**: Every query is wrapped in parentheses "(operator args...)" 2. **Quotes**: All string values use double quotes ""value"" 3. **Nesting**: Queries can be nested for complex logic 4. **Case Sensitivity**: - Name searches ("by.name") are **case- insensitive** - Metadata keys and values ("by.metakey", "by.metaitem") are **case-sensitive** - Formula content searches ("by.formulacontents") are **case-sensitive** Basic Search Operators ====================== Name-based Search: "by.name" ---------------------------- Searches for series whose names contain the given words, in order (case-insensitive). A space-separated string is a sequence of substrings that must all appear, in that order, anything in between: ;; Basic name search (by.name "temperature") ;; Matches: "room_temperature", "temperature_sensor" ;; Multiple words: substrings in order (by.name "power capacity") ;; Matches: "power.fr.capacity.mw" Metadata Key Search: "by.metakey" --------------------------------- Finds series that have a specific metadata key (regardless of the value): ;; Basic metadata key search (by.metakey "region") ;; All series that have a "region" metadata field Metadata Value Search: "by.metavalue" ------------------------------------- Finds series that have a specific metadata value (regardless of the key): ;; Basic metadata value search (by.metavalue "europe") ;; All series that have a metadata field equal to "europe" Everything Operator: "by.everything" ------------------------------------ Returns all series without any filtering: (by.everything) ;; Returns all series Timezone-Aware Filter: "by.tzaware" ----------------------------------- Filters for timezone-aware series: (by.tzaware) ;; Only timezone-aware series Metadata Value Search: "by.metaitem" ------------------------------------ Finds series where a specific metadata key has a specific value. The point of examples here is also to illustrate use of the series matadata: ;; Basic metadata value search (by.metaitem "region" "europe") ;; Series in Europe region (by.metaitem "data_type" "financial") ;; Financial data ;; Quality and status (by.metaitem "quality" "high") ;; High quality data (by.metaitem "confidence" "reliable") ;; Reliable data Internal Metadata Search: "by.internal-metaitem" ------------------------------------------------ Searches internal metadata (system-managed metadata): (by.internal-metaitem "supervision_status" "supervised") ;; Find supervised series ;; Separate string series from numeric ones: the internal ;; value_type key holds the storage dtype (by.internal-metaitem "value_type" "object") ;; String series only (by.internal-metaitem "value_type" "float64") ;; Numeric series only Internal Metadata Key Search: "by.internal-metakey" --------------------------------------------------- Finds series that have a specific internal metadata key, regardless of the value — the internal counterpart of "by.metakey": ;; Series stored on the fs1 backend (only those carry a path key) (by.internal-metakey "path") ;; Its negation: series stored in postgres (by.not (by.internal-metakey "path")) Basket Search: "by.basket" -------------------------- Expands to the query saved in a basket (see Saving Searches with Baskets), composable with any other predicate: (by.basket "energy_sensors") ;; Restrict a basket further (by.and (by.basket "energy_sensors") (by.name "daily")) Supervision status shortcuts ---------------------------- Two convenience operators match on the supervision status directly (equivalent to the "by.internal-metaitem" form above): (by.supervised) ;; series with both provider updates and manual corrections (by.handcrafted) ;; series maintained entirely by hand Value Comparison Search ----------------------- Finds series where a metadata value meets a numerical condition. Warning: It must be noted that while these filters can be used by ".find" and in baskets, they are also available in the formula language, as input parameters of the "findseries" operator. In the later case, the notation can be different. This is the case for values filtering.In any case, the Web interface provides structured editors that can help you discover the available operators in the relevant context (find / basket / findseries) and will guide you write correct filters. ;; Direct comparison operators (for use with find) (< "temperature" 25) ;; Temperature metadata < 25 (<= "confidence" 0.8) ;; Confidence <= 80% (> "threshold" 100) ;; Threshold > 100 (>= "count" 10) ;; Count >= 10 (= "status" "active") ;; Status equals "active" Logical Operators ================= AND Operator: "by.and" ---------------------- Combines multiple conditions where ALL must be true: (by.and (by.name "temperature") (by.metaitem "region" "europe")) OR Operator: "by.or" -------------------- Combines multiple conditions where ANY can be true: (by.or (by.metaitem "region" "europe") (by.metaitem "region" "asia")) ;; Multiple regions (by.or (by.metaitem "country" "france") (by.metaitem "country" "germany") (by.metaitem "country" "italy")) NOT Operator: "by.not" ---------------------- Excludes series matching the specified condition: (by.not (by.metaitem "status" "deprecated")) ;; Exclude deprecated series (by.not (by.name "test")) ;; Exclude test series Formula-Specific Operators ========================== Formula Detection: "by.formula" ------------------------------- Finds all computed (formula) series: ;; Find all formula series (by.formula) ;; Combine with other conditions (by.and (by.formula) (by.metaitem "department" "finance")) ;; Finance formulas (by.and (by.formula) (by.metaitem "type" "kpi")) ;; KPI formulas ;; Find non-formula (stored) series (by.not (by.formula)) Formula Content Search: "by.formulacontents" -------------------------------------------- Note: This is in practice quite useful to find a specific formula when we don't know its name (or other properties) but we have an idea of what it does. Searches within the actual formula expressions: ;; Find formulas using specific functions (by.formulacontents "resample") ;; Formulas with resampling (by.formulacontents "priority") ;; Formulas using priority operator ;; Find formulas referencing specific series (by.formulacontents "stock_price") ;; Formulas using directly stock_price series ;; Advanced operations (by.formulacontents "findseries") ;; Dynamic series discovery formulas (by.formulacontents "constant") ;; Formulas creating constants Tree and Cache Operators ======================== Tree Path Filters ----------------- Filter series based on their organization in tree paths: (by.without-path) ;; Series not in any tree path (by.at-path "energy.solar") ;; Series at this exact path (by.at-path "energy" #:children #t) ;; Series at this path and its children Cache Policy Filters -------------------- Filter by cache policy (refinery-specific): (by.cache) ;; Has any cache policy (by.cachepolicy "daily_cache") ;; Has specific cache policy =============================================================================== # https://refinery.docs.pythonian.fr/core_components/rework.html =============================================================================== Task Manager Operations *********************** *rework* is a distributed execution system for the execution of tasks that can belong to independent python environments and code bases, even hosted on different computers. The only constraint is that postgres must be accessible from all nodes of a given *rework* installation. Rework might interest people who: * want Postgres (and *only Postgres*) as a Task Queue Manager, Input/Output store and task log store * have Python long-running tasks to run, with the ability to preemptively kill tasks * want a tiny, self-contained tool with great functional test abilities (writing tests for tasks is easy) Rework provides a rich command line utility to diagnose the state of the system. Introduction ============ Overview -------- To use it properly one has to understand the following concepts: *operation* A python function decorated with the *task* decorator. The function has a single *task* parameter that allows to communicate with the system (for the purposes of input and output management, and log capture). It is defined within a *domain* and on a specific *host*. *task* A concrete execution of an operation. Also, name of the decorator that indicates an *operation*. The task can indicate its state and be aborted if needed. It can provide access to the captured logs, input and output. *worker* A python process spawned by a *monitor*, that will execute *tasks*. It is always associated with a *domain* on a specific *host*. *domain* A label associated with *operations*, *tasks* and *workers*, which can be used to map operations to virtual environments or just help organize a logical separation of operations (and the associated pools of workers). *monitor* A python process which is responsible for the management of workers (start, stop and abort), whose precise amount is configurable, within a *domain*. They will be illustrated further in the documentation. Installation ------------ $ pip install rework Quick start ----------- Let's have a look at a simple example. We need to set up a database first, which we'll name *jobstore*. $ createdb jobstore Rework will install its tables into its own namespace schema, so you can use either a dedicated database (like we're doing right now) or an existing one, with little risk of conflict. Now we must set up the rework schema: rework init-db postgres://babar:password@localhost/jobstore This being done, we can start writing our first task: from rework import api, io from sqlhelp.pgapi import pgdb as create_engine @api.task( inputs=(io.number('value'),), outputs=(io.number('doubled'),) ) def my_first_task(task): with task.capturelogs(std=True): print('I am running') task.save_output({'doubled': task.input['value'] * 2}) print('I am done') def main(uri): engine = create_engine( 'postgres://babar:password@localhost/jobstore' ) # now, schedule tasks t1 = api.schedule(engine, 'my_first_task', {'value': 21}) t2 = api.schedule(engine, 'my_first_task', {'value': 100}) # wait til they are completed t1.join() t2.join() assert t1.output == {'doubled': 42} assert t2.output == {'doubled': 200} if __name__ == '__main__': main('postgres://babar:password@localhost:5432/jobstore') Here we have defined a dummy task that will print a bunch of sentences, double the input value and save a result back. This has to be put into a python module, e.g. *test_rework.py* At this point, the rework system knows *nothing* of the task. We must register it, as follows: $ rework register-operations postgres://babar:password@localhost/jobstore test_rework.py registered 1 new operation (0 already known) From this point, we can check it is indeed registered: $ rework list-operations postgres://babar:password@localhost/jobstore 1 host(1) `10.211.55.3` path(my_first_task) Now, let's execute our script: $ python test_rework.py It will start and hang indefinitely on the first *join* call. Indeed we are missing an important step: providing *workers* that will execute the tasks. This should be made in a separate shell, since it is a blocking operation: $ rework monitor postgres://babar:password@localhost/jobstore Then, the script will quickly terminate, as both tasks have been executed. Congratulations ! You just fired your first rework tasks. We can finish this chapter with a few command line goodies. First we'll want to know about the existing tasks: $ rework list-tasks postgres://babar:password@localhost/jobstore 1 my_first_task done [2018-11-28 16:07:51.672672+01] → [2018-11-28 16:08:27.974392+01] → [2018-11-28 16:08:27.985432+01] 2 my_first_task done [2018-11-28 16:07:51.676981+01] → [2018-11-28 16:08:27.974642+01] → [2018-11-28 16:08:27.985502+01] It is possible to monitor the output of a given task: $ rework log-task postgres://babar:password@localhost/jobstore 1 stdout:INFO: 2018-11-28 16:08:27: I am running stdout:INFO: 2018-11-28 16:08:27: I am done The last argument *1* is the task identifier as was shown by the *list-tasks* command. Notice how we capture the standard output (print calls) using the *task.capturelogs* context manager. This is completely optional of course but quite handy. The line shown above actually capture *standard output*, *standard error* and *all logs*. It accepts a *level* parameter, like e.g. *capturelogs(level=logging.INFO)*. Lastly, *list-workers* will show the currently running workers: $ rework list-workers postgres://babar:password@localhost/jobstore 1 4124@10.211.55.3 43 Mb [running (idle)] [2018-11-28 16:08:27.438491+01] → [2018-11-28 15:08:27.967432+01] 2 4125@10.211.55.3 43 Mb [running (idle)] [2018-11-28 16:08:27.442869+01] → [2018-11-28 15:08:27.967397+01] It is now possible to stop the *monitor* on its separate console, with a plain *ctrl-c*. After this, *list-workers* will provide an updated status: $ rework list-workers postgres://aurelien:aurelien@localhost/rework 1 4124@10.211.55.3 43 Mb [dead] [2018-11-28 16:08:27.438491+01] → [2018-11-28 15:08:27.967432+01] → [2018-11-28 16:11:09.668587+01] monitor exit 2 4125@10.211.55.3 43 Mb [dead] [2018-11-28 16:08:27.442869+01] → [2018-11-28 15:08:27.967397+01] → [2018-11-28 16:11:09.668587+01] monitor exit =============================================================================== # https://refinery.docs.pythonian.fr/core_components/tswatch.html =============================================================================== Time Series Monitoring ********************** Tswatch is a monitoring and alerting system for time series data that helps you track the health and update frequency of your most important time series. It provides a web-based tool to monitor data freshness and identify potential issues with data pipelines. Note: This is an essential feature of the Timeseries Refinery. It is only accessible within the pro version. If interested, please visit: https://timeseries.pythonian.fr/#ouroffer. Overview ======== Tswatch continuously monitors your critical time series and provides: * **Real-time status monitoring** of time series update frequency * **Priority-based alerting** with color-coded status indicators * **Dependency tracking** for formula series * **Investigation tools** to trace data pipeline issues * **Configurable monitoring namespaces** for organizing series The system automatically calculates expected update frequencies based on historical patterns and alerts you when series deviate from their normal update schedule. [image] Adding Series to Monitor ======================== 1. **Search for series**: Use the search bar to find series by keywords separated by spaces 2. **Select series**: Click on the series you want to monitor 3. **Register changes**: Click "Register changes" to save your selections The system will begin monitoring these series immediately. Understanding the Status Dashboard ================================== The main dashboard displays your monitored series in order of priority, with the most problematic series at the top. Status Indicators ----------------- * **Green check ✓**: Normal situation - last update conforms to expected frequency * **Orange square ■**: Small deviation observed - should be monitored * **Red alert ⚠**: Important deviation - requires immediate attention * **Black cross ✖**: Ghost series - either doesn't exist or hasn't been updated in 2+ months The status calculation considers: * Historical update patterns * Expected update frequency * Time since last update * Deviation from normal schedule Monitoring Features =================== Series Information ------------------ For each monitored series, the dashboard shows: * **Series name** (clickable link to tsinfo page) * **Status indicator** with color coding * **Last update timestamp** * **Expected update frequency** * **Deviation metrics** Formula Dependencies -------------------- For formula series: * The status displayed is the **maximum score** of all dependencies * Click the **▶** symbol next to "formula" to see individual dependency status * Trace problematic dependencies to identify root causes Namespaces and Organization --------------------------- Tswatch supports organizing series into namespaces: * **Manual namespaces**: User-defined groups of series * **Basket namespaces**: Automatically synchronized with series baskets * **Mixed monitoring**: Combine different types of series in custom namespaces Removing Series --------------- **Via web interface**: Check the checkbox in the first column and click "Register changes" =============================================================================== # https://refinery.docs.pythonian.fr/core_components/exceladdin.html =============================================================================== Excel addin *********** What ==== This is an Excel client for the Timeseries Refinery. Install Xlwings client ====================== pip install tshistory_xl pip install xlwings Close Excel. Make sure that xlwings is registered: xlwings addin install Then type: $ tsh xl-addin install In Excel you should see two new tabs: "xlwings" and "TSHISTORY_CFG". Uninstall Xlwings client ======================== Just type $ tsh xl-addin uninstall Base use ======== On a brand new Excel sheet, you need initially a tab named "TSHISTORY_CFG", which must contain: * in A1: "webapi" * in B1: http://uri-of-the-tshistory-instance +----+--------+--------------------------------------+ | | A | B | |====|========|======================================| | 1 | webapi | http://uri-of-the-tshistory-instance | +----+--------+--------------------------------------+ Then, let's create the conditions to play with one series. * *Push data (Save them server side)* *Warning: with this addin one can easly push data in production database. Be sure to only use series begining with "test" until you completly master the process to not make a mess.* +------------+---------------+---------------+ | | test.series.1 | test.series.2 | |============|===============|===============| | 2020-09-01 | 1 | 6 | +------------+---------------+---------------+ | 2020-09-02 | 2 | 5 | +------------+---------------+---------------+ | 2020-09-03 | 3 | 4 | +------------+---------------+---------------+ | 2020-09-04 | 4 | 3 | +------------+---------------+---------------+ | 2020-09-05 | 5 | 2 | +------------+---------------+---------------+ | 2020-09-06 | 6 | 1 | +------------+---------------+---------------+ In a **new sheet**, let's go to "B1". There we type the name of a series (e.g. "test.series.1"), same thing for"B2". From "A2" to "A7", type timestamps e.g. "2020-9-1", ..., "2020-9-6". Make sure Excel really understand those as dates. Then you have to create a "name" (e.g. using the name manager accessible from the "formula" tab) for the range "B2:C7", whose name is e.g. "rwc_test_zone" (it is crucial that we have a prefix like "rwc_". **Only the data must be included into the range of the name. The margins (series name and date) must be adjacent to its range.** Finally in the "TSHISTORY_CFG" tab, click on "Save Tab" You can watch the result in base with the adapted url : http://uri-of- the-tshistory-instance/tsinfo?name=test.series.1 * *Pull data (Get them, client side)* On another sheet, you can recreate the previous step: * build a name (with the correct suffix) * write in the margin the names of the series and the date Click on "Get Tab" to obtain the data in Excel. Configuration options ===================== Prefix name options ------------------- In the example, we use a "rwc_" prefix. Each caracter has its meaning and can be omited. * "r": indicate that the zone can read data from the database. Could be omited if this excel zone is only used for manual entry * "w": allows to write in the database. Should be omited if the zone is only used for data consultation * "c": put some colors in the name range. Very useful: most of the errors that you will encounter are linked to the size of the range that does not fit the data. Specific request options: ~~~~~~~~~~~~~~~~~~~~~~~~~ * "f": will fill the trailing empty values with the last non-empty one * If "r_" name would return such data: +------------+---------------+ | | test.series.3 | |============|===============| | 2020-09-01 | 1 | +------------+---------------+ | 2020-09-02 | | +------------+---------------+ | 2020-09-03 | 3 | +------------+---------------+ | 2020-09-04 | | +------------+---------------+ | 2020-09-05 | | +------------+---------------+ * "rf_" name would return : +------------+---------------+ | | test.series.3 | |============|===============| | 2020-09-01 | 1 | +------------+---------------+ | 2020-09-02 | | +------------+---------------+ | 2020-09-03 | 3 | +------------+---------------+ | 2020-09-04 | 3 | +------------+---------------+ | 2020-09-05 | 3 | +------------+---------------+ * "_month_": when pushing data, a value defined on one date of the month will be extrapolated (daily) for the whole month Layout ------ The name range can be placed anywhere on the sheet, as long as the margins (with series and dates) are placed adjacently. One could play a little with the data layout by letting empty cells in the margins: +------------+---------------+---------+---------------+ | | test.series.1 | | test.series.2 | |============|===============|=========|===============| | | | I can | | +------------+---------------+---------+---------------+ | 2020-09-01 | 1 | put | 6 | +------------+---------------+---------+---------------+ | 2020-09-02 | 2 | any | 5 | +------------+---------------+---------+---------------+ | 2020-09-03 | 3 | comment | 4 | +------------+---------------+---------+---------------+ | 2020-09-04 | 4 | in | 3 | +------------+---------------+---------+---------------+ | 2020-09-05 | 5 | this | 2 | +------------+---------------+---------+---------------+ | | also | | here | +------------+---------------+---------+---------------+ | 2020-09-06 | 6 | column | 1 | +------------+---------------+---------+---------------+ Revisiting the past ------------------- All the series are versionned, which mean anyone can access to a previous version of the series. There are two ways to access it: Whole name ~~~~~~~~~~ After the first pulling of data, the sheet should be decorated in the upper left corner of the name range with a comment "ASOF". A date (*recognized as such by excel*) placed in this corner will allow to view the state of all series at this given time. Finally in the "TSHISTORY_CFG" tab, click on "Get All", and see the values coming. +----------------------------+---------------+ | *ASOF date should be here* | test.series.3 | |============================|===============| | 2020-09-01 | 1 | +----------------------------+---------------+ Note that when such a date is given, **the data can not be pushed in the database** (iow you cannot rewrite the past from the excel client). By column ~~~~~~~~~ Sometimes you want to be able to see side by side the same series at different times. You can display such data with additionnal argument "asof"in the upper margin +------------+---------------+----------------------------------+ | | test.series.4 | test.series.4(asof = 2020-09-03) | |============|===============|==================================| | 2020-09-01 | 1 | 1 | +------------+---------------+----------------------------------+ | 2020-09-02 | 2 | 2 | +------------+---------------+----------------------------------+ | 2020-09-03 | 3 | 3 | +------------+---------------+----------------------------------+ | 2020-09-04 | 4 | #N/A | +------------+---------------+----------------------------------+ | 2020-09-05 | 5 | #N/A | +------------+---------------+----------------------------------+ | 2020-09-06 | 6 | #N/A | +------------+---------------+----------------------------------+ *Note: date must be in ISO format YYYY-MM-DD* As before, the series with the "asof" parameter will not be pushed when pressing the "Save tab" or "Save all" button. Model backtest ~~~~~~~~~~~~~~ When backtesting a forecast model, one will need a "staircase" request, i.e. a request where the selected value dates are linked to the insertion dates. It allows to evaluate the validity of a model given a forecast horizon. For this, one has to use in the upper left corner a new keyword "asofdelta=" where the number is the forecast horizon in hours. +--------------+---------------+ | asofdelta=24 | test.series.3 | |==============|===============| | 2020-09-01 | 1 | +--------------+---------------+ "Not A Number" handling ----------------------- By default, when the data are missing at a given date, the corresponding cell will be filled with "#N/A". This default behaviour can be altered with some more columns options "(blank=empy)", "(blank=prev)", "(blank=3.14)" +--------------+----------+----------------------+----------------------+----------------------+-------------------+ | | test.5 | test.6(blank=empy) | test.7(blank=prev) | test.8(blank=3.14) | test.9(blank=0) | |==============|==========|======================|======================|======================|===================| | 2020-09-01 | 1 | 1 | 1 | 1 | 1 | +--------------+----------+----------------------+----------------------+----------------------+-------------------+ | 2020-09-02 | #N/A | | 1 | 3.14 | 0 | +--------------+----------+----------------------+----------------------+----------------------+-------------------+ | 2020-09-03 | 3 | 3 | 3 | 3 | 3 | +--------------+----------+----------------------+----------------------+----------------------+-------------------+ | 2020-09-04 | #N/A | | 3 | 3.14 | 0 | +--------------+----------+----------------------+----------------------+----------------------+-------------------+ Resampling ---------- The excel addin allows to resample the data when pulling them with the option "(agg = )" where the method can be "mean", "sum","max", "min". The resampling algorithm uses the dates given by the user in the left margin as intervals for the resampling wich will led to an empty cell at the end. +------------+----------------+ | | test.series.10 | |============|================| | 2020-09-01 | 1 | +------------+----------------+ | 2020-09-02 | 1 | +------------+----------------+ | 2020-09-03 | 1 | +------------+----------------+ | 2020-09-04 | 1 | +------------+----------------+ | 2020-09-05 | 1 | +------------+----------------+ +--------------+---------------------------+-----------------------------+ | | test.series.10(agg=sum) | *comments* | |==============|===========================|=============================| | 2020-09-01 | 3 | *Sum from date >= | | | | 2020-09-01 and date < | | | | 2020-09-04* | +--------------+---------------------------+-----------------------------+ | 2020-09-04 | 2 | *Sum from date >= | | | | 2020-09-04 and date < | | | | 2020-09-07* | +--------------+---------------------------+-----------------------------+ | 2020-09-07 | | *No computation here * | +--------------+---------------------------+-----------------------------+ Notes on resampling ~~~~~~~~~~~~~~~~~~~ * The last cell will be empty, in any case * The resampled data won't be pushed * If such resampling is reoccuring, we strongly advise to define a new resampled series, *server side*, with the formula system of tshistory Common pitfalls --------------- * Most of your errors will come from a range name with an incorrect form. Check it thoroughly. Check that all your left margin are dates, and that the upper margin does not have the same series called twice (with an exception when the series are asociated with an "asof" option) * The error returned by "xlwings" are quite a mouthful. However, most of the error that will raise will provide a usefull comment bury somewhere, provided by the developpers of this addin. Your eyes may bleed because of it, but the solution might be there. * It is quite easy to push some data in the database, it is also very simple to prevent it (use the "rc_" prefix) * If one change the value of series inserted by a different process, the rule of updating such data might surprise you at first look. More information here Testing ******* While the environment works well with recent versions of python and pandas, it might not work under the most recent versions of Excel (timedelta tests fail under V16.0). Keep that in mind whenever you're launching pytest. Some tests might not work especially if you haven't configured your local Postgres instance, since "sqlhelp" requires commands from your local Postgres installation, like "initdb". # For Linux, add the following to your .profile export PATH=$PATH:/usr/lib/postgresql/{version_number}/bin/ source ~/.profile # For Windows [System.Environment]::SetEnvironmentVariable('path', $Env:Programfiles + ".\PostgreSQL\{version_number}\bin;" + [System.Environment]::GetEnvironmentVariable('path', "User"),"User") # Or add the variables manually through your panel =============================================================================== # https://refinery.docs.pythonian.fr/core_components/dashboard.html =============================================================================== Dashboards ********** The dashboards are sophisticated, interactive visualizations built from time series data. They feature a browser-based interface that lets users assemble complex analytical dashboards without programming, while still supporting advanced configuration for power users. Note: This is an essential feature of the Timeseries Refinery. It is only accessible within the pro version. If interested, please visit: https://timeseries.pythonian.fr/#ouroffer. [image] The dashboard admin page ======================== All dashboard authoring happens in a single place, the *dashboard admin page*, a web application available under the "/dashboards" segment inside the refinery. It is organised in three tabs: * **Dashboards** — a grid of the existing dashboards, plus an *Add a new dashboard* card. Each card links to the dashboard viewer and to its editor. * **Figures** — the figure catalog, with a *new figure* button. Figures can be filtered by name and type, and opened in the figure editor. * **Balances** — the balance catalog, with a *new balance* link. Balances are a reusable, first-class building block (see Balances below). Managing dashboards =================== A dashboard is a named tree of navigation nodes, sections and figures. The dashboard editor (reached from a card on the *Dashboards* tab, or from the *Add a new dashboard* card) lets you build that tree interactively: 1. **General information** — set the dashboard *title* and a short *description*. The name is checked live for availability; a conflict is reported before you can save. 2. **Build the tree** — add navigation nodes, sections under them, and figures inside sections. Nodes, sections and figures can be renamed, reordered (move up / move down), collapsed and deleted in place. There is no fixed menu depth to declare up front — you grow the tree as needed. 3. **Section forms** — a section can be associated with a *form* so that, in the viewer, the reader may request a custom time period (or other parameters) for the figures of that section. 4. **Figures** — figures are picked from the existing catalog and placed into sections. A single figure can appear in several dashboards. 5. **Save or clone** — save the dashboard, or clone it under a new name to start a variant. A *View* button opens the resulting dashboard; you can share its link with your team. [image] Warning: Mandatory fields are flagged in the editor. You cannot save while a mandatory field is empty or while the name is in conflict. Viewing a dashboard =================== The dashboard viewer presents the saved tree to the reader. [image] **Menu** A collapsible tree menu on the left side reflects the dashboard structure. It can be folded away with the "☰" toggle to give the figures more room. Intermediate nodes group the tree; leaf entries are clickable and select the page to display. **Sections** The selected page is made of named sections. Each section carries a title banner and the figures assigned to it. **Forms** A section may expose a date (or dropdown) form. Submitting it re- runs the section's figures for the requested period or selection. **Per-figure info panel** Every figure has an info panel offering a *standalone view* link, an *edit* link back to the admin page (for users with write access), and deep links to the underlying series or groups. Figures are rendered according to their kind: ordinary figures are drawn as interactive plots, while a *balance* figure is shown through an embedded balance viewer (see Balances). Managing figures ================ A figure is a named, reusable plot definition. The figure editor (reached from the *Figures* tab, or from the *new figure* button) is a single panelled interface with a live preview: * **Name and type** — give the figure a name (checked for availability) and pick its type from the list of Figure Types. * **Configuration panels** — the parameters are grouped in collapsible panels (horizon, data range, traces, and type-specific panels such as transform, layout and style). Contextual help is available on the parameters. * **Traces** — a figure can carry several traces; each trace targets a series or a group and has its own styling. Traces can be added, re- targeted (rebind) and removed. * **Live preview** — the preview pane renders the figure as you configure it, without saving, which is handy for tuning parameters. * **Save, clone or delete** — save the figure, clone it under a new name, or delete it. Deletion is aware of the dashboards that reference the figure. [image] Warning: Mandatory fields are flagged in the editor; you cannot save a figure while a mandatory field is empty. A *balance* figure is a special case: instead of traces, it is edited as a **reference** to a balance definition from the registry — you choose the target balance, its period, and a mode (see Balances). Balance figures are not previewed server-side. Figure Types ============ Timeseries ---------- This is the most common figure type, ideal for displaying time- oriented data with a horizontal date axis. It is particularly useful for monitoring trends, detecting patterns, and comparing multiple series over time. **Common use cases:** * Energy production and consumption monitoring * Price evolution tracking * Temperature and weather data visualization * Financial metrics and KPIs evolution **Available options:** * Line plots with multiple styling options * Area charts with configurable fill modes * Stacked and grouped bar configurations * Seasonal overlays for year-over-year analysis * Multi-axis support for different data scales [image] Table ----- The table figure displays time series data in a tabular format, making it easy to read precise values and compare data across different time periods. This format is particularly useful for reports, data exports, and when exact numerical values are more important than visual trends. **Common use cases:** * Detailed numerical reports with exact values * Comparison of multiple series side by side * Monthly or yearly aggregated summaries [image] Scatter ------- The scatter plot is essential for analyzing correlations and relationships between two variables. When hovering over the correlation line, statistical information appears including the correlation coefficient. **Common use cases:** * Correlation analysis between two time series * Identifying linear or non-linear relationships * Detecting outliers and anomalies * Validating model predictions against observations [image] Daily Profile ------------- The daily profile plot is specialized for intraday data, presenting the daily pattern hour by hour across multiple days. This visualization highlights recurring patterns and helps identify typical daily cycles. **Common use cases:** * Electricity load profiles (daily consumption patterns) * Intraday trading activity analysis * Temperature variations throughout the day * Traffic or usage patterns during business hours [image] Candlestick ----------- The candlestick figure is a standard visualization for market data, displaying Open, Close, High, and Low values for each time period. Each candlestick provides a comprehensive view of price movement within a single time interval. **Common use cases:** * Stock price visualization * Commodity market analysis * Energy market price movements [image] Balance ------- The balance figure presents hierarchical data in a table format where values are organized by aggregation levels. Each lower level can be summed up to obtain the corresponding upper level, ensuring data consistency and making it easy to drill down from aggregated to detailed values. It comes in two modes. See Balances for defining and managing the underlying balance. *Values mode* The default mode shows the balance as described above. **Common use cases:** * Financial statements (revenue breakdown by category) * Energy balance sheets (production by source) * Hierarchical budget tracking and reconciliation [image] *Delta mode* The delta mode highlights the differences between two versions of the table (a *then* and a *now* revision), making it easy to see what changed between two periods or data revisions. **Common use cases:** * Month-over-month or year-over-year comparison * Budget vs actual variance analysis * Revision tracking and change detection [image] Group ----- The group figure is designed to visualize "group" data, which represents multiple related scenarios or ensemble forecasts. It displays all members of a group simultaneously, allowing comparison between different scenarios or model runs. **Common use cases:** * Weather ensemble forecasts (multiple model runs) * Monte Carlo simulation results * Scenario planning (best case, worst case, expected) * Uncertainty quantification in predictions [image] Waterfall --------- The waterfall figure shows how a starting value is built up (or drawn down) by a sequence of positive and negative contributions to a final total. Each trace is a contributing term. **Common use cases:** * P&L bridges from one period to the next * Contribution and breakdown analysis * Decomposing a total into its sources [image] History heatmap --------------- The history heatmap displays a single series' history as a grid: value dates on one axis, revision (insertion) dates on the other, and colour encoding the value. It reveals how the series changed from one revision to the next. **Common use cases:** * Watching a forecast evolve across successive revisions * Spotting when and where values were revised * Data-quality and stability review [image] History curve ------------- The history curve overlays chosen revisions of a single series as ordinary curves: pick the insertion dates (each with an optional label) and get one curve per revision. It is the curve counterpart of the history heatmap, for when a handful of well-chosen revisions tell the story better than the full grid. **Common use cases:** * Comparing yesterday's and today's forecast on one plot * Showing how successive forecasts converged towards the outcome Model statistics ---------------- The model statistics figure evaluates one or more model series against a reference series and reports error metrics (MAE, RMSE, ...). Each model trace is compared to the reference over the window; it needs a reference series and a list of metrics. **Common use cases:** * Forecast accuracy assessment * Comparing competing models against observations * Monitoring model error over time [image] Note: Keep in mind that new figure types can be easily added. Balances ======== A *balance* is a first-class, reusable object stored in a registry, separate from figures. It captures a hierarchical, aggregable table — values that sum up level by level. Figures then reference a balance rather than redefining it. **Definition versus reference** A balance *definition* lives in the balance registry under a name, as a typed expression in the balance language. A figure only stores a *reference* to that name (with a period and, optionally, a mode). Renaming a balance propagates to the figures that reference it. **The balance language** A definition is a "(balance "label" (config ...) ...)" expression. Its pieces: *config and periods* — "config" sets the timezone and one or more named periods. Each "period" gives a label, a frequency, a from/to range, and, for the Delta mode, optional "#:delta_then" / "#:delta_now" revisions. The frequency has three spellings: a bare offset ""h"", "(freq "D")", or a multiple "(n_freq 15 "min")". The range bounds are date expressions built from "(now)", "(today)", the period projections "monthstart", "yearstart" and "yearend", the "deltahours" / "deltadays" / "deltamonths" / "deltayears" shifts, and the multi-keyword "shifted". (config "CET" (period "15min" (n_freq 15 "min") (shifted (today) #:days 1) (shifted (today) #:days 2)) (period "Daily" (freq "D") (shifted (today) #:days 1) (shifted (today) #:days 14) #:delta_then (shifted (today) #:days -1))) *rows: sum, series, basket* — the rows form a tree. Branch nodes are "sum" (they total their children); leaf nodes are "series" (a series name and an aggregation method) or "basket" (a saved query, a method, and a metadata key used to label its members). Each level equals the sum of its descendants, which is what keeps the table internally consistent. (sum (draw_option "Production" #:unit "MW") (series "prod.plant_a.mw.fcst.15min" "mean" (draw_option "Plant A")) (basket "plant_b.machines.fcst" "mean" "label" (draw_option "Plant B"))) "sum" accepts two options: "#:override" supplies a directly measured series standing for the subtotal, when it is more reliable than summing the parts, and "#:fill_0" treats missing points as zero ("basket" also accepts it): (sum (draw_option "Production" #:unit "MW") (series "prod.plant_a.mw" "sum" (draw_option "Plant A")) (series "prod.plant_b.mw" "sum" (draw_option "Plant B")) #:override (series "prod.total.measured.mw" "sum" (draw_option "Production")) #:fill_0 #t) *draw_option* — every node carries a "draw_option" setting its display label and formatting: "#:unit", "#:decimal", "#:coefficient" to scale a term (e.g. "-1" to subtract it), and "#:exclude" to show the row for information while leaving it out of the aggregate. (draw_option "Net position" #:unit "MW" #:decimal 0 #:coefficient -1) (draw_option "Memo: imports" #:exclude #t) The "update_balance" client method accepts exactly this text; see Programmatic access for a complete definition and how to submit it. **Balance editor** Reached from the admin page's *Balances* tab, the balance editor is a dedicated editor for the balance language. You give the definition a name and create or update it; the definition is validated (types, and existence of the referenced series and baskets) before being saved. A *View balance* link opens the viewer once the balance is saved. [image] **Balance viewer** The balance viewer renders a balance as a table. It offers a *Values* mode and a *Delta* mode (comparing a *then* and a *now* revision), plus an *expanded* control governing how far the hierarchy is unfolded by default. [image] Programmatic access =================== Beyond the admin page, figures, dashboards and balances can be managed programmatically through the "tsa.dashboard" client — a "DashboardAPI" instance carried by every refinery API handle. This is convenient for automated or bulk generation. **Figures** A figure is assembled from model objects: a *variant* (the plot type, which holds the traces) plus optional "query" (value window and timezone), "transform", "layout" and "style" blocks. Each "Trace" targets a series (or a group) and carries its own styling: from tshistory.api import timeseries from tshistory_formula.helper import seriesname from tshistory_dashboard.model import ( Figure, Timeseries, Trace, Query, FigureLayout, ) tsa = timeseries() tsa.dashboard.update_figure( Figure( name='plant.generation', variant=Timeseries( traces=[ Trace( name=seriesname('prod.plant.mw.actual.15min'), legend_name='actual', color='#1f77b4', ), Trace( name=seriesname('prod.plant.mw.forecast.15min'), legend_name='forecast', color='#1f77b4', line_style='dot', from_date='(deltadays (today) -1)', ), ], ), query=Query(tz='Europe/Paris'), layout=FigureLayout( title='Plant generation (MW)', yaxis_title='MW', today_line=True, ), ) ) "update_figure" both creates and updates (upsert by name). The figure is validated before being stored; an invalid one (missing series, no trace, ...) raises "FigureError". Other variants live in the same module — "Table", "Group", "Scatter", "DailyProfile", "Candlestick", "Waterfall", "HistoryHeatmap", "HistoryCurve" and "ModelStatistics" — each assembled the same way (a few take extra fields, e.g. "ModelStatistics" needs a "reference" series and "metrics"). Balance figures are a special case, covered below. Reading, listing, renaming and deleting: fig = tsa.dashboard.get_figure('plant.generation') # Figure or None catalog = tsa.dashboard.figures_catalog() # {variant: [names]} tsa.dashboard.rename_figure('plant.generation', 'plant.gen') tsa.dashboard.delete_figure('plant.gen') # force=True to # dereference first **Dashboards** A dashboard is a tree of navigation nodes and sections referencing figures. Nodes nest to any depth; a section holds ordered figure references and an optional form. Build it from the model objects: from tshistory_dashboard.model import ( Dashboard, TreeNode, Section, FigureRef, ) tsa.dashboard.update_dashboard( Dashboard( name='generation', title='Generation', description='Fleet overview', tree=[ TreeNode(label='Overview', children=[ Section(label='Recent', form='bounds.d-w', figures=[ FigureRef('plant.generation'), ]), ]), ], ) ) or from its lisp source, which is handy for round-tripping an existing definition ("get_dashboard" hands you back the same syntax): dash = Dashboard.fromlisp( '(def-dashboard "generation" #:title "Generation"' ' (tree (node "Overview"' ' (section "Recent" #:form "bounds.d-w" (figure "plant.generation")))))' ) tsa.dashboard.update_dashboard(dash) # list of {name, title, description} catalog = tsa.dashboard.dashboards_catalog() tsa.dashboard.rename_dashboard('generation', 'fleet') tsa.dashboard.delete_dashboard('fleet') **Balances** A balance definition is stored in the registry under a name and referenced by balance figures (see Balances). It is created or updated from its source expression — the same balance language as in the balance editor: a "config" (timezone and one or more periods) followed by "sum" rows whose leaves are "series" or "basket" nodes. The registry name is independent of the internal "(balance "...")" label: balance_source = ''' (balance "Portfolio" (config "CET" (period "15min" (n_freq 15 "min") (shifted (today) #:days 1) (shifted (today) #:days 2))) (sum (draw_option "Production" #:unit "MW") (series "prod.plant_a.mw.fcst.15min" "mean" (draw_option "Plant A")) (series "prod.plant_b.mw.fcst.15min" "mean" (draw_option "Plant B")))) ''' tsa.dashboard.update_balance('portfolio_schedule', balance_source) tsa.dashboard.balance_list() # list of names tsa.dashboard.balance_formula('portfolio_schedule') # the source, or None tsa.dashboard.rename_balance('portfolio_schedule', 'portfolio') tsa.dashboard.delete_balance('portfolio') "update_balance" validates the definition (types, and existence of the referenced series and baskets) and raises "FigureError" on failure. Access and deployment ===================== By default the dashboards are served by the refinery itself, under the "/dashboards" segment, reusing the refinery's authentication. This is the admin page and viewer described above, and needs no extra configuration. The **viewer** can additionally be deployed as a separate, view-only application — useful when the audience of the dashboards is not the same as the refinery's users. It exposes the dashboard viewer (and the read endpoints it needs) but none of the editors: authoring always happens back on the refinery's dashboard admin page. Two flavours are available: * a **public** app: no authentication, every visitor is granted the read-only ("ro") role; * a **private** app behind OAuth2, resolving each authenticated user's role. Roles gate access the same way everywhere: reading requires the "ro", "rw" or "admin" role, and editing (only ever on the integrated admin page) requires "rw" or "admin". Separate deployments are wired together through the "[dashboard]" section of "tshistory.cfg": [dashboard] refinery = https://refinery.example.com/ dashboards = https://dashboard.example.com/ * "refinery" — where the refinery lives. The standalone viewer uses it to aim its *edit* links (back at the dashboard admin page) and its series/group *info* links at the refinery. * "dashboards" — where the dashboard service lives. The refinery's own navigation uses it to link out to the dashboards (defaults to "/dashboards"). =============================================================================== # https://refinery.docs.pythonian.fr/core_components/mcp.html =============================================================================== The MCP server ************** The refinery ships an **MCP server** — a Model Context Protocol endpoint that lets AI assistants and other MCP clients work directly against a refinery instance. Through it an assistant can explore the catalog, sample and analyse data, read and author formulas, and manage dashboards, figures and balances — all under the same authentication and role model as the rest of the refinery. Note: This is a pro-only feature of the Timeseries Refinery. If interested, please visit: https://timeseries.pythonian.fr/#ouroffer. What it exposes =============== Its capabilities fall into a few families: * **discovery** — search the catalog, read metadata, follow dependencies; * **data** — sample recent values, compute statistics, detect regimes and anomalies; * **formulas** — read, expand, test and (with write access) author computed series and groups; * **dashboards, figures and balances** — browse and manage the visualization objects; * **baskets** — reusable saved queries * **notes** - a per-user knowledge base; * **tasks** — inspect the scheduler and recent runs; * **guides** — built-in, self-documenting help (see Built-in guidance). Connecting a client =================== Any MCP client that speaks streamable HTTP can connect to the server's "/mcp" endpoint. There is nothing to install on the client side — point your client at the endpoint URL as a remote HTTP MCP server. Clients are configured either through a command that registers a remote HTTP server, or through a JSON configuration such as: { "mcpServers": { "refinery": { "type": "http", "url": "https://your-refinery-host/mcp" } } } On first use the client performs an OAuth2 login in your browser against the refinery's identity provider; subsequent requests carry the resulting token. Authentication and roles ======================== Access uses the same OAuth2/OpenID Connect authentication as the rest of the pro application, and the same four-tier role ladder — "guest" < "ro" < "rw" < "admin" (see Securing the Refinery). Your role governs what the assistant may do on your behalf: * **read** tools require the "ro", "rw" or "admin" role; * **write** tools — anything that creates, modifies or deletes an object — require "rw" or "admin"; * "guest" can call nothing. In the tool catalog below, write tools are marked *(write)*; all others are read-only. Built-in guidance ================= The server is designed to onboard an assistant on its own. The "get_*_guide" and "get_*_spec" tools listed below return structured instructions and language references, so a client can learn the query language, the formula operators, or the figure/dashboard/balance syntaxes on demand rather than from external documentation. A single MCP **prompt**, "welcome", acts as the entry point: it presents a menu of activities — discover the instance, analyse the gold layer, detect anomalies, create and test formulas, explore and manage dashboards — and routes the assistant to the relevant guide. It also reminds the assistant to check the notes at the start of a session, and to ask for explicit confirmation before creating or changing anything. Available tools =============== **Notes** — a knowledge base the server keeps for the instance. * "list_notes" — list all note keys. * "get_note" — read one note. * "save_note" *(write)* — create or update a note (full replace, append, or find-and-replace). * "delete_note" *(write)* — remove a note. Note: The notes are a collaborative, instance-wide knowledge base: everyone using the instance shares the same notes. An assistant can record what it learns about the refinery — naming conventions, a domain map, recurring gotchas — and that knowledge persists across sessions and is available to every other user. **Discovery, search** * "find_items" — search series or groups with the lisp query language. * "get_info" — global statistics about the instance. * "list_metadata_keys" — all metadata keys in use. * "get_dependents" — what depends on an item (impact analysis). * "troubleshoot_item" — metadata, internal metadata and, for formulas, depth and dependencies. **Series** * "get_fresh_data_sample" — a recent data window with statistics (auto-downsampled to a readable size). * "analyze_series" — change-point segmentation into regimes (flat, ascending, oscillating, ...). * "get_activity_summary" — per-series statistics across a basket, ranked by variability. * "rename_series" *(write)* — rename a series or formula, propagating into referencing formulas. * "delete_series" *(write)* — delete a series or formula (irreversible). * "update_metadata" *(write)* — merge metadata onto a series or group. **Formulas** * "get_formula" — a formula definition, optionally expanded. * "eval_formula" — evaluate an ad-hoc formula without saving it. * "get_gold_layer_formulas" — the deepest, highest-value formulas (expensive). * "register_formula" *(write)* — create or replace a series or group formula. **Baskets** * "list_baskets" — saved queries and their definitions. * "resolve_basket" — evaluate a basket to its matching items. * "register_basket" *(write)* — create or update a basket. * "delete_basket" *(write)* — remove a basket. **Figures** * "list_figures" — the figure catalog (paginated, by variant). * "figure_dependents" — dashboards referencing a figure. * "get_figure" — a figure's canonical definition. * "update_figure" *(write)* — create or replace a figure (validated before saving). * "rename_figure" *(write)* — rename, rewriting referring dashboards. * "delete_figure" *(write)* — delete a figure (refuses if referenced, unless forced). **Dashboards** * "list_dashboards" — dashboards with title and description. * "get_dashboard" — a dashboard's canonical definition. * "update_dashboard" *(write)* — create or replace a dashboard. * "rename_dashboard" *(write)* — rename a dashboard. * "delete_dashboard" *(write)* — delete a dashboard. **Balances** * "list_balances" — balance definitions in the registry. * "get_balance" — a balance's canonical definition. * "update_balance" *(write)* — create or replace a balance (validated). * "rename_balance" *(write)* — rename, rewriting referencing figures. * "delete_balance" *(write)* — delete a balance (refuses if referenced). **Tasks and schedules** * "list_operations" — registered scheduler operations. * "list_schedules" — scheduled rules and their decoded inputs. * "list_tasks" — recent task runs with their status. **Guides and help** * "get_query_syntax" — the search query language. * "get_operators" — the formula operators (introspected from the registry). * "get_discovery_guide" — a methodology to map an unfamiliar refinery. * "get_formula_building_guide" — a step-by-step formula authoring workflow. * "get_gold_layer_analysis_guide" — how to find and read the highest- value formulas. * "get_anomaly_detection_guide" — a workflow for spotting outliers and gaps. * "get_figure_spec" — the figure definition language. * "get_dashboard_spec" — the dashboard definition language. * "get_balance_spec" — the balance definition language. =============================================================================== # https://refinery.docs.pythonian.fr/core_components/http_api.html =============================================================================== The HTTP API ************ Every refinery instance serves its complete API over HTTP, under the "/api" prefix. This is the very protocol the Python client speaks: pointing "timeseries()" at an http uri yields the same api points with the same semantics as a direct postgres uri (see API documentation) — and it is also how federated instances consume each other (see Securing the Refinery). Interactive reference ===================== The authoritative, always up-to-date endpoint reference is served by the instance itself: browse "https:///api" for the interactive (swagger) console, listing every endpoint with its parameters, responses, and a try-it-out mode. Speaking to it directly ======================= Requests authenticate with a bearer token (see Securing the Refinery). Most endpoints speak plain json: $ curl -H "Authorization: Bearer $TOKEN" \ 'https://refinery.example.com/api/series/metadata?name=banana-spot-price' {"unit": "eur", "source": "market"} The series *values* endpoints ("/series/state", "/series/history" and friends) return a compact binary encoding rather than json: they are designed for the Python and Excel clients. For scripted access to the values, prefer the Python client over an http uri: from tshistory.api import timeseries tsa = timeseries('https://refinery.example.com/api') ts = tsa.get('banana-spot-price') =============================================================================== # https://refinery.docs.pythonian.fr/core_components/security.html =============================================================================== Securing the Refinery ********************* Two modalities are available: * Http basic-auth * OpenID Connect Setting up basic auth ====================== Most of the setup happens with your web frontend, for instance Nginx. You should consult the relevant documentation to set up the basic auth mechanism. This will provide a simple login + password form in front of the application. Now, if you want to make a data-mesh configuration work in this context, you have to provide the credentials of the target instance in the *tshistory.cfg* configuration file. Let's start with a sample configuration with one external source: [dburi] power = postgresql://power [sources] power.meteo = https://refinery.meteo.acme.com/api, tsh This describes a *power* refinery that also consumes meteorogical data from a *meteo* instance. Now if you just put http basic auth in front of your *meteo* instance, you have to add a new section like the following: [auth] meteo.uri = https://refinery.meteo.acme.com/api meteo.login = meteo.password = Working with OpenID Connect --------------------------- Note: The OpenID Connect adapter for the Refinery is an Open Source component, like the rest of the Refinery, but it is provided in the context of Pythonian's paying plans like e.g. the Premium Subscription. This enables a very secure and seamless SSO experience for users if well configured. Moreover, it permits to assign roles to the users. Let's describe them right away, and their associated capabilities: * *guest* : user is vetted by the OIDC platform but is unknown from the Refinery. Can't do nor see anything. * *ro* : with this role, everything can be read, nothing can be created, updated or deleted. * *rw* : this role allows all CRUD operations except specific things only Admins can do. * *admin* : this role can do everything. Now, back to the setup. In the next chapter, we will see how to configure one particular OIDC platform (KeyCloack) for a Refinery instance. And in the subsequent chapter, we will configure the Refinery instance for all the relevant access scenarios. Indeed, we need these 3 scenarios to be handled: * web access by end users * robot access (or *machine to machine*) by e.g. scrapers, models, etc. * scripting access for end users Though the last two ones look similar, their security context is different: * the robots are hosted on a secure server platform, * the end user runs from an a priori unsecure computer. Setting up KeyCloak clients =========================== KeyCloak is a powerful and also complex platform and we don't cover its use at all. We are only concerned with the configuration of what is commonly used an OIDC *client*. To properly secure all access modes to a Refinery, we need two such clients and we will show how to configure them. Such configuration steps should be similar with other OIDC platforms. Confidential client ------------------- The first client is said to be confidential (or private) because its users (except for the web browser) are deemed secure enough to host the necessary credentials (especially the *client_secret*). It covers the need of: * web access, * robot access. Let's configure one for our *eFlower* application (which is about power prices and marginal co2 forecasts). In KeyCloak, we create the client as such: [image] The *client_id* is the most important piece of information there. Note it for later. Then, the second part: [image] The *client authentication flow* flag is of utmost importance, since it indicates this is a confidential or *public* client. Now we have to setup a number of uris: [image] These of course must match the specific uris you have for the *eFlower* application. As a last step, we are going to configure an *audience* for our application. First, go to the *Client Scopes* tab. [image] There, you will see a list of *scopes* including one that has been created automatically for you (at client creation time) named *eflower-dev-dedicated*. Click on this item. You land in a section to manage *mappers* and *scope* for *eflower-dev-dedicated*. Chose *Add a mapper* (by configuration), and fill this: [image] Public client ------------- Create a new client with the *eflower-dev-public* id. The important differences with the confidential client lie in: * an additional *Valid redirect URI*: http://localhost:5000/* * the client authentication field must be set to *off* [image] The audience, like other parameters, must be set up the same way as with the confidential client. Setting up the Refinery configuration files =========================================== The *tshistory.cfg* must be augmented in the *[auth]* sections. For a machine to machine configuration (typically server side), you want something like this (with the *secret* set to an appropriate value): [auth] eflower_dev.uri = https://refinery.eflower-dev.pythonian.fr/api eflower_dev.client_id = eflower-dev eflower_dev.client_secret = eflower_dev.domain = id.cloudcrane.tech.scaleway.cloudcrane.io/realms/pythonian For Python users on their laptops, using a Python shell or Notebook, the *pkce* option is needed. In this case, the personal computer is considered too insecure to host the *client_secret* and we also want to identify exactly who is making the calls. So we don't have a *client_secret* entry but a *pkce* key with no value. Also we must refer to the right client (the *public* one). Here is how it goes: [auth] eflower_dev.uri = https://refinery.eflower-dev.pythonian.fr/api eflower_dev.pkce = eflower_dev.client_id = eflower-dev-public eflower_dev.domain = id.cloudcrane.tech.scaleway.cloudcrane.io/realms/pythonian Lastly, the server side has to be also configured: [server-auth] client_id = eflower-dev-public client_secret = domain = id.cloudcrane.tech.scaleway.cloudcrane.io/realms/pythonian authorize_uri = https://refinery.eflower-dev.pythonian.fr/authorize audience = https://refinery.eflower-dev.pythonian.fr/api Associating roles to users ========================== Any user that connects to a Refinery instance through OIDC will be default have the *guest* status. Guests are unknown from the Refinery and can't see nor do anything at all. So one needs to assign users to relevant roles: *admin*, *rw* or *ro*. Using the command line ---------------------- Two command line operations are provided to manage this: * tsh list-users-roles * tsh set-user-role The first lists the existing user / role mapping. Here's how it works. $ tsh list-users-roles No user is currently known. $ tsh set-user-role babar@timeseries.fr admin $ tsh set-user-role celeste@timeseries.fr rw $ tsh list-users-roles Found 2 users. admin → ['babar@timeseries.fr'] rw → ['celeste@timeseries.fr'] $ tsh set-user-role babar@timeseries.fr guest $ tsh list-users-roles Found 1 users. rw → ['celeste@timeseries.fr'] Using the settings section -------------------------- A web ui in the *settings* section is also provided to help perform these operations : [image] Note: The initial *admin* role will still have to be set up through the command line. =============================================================================== # https://refinery.docs.pythonian.fr/core_components/settings.html =============================================================================== Settings ******** The "/settings" page groups a few preferences in three tabs. These settings are **shared by everyone using the instance**: a change made here applies to all users, not just to you. Horizons ======== A *horizon* is a named time window — a label with a *from* and a *to* date expression. Horizons defined here populate the time-window selector used across the interface (the quick view and the series editor), so a reader can switch to a familiar range in one click instead of typing dates. [image] Timezones ========= The timezone dropdowns throughout the interface do not list the whole timezone database; they show a curated shortlist maintained here. Each entry is a timezone name plus an optional friendly label. Add the timezones your team works with and they appear in every timezone dropdown of the interface — notably in the formula constructor and in the horizon widget. [image] Users ===== In the pro version, the *Users* tab lists the instance's users and lets an administrator set each one's role — **Reader**, **Editor** or **Admin** — which controls who can read versus edit. See Securing the Refinery for what each role grants. [image] =============================================================================== # https://refinery.docs.pythonian.fr/operations/index.html =============================================================================== Operations Guide **************** This section covers operational aspects of running and maintaining a Refinery installation. Contents: ^^^^^^^^^ * Configuration Reference * Configuration File Locations and Discovery * Required Sections * Optional Sections * Examples and Use Cases * Best Practices * Naming Conventions * Data Governance * Formula Development * Dual Storage Operations * Configuration * Moving series between backends * Operational notes * Series replication * replicate_series_from_refinery * replicate_from_basket * Formula Debugging Techniques * Table of Contents * Common Semantic Issues * Progressive Debugging Strategy * Data Alignment Problems * Missing Data Propagation * Date and Time Semantic Errors * Formula Dependencies and Recursion * Troubleshooting * Installation Issues * Runtime Issues * Web Interface Issues * Performance Issues * Migration Issues * Getting Help =============================================================================== # https://refinery.docs.pythonian.fr/configuration_reference.html =============================================================================== Configuration Reference *********************** This reference guide covers all configuration options available in the "tshistory.cfg" configuration file. Configuration File Locations and Discovery ========================================== **Configuration File Search Order** The tshistory configuration system searches for "tshistory.cfg" in the following order: 1. **Environment Variable Path**: "$TSHISTORYCFGPATH" - if set and file exists 2. **Current Directory**: "./tshistory.cfg" 3. **Home Directory**: "~/tshistory.cfg" 4. **XDG Config Directory**: "$XDG_CONFIG_HOME/tshistory.cfg" (defaults to "~/.config/tshistory.cfg") **Environment Variables:** * "TSHISTORYCFGPATH" - Override default configuration file path * "XDG_CONFIG_HOME" - XDG base directory specification (defaults to "~/.config") **File Format:** * INI-style configuration file * Case-sensitive section and key names * Comments start with "#" Required Sections ================= [dburi] Section - Database Connections -------------------------------------- The "[dburi]" section is mandatory and defines database connections for your refinery instances. **Basic Format:** [dburi] instance_name = database_uri **Supported URI Formats:** * **PostgreSQL**: "postgresql:///database_name" (local connection) * **PostgreSQL with credentials**: "postgresql://user:password@host:port/database" * **HTTP API**: "https://refinery.example.com/api" (remote refinery instance) **Examples:** [dburi] # Local PostgreSQL database refinery = postgresql:///refinery # PostgreSQL with full connection details test_instance = postgresql://user:password@db.company.com:5432/refinery_dev # Remote refinery instance via HTTP API external = https://refinery.partner.com/api # Multiple instances local = postgresql:///refinery_local staging = postgresql:///refinery_staging prod = postgresql:///refinery_prod **API Usage:** * **Default connection**: "tsa = timeseries()" uses the first entry in the "[dburi]" section * **Direct URI**: "tsa = timeseries('postgresql:///mydb')" bypasses configuration **Production vs Development:** * **Production deployments** typically use a single database URI * **Multiple entries** are primarily useful for developers and testers who need to work with different environments * The first entry serves as the default when no specific instance is requested [storage] Section - Series Storage Backends ------------------------------------------- Since version 1.0 the series values can live on either of two storage backends: * "pg" - the historical PostgreSQL backend (the default) * "fs1" - a high-performance filesystem backend The "[storage]" section configures them. **Format:** [storage] instance_name.path = /path/to/storage instance_name.default = pg **Parameters:** * "instance_name.path" - root directory of the fs1 backend. **Mandatory with the pro edition** (refinery_pro >= 0.5.0): the application refuses to start without it. The directory must be writable by the application user. * "instance_name.default" - the backend assigned to newly created series: "pg" (the default) or "fs1". Each series individually carries its backend; changing this setting does not affect existing series. **Example:** [dburi] refinery = postgresql:///refinery [storage] # the fs1 storage root (mandatory with refinery_pro) refinery.path = /data/refinery/storage # optional: create new series on fs1 rather than postgres # refinery.default = fs1 **Key Points:** * The catalog and the metadata always stay in PostgreSQL; only the series values live on the chosen backend. * Series can be moved between backends while the platform runs, with the "change-backend" command (pro edition), completely or selectively. * The fs1 directory belongs in your backup perimeter, next to the database dumps. Optional Sections ================= [sources] Section - Data Mesh Configuration ------------------------------------------- The "[sources]" section configures external refinery instances for data mesh setups, allowing your refinery to consume data from other refineries. **Format:** [sources] local_instance.source_name = remote_uri, namespace **Parameters:** * "local_instance" - Must match a name from the "[dburi]" section * "source_name" - Local alias for the remote source * "remote_uri" - HTTP API endpoint of the remote refinery * "namespace" - Always use "tsh" for refinery sources **Examples:** [sources] # Connect to external weather data refinery.meteo = https://refinery.meteo.company.com/api, tsh # Multiple external sources refinery.weather = https://refinery.weather.partner.com/api, tsh refinery.market = https://refinery.market.partner.com/api, tsh **Usage:** * External series become available as if they were local * Transparent access through normal API calls * Requires proper authentication (see "[auth]" section) [auth] Section - Authentication for External Sources ---------------------------------------------------- The "[auth]" section provides authentication credentials for accessing external refinery instances defined in the "[sources]" section. **Authentication Methods:** **HTTP Basic Authentication:** [auth] source_name.uri = https://external.refinery.com/api source_name.login = username source_name.password = password **OpenID Connect - Client Credentials (M2M):** [auth] source_name.uri = https://external.refinery.com/api source_name.client_id = your_client_id source_name.client_secret = your_client_secret source_name.domain = openid.provider.com/realms/realm_name **OpenID Connect - PKCE (Public Client):** [auth] source_name.uri = https://external.refinery.com/api source_name.pkce = source_name.client_id = public_client_id source_name.domain = openid.provider.com/realms/realm_name **Real-World Examples:** [auth] # Basic authentication meteo.uri = https://refinery.meteo.pythonian.fr/api meteo.login = username meteo.password = password # Machine-to-Machine OAuth partner_api.uri = https://refinery.partner.pythonian.fr/api partner_api.client_id = partner_client partner_api.client_secret = your_client_secret partner_api.domain = openid.pythonian.space/realms/pythonian # PKCE flow (public client) public_api.uri = https://refinery.public.pythonian.fr/api public_api.pkce = public_api.client_id = public-client public_api.domain = openid.pythonian.space/realms/pythonian **Key Points:** * Authentication is required for most external refinery instances * Basic auth is simpler but less secure than OpenID Connect * PKCE flow is for public clients (no client secret) * M2M (client credentials) flow is for server-to-server communication * "[auth]" entries must correspond to sources defined in "[sources]" or "[dburi]" * "[auth]" is for authenticating as a **client** to external services [server-auth] Section - Server Authentication Configuration ----------------------------------------------------------- The "[server-auth]" section configures authentication for the refinery server itself (when acting as a server rather than client). **Format:** [server-auth] client_id = your_server_client_id client_secret = your_server_client_secret domain = openid.provider.com/realms/realm_name authorize_uri = https://your.refinery.com/authorize audience = https://your.refinery.com/api # optional issuer = https://openid.provider.com/realms/realm_name m2m_allowed_clients = client-a, client-b session_lifetime_seconds = 3600 The optional keys: "issuer" overrides the OpenID issuer used for the .well-known discovery (derived from "domain" otherwise); "m2m_allowed_clients" lists the client ids allowed to authenticate machine-to-machine; "session_lifetime_seconds" bounds the browser session lifetime when the identity provider does not supply a refresh expiry. **Usage:** * Used when your refinery instance needs to authenticate incoming requests * Configures your refinery instance as an OAuth2 **server** * Used when other clients need to authenticate to access your refinery * Different from "[auth]" which is for authenticating to external services Note: For detailed server authentication setup, see the Securing the Refinery chapter. [dashboard] Section - External Dashboard URL Configuration ---------------------------------------------------------- The "[dashboard]" section defines URLs for external dashboard services and integrations. **Important Note:** * This section is only for **external** dashboard services * The built-in dashboard (available at "http://your- refinery/dashboards") requires no configuration * Most installations do not need this section **Format:** [dashboard] refinery = external_dashboard_url dashboards = external_dashboards_service_url **Examples:** [dashboard] # External dashboard service for the refinery instance refinery = http://external-dashboard.company.com # Dedicated external dashboards service dashboards = http://external-dashboards.company.com/dashboards/ **Usage:** * Configures integration with external dashboard services (not the built-in one) * Used by the refinery web interface to link to external dashboard systems * Allows for integration with third-party dashboard platforms **Key Points:** * Optional section - only needed if integrating with external dashboard services * The built-in dashboard system requires no configuration entries * URLs should point to accessible external dashboard service endpoints Examples and Use Cases ====================== Basic Single Instance Setup --------------------------- The minimal configuration for a local development setup: [dburi] refinery = postgresql:///refinery This configuration: * Uses local PostgreSQL database named "refinery" * No external sources or authentication * Built-in dashboard available at "http://localhost:5000/dashboards" Multi-Database Development Setup -------------------------------- Configuration for developers working with multiple environments: [dburi] local = postgresql:///refinery_local staging = postgresql:///refinery_staging prod = postgresql:///refinery_prod Usage: * "tsa = timeseries()" connects to "local" (first entry) * "tsa = timeseries('postgresql:///refinery_staging')" connects to staging * Useful for testing migrations and data comparisons Data Mesh Configuration with Client Authentication -------------------------------------------------- Production setup consuming data from external refinery instances: [dburi] company = postgresql://user:password@db.company.com:5432/refinery [sources] company.weather = https://refinery.weather.partner.com/api, tsh company.market = https://refinery.market.partner.com/api, tsh [auth] # Authentication for weather source (matches [sources] entry) weather.uri = https://refinery.weather.partner.com/api weather.login = your_username weather.password = your_password # Authentication for market source (matches [sources] entry) market.uri = https://refinery.market.partner.com/api market.client_id = your_client_id market.client_secret = your_client_secret market.domain = auth.market.partner.com/realms/api **Key Point:** * "[auth]" entries must correspond to sources defined in "[sources]" or "[dburi]" * "[auth]" is for authenticating as a **client** to external services Enterprise Server Authentication Configuration ---------------------------------------------- Configuration for when your refinery acts as a **server** requiring authentication: [dburi] production = postgresql://user:password@db.internal:5432/refinery_prod [server-auth] client_id = your_server_client_id client_secret = your_server_client_secret domain = openid.yourprovider.com/realms/your_realm authorize_uri = https://your.refinery.com/authorize audience = https://your.refinery.com/api **Key Points:** * "[server-auth]" configures your refinery instance as an OAuth2 **server** * Used when other clients need to authenticate to access your refinery * Different from "[auth]" which is for authenticating to external services * Requires OpenID Connect provider configuration =============================================================================== # https://refinery.docs.pythonian.fr/best_practices.html =============================================================================== Best Practices ************** This guide provides best practices for working effectively with the Timeseries Refinery, based on production experience and proven patterns. Naming Conventions ================== **Series Naming Guidelines** Follow a consistent hierarchical structure using dots as separators: domain.category.subcategory.source.location.unit.frequency Examples: * "energy.electricity.price.spot.france.eur_mwh.h" - Hourly French electricity spot prices * "weather.temperature.air.meteo_france.paris.celsius.d" - Daily temperature in Paris * "finance.fx.rate.ecb.eur_usd.rate.d" - Daily EUR/USD exchange rate **Guidelines:** * Use lowercase letters and underscores for multi-word components * Keep names descriptive but concise (aim for 6-8 components maximum) * Place most general categories first, most specific last * Include units and frequency when relevant * Avoid abbreviations unless they are standard in your domain **Metadata Naming Standards** Use consistent metadata keys across your organization: **Standard Keys:** * "source" - Data provider or system of origin * "unit" - Measurement unit * "frequency" - Native data frequency * "geography" - Geographic scope or location * "category" - Business domain classification * "quality" - Data quality indicators * "contact" - Responsible person or team **Guidelines:** * Use snake_case for metadata keys * Prefer established vocabularies when possible * Document your metadata schema * Keep values consistent (use controlled vocabularies) * **Metadata keys should remain stable** - avoid frequently changing metadata values * Use "tsa.insertion_dates()" to get timing information instead of storing it in metadata * Per update metadata can be provided: "tsa.update(name, series, author, metadata={...})" to document what is going on in a specific revision Data Governance =============== **Team Collaboration Guidelines** **Establish Clear Ownership:** * Assign data stewards for each domain or category * Use the "contact" metadata field: "tsa.update_metadata('series', {'contact': 'energy.team@company.com'})" * Find series by owner: "tsa.find('(by.metaitem "contact" "energy.team")')" * Use supervision with "tsa.update('series', data, 'author', manual=True)" to track manual interventions **Communication Protocols:** * All formula changes, updates, and metadata modifications are automatically logged * Use "tsa.history('series', diffmode=True)" (sparingly though, it is an expensive api point) to see what changed between versions * Set up "tswatch" alerts for critical series that stop updating * Use the web UI's series browser to explore dependencies before changes **Change Management Process** **Before Making Changes:** * Test formulas with "tsa.eval_formula('(+ (series "a") (series "b"))')" before registering * Check "tsa.formula_depth('complex_formula')" to understand computational complexity * Use the formula editor in the web UI for validation and testing **Implementation:** * Formula registration is automatically versioned: "tsa.register_formula('name', 'new_formula')" * Use cache policies for performance - see Formulas: when to use a cache/materialized view * Leverage the rework task system for scheduled updates - see Tasks system: how to organize and schedule tasks * Use the mini scraping framework to link scrapers to tasks and series (see "scrap.py" and "refresh" task) **After Changes:** * Use "tsa.get('series', revision_date=timestamp)" to compare before/after states * Update dashboard configurations if series structure changed * Monitor cache performance and policies **Data Quality Standards** **Validation Using Refinery Features:** * Use "tsa.supervision_status('series')" to check if manual overrides exist * Implement quality checks in rework tasks that run on schedule * Use "tsa.edited('series')" to identify series with manual interventions * Store quality indicators in metadata: "{'quality': 'validated', 'source': 'verified'}" **Audit Trail Management:** * Every "tsa.update()", "tsa.update_metadata()", and "tsa.replace_metadata()" is automatically logged * Use "tsa.log('series')" to see change history (and per-update metadata) * Supervision via "manual=True" maintains audit trail for corrections * "tsa.insertion_dates('series')" shows when data was added to the system **Data Lineage:** * Use "tsa.formula('computed_series')" to see formula definition * "tsa.source('series')" identifies the database source * Formula dependencies are tracked automatically * Web UI provides visual dependency graphs for complex formulas Formula Development =================== **Data Medallion Architecture for Formulas** **Bronze Layer - Raw Ingestion:** * Direct from sources: "energy.prices.nordpool.raw", "weather.meteo.paris.raw" * No processing, preserve original structure and timestamps **Silver Layer - Cleaned and Standardized:** * Handle data quality issues here: missing values, duplicates, basic validation * Resampling to standard frequencies happens here, example: "(resample (series "energy.prices.raw") "H")" * Standardized units, timezone-aware, validated * Outlier removal, gap filling * Business rule application: "(slice ... #:from (date "2020-01-01"))" for data quality cutoffs **Gold Layer - Business Logic:** * "energy.daily_average_price" - business KPIs and aggregations, ML model inputs * "energy.price_forecast" - ML model outputs * "trading.settlement_prices" - complex business calculations * Cross-domain joins and enrichment **Platinum Layer - Presentation:** * "dashboard.energy.price_summary" - optimized for specific dashboards * "api.energy.latest_prices" - formatted for external APIs * User-specific views and permissions **Formula Composition Strategies by Layer** **Bronze → Silver Transformations:** * Focus on data quality * Standardization * Basic gap filling * Resampling for stable time granularities **Silver → Gold Business Logic:** * Domain calculations: "(/ (+ (series "clean.demand") (series "clean.losses")) (series "clean.capacity"))" * Aggregations (by geography or other domains) at different levels * Cross-referencing: "(priority (series "validated") (series "estimated"))" **Gold → Platinum Optimization:** * Performance caching for heavy calculations * User-specific filters and permissions * Dashboard-optimized time ranges and granularity **Production Architecture Patterns** **The "Source of Truth" Pattern:** * Each business concept has ONE gold-layer source of truth * All downstream uses reference this canonical series * Example: "computed.energy.official_price" used by all trading, reporting, and billing systems **The "Temporal Consistency" Pattern:** * Maintain consistent time horizons across related series * "computed.energy.rolling_30d_average" and "computed.energy.rolling_30d_volatility" * Use shared time windows: "(rolling ... #:window "30D" #:center False)" **The "Lineage Preservation" Pattern:** * Embed source attribution in formula names * "computed.energy.price.from_nordpool_entsoe" vs "computed.energy.price.from_local_market" * Completes tracking data provenance through the medallion layers **Anti-Patterns from Production Experience** **The "Layer Bypass" Anti-Pattern:** * Gold formulas directly reading raw data: "(series "messy_data.raw")" * Skips cleaning and validation, leads to hazardous results * Always flow through the medallion: raw → clean → computed **The "Mixed-Layer Formula" Anti-Pattern:** * One formula mixing concerns: cleaning + business logic + presentation formatting * Makes debugging and maintenance difficult * Keep each formula focused on one medallion layer's responsibilities =============================================================================== # https://refinery.docs.pythonian.fr/operations/dual_storage.html =============================================================================== Dual Storage Operations *********************** Since version 1.0 (pro edition), the series values can live on either of two storage backends: * "pg" - the historical PostgreSQL backend * "fs1" - a high-performance filesystem backend Each series individually carries its backend. The catalog and the metadata always stay in PostgreSQL, whatever the backend of the values. Both kinds coexist transparently in the same instance: the API, the formulas, the supervision and the web interface behave identically. Configuration ============= The "[storage]" section of "tshistory.cfg" provides the fs1 root directory (mandatory) and optionally the default backend for newly created series. See the configuration reference. Moving series between backends ============================== The "change-backend" command schedules the migration of every series and group of a namespace towards a target backend: tsh change-backend --target fs1 The migration runs through the task queue, series by series, in a chunk-streamed fashion: the platform keeps running while it proceeds, and several workers can consume the queue concurrently. Series already migrated (or already queued) are skipped, so the command can be re-run safely to resume an interrupted campaign. A pending campaign can be cancelled - series not yet picked by a worker stay on their current backend: tsh cancel-backend-change A cancelled campaign can be relaunched later with "tsh resume-backend- change ". Operational notes ================= * The fs1 directory belongs in your backup perimeter, next to the database dumps. * fs1 files are append-only; revisions are fsynced before being committed. * The "--target" option also accepts "pg", so a campaign can be reversed. =============================================================================== # https://refinery.docs.pythonian.fr/operations/replication.html =============================================================================== Series replication ****************** Federation (the "[sources]" mechanism, see Securing the Refinery) reads remote series live, on every request. Replication is the *pull* alternative: two rework tasks copy series from a remote refinery — revision history included — into the local instance, where they become ordinary primary series. replicate_series_from_refinery ============================== Replicates a single series. Its inputs: * "url_refinery_origin" — the http uri of the origin instance; * "seriesname_origin" — the series to replicate; * "seriesname_target" — the local name (defaults to the origin name); * "from_insertion_date" — where to start in the origin revision history (optional; a moment expression); * "insertion_freq_offset" — thin the replicated history to at most one revision per given pandas offset (e.g. "D"), when the origin history is denser than needed (optional). The task is incremental: a re-run only fetches the revisions newer than the last replicated one. Scheduling it hence keeps the local copy current. The origin metadata is copied over as well, replacing whatever the local copy carried: metadata edited on the copy does not survive the next run. replicate_from_basket ===================== The same, for every series yielded by a basket of the origin instance: * "url_refinery_origin" and "basket_name" select the series; * "prefix" / "suffix" decorate the local names, to give the copies a namespace of their own; * "from_insertion_date" and "insertion_freq_offset" behave as above. Both tasks run in the "timeseries" domain and can be launched or scheduled from the tasks board like any other task (see To the next level). =============================================================================== # https://refinery.docs.pythonian.fr/operations/formula_debugging.html =============================================================================== Formula Debugging Techniques **************************** While the Refinery formula system catches syntax and type errors automatically, the most challenging debugging scenarios involve **semantic issues** - formulas that are syntactically correct but produce unexpected or wrong results. This guide focuses on systematic approaches to diagnose and fix these semantic problems. Table of Contents ================= * Common Semantic Issues * Progressive Debugging Strategy * Data Alignment Problems * Missing Data Propagation * Date and Time Semantic Errors * Formula Dependencies and Recursion * Understanding the Reactive System Common Semantic Issues ====================== **The formula works but the results are wrong.** Here are the most frequent semantic problems: 1. **Data Alignment Issues** - Series with different frequencies or date ranges 2. **Missing Data Propagation** - Holes in data becoming NaNs when series combine 3. **TZ-Aware vs TZ-Naive Incompatibility** - Cannot mix timestamped and date-based series 4. **Time Shifting for Business Periods** - Business days don't align with calendar boundaries 5. **Circular Dependencies** - Formulas depending on each other creating loops 6. **Deep Formula Performance** - Complex formulas becoming slow or hard to debug Progressive Debugging Strategy ============================== Using the Formula Editor ------------------------ **The formula editor is your primary tool for incremental formula construction and debugging.** It provides real-time validation, immediate visual feedback, and powerful debugging capabilities that make semantic issue detection much easier. [image] **Key Capabilities:** **Dual Editor Interface** The formula editor provides both a text editor with ACE editor integration for code-based formula editing and a visual tree editor for graphical formula construction. Changes in either editor automatically update the other in real-time, allowing you to choose the interface that best fits your workflow while maintaining perfect synchronization between both views. **Real-time Formula Evaluation** Formulas are automatically evaluated 2 seconds after your last edit, with results plotted immediately as you type. Syntax and evaluation errors are highlighted inline using ACE editor annotations, while loading states provide visual feedback during formula evaluation. This creates an immediate feedback loop that accelerates the debugging process. **Interactive Formula Building** You can build complex formulas incrementally, with each change immediately reflected in the plot visualization. The tree editor provides structured formula building with context-aware editing assistance, while real-time syntax checking and type validation catch errors as you work. **Advanced Editor Features** The editor includes full undo/redo history tracking for both text and tree editors, allowing you to experiment freely and revert changes when needed. You can save and update formulas directly from the editor, with separate modes for series and group formula types. Protected formulas can be viewed in read-only mode for reference without modification risk. **Incremental Building Workflow in the formula editor:** ;; Example: Building a complex financial indicator step by step ;; Step 1: Start with basic data access (plot immediately visible) (series "close_price") ;; Step 2: Add simple transformation (see the shift effect) (time-shifted (series "close_price") #:days -1) ;; Step 3: Calculate returns (observe the scale change) (div (sub (series "close_price") (time-shifted (series "close_price") #:days -1)) (time-shifted (series "close_price") #:days -1)) ;; Step 4: Add smoothing (see noise reduction) (rolling (div (sub (series "close_price") (time-shifted (series "close_price") #:days -1)) (time-shifted (series "close_price") #:days -1)) 5) ;; Step 5: Final indicator with parameters you can adjust live (rolling (div (sub (series "close_price") (time-shifted (series "close_price") #:days -1)) (time-shifted (series "close_price") #:days -1)) 20) **Visual Debugging Benefits:** **Immediate Pattern Recognition** * Spot frequency mismatches by observing plot density * Detect missing data gaps visually * Identify outliers and anomalies in real-time * See aggregation effects on data distribution **Temporal Alignment Verification** * Overlay multiple series to check alignment * Zoom into specific periods to verify time shifts * Visual confirmation of resampling effects * Holiday/weekend gap analysis **Scale and Range Validation** * Immediate feedback on value ranges * Detect unit conversion issues * Spot normalization problems * Verify percentage vs ratio calculations **Common Debugging Patterns in the formula editor:** ;; Pattern 1: Missing data investigation (series "sensor_data") ;; See the gaps (series "sensor_data" #:fill "ffill") ;; See fill effect immediately ;; Pattern 2: Aggregation verification (resample (series "hourly_data") "D") ;; Check daily aggregation visually (resample (series "hourly_data") "D" #:method "sum") ;; Compare sum vs mean ;; Pattern 3: Rolling operation tuning (rolling (series "noisy_signal") 5) ;; Too little smoothing? (rolling (series "noisy_signal") 20) ;; Better smoothing (rolling (series "noisy_signal") 50) ;; Too much smoothing? Inspect Intermediate Results with python ---------------------------------------- # Check data properties at each step def debug_series(series, name): print(f"\n=== {name} ===") print(f"Length: {len(series)}") print(f"Date range: {series.index.min()} to {series.index.max()}") print(f"Frequency: {series.index.freq}") print(f"NaN count: {series.isna().sum()}") print(f"Value range: {series.min():.4f} to {series.max():.4f}") print(f"Sample values:\n{series.head()}") # Apply to each step returns = tsa.eval_formula('(series "daily_returns")') debug_series(returns, "Daily Returns") rolling_mean = tsa.eval_formula('(rolling (series "daily_returns") 252 #:method "mean")') debug_series(rolling_mean, "252-day Rolling Mean") Data Alignment Problems ======================= Series with Different Frequencies --------------------------------- **Problem**: Combining daily and monthly data without proper alignment. # ❌ PROBLEM: This will cause unexpected results problematic_formula = '(add (series "daily_sales") (series "monthly_budget"))' # The monthly budget gets broadcast/aligned in unpredictable ways result = tsa.eval_formula(problematic_formula) # Result: Data loss - operation succeeds but produces meaningless results **Debugging approach:** # Inspect the frequency mismatch daily_sales = tsa.eval_formula('(series "daily_sales")') monthly_budget = tsa.eval_formula('(series "monthly_budget")') print(f"Daily sales frequency: {daily_sales.index.freq}") print(f"Monthly budget frequency: {monthly_budget.index.freq}") print(f"Daily sales length: {len(daily_sales)}") print(f"Monthly budget length: {len(monthly_budget)}") **Solutions:** ;; Solution: Aggregate daily to monthly (downsampling) (add (resample (series "daily_sales") "M" #:method "sum") (series "monthly_budget")) ;; Downsampling is reliable - aggregation works within known boundaries ;; ❌ Solution 3: NEVER DO THIS - Calendar approximation is dangerous! (add (series "daily_sales") (/ (series "monthly_budget") 30)) ;; Calendars are tricky: NEVER APPROXIMATE THEM! ;; This fails because months have different lengths (28-31 days) ;; February gets wrong allocation, leap years break it, etc. ;; ALWAYS use calendar-preserving operations like resample! Different Date Ranges --------------------- **Problem**: Series covering different time periods. # Debugging date range mismatches discontinued = tsa.eval_formula('(series "bloomberg_prices")') # 2010-2023, stopped feeding current = tsa.eval_formula('(series "reuters_prices")') # 2022-2024, actively updated print(f"Discontinued source: {discontinued.index.min()} to {discontinued.index.max()}") print(f"Current source: {current.index.min()} to {current.index.max()}") print(f"Overlap period: {max(discontinued.index.min(), current.index.min())} to {min(discontinued.index.max(), current.index.max())}") **Solution:** ;; Use priority to create seamless time series ;; Old source first, then newer source patches/extends it (priority (series "bloomberg_prices") (series "reuters_prices")) Date Range Slicing for Optimization ----------------------------------- Slicing can optimize data retrieval and avoid unnecessary overlap: ;; Efficient source combination: use Reuters only after Bloomberg ends ;; This minimizes database queries by avoiding the 2022-2023 overlap (priority (series "bloomberg_prices") (slice (series "reuters_prices") #:fromdate (date "2023-01-01"))) ;; Result: Bloomberg data through 2022, Reuters from 2023 onwards ;; Avoids fetching overlapping data from both sources Missing Data Propagation ======================== Understanding Holes propagation ------------------------------- **Problem**: Holes in upstream data are propagated when series are combined. # Reality: "Holes" (missing timestamps) are the main issue, not stored NaNs sensor_a = tsa.eval_formula('(series "sensor_a")') # Has data Mon-Fri sensor_b = tsa.eval_formula('(series "sensor_b")') # Has data every day # When combined, holes in sensor_a become holes in the result combined = tsa.eval_formula('(add (series "sensor_a") (series "sensor_b"))') print(f"Sensor A points: {len(sensor_a)}") # e.g., 250 (weekdays only) print(f"Sensor B points: {len(sensor_b)}") # e.g., 365 (every day) print(f"Combined points: {len(combined)}") # 250 points # Note: Some operations can create holes: # - div by zero, log of negative values, rolling std with insufficient data # But the main challenge is managing holes in upstream series **Solutions:** ;; Use fill options strategically (add (series "a" #:fill "ffill") (series "b" #:fill "ffill")) ;; Use priority for fallback (priority (add (series "a") (series "b")) (series "backup_calculation")) ;; Control aggregation NaN handling (row-mean (series "sensor_1") (series "sensor_2") #:skipna #t) Date and Time Semantic Errors ============================= TZ-Aware vs TZ-Naive: A Fundamental Distinction ----------------------------------------------- **Context**: The Refinery uses two types of series for different purposes: * **TZ-aware series**: Timestamped measurements (sensors, market data) - stored in UTC * **TZ-naive series**: Period-based reference values (regulatory limits, accounting data) **Problem**: Comparing real-time measurements against daily regulatory thresholds. # Real scenario: pollution monitoring vs regulatory limits no2_readings = tsa.eval_formula('(series "no2_sensor_readings")') # TZ-aware (µg/m³ every hour) daily_limit = tsa.eval_formula('(series "no2_regulatory_limit")') # TZ-naive (daily threshold) print(f"Sensor readings tz: {no2_readings.index.tz}") # UTC print(f"Regulatory limit tz: {daily_limit.index.tz}") # None - just dates # This fails - cannot compare timestamped data with date-based limits try: exceedances = tsa.eval_formula( '(sub (series "no2_sensor_readings") (series "no2_regulatory_limit"))') except TypeError as e: print("Cannot mix tz-aware and tz-naive series!") # The semantic question: which timezone should define "a day" for the limit? **Solutions:** ;; Solution 1: Convert sensor data to naive ;; This interprets "a day" according to local calendar (sub (naive (series "no2_sensor_readings") "Europe/Paris") (series "no2_regulatory_limit")) ;; Solution 2: Aggregate hourly data to daily before comparison ;; This preserves tz-awareness but creates daily averages (sub (resample (series "no2_sensor_readings") "D" #:method "mean") (tzaware (series "no2_regulatory_limit") "Europe/Paris")) ;; The choice depends on regulatory interpretation: ;; - Option 1: Local calendar days ;; - Option 2: 24-hour periods in consistent time reference Time Shifting for Business Period Alignment ------------------------------------------- **Problem**: Business periods often don't align with calendar boundaries. ;; Example: Gas day runs 06:00 to 06:00 ;; Hourly gas flow data needs alignment to calendar days ;; Raw hourly data (series "gas_flow_m3h") ;; 2024-01-15 00:00 belongs to gas day 2024-01-14! ;; 2024-01-15 06:00 belongs to gas day 2024-01-15 ;; Shift by -6 hours to align gas day with calendar day (time-shifted (series "gas_flow_m3h") #:hours -6) ;; Now 2024-01-15 00:00 contains flow from 06:00 ;; Perfect for daily aggregation by calendar day Formula Dependencies and Recursion ================================== Deep Formula Performance ------------------------ **Problem**: Deeply nested formulas can be slow and hard to debug. # Check formula depth depth = tsa.formula_depth('complex_formula') if depth > 20: print(f"Warning: depth {depth} may impact performance") # For debugging, test at different expansion levels for level in [5, 10, 15]: partial = tsa.formula('complex_formula', level=level) result = tsa.eval_formula(partial) print(f"Level {level}: OK" if result is not None else f"Level {level}: Failed") **Debugging approach:** Deep formulas are naturally composed of many simpler formulas, each doing one job well. Use */tsinfo* interface to navigate the dependency tree and spot where problems occur (typically bad values or data holes at specific formula levels). **Performance Optimization** For formulas with performance issues, the Refinery provides a caching system that can materialize intermediate results. See Formulas: when to use a cache/materialized view for details on setting up cached series. **Using TSInfo Interface for Formula Analysis** For a deep formula: 1. Navigate to the series via the catalog search 2. Check the inline plot visualization - toggle "inferred freq" to see data holes as points 3. Review the Statistics panel for data quality metrics 4. Use the Cache tab to see cache status and policies 5. Enable "History mode" to compare revisions and debug temporal issues 6. Use the formula depth selector to examine different expansion levels Understanding the Reactive System --------------------------------- The Refinery implements functional reactive programming where formulas are pure functions computed lazily on demand. Understanding this helps debug unexpected behavior. **Key Properties:** **1. Lazy Evaluation** Updates store data but don't trigger computation: # Update primary series tsa.update('gas_injection_facility_001', new_data, 'operator') # Data is stored, but no formulas compute yet # Get a formula that depends on this data result = tsa.get('eu_energy_balance_optimized') # NOW the entire dependency graph computes # (assuming eu_energy_balance_optimized uses gas_injection_facility_001) **2. Time-Dependent Behavior** Formulas appear pure but *get()* results can change: result1 = tsa.get('deep_formula') tsa.update('base_series', new_data, 'operator') result2 = tsa.get('deep_formula') # result1 != result2 (underlying data changed) # But see point 4 - historical data eventually becomes immutable **3. The "Today" Operator** The *(today)* operator returns: - Current date when called without *revision_date* - The *revision_date* when provided ;; Formula taking last 30 days (rolling (slice (series "data") #:fromdate (shifted (today) #:days -30)) 7) This creates a moving window that follows either wall-clock time or revision time. **4. Eventual Purity** Historical data becomes immutable once "settled": # For audit date 2023-12-31, if no more data can be inserted before it: year_end = tsa.get('financial_metric', revision_date='2023-12-31') # This result is now permanent - perfect for compliance =============================================================================== # https://refinery.docs.pythonian.fr/troubleshooting.html =============================================================================== Troubleshooting *************** This section covers common issues and their solutions when working with the Timeseries Refinery. Installation Issues =================== **PostgreSQL Connection Failed** psycopg.OperationalError: could not connect to server * Verify PostgreSQL is running: "sudo systemctl status postgresql" * Check connection parameters in "tshistory.cfg" * Ensure database exists: "createdb my_time_series" * Test connection: "psql -d my_time_series" **Missing Dependencies** ImportError: No module named 'tshistory' * Ensure all dependencies are installed: "pip install tshistory_refinery --upgrade" * Check Python version compatibility (>= 3.10) * For older versions, see the installation guide for specific requirements Runtime Issues ============== **Series Not Found** * Series returns "None" when it doesn't exist * Verify series exists: "tsa.exists('my_series')" * Check for typos in series name * Use "tsa.find()" to find series * Ensure you're connected to the correct database **Formula Parsing Errors** SyntaxError: unbalanced parentheses * Check formula syntax - parentheses must be balanced * Verify operator names are correct * Use the formula editor in the web UI for validation * Check that referenced series exist Web Interface Issues ==================== **Authentication Failures** * Verify credentials in "tshistory.cfg" under "[auth]" section * For OAuth/OpenID Connect, contact Pythonian for enterprise deployment plans * Clear browser cache and cookies * Check server-side authentication logs Performance Issues ================== **Slow Formula Evaluation** * Enable caching for complex formulas by assigning them to a cache policy: "tsa.set_cache_policy('my_policy', ['my_formula'])" * See Formula cache for details * Check formula depth: "tsa.formula_depth('my_formula')" * Consider breaking complex formulas into simpler components **High Memory Usage** * Limit time ranges in queries * Use streaming operations for large datasets * Monitor Python process memory usage * Consider upgrading hardware for large deployments Migration Issues ================ **Migration Issues** * Always backup database before migration * Test migrations on staging environment first * Follow the step-by-step migration guide in the installation documentation * For complex scenarios, contact Pythonian for support Getting Help ============ If you encounter issues not covered here: 1. Check the server logs for detailed error messages 2. Visit the project documentation at https://refinery.docs.pythonian.fr 3. Contact support here https://heptapod.host/groups/pythonian /timeseries-refinery/-/issues, with: * Error message details * System configuration * Steps to reproduce the issue * Refinery version: "pip show tshistory_refinery" =============================================================================== # https://refinery.docs.pythonian.fr/api/index.html =============================================================================== API documentation ***************** All the api points are available through the *timeseries* object. As in: from tshistory.api import timeseries tsa = timeseries('http://refinery.datascience.com') ts = tsa.get('banana-spot-price') The available methods are the same and behave the same wether you use an http uri or a direct postgres uri. The methods description below appear to belong to the *mainsource* object, which talks directly to postgres. This is an unimportant implementation detail. * Base Series Operations * Search and Baskets * Groups * Supervision * Formulas * Excel * Dashboards * Formula cache * Tasks * Administration =============================================================================== # https://refinery.docs.pythonian.fr/api/base_series.html =============================================================================== Base Series Operations ********************** This constitutes the fundamental API to deal with series on an individual basis. +------------+--------------------------------------------------------------------------------------------+ | "get" | Get a series by name. | +------------+--------------------------------------------------------------------------------------------+ | "update" | Update a series named by with the input pandas series. | +------------+--------------------------------------------------------------------------------------------+ | "multi_up | Update several series at once. | | date" | | +------------+--------------------------------------------------------------------------------------------+ | "replace" | Replace a series named by with the input pandas series. | +------------+--------------------------------------------------------------------------------------------+ | "exists" | Checks the existence of a series with a given name. | +------------+--------------------------------------------------------------------------------------------+ | "source" | Provide the source name of a series. | +------------+--------------------------------------------------------------------------------------------+ | "delete" | Override to add dependency checking. | +------------+--------------------------------------------------------------------------------------------+ | "rename" | Rename a series. | +------------+--------------------------------------------------------------------------------------------+ | "type" | Return the type of a series, for instance 'primary' or 'formula'. | +------------+--------------------------------------------------------------------------------------------+ | "interval" | Return a pandas interval object which provides the smallest and highest value date of a | | | series. | +------------+--------------------------------------------------------------------------------------------+ | "inferred | Return a tuple of timedelta, float (between 0 and 1). | | _freq" | | +------------+--------------------------------------------------------------------------------------------+ | "insertio | Get the list of all insertion dates (as pandas timestamps). | | n_dates" | | +------------+--------------------------------------------------------------------------------------------+ | "history" | Get all versions of a series in the form of a dict from insertion dates to series version. | +------------+--------------------------------------------------------------------------------------------+ | "staircas | Compute a series whose value dates are the most recent constrained to be *delta* time | | e" | after the insertion dates of the series. | +------------+--------------------------------------------------------------------------------------------+ | "block_st | Staircase a series by block | | aircase" | | +------------+--------------------------------------------------------------------------------------------+ | "strip" | Remove revisions after a specific insertion date. | +------------+--------------------------------------------------------------------------------------------+ | "log" | Return a list of revisions for a given series, in reverse chronological order, with | | | filters. | +------------+--------------------------------------------------------------------------------------------+ | "metadata" | Return a series metadata dictionary. | +------------+--------------------------------------------------------------------------------------------+ | "internal | Return a series internal metadata dictionary. | | _metadata" | | +------------+--------------------------------------------------------------------------------------------+ | "replace_ | Replace a series metadata with a dictionary from strings to anything json-serializable. | | metadata" | | +------------+--------------------------------------------------------------------------------------------+ | "update_m | Update a series metadata with a dictionary from strings to anything json-serializable. | | etadata" | | +------------+--------------------------------------------------------------------------------------------+ | "old_meta | Get a list of the the older versions of the metadata. | | data" | | +------------+--------------------------------------------------------------------------------------------+ | "list_met | List available metadata keys | | adata_key | | | s" | | +------------+--------------------------------------------------------------------------------------------+ | "list_int | List available internal metadata keys | | ernal_met | | | adata_key | | | s" | | +------------+--------------------------------------------------------------------------------------------+ | "tree" | | +------------+--------------------------------------------------------------------------------------------+ | "path_ser | | | ies" | | +------------+--------------------------------------------------------------------------------------------+ | "series_p | | | ath" | | +------------+--------------------------------------------------------------------------------------------+ | "set_seri | | | es_path" | | +------------+--------------------------------------------------------------------------------------------+ | "delete_p | | | ath" | | +------------+--------------------------------------------------------------------------------------------+ | "rename_p | | | ath" | | +------------+--------------------------------------------------------------------------------------------+ | "info" | | +------------+--------------------------------------------------------------------------------------------+ | "sources" | | +------------+--------------------------------------------------------------------------------------------+ * tsa.get * "mainsource.get()" * tsa.update * "mainsource.update()" * tsa.multi_update * "mainsource.multi_update()" * tsa.replace * "mainsource.replace()" * tsa.exists * "mainsource.exists()" * tsa.source * "mainsource.source()" * tsa.delete * "mainsource.delete()" * tsa.rename * "mainsource.rename()" * tsa.type * "mainsource.type()" * tsa.interval * "mainsource.interval()" * tsa.inferred_freq * "mainsource.inferred_freq()" * tsa.insertion_dates * "mainsource.insertion_dates()" * tsa.history * "mainsource.history()" * tsa.staircase * "mainsource.staircase()" * tsa.block_staircase * "mainsource.block_staircase()" * tsa.strip * "mainsource.strip()" * tsa.log * "mainsource.log()" * tsa.metadata * "mainsource.metadata()" * tsa.internal_metadata * "mainsource.internal_metadata()" * tsa.replace_metadata * "mainsource.replace_metadata()" * tsa.update_metadata * "mainsource.update_metadata()" * tsa.old_metadata * "mainsource.old_metadata()" * tsa.list_metadata_keys * "mainsource.list_metadata_keys()" * tsa.list_internal_metadata_keys * "mainsource.list_internal_metadata_keys()" * tsa.tree * "mainsource.tree()" * tsa.path_series * "mainsource.path_series()" * tsa.series_path * "mainsource.series_path()" * tsa.set_series_path * "mainsource.set_series_path()" * tsa.delete_path * "mainsource.delete_path()" * tsa.rename_path * "mainsource.rename_path()" * tsa.info * "mainsource.info()" * tsa.sources * "mainsource.sources()" =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.get.html =============================================================================== tsa.get ******* mainsource.get(name, revision_date=None, from_value_date=None, to_value_date=None, inferred_freq=False, keepnans=False, **kw) Get a series by name. By default one gets the latest version. By specifying *revision_date* one can get the closest version matching the given date. The *from_value_date* and *to_value_date* parameters permit to specify a narrower date range (by default all points are provided). If the series does not exists, a None is returned. >>> tsa.get('my_series') ... 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 3.0 Name: my_series, dtype: float64 Parameters: * **name** (*str*) * **revision_date** (*datetime** | **None*) * **from_value_date** (*datetime** | **None*) * **to_value_date** (*datetime** | **None*) * **inferred_freq** (*bool*) * **keepnans** (*bool*) Return type: *Series* | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.update.html =============================================================================== tsa.update ********** mainsource.update(name, updatets, author, metadata=None, insertion_date=None, keepnans=False, **kw) Update a series named by with the input pandas series. This creates a new version of the series. Only the _changes_ between the last version and the provided series are part of the new version. A series made of the changed points is returned. If there was no change, an empty series is returned and no new version is created. New points are added, changed points are changed, points with NaN are dropped if *keepnans* is False (by default) or _erased_ if True. The *author* is mandatory. The *metadata* dictionary allows to associate any metadata with the new series revision. It is possible to force an *insertion_date*, which can only be higher than the previous *insertion_date*. >>> import pandas as pd >>> from tshistory.api import timeseries >>> >>> tsa = timeseries('postgres://me:password@localhost/mydb') >>> >>> series = pd.Series([1, 2, 3], ... pd.date_range(start=pd.Timestamp(2017, 1, 1), ... freq='D', periods=3)) # db insertion >>> tsa.update('my_series', series, 'babar@pythonian.fr') ... 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 3.0 Freq: D, Name: my_series, dtype: float64 Parameters: * **name** (*str*) * **updatets** (*Series*) * **author** (*str*) * **metadata** (*dict** | **None*) * **insertion_date** (*datetime** | **None*) * **keepnans** (*bool** | **None*) Return type: *Series* | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.multi_update.html =============================================================================== tsa.multi_update **************** mainsource.multi_update(author, updatemap, metadata=None, insertion_date=None, keepnans=False) Update several series at once. Each series in *updatemap* is independently updated: new points are added, changed points are changed, unchanged points are ignored. This is a bulk ingestion entry point, not a federation one: unlike *update*, it does not ask the secondary sources whether a series may be created here. *updatemap* is a dict of {name: pd.Series}. The *author* is mandatory and shared by all updates. The whole batch shares a single transaction: it lands entirely or not at all. >>> tsa.multi_update('babar@pythonian.fr', { ... 'temperature': temp_series, ... 'pressure': press_series, ... }) {} Parameters: * **author** (*str*) * **updatemap** (*dict*) * **metadata** (*dict** | **None*) * **insertion_date** (*datetime** | **None*) * **keepnans** (*bool** | **None*) Return type: dict =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.replace.html =============================================================================== tsa.replace *********** mainsource.replace(name, replacets, author, metadata=None, insertion_date=None, **kw) Replace a series named by with the input pandas series. This creates a new version of the series. The series is completely replaced with the provided values. The *author* is mandatory. The *metadata* dictionary allows to associate any metadata with the new series revision. It is possible to force an *insertion_date*, which can only be higher than the previous *insertion_date*. Parameters: * **name** (*str*) * **replacets** (*Series*) * **author** (*str*) * **metadata** (*dict** | **None*) * **insertion_date** (*datetime** | **None*) Return type: *Series* | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.exists.html =============================================================================== tsa.exists ********** mainsource.exists(name) Checks the existence of a series with a given name. Parameters: **name** (*str*) Return type: bool =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.source.html =============================================================================== tsa.source ********** mainsource.source(name) Provide the source name of a series. When coming from the main source, it returns 'local'. Parameters: **name** (*str*) Return type: str | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.delete.html =============================================================================== tsa.delete ********** mainsource.delete(name, force=False) Override to add dependency checking. Delete a series. By default, refuses to delete if the series is referenced by formulas. Use force=True to override this safety check. Parameters: * **name** (*str*) * **force** (*bool*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.rename.html =============================================================================== tsa.rename ********** mainsource.rename(name, newname, propagate=True, **_kw) Rename a series. The target name must be available. Parameters: * **name** (*str*) * **newname** (*str*) * **propagate** (*bool*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.type.html =============================================================================== tsa.type ******** mainsource.type(name) Return the type of a series, for instance 'primary' or 'formula'. Parameters: **name** (*str*) Return type: str =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.interval.html =============================================================================== tsa.interval ************ mainsource.interval(name) Return a pandas interval object which provides the smallest and highest value date of a series. Parameters: **name** (*str*) Return type: *Interval* =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.inferred_freq.html =============================================================================== tsa.inferred_freq ***************** mainsource.inferred_freq(name, revision_date=None, from_value_date=None, to_value_date=None) Return a tuple of timedelta, float (between 0 and 1). The timedelta represents the period (or 'freq' in pandas parlance) and the number the quality of the period, which may vary because of the irregularity of the series. Parameters: * **name** (*str*) * **revision_date** (*Timestamp** | **None*) * **from_value_date** (*Timestamp** | **None*) * **to_value_date** (*Timestamp** | **None*) Return type: *Tuple*[*Timedelta*, float] | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.insertion_dates.html =============================================================================== tsa.insertion_dates ******************* mainsource.insertion_dates(name, from_insertion_date=None, to_insertion_date=None, from_value_date=None, to_value_date=None, limit=None, **kw) Get the list of all insertion dates (as pandas timestamps). Parameters: * **name** (*str*) * **from_insertion_date** (*datetime** | **None*) * **to_insertion_date** (*datetime** | **None*) * **from_value_date** (*datetime** | **None*) * **to_value_date** (*datetime** | **None*) * **limit** (*int** | **None*) =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.history.html =============================================================================== tsa.history *********** mainsource.history(name, from_insertion_date=None, to_insertion_date=None, from_value_date=None, to_value_date=None, diffmode=False, keepnans=False, **kw) Get all versions of a series in the form of a dict from insertion dates to series version. It is possible to restrict the versions range by specifying *from_insertion_date* and *to_insertion_date*. It is possible to restrict the values range by specifying *from_value_date* and *to_value_date*. If *diffmode* is set to True, we don't get full series values between two consecutive insertion date but only the difference series (with new points, updated points and deleted points). This is typically more costly to compute but can be much more compact, and it encodes the same information as with *diffmode* set to False. >>> history = tsa.history('my_series') ... >>> >>> for idate, series in history.items(): # it's a dict ... print('insertion date:', idate) ... print(series) ... insertion date: 2018-09-26 17:10:36.988920+02:00 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 3.0 Name: my_series, dtype: float64 insertion date: 2018-09-26 17:12:54.508252+02:00 2017-01-01 1.0 2017-01-02 2.0 2017-01-03 7.0 2017-01-04 8.0 2017-01-05 9.0 Name: my_series, dtype: float64 Parameters: * **name** (*str*) * **from_insertion_date** (*datetime** | **None*) * **to_insertion_date** (*datetime** | **None*) * **from_value_date** (*datetime** | **None*) * **to_value_date** (*datetime** | **None*) * **diffmode** (*bool*) * **keepnans** (*bool*) Return type: *Dict*[*datetime*, *Series*] | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.staircase.html =============================================================================== tsa.staircase ************* mainsource.staircase(name, delta, from_value_date=None, to_value_date=None, revision_date=None, keepnans=False) Compute a series whose value dates are the most recent constrained to be *delta* time after the insertion dates of the series. This kind of query typically makes sense for forecast series where the relationship between insertion date and value date is sound. Parameters: * **name** (*str*) * **delta** (*timedelta*) * **from_value_date** (*datetime** | **None*) * **to_value_date** (*datetime** | **None*) * **revision_date** (*datetime** | **None*) * **keepnans** (*bool*) Return type: *Series* | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.block_staircase.html =============================================================================== tsa.block_staircase ******************* mainsource.block_staircase(name, from_value_date=None, to_value_date=None, revision_freq=None, revision_time=None, revision_tz='UTC', maturity_offset=None, maturity_time=None, revision_date=None, keepnans=False) Staircase a series by block This is a more sophisticated and controllable version of the *staircase* method. Computes a series rebuilt from successive blocks of history, each linked to a distinct revision date. The revision dates are taken at regular time intervals determined by *revision_freq*, *revision_time* and *revision_tz*. The time lag between revision dates and value dates of each block is determined by *maturity_offset* and *maturity_time*. name: str unique identifier of the series from_value_date: pandas.Timestamp from which values are retrieved to_value_date: pandas.Timestamp to which values are retrieved revision_freq: dict giving revision frequency, of which keys must be taken from ['years', 'months', 'weeks', 'bdays', 'days', 'hours', 'minutes', 'seconds'] and values as integers. Default is daily frequency, i.e. {'days': 1} revision_time: dict giving revision time, of which keys should be taken from ['year', 'month', 'day', 'weekday', 'hour', 'minute', 'second'] and values must be integers. It is only used for revision date initialisation. The next revision dates are then obtained by successively adding *revision_freq*. Default is {'hour': 0} revision_tz: str giving time zone in which revision date and time are expressed. Default is 'UTC' maturity_offset: dict giving time lag between each revision date and start time of related block values. Its keys must be taken from ['years', 'months', 'weeks', 'bdays', 'days', 'hours', 'minutes', 'seconds'] and values as integers. Default is {}, i.e. the revision date is the block start date maturity_time: dict fixing start time of each block, of which keys should be taken from ['year', 'month', 'day', 'weekday', 'hour', 'minute', 'second'] and values must be integers. The start date of each block is thus obtained by adding *maturity_offset* to revision date and then applying *maturity_time*. Default is {}, i.e. block start date is just the revision date shifted by *maturity_offset* Parameters: * **from_value_date** (*datetime** | **None*) * **to_value_date** (*datetime** | **None*) * **revision_freq** (*Dict**[**str**, **int**] **| **None*) * **revision_time** (*Dict**[**str**, **int**] **| **None*) * **revision_tz** (*str*) * **maturity_offset** (*Dict**[**str**, **int**] **| **None*) * **maturity_time** (*Dict**[**str**, **int**] **| **None*) * **revision_date** (*datetime** | **None*) * **keepnans** (*bool*) =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.strip.html =============================================================================== tsa.strip ********* mainsource.strip(name, insertion_date) Remove revisions after a specific insertion date. This is an irreversible operation. Parameters: * **name** (*str*) * **insertion_date** (*datetime*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.log.html =============================================================================== tsa.log ******* mainsource.log(name, limit=None, fromdate=None, todate=None) Return a list of revisions for a given series, in reverse chronological order, with filters. Revisions are dicts of: * rev: revision id (int) * author: author name * date: timestamp of the revision * meta: the revision metadata Parameters: * **name** (*str*) * **limit** (*int** | **None*) * **fromdate** (*Timestamp** | **None*) * **todate** (*Timestamp** | **None*) Return type: *List*[*Dict*[str, *Any*]] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.metadata.html =============================================================================== tsa.metadata ************ mainsource.metadata(name) Return a series metadata dictionary. Parameters: **name** (*str*) Return type: *Dict*[str, *Any*] | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.internal_metadata.html =============================================================================== tsa.internal_metadata ********************* mainsource.internal_metadata(name) Return a series internal metadata dictionary. Parameters: **name** (*str*) Return type: *Dict*[str, *Any*] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.replace_metadata.html =============================================================================== tsa.replace_metadata ******************** mainsource.replace_metadata(name, metadata, user='no-user') Replace a series metadata with a dictionary from strings to anything json-serializable. Parameters: * **name** (*str*) * **metadata** (*dict*) * **user** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.update_metadata.html =============================================================================== tsa.update_metadata ******************* mainsource.update_metadata(name, metadata, user='no-user') Update a series metadata with a dictionary from strings to anything json-serializable. Parameters: * **name** (*str*) * **metadata** (*dict*) * **user** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.old_metadata.html =============================================================================== tsa.old_metadata **************** mainsource.old_metadata(name) Get a list of the the older versions of the metadata. Parameters: **name** (*str*) Return type: *List*[*Tuple*[*Timestamp*, dict, str]] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.list_metadata_keys.html =============================================================================== tsa.list_metadata_keys ********************** mainsource.list_metadata_keys() List available metadata keys Return type: *List*[str] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.list_internal_metadata_keys.html =============================================================================== tsa.list_internal_metadata_keys ******************************* mainsource.list_internal_metadata_keys() List available internal metadata keys Return type: *List*[str] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.tree.html =============================================================================== tsa.tree ******** mainsource.tree() Return type: *List*[str] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.path_series.html =============================================================================== tsa.path_series *************** mainsource.path_series(path) Parameters: **path** (*str*) Return type: str | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.series_path.html =============================================================================== tsa.series_path *************** mainsource.series_path(name) Parameters: **name** (*str*) Return type: str | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.set_series_path.html =============================================================================== tsa.set_series_path ******************* mainsource.set_series_path(name, path) Parameters: * **name** (*str*) * **path** (*str** | **None*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.delete_path.html =============================================================================== tsa.delete_path *************** mainsource.delete_path(path) Parameters: **path** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.rename_path.html =============================================================================== tsa.rename_path *************** mainsource.rename_path(path, newpath) Parameters: * **path** (*str*) * **newpath** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.info.html =============================================================================== tsa.info ******** mainsource.info() =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.sources.html =============================================================================== tsa.sources *********** mainsource.sources() =============================================================================== # https://refinery.docs.pythonian.fr/api/search_and_baskets.html =============================================================================== Search and Baskets ****************** These methods permit to enumerate all know series, find them using sophisticated search criteria (by name, metadata key/value, source), and save searches as baskets. +------------+--------------------------------------------------------------------------------------------+ | "find" | Return a list of series descriptors matching the query. | +------------+--------------------------------------------------------------------------------------------+ | "catalog" | Produces a catalog of all series in the form of a mapping from source to a list of (name, | | | kind) pair. | +------------+--------------------------------------------------------------------------------------------+ | "register | Register a dynamic series basket using a search query. | | _basket" | | +------------+--------------------------------------------------------------------------------------------+ | "basket" | Returns the list of series descriptors associated with a basket. | +------------+--------------------------------------------------------------------------------------------+ | "basket_d | Returns the query string associated with a basket. | | efinition" | | +------------+--------------------------------------------------------------------------------------------+ | "list_bas | Return the list of available basket names. | | kets" | | +------------+--------------------------------------------------------------------------------------------+ | "rename_b | Rename a basket. | | asket" | | +------------+--------------------------------------------------------------------------------------------+ | "delete_b | Delete a basket. | | asket" | | +------------+--------------------------------------------------------------------------------------------+ * tsa.find * "mainsource.find()" * tsa.catalog * "mainsource.catalog()" * tsa.register_basket * "mainsource.register_basket()" * tsa.basket * "mainsource.basket()" * tsa.basket_definition * "mainsource.basket_definition()" * tsa.list_baskets * "mainsource.list_baskets()" * tsa.rename_basket * "mainsource.rename_basket()" * tsa.delete_basket * "mainsource.delete_basket()" =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.find.html =============================================================================== tsa.find ******** mainsource.find(query=None, limit=None, meta=False, sources=[], _source='local') Return a list of series descriptors matching the query. A series descriptor is a string-like object (exhibiting the series name) with additional attributes. If *meta* has been set to True, the .meta (for normal metadata) and .imeta (for internal metadata) fields will be populated (non None). Lastly, the .source and .kind attributes provides the series source and kind. Here is an example: tsa.find( '(by.and ' ' (by.tzaware)' ' (by.name "power capacity") ' ' (by.metakey "plant")' ' (by.not (by.or ' ' (by.metaitem "plant_type" "oil")' ' (by.metaitem "plant_type" "coal")))' ' (by.metaitem "unit" "mwh")' ' (by.metaitem "country" "fr"))' ) This builds a query for timezone aware series about french power plants (in mwh) which are not of the coal or oil fuel type. The following filters can be used from the search module: * by.tzaware: no parameter, yields time zone aware series names * by.name : takes a space separated string of word, yields series names containing the substrings (in order) * by.metakey : takes a string, strictly matches all series having this metadata key * by.metaitems : takes a string (key) and an str (or numerical) value and yields all series strictly matching this metadata item * by.and: takes a variable number of filters as above to combine them * by.or: takes a variable number of filters as above to combine them * by.not: produce the negation of a filter Also inequalities on metadata values can be used: * <, <=, >, >=, =: take a string key, a value (str or num) As in *(<= "max_capacity" 900)* Parameters: * **query** (*str** | **None*) * **limit** (*int** | **None*) * **meta** (*int** | **None*) * **sources** (*List**[**str**]*) * **_source** (*str** | **None*) Return type: *List*[*ts*] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.catalog.html =============================================================================== tsa.catalog *********** mainsource.catalog(allsources=True) Produces a catalog of all series in the form of a mapping from source to a list of (name, kind) pair. By default it provides the series from all sources. If *allsources* is False, only the main source is listed. Parameters: **allsources** (*bool*) Return type: *Dict*[*Tuple*[str, str], *List*[*Tuple*[str, str]]] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.register_basket.html =============================================================================== tsa.register_basket ******************* mainsource.register_basket(name, query, group=False) Register a dynamic series basket using a search query. The search query has the same specification as the .find(..., query) api call. Parameters: * **name** (*str*) * **query** (*str*) * **group** (*bool*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.basket.html =============================================================================== tsa.basket ********** mainsource.basket(name, limit=None, meta=False, sources=[], group=False) Returns the list of series descriptors associated with a basket. A series descriptor is a string-like object (exhibiting the series name) with additional attributes. If *meta* has been set to True, the .meta (for normal metadata) and .imeta (for internal metadata) fields will be populated (non None). Lastly, the .source and .kind attributes provides the series source and kind. Parameters: * **name** (*str*) * **limit** (*int** | **None*) * **meta** (*int** | **None*) * **sources** (*List**[**str**]*) * **group** (*bool*) Return type: *List*[*ts*] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.basket_definition.html =============================================================================== tsa.basket_definition ********************* mainsource.basket_definition(name, group=False) Returns the query string associated with a basket. Parameters: * **name** (*str*) * **group** (*bool*) Return type: str =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.list_baskets.html =============================================================================== tsa.list_baskets **************** mainsource.list_baskets(group=False) Return the list of available basket names. Parameters: **group** (*bool*) Return type: *List*[str] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.rename_basket.html =============================================================================== tsa.rename_basket ***************** mainsource.rename_basket(name, newname, group=False) Rename a basket. Parameters: * **name** (*str*) * **newname** (*str*) * **group** (*bool*) =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.delete_basket.html =============================================================================== tsa.delete_basket ***************** mainsource.delete_basket(name, group=False) Delete a basket. Parameters: **group** (*bool*) =============================================================================== # https://refinery.docs.pythonian.fr/api/groups.html =============================================================================== Groups ****** A group is a set of related series (e.g. the scenarios of a same signal) manipulated as a single object. Most operations available on individual series have a group counterpart, prefixed with "group_". +------------+--------------------------------------------------------------------------------------------+ | "group_ex | Checks the existence of a group with a given name. | | ists" | | +------------+--------------------------------------------------------------------------------------------+ | "group_ge | Get a group by name. | | t" | | +------------+--------------------------------------------------------------------------------------------+ | "group_up | Update a group named by with the input dataframe. | | date" | | +------------+--------------------------------------------------------------------------------------------+ | "group_re | Replace a group named by with the input dataframe. | | place" | | +------------+--------------------------------------------------------------------------------------------+ | "group_de | Override to add dependency checking. | | lete" | | +------------+--------------------------------------------------------------------------------------------+ | "group_re | Rename a group. | | name" | | +------------+--------------------------------------------------------------------------------------------+ | "group_ty | Return the type of a group, for instance 'primary', 'formula' or 'bound' | | pe" | | +------------+--------------------------------------------------------------------------------------------+ | "group_so | Provide the source name of a group. | | urce" | | +------------+--------------------------------------------------------------------------------------------+ | "group_in | Return the value date range of a group. | | terval" | | +------------+--------------------------------------------------------------------------------------------+ | "group_in | Return the inferred frequency of a group. | | ferred_fr | | | eq" | | +------------+--------------------------------------------------------------------------------------------+ | "group_in | Get the list of all insertion dates for any given group | | sertion_d | | | ates" | | +------------+--------------------------------------------------------------------------------------------+ | "group_hi | Get all versions of a group in the form of a dict from insertion dates to dataframe. | | story" | | +------------+--------------------------------------------------------------------------------------------+ | "group_lo | | | g" | | +------------+--------------------------------------------------------------------------------------------+ | "group_me | Return a group metadata dictionary. | | tadata" | | +------------+--------------------------------------------------------------------------------------------+ | "group_in | Return a group internal metadata dictionary. | | ternal_me | | | tadata" | | +------------+--------------------------------------------------------------------------------------------+ | "group_ol | Get a list of the the older versions of the metadata. | | d_metadat | | | a" | | +------------+--------------------------------------------------------------------------------------------+ | "update_g | Update a group metadata with a dictionary from strings to anything json-serializable. | | roup_meta | | | data" | | +------------+--------------------------------------------------------------------------------------------+ | "replace_ | Replace a group metadata with a dictionary from strings to anything json-serializable. | | group_met | | | adata" | | +------------+--------------------------------------------------------------------------------------------+ | "group_fi | Return a list of group descriptors matching the query. | | nd" | | +------------+--------------------------------------------------------------------------------------------+ | "group_ca | Produces a catalog of all groups in the form of a mapping from source to a list of (name, | | talog" | kind) pair. | +------------+--------------------------------------------------------------------------------------------+ * tsa.group_exists * "mainsource.group_exists()" * tsa.group_get * "mainsource.group_get()" * tsa.group_update * "mainsource.group_update()" * tsa.group_replace * "mainsource.group_replace()" * tsa.group_delete * "mainsource.group_delete()" * tsa.group_rename * "mainsource.group_rename()" * tsa.group_type * "mainsource.group_type()" * tsa.group_source * "mainsource.group_source()" * tsa.group_interval * "mainsource.group_interval()" * tsa.group_inferred_freq * "mainsource.group_inferred_freq()" * tsa.group_insertion_dates * "mainsource.group_insertion_dates()" * tsa.group_history * "mainsource.group_history()" * tsa.group_log * "mainsource.group_log()" * tsa.group_metadata * "mainsource.group_metadata()" * tsa.group_internal_metadata * "mainsource.group_internal_metadata()" * tsa.group_old_metadata * "mainsource.group_old_metadata()" * tsa.update_group_metadata * "mainsource.update_group_metadata()" * tsa.replace_group_metadata * "mainsource.replace_group_metadata()" * tsa.group_find * "mainsource.group_find()" * tsa.group_catalog * "mainsource.group_catalog()" =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_exists.html =============================================================================== tsa.group_exists **************** mainsource.group_exists(name) Checks the existence of a group with a given name. Parameters: **name** (*str*) Return type: bool =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_get.html =============================================================================== tsa.group_get ************* mainsource.group_get(name, revision_date=None, from_value_date=None, to_value_date=None) Get a group by name. By default one gets the latest version. By specifying *revision_date* one can get the closest version matching the given date. The *from_value_date* and *to_value_date* parameters permit to specify a narrower date range (by default all points are provided). If the group does not exists, a None is returned. Parameters: * **name** (*str*) * **revision_date** (*Timestamp** | **None*) * **from_value_date** (*Timestamp** | **None*) Return type: *DataFrame* | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_update.html =============================================================================== tsa.group_update **************** mainsource.group_update(name, df, author, metadata=None, insertion_date=None) Update a group named by with the input dataframe. This creates a new version of the group. The *author* is mandatory. The *metadata* dictionary allows to associate any metadata with the new group revision. It is possible to force an *insertion_date*, which can only be higher than the previous *insertion_date*. Parameters: * **name** (*str*) * **df** (*DataFrame*) * **author** (*str*) * **metadata** (*dict** | **None*) * **insertion_date** (*Timestamp** | **None*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_replace.html =============================================================================== tsa.group_replace ***************** mainsource.group_replace(name, df, author, metadata=None, insertion_date=None) Replace a group named by with the input dataframe. This creates a new version of the group. The group is completely replaced with the provided values. The *author* is mandatory. The *metadata* dictionary allows to associate any metadata with the new group revision. It is possible to force an *insertion_date*, which can only be higher than the previous *insertion_date*. Parameters: * **name** (*str*) * **df** (*DataFrame*) * **author** (*str*) * **metadata** (*dict** | **None*) * **insertion_date** (*Timestamp** | **None*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_delete.html =============================================================================== tsa.group_delete **************** mainsource.group_delete(name, force=False) Override to add dependency checking. Delete a group. By default, refuses to delete if the group is referenced by group formulas. Use force=True to override this safety check. Parameters: * **name** (*str*) * **force** (*bool*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_rename.html =============================================================================== tsa.group_rename **************** mainsource.group_rename(name, newname, propagate=True, **_kw) Rename a group. The target name must be available. Parameters: * **name** (*str*) * **newname** (*str*) * **propagate** (*bool*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_type.html =============================================================================== tsa.group_type ************** mainsource.group_type(name) Return the type of a group, for instance 'primary', 'formula' or 'bound' Parameters: **name** (*str*) Return type: str =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_source.html =============================================================================== tsa.group_source **************** mainsource.group_source(name) Provide the source name of a group. When coming from the main source, it returns 'local'. Parameters: **name** (*str*) Return type: str | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_interval.html =============================================================================== tsa.group_interval ****************** mainsource.group_interval(name) Return the value date range of a group. Parameters: **name** (*str*) Return type: *Interval* =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_inferred_freq.html =============================================================================== tsa.group_inferred_freq *********************** mainsource.group_inferred_freq(name, revision_date=None, from_value_date=None, to_value_date=None) Return the inferred frequency of a group. Parameters: * **name** (*str*) * **revision_date** (*Timestamp** | **None*) * **from_value_date** (*Timestamp** | **None*) * **to_value_date** (*Timestamp** | **None*) Return type: *Tuple*[*Timedelta*, float] | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_insertion_dates.html =============================================================================== tsa.group_insertion_dates ************************* mainsource.group_insertion_dates(name, from_insertion_date=None, to_insertion_date=None) Get the list of all insertion dates for any given group Parameters: * **name** (*str*) * **from_insertion_date** (*Timestamp** | **None*) * **to_insertion_date** (*Timestamp** | **None*) Return type: *List*[*Timestamp*] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_history.html =============================================================================== tsa.group_history ***************** mainsource.group_history(name, from_value_date=None, to_value_date=None, from_insertion_date=None, to_insertion_date=None) Get all versions of a group in the form of a dict from insertion dates to dataframe. It is possible to restrict the versions range by specifying *from_insertion_date* and *to_insertion_date*. It is possible to restrict the values range by specifying *from_value_date* and *to_value_date*. Parameters: * **name** (*str*) * **from_value_date** (*Timestamp** | **None*) * **to_value_date** (*Timestamp** | **None*) * **from_insertion_date** (*Timestamp** | **None*) * **to_insertion_date** (*Timestamp** | **None*) Return type: *Dict*[*Timestamp*, *DataFrame*] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_log.html =============================================================================== tsa.group_log ************* mainsource.group_log(name, limit=None, fromdate=None, todate=None) Parameters: * **name** (*str*) * **limit** (*int** | **None*) * **fromdate** (*Timestamp** | **None*) * **todate** (*Timestamp** | **None*) Return type: *List*[*Dict*[str, *Any*]] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_metadata.html =============================================================================== tsa.group_metadata ****************** mainsource.group_metadata(name) Return a group metadata dictionary. Parameters: **name** (*str*) Return type: *Dict*[str, *Any*] | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_internal_metadata.html =============================================================================== tsa.group_internal_metadata *************************** mainsource.group_internal_metadata(name) Return a group internal metadata dictionary. Parameters: **name** (*str*) Return type: *Dict*[str, *Any*] | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_old_metadata.html =============================================================================== tsa.group_old_metadata ********************** mainsource.group_old_metadata(name) Get a list of the the older versions of the metadata. Parameters: **name** (*str*) Return type: *List*[*Tuple*[*Timestamp*, dict, str]] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.update_group_metadata.html =============================================================================== tsa.update_group_metadata ************************* mainsource.update_group_metadata(name, meta, user='no-user') Update a group metadata with a dictionary from strings to anything json-serializable. Parameters: * **name** (*str*) * **meta** (*Dict**[**str**, **Any**]*) * **user** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.replace_group_metadata.html =============================================================================== tsa.replace_group_metadata ************************** mainsource.replace_group_metadata(name, meta, user='no-user') Replace a group metadata with a dictionary from strings to anything json-serializable. Parameters: * **name** (*str*) * **meta** (*Dict**[**str**, **Any**]*) * **user** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_find.html =============================================================================== tsa.group_find ************** mainsource.group_find(query=None, limit=None, meta=False, sources=[], _source='local') Return a list of group descriptors matching the query. A series descriptor is a string-like object (exhibiting the series name) with additional attributes. If *meta* has been set to True, the .meta (for normal metadata) and .imeta (for internal metadata) fields will be populated (non None). Lastly, the .source and .kind attributes provides the series source and kind. Here is an example: tsa.group_find( '(by.and ' ' (by.tzaware)' ' (by.name "power capacity") ' ' (by.metakey "plant")' ' (by.not (by.or ' ' (by.metaitem "plant_type" "oil")' ' (by.metaitem "plant_type" "coal")))' ' (by.metaitem "unit" "mwh")' ' (by.metaitem "country" "fr"))' ) The following filters can be used from the search module: * by.tzaware: no parameter, yields time zone aware series names * by.name : takes a space separated string of word, yields series names containing the substrings (in order) * by.metakey : takes a string, strictly matches all series having this metadata key * by.metaitems : takes a string (key) and an str (or numerical) value and yields all series strictly matching this metadata item * by.and: takes a variable number of filters as above to combine them * by.or: takes a variable number of filters as above to combine them * by.not: produce the negation of a filter Also inequalities on metadata values can be used: * <, <=, >, >=, =: take a string key, a value (str or num) As in *(<= "max_capacity" 900)* Parameters: * **query** (*str** | **None*) * **limit** (*int** | **None*) * **meta** (*int** | **None*) * **sources** (*List**[**str**]*) * **_source** (*str** | **None*) Return type: *List*[*ts*] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.group_catalog.html =============================================================================== tsa.group_catalog ***************** mainsource.group_catalog(allsources=True) Produces a catalog of all groups in the form of a mapping from source to a list of (name, kind) pair. Parameters: **allsources** (*bool*) Return type: *Dict*[*Tuple*[str, str], *List*[*Tuple*[str, str]]] =============================================================================== # https://refinery.docs.pythonian.fr/api/supervision.html =============================================================================== Supervision *********** The supervision feature exposes two API points for stored series. +------------+--------------------------------------------------------------------------------------------+ | "edited" | Returns the base series and a second boolean series whose entries indicate if an override | | | has been made or not. | +------------+--------------------------------------------------------------------------------------------+ | "supervis | Returns the supervision status of a series. | | ion_statu | | | s" | | +------------+--------------------------------------------------------------------------------------------+ * tsa.edited * "mainsource.edited()" * tsa.supervision_status * "mainsource.supervision_status()" =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_supervision.api.mainsource.edited.html =============================================================================== tsa.edited ********** mainsource.edited(name, revision_date=None, from_value_date=None, to_value_date=None, inferred_freq=False, keepnans=False) Returns the base series and a second boolean series whose entries indicate if an override has been made or not. Parameters: * **name** (*str*) * **revision_date** (*Timestamp** | **None*) * **from_value_date** (*Timestamp** | **None*) * **to_value_date** (*Timestamp** | **None*) * **inferred_freq** (*bool** | **None*) * **keepnans** (*bool*) Return type: *Tuple*[*Series*, *Series*] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_supervision.api.mainsource.supervision_status.html =============================================================================== tsa.supervision_status ********************** mainsource.supervision_status(name) Returns the supervision status of a series. Possible values are *unsupervised*, *handcrafted* and *supervised*. Parameters: **name** (*str*) Return type: str =============================================================================== # https://refinery.docs.pythonian.fr/api/formulas.html =============================================================================== Formulas ******** The formulas adds computed series to the system ; most previously seen API points work with them. What does not: update and replace (obviously, since formula are by construction a read-only) feature. In the future it is possible that these methods will be implemented with *override* semantics. +------------+--------------------------------------------------------------------------------------------+ | "register | Define a series as a named formula. | | _formula" | | +------------+--------------------------------------------------------------------------------------------+ | "formula" | Get the formula associated with a name. | +------------+--------------------------------------------------------------------------------------------+ | "oldformu | | | las" | | +------------+--------------------------------------------------------------------------------------------+ | "eval_for | Execute a formula on the spot. | | mula" | | +------------+--------------------------------------------------------------------------------------------+ | "formula_ | Compute the depth of a formula. | | depth" | | +------------+--------------------------------------------------------------------------------------------+ | "formula_ | Compute a mapping from series name (defined as formulas) to the names of the component | | component | series. | | s" | | +------------+--------------------------------------------------------------------------------------------+ | "depends" | | +------------+--------------------------------------------------------------------------------------------+ | "register | Define a group by association of an existing series formula and a *bindings* object. | | _formula_ | | | bindings" | | +------------+--------------------------------------------------------------------------------------------+ | "bindings | | | _for" | | +------------+--------------------------------------------------------------------------------------------+ | "register | Define a group as a named formula. | | _group_fo | | | rmula" | | +------------+--------------------------------------------------------------------------------------------+ | "group_fo | Get the group formula associated with a name. | | rmula" | | +------------+--------------------------------------------------------------------------------------------+ | "group_fo | | | rmula_dep | | | th" | | +------------+--------------------------------------------------------------------------------------------+ | "group_ev | Execute a group formula on the spot. | | al_formul | | | a" | | +------------+--------------------------------------------------------------------------------------------+ | "group_de | | | pends" | | +------------+--------------------------------------------------------------------------------------------+ * tsa.register_formula * "mainsource.register_formula()" * tsa.formula * "mainsource.formula()" * tsa.oldformulas * "mainsource.oldformulas()" * tsa.eval_formula * "mainsource.eval_formula()" * tsa.formula_depth * "mainsource.formula_depth()" * tsa.formula_components * "mainsource.formula_components()" * tsa.depends * "mainsource.depends()" * tsa.register_formula_bindings * "mainsource.register_formula_bindings()" * tsa.bindings_for * "mainsource.bindings_for()" * tsa.register_group_formula * "mainsource.register_group_formula()" * tsa.group_formula * "mainsource.group_formula()" * tsa.group_formula_depth * "mainsource.group_formula_depth()" * tsa.group_eval_formula * "mainsource.group_eval_formula()" * tsa.group_depends * "mainsource.group_depends()" =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_formula.api.mainsource.register_formula.html =============================================================================== tsa.register_formula ******************** mainsource.register_formula(name, formula, reject_unknown=True, user='no-user') Define a series as a named formula. tsa.register_formula('sales.eu', '(add (series "sales.fr") (series "sales.be"))') Parameters: * **name** (*str*) * **formula** (*str*) * **reject_unknown** (*bool*) * **user** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_formula.api.mainsource.formula.html =============================================================================== tsa.formula *********** mainsource.formula(name, display=True, expanded=False, remote=True, level=-1) Get the formula associated with a name. tsa.formula('sales.eu') ... '(add (series "sales.fr") (series "sales.be"))') Expanding means replacing all *series* expressions that are formulas with the formula contents. It can be all-or-nothing with the expanded parameter or asked for a defined level (stopping the expansion process). The maximum level can be obtained through the *formula_depth* api call. Parameters: * **name** (*str*) * **display** (*bool*) * **expanded** (*bool*) * **remote** (*bool*) * **level** (*int*) Return type: str | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_formula.api.mainsource.oldformulas.html =============================================================================== tsa.oldformulas *************** mainsource.oldformulas(name) Parameters: **name** (*str*) Return type: *List*[*Tuple*[str, *Timestamp*, str]] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_formula.api.mainsource.eval_formula.html =============================================================================== tsa.eval_formula **************** mainsource.eval_formula(formula, revision_date=None, from_value_date=None, to_value_date=None, tz=None, nocache=False) Execute a formula on the spot. tsa.eval_formula('(add (series "sales.fr") (series "sales.be"))') Parameters: * **formula** (*str*) * **revision_date** (*Timestamp** | **None*) * **from_value_date** (*Timestamp** | **None*) * **to_value_date** (*Timestamp** | **None*) * **nocache** (*bool*) Return type: *Series* =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_formula.api.mainsource.formula_depth.html =============================================================================== tsa.formula_depth ***************** mainsource.formula_depth(name) Compute the depth of a formula. The depth is the maximum number of formula series sub expressions that have to be traversed to get to the bottom. Parameters: **name** (*str*) =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_formula.api.mainsource.formula_components.html =============================================================================== tsa.formula_components ********************** mainsource.formula_components(name, expanded=False) Compute a mapping from series name (defined as formulas) to the names of the component series. If *expanded* is true, it will expand the formula before computing the components. Hence only "ground" series (stored or autotrophic formulas) will show up in the leaves. >>> tsa.formula_components('my-series') {'my-series': ['component-a', 'component-b']} >>> tsa.formula_components('my-series-2', expanded=True) {'my-series-2': [{'sub-component-1': ['component-a', 'component-b']}, 'component-b']} Parameters: * **name** (*str*) * **expanded** (*bool*) Return type: *Dict*[str, list] | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_formula.api.mainsource.depends.html =============================================================================== tsa.depends *********** mainsource.depends(name, direct=False, reverse=False, meta=False) Parameters: **name** (*str*) Return type: *List*[*ts*] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_formula.api.mainsource.register_formula_bindings.html =============================================================================== tsa.register_formula_bindings ***************************** mainsource.register_formula_bindings(groupname, formulaname, bindings) Define a group by association of an existing series formula and a *bindings* object. The designated series formula will be then interpreted as a group formula. And the bindings object provides mappings that tell which components of the formula are to be interpreted as groups. Given a formula named "form1": (add (series "foo") (series "bar") (series "quux")) ... where one wants to treat "foo" and "bar" as groups. The binding is expressed as a dataframe: binding = pd.DataFrame( [ ['foo', 'foo-group', 'group'], ['bar', 'bar-group', 'group'], ], columns=('series', 'group', 'family') ) The complete registration looks like: register_formula_bindings( 'groupname', 'form1', pd.DataFrame( [ ['foo', 'foo-group', 'group'], ['bar', 'bar-group', 'group'], ], columns=('series', 'group', 'family') )) Within a given family, all groups must have the same number of members (series) and the member roles are considered equivalent (e.g. meteorological scenarios). Parameters: * **groupname** (*str*) * **formulaname** (*str*) * **bindings** (*DataFrame*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_formula.api.mainsource.bindings_for.html =============================================================================== tsa.bindings_for **************** mainsource.bindings_for(name) Parameters: **name** (*str*) =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_formula.api.mainsource.register_group_formula.html =============================================================================== tsa.register_group_formula ************************** mainsource.register_group_formula(name, formula) Define a group as a named formula. You can use any operator (including those working on series) provided the top-level expression is a group. Parameters: * **name** (*str*) * **formula** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_formula.api.mainsource.group_formula.html =============================================================================== tsa.group_formula ***************** mainsource.group_formula(name, display=True, expanded=False, remote=True, level=-1) Get the group formula associated with a name. Parameters: * **name** (*str*) * **display** (*bool*) * **expanded** (*bool*) * **remote** (*bool*) * **level** (*int*) Return type: str | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_formula.api.mainsource.group_formula_depth.html =============================================================================== tsa.group_formula_depth *********************** mainsource.group_formula_depth(name) Parameters: **name** (*str*) Return type: int =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_formula.api.mainsource.group_eval_formula.html =============================================================================== tsa.group_eval_formula ********************** mainsource.group_eval_formula(formula, revision_date=None, from_value_date=None, to_value_date=None, tz=None) Execute a group formula on the spot. tsa.group_eval_formula('(group-add (group "group1") (group "group2"))') Parameters: * **formula** (*str*) * **revision_date** (*Timestamp** | **None*) * **from_value_date** (*Timestamp** | **None*) * **to_value_date** (*Timestamp** | **None*) Return type: *Series* =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_formula.api.mainsource.group_depends.html =============================================================================== tsa.group_depends ***************** mainsource.group_depends(name, direct=False, reverse=False, meta=False) Parameters: **name** (*str*) Return type: *List*[*ts*] =============================================================================== # https://refinery.docs.pythonian.fr/api/excel.html =============================================================================== Excel ***** The API points listed there are mostly for use by the Excel client. +------------+--------------------------------------------------------------------------------------------+ | "values_m | | | arkers_or | | | igins" | | +------------+--------------------------------------------------------------------------------------------+ * tsa.values_markers_origins * "mainsource.values_markers_origins()" =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_xl.api.mainsource.values_markers_origins.html =============================================================================== tsa.values_markers_origins ************************** mainsource.values_markers_origins(name, revision_date=None, from_value_date=None, to_value_date=None, delta=None) =============================================================================== # https://refinery.docs.pythonian.fr/api/dashboards.html =============================================================================== Dashboards ********** Note: This is an essential feature of the Timeseries Refinery. It is only accessible within the pro version. If interested, please visit: https://timeseries.pythonian.fr/#ouroffer. Dashboards let users assemble interactive visualizations (figures, dashboards, balances) from time series data. These API points are reachable through the "dashboard" attribute of the api object, e.g. "tsa.dashboard.get_dashboard(...)". +------------+--------------------------------------------------------------------------------------------+ | "update_f | Create or update a figure configuration. | | igure" | | +------------+--------------------------------------------------------------------------------------------+ | "figures_ | Return all figure names grouped by variant. | | catalog" | | +------------+--------------------------------------------------------------------------------------------+ | "figure_e | Return True if a figure with the given *name* exists. | | xists" | | +------------+--------------------------------------------------------------------------------------------+ | "get_figu | Return the Figure identified by *name*, or None if absent. | | re" | | +------------+--------------------------------------------------------------------------------------------+ | "figure_d | Return the names of dashboards that reference *name*. | | ependents" | | +------------+--------------------------------------------------------------------------------------------+ | "delete_f | Delete the figure identified by *name*. | | igure" | | +------------+--------------------------------------------------------------------------------------------+ | "rename_f | Rename a figure. | | igure" | | +------------+--------------------------------------------------------------------------------------------+ | "dashboar | Return a DashboardSummary list for all dashboards. | | ds_catalo | | | g" | | +------------+--------------------------------------------------------------------------------------------+ | "update_d | Create or update a dashboard configuration. | | ashboard" | | +------------+--------------------------------------------------------------------------------------------+ | "dashboar | Return True if a dashboard with the given *name* exists. | | d_exists" | | +------------+--------------------------------------------------------------------------------------------+ | "get_dash | Return the Dashboard identified by *name*, or None if absent. | | board" | | +------------+--------------------------------------------------------------------------------------------+ | "delete_d | Delete the dashboard identified by *name*; no-op if absent. | | ashboard" | | +------------+--------------------------------------------------------------------------------------------+ | "rename_d | Rename a dashboard. | | ashboard" | | +------------+--------------------------------------------------------------------------------------------+ | "balance_ | Return the names of all registered balance definitions, sorted. | | list" | | +------------+--------------------------------------------------------------------------------------------+ | "balance_ | Return the lisp source of the balance definition identified by *name*, or None if absent. | | formula" | | +------------+--------------------------------------------------------------------------------------------+ | "balance_ | Return True if a balance definition with the given *name* exists. | | exists" | | +------------+--------------------------------------------------------------------------------------------+ | "update_b | Create or update a balance definition. | | alance" | | +------------+--------------------------------------------------------------------------------------------+ | "rename_b | Rename a balance definition. | | alance" | | +------------+--------------------------------------------------------------------------------------------+ | "delete_b | Delete the balance definition identified by *name*; no-op if absent. | | alance" | | +------------+--------------------------------------------------------------------------------------------+ * tsa.dashboard.update_figure * "DashboardAPI.update_figure()" * tsa.dashboard.figures_catalog * "DashboardAPI.figures_catalog()" * tsa.dashboard.figure_exists * "DashboardAPI.figure_exists()" * tsa.dashboard.get_figure * "DashboardAPI.get_figure()" * tsa.dashboard.figure_dependents * "DashboardAPI.figure_dependents()" * tsa.dashboard.delete_figure * "DashboardAPI.delete_figure()" * tsa.dashboard.rename_figure * "DashboardAPI.rename_figure()" * tsa.dashboard.dashboards_catalog * "DashboardAPI.dashboards_catalog()" * tsa.dashboard.update_dashboard * "DashboardAPI.update_dashboard()" * tsa.dashboard.dashboard_exists * "DashboardAPI.dashboard_exists()" * tsa.dashboard.get_dashboard * "DashboardAPI.get_dashboard()" * tsa.dashboard.delete_dashboard * "DashboardAPI.delete_dashboard()" * tsa.dashboard.rename_dashboard * "DashboardAPI.rename_dashboard()" * tsa.dashboard.balance_list * "DashboardAPI.balance_list()" * tsa.dashboard.balance_formula * "DashboardAPI.balance_formula()" * tsa.dashboard.balance_exists * "DashboardAPI.balance_exists()" * tsa.dashboard.update_balance * "DashboardAPI.update_balance()" * tsa.dashboard.rename_balance * "DashboardAPI.rename_balance()" * tsa.dashboard.delete_balance * "DashboardAPI.delete_balance()" =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.update_figure.html =============================================================================== tsa.dashboard.update_figure *************************** DashboardAPI.update_figure(figure) Create or update a figure configuration. The figure is validated before being persisted: first by a structural check via *validate_for_render*, then for non-balance variants by rendering them through the plotter to catch deeper problems (missing series, bad parameters, ...). A *FigureError* is raised on any validation failure. tsa.dashboard.update_figure( Figure( name='french_prices', variant='timeseries', traces=[Trace(name='power.fr', style=Style(color='blue'))], layout=FigureLayout(title='French prices'), ) ) Parameters: **figure** (*Figure*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.figures_catalog.html =============================================================================== tsa.dashboard.figures_catalog ***************************** DashboardAPI.figures_catalog() Return all figure names grouped by variant. Return type: dict[str, list[str]] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.figure_exists.html =============================================================================== tsa.dashboard.figure_exists *************************** DashboardAPI.figure_exists(name) Return True if a figure with the given *name* exists. Parameters: **name** (*str*) Return type: bool =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.get_figure.html =============================================================================== tsa.dashboard.get_figure ************************ DashboardAPI.get_figure(name) Return the Figure identified by *name*, or None if absent. Parameters: **name** (*str*) Return type: *Figure* | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.figure_dependents.html =============================================================================== tsa.dashboard.figure_dependents ******************************* DashboardAPI.figure_dependents(name) Return the names of dashboards that reference *name*. Parameters: **name** (*str*) Return type: list[str] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.delete_figure.html =============================================================================== tsa.dashboard.delete_figure *************************** DashboardAPI.delete_figure(name, force=False) Delete the figure identified by *name*. If *force* is False (default), raises FigureError when the figure is referenced by any dashboard. If *force* is True, dereferences the figure from every referrer dashboard (rewriting their stored source) before deleting the figure. Parameters: * **name** (*str*) * **force** (*bool*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.rename_figure.html =============================================================================== tsa.dashboard.rename_figure *************************** DashboardAPI.rename_figure(old_name, new_name) Rename a figure. Propagates to dashboards that reference it (rewriting their stored source). Raises FigureError if *old_name* does not exist or *new_name* already exists. Parameters: * **old_name** (*str*) * **new_name** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.dashboards_catalog.html =============================================================================== tsa.dashboard.dashboards_catalog ******************************** DashboardAPI.dashboards_catalog() Return a DashboardSummary list for all dashboards. Return type: list[*DashboardSummary*] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.update_dashboard.html =============================================================================== tsa.dashboard.update_dashboard ****************************** DashboardAPI.update_dashboard(dashboard) Create or update a dashboard configuration. tsa.dashboard.update_dashboard( Dashboard( name='prices', title='Prices', tree=[...], ) ) Parameters: **dashboard** (*Dashboard*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.dashboard_exists.html =============================================================================== tsa.dashboard.dashboard_exists ****************************** DashboardAPI.dashboard_exists(name) Return True if a dashboard with the given *name* exists. Parameters: **name** (*str*) Return type: bool =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.get_dashboard.html =============================================================================== tsa.dashboard.get_dashboard *************************** DashboardAPI.get_dashboard(name) Return the Dashboard identified by *name*, or None if absent. Parameters: **name** (*str*) Return type: *Dashboard* | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.delete_dashboard.html =============================================================================== tsa.dashboard.delete_dashboard ****************************** DashboardAPI.delete_dashboard(name) Delete the dashboard identified by *name*; no-op if absent. Parameters: **name** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.rename_dashboard.html =============================================================================== tsa.dashboard.rename_dashboard ****************************** DashboardAPI.rename_dashboard(old_name, new_name) Rename a dashboard. Rewrites the stored source so its *def- dashboard* head matches the new name. Raises DashboardError if *old_name* does not exist or *new_name* already exists. Parameters: * **old_name** (*str*) * **new_name** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.balance_list.html =============================================================================== tsa.dashboard.balance_list ************************** DashboardAPI.balance_list() Return the names of all registered balance definitions, sorted. Return type: list[str] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.balance_formula.html =============================================================================== tsa.dashboard.balance_formula ***************************** DashboardAPI.balance_formula(name) Return the lisp source of the balance definition identified by *name*, or None if absent. Parameters: **name** (*str*) Return type: str | None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.balance_exists.html =============================================================================== tsa.dashboard.balance_exists **************************** DashboardAPI.balance_exists(name) Return True if a balance definition with the given *name* exists. Parameters: **name** (*str*) Return type: bool =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.update_balance.html =============================================================================== tsa.dashboard.update_balance **************************** DashboardAPI.update_balance(name, definition) Create or update a balance definition. The definition is validated before being persisted: parsed, type- checked against the balance DSL, and checked for dangling series/basket references. A *FigureError* is raised on any validation failure. tsa.dashboard.update_balance( 'test_balance', '(balance "Test"' ' (config "UTC" (period "Daily" "D" (today) (today)))' ' (series "power.fr" sum (draw_option "power")))' ) A figure then *references* the definition by name, rather than embedding it: tsa.dashboard.update_figure( Figure( name='balance_view', variant=Balance(source='(balance "test_balance" #:period "Daily")'), ) ) Parameters: * **name** (*str*) * **definition** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.rename_balance.html =============================================================================== tsa.dashboard.rename_balance **************************** DashboardAPI.rename_balance(old_name, new_name) Rename a balance definition. Propagates to figures that reference it (rewriting their stored *(balance ...)* source). Raises FigureError if *old_name* does not exist or *new_name* already exists. tsa.dashboard.rename_balance('old_bal', 'new_bal') Parameters: * **old_name** (*str*) * **new_name** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_dashboard.api.DashboardAPI.delete_balance.html =============================================================================== tsa.dashboard.delete_balance **************************** DashboardAPI.delete_balance(name) Delete the balance definition identified by *name*; no-op if absent. Raises FigureError if any figure still references it — those figures must be dereferenced or deleted first. Parameters: **name** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/formula_cache.html =============================================================================== Formula cache ************* The formula system allows to grow *very complicated* computed series (by building them bottom-up), which are by default computed on the fly. The downside can be sluggish performance as complex formulas read hundreds of base series and does computations on them. Hence it can be useful to put them into a "cache". +------------+--------------------------------------------------------------------------------------------+ | "new_cach | Create a cache policy. | | e_policy" | | +------------+--------------------------------------------------------------------------------------------+ | "edit_cac | Modify an existing cache policy (by name). | | he_policy" | | +------------+--------------------------------------------------------------------------------------------+ | "delete_c | Delete a cache policy (by name). | | ache_poli | | | cy" | | +------------+--------------------------------------------------------------------------------------------+ | "set_cach | Associate series with a cache policy. | | e_policy" | | +------------+--------------------------------------------------------------------------------------------+ | "unset_ca | Dis-associate series from a cache policy. | | che_polic | | | y" | | +------------+--------------------------------------------------------------------------------------------+ | "cache_fr | List the series that are available for association with a cache policy. | | ee_series" | | +------------+--------------------------------------------------------------------------------------------+ | "cache_po | Return a list of cache policies names. | | licies" | | +------------+--------------------------------------------------------------------------------------------+ | "cache_po | Return the list of series associated with a cache policy. | | licy_seri | | | es" | | +------------+--------------------------------------------------------------------------------------------+ | "cache_se | Return the cache policy of a series. | | ries_poli | | | cy" | | +------------+--------------------------------------------------------------------------------------------+ | "has_cach | Predicate to verify is a series formula has a cache. | | e" | | +------------+--------------------------------------------------------------------------------------------+ | "delete_c | Purge the cache of a formula. | | ache" | | +------------+--------------------------------------------------------------------------------------------+ | "refresh_ | | | series_po | | | licy_now" | | +------------+--------------------------------------------------------------------------------------------+ * tsa.new_cache_policy * "mainsource.new_cache_policy()" * tsa.edit_cache_policy * "mainsource.edit_cache_policy()" * tsa.delete_cache_policy * "mainsource.delete_cache_policy()" * tsa.set_cache_policy * "mainsource.set_cache_policy()" * tsa.unset_cache_policy * "mainsource.unset_cache_policy()" * tsa.cache_free_series * "mainsource.cache_free_series()" * tsa.cache_policies * "mainsource.cache_policies()" * tsa.cache_policy_series * "mainsource.cache_policy_series()" * tsa.cache_series_policy * "mainsource.cache_series_policy()" * tsa.has_cache * "mainsource.has_cache()" * tsa.delete_cache * "mainsource.delete_cache()" * tsa.refresh_series_policy_now * "mainsource.refresh_series_policy_now()" =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_refinery.api.mainsource.new_cache_policy.html =============================================================================== tsa.new_cache_policy ******************** mainsource.new_cache_policy(name, initial_revdate, look_before, look_after, revdate_rule, schedule_rule) Create a cache policy. Parameters: * **name** (*str*) * **initial_revdate** (*str*) * **look_before** (*str*) * **look_after** (*str*) * **revdate_rule** (*str*) * **schedule_rule** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_refinery.api.mainsource.edit_cache_policy.html =============================================================================== tsa.edit_cache_policy ********************* mainsource.edit_cache_policy(name, initial_revdate, look_before, look_after, revdate_rule, schedule_rule) Modify an existing cache policy (by name). Parameters: * **name** (*str*) * **initial_revdate** (*str*) * **look_before** (*str*) * **look_after** (*str*) * **revdate_rule** (*str*) * **schedule_rule** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_refinery.api.mainsource.delete_cache_policy.html =============================================================================== tsa.delete_cache_policy *********************** mainsource.delete_cache_policy(name) Delete a cache policy (by name). Parameters: **name** (*str*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_refinery.api.mainsource.set_cache_policy.html =============================================================================== tsa.set_cache_policy ******************** mainsource.set_cache_policy(policyname, seriesnames) Associate series with a cache policy. Parameters: * **policyname** (*str*) * **seriesnames** (*List**[**str**]*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_refinery.api.mainsource.unset_cache_policy.html =============================================================================== tsa.unset_cache_policy ********************** mainsource.unset_cache_policy(seriesnames) Dis-associate series from a cache policy. Parameters: **seriesnames** (*List**[**str**]*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_refinery.api.mainsource.cache_free_series.html =============================================================================== tsa.cache_free_series ********************* mainsource.cache_free_series(allsources=True) List the series that are available for association with a cache policy. Parameters: **allsources** (*bool*) =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_refinery.api.mainsource.cache_policies.html =============================================================================== tsa.cache_policies ****************** mainsource.cache_policies() Return a list of cache policies names. =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_refinery.api.mainsource.cache_policy_series.html =============================================================================== tsa.cache_policy_series *********************** mainsource.cache_policy_series(policyname) Return the list of series associated with a cache policy. Parameters: **policyname** (*str*) =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_refinery.api.mainsource.cache_series_policy.html =============================================================================== tsa.cache_series_policy *********************** mainsource.cache_series_policy(seriesname) Return the cache policy of a series. Parameters: **seriesname** (*str*) =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_refinery.api.mainsource.has_cache.html =============================================================================== tsa.has_cache ************* mainsource.has_cache(seriesname) Predicate to verify is a series formula has a cache. Parameters: **seriesname** (*str*) =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_refinery.api.mainsource.delete_cache.html =============================================================================== tsa.delete_cache **************** mainsource.delete_cache(seriesname) Purge the cache of a formula. Parameters: **seriesname** (*str*) =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory_refinery.api.mainsource.refresh_series_policy_now.html =============================================================================== tsa.refresh_series_policy_now ***************************** mainsource.refresh_series_policy_now(policyname) Parameters: **policyname** (*str*) =============================================================================== # https://refinery.docs.pythonian.fr/api/tasks.html =============================================================================== Tasks ***** Note: This is an essential feature of the Timeseries Refinery. It is only accessible within the pro version. If interested, please visit: https://timeseries.pythonian.fr/#ouroffer. Tasks let one define and schedule background jobs (e.g. scrapers, formula cache refreshes) running under the rework task queue. These API points are reachable through the "task" attribute of the api object, e.g. "tsa.task.schedule(...)". +------------+--------------------------------------------------------------------------------------------+ | "schedule" | Schedule an operation for immediate execution by a worker. | +------------+--------------------------------------------------------------------------------------------+ | "prepare" | Schedule an operation to be run periodically. | +------------+--------------------------------------------------------------------------------------------+ | "unprepar | Remove a periodically scheduled operation. | | e" | | +------------+--------------------------------------------------------------------------------------------+ | "tasks" | List the tasks, most recent first. | +------------+--------------------------------------------------------------------------------------------+ | "prepared" | List the periodically scheduled operations. | +------------+--------------------------------------------------------------------------------------------+ * tsa.task.schedule * "TaskAPI.schedule()" * tsa.task.prepare * "TaskAPI.prepare()" * tsa.task.unprepare * "TaskAPI.unprepare()" * tsa.task.tasks * "TaskAPI.tasks()" * tsa.task.prepared * "TaskAPI.prepared()" =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/refinery_pro.api.TaskAPI.schedule.html =============================================================================== tsa.task.schedule ***************** TaskAPI.schedule(opname, inputdata=None, domain=None, hostid=None, metadata=None) Schedule an operation for immediate execution by a worker. tsa.task.schedule( 'migrate_series_storage', {'series_name': 'a-series', 'target_backend': 'fs1'} ) File inputs are provided as bytes values. A *TaskError* is raised if the operation is unknown or the inputs do not match its specification. Parameters: * **opname** (*str*) * **inputdata** (*dict** | **None*) * **domain** (*str** | **None*) * **hostid** (*str** | **None*) * **metadata** (*dict** | **None*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/refinery_pro.api.TaskAPI.prepare.html =============================================================================== tsa.task.prepare **************** TaskAPI.prepare(opname, rule, inputdata=None, domain='default', hostid=None, metadata=None) Schedule an operation to be run periodically. tsa.task.prepare( 'migrate_series_storage', '5 * * * *', {'series_name': 'a-series', 'target_backend': 'fs1'} ) The rule is a standard five fields cron expression (minute, hour, day of the month, month, day of the week). The task fires on the zeroth second of the matching minute: rework takes a sixth leading field for the seconds but we do not expose it. A *TaskError* is raised if the operation is unknown, the rule is invalid or the inputs do not match the operation specification. Parameters: * **opname** (*str*) * **rule** (*str*) * **inputdata** (*dict** | **None*) * **domain** (*str*) * **hostid** (*str** | **None*) * **metadata** (*dict** | **None*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/refinery_pro.api.TaskAPI.unprepare.html =============================================================================== tsa.task.unprepare ****************** TaskAPI.unprepare(sid) Remove a periodically scheduled operation. tsa.task.unprepare(3) The id is the one carried by *prepared*. A *TaskError* is raised if no scheduled operation bears it. Parameters: **sid** (*int*) Return type: None =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/refinery_pro.api.TaskAPI.tasks.html =============================================================================== tsa.task.tasks ************** TaskAPI.tasks(domain=None, operation=None, status=None, limit=50) List the tasks, most recent first. tsa.task.tasks(operation='migrate_series_storage', status='failed') The status is one of *queued*, *running*, *done*, *failed*, *aborting* or *aborted*. It is computed from several columns, hence applied after the query: it filters the *limit* most recent tasks rather than selecting the *limit* most recent ones having that status. The *input* and *output* carry the non-file values only. Use the dedicated endpoints to fetch the file ones. Parameters: * **domain** (*str** | **None*) * **operation** (*str** | **None*) * **status** (*str** | **None*) * **limit** (*int*) Return type: list[dict] =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/refinery_pro.api.TaskAPI.prepared.html =============================================================================== tsa.task.prepared ***************** TaskAPI.prepared(domain=None, operation=None) List the periodically scheduled operations. tsa.task.prepared(operation='migrate_series_storage') The *id* is the one to be handed over to *unprepare*. The *input* carries the non-file values only. Parameters: * **domain** (*str** | **None*) * **operation** (*str** | **None*) Return type: list[dict] =============================================================================== # https://refinery.docs.pythonian.fr/api/administration.html =============================================================================== Administration ************** Note: These API points belong to the pro version. If interested, please visit: https://timeseries.pythonian.fr/#ouroffer. User roles (see Securing the Refinery) and storage backend management (see Dual Storage Operations). +------------+--------------------------------------------------------------------------------------------+ | "set_user | Assign a role to a user. | | _role" | | +------------+--------------------------------------------------------------------------------------------+ | "user_rol | Return the known users and their roles, as a list of (email, role) pairs sorted by email. | | es" | | +------------+--------------------------------------------------------------------------------------------+ | "change_b | Schedule the migration of every stored series and group towards the target storage backend | | ackend" | (*fs1* or *pg*). | +------------+--------------------------------------------------------------------------------------------+ * tsa.set_user_role * "mainsource.set_user_role()" * tsa.user_roles * "mainsource.user_roles()" * tsa.change_backend * "mainsource.change_backend()" =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.set_user_role.html =============================================================================== tsa.set_user_role ***************** mainsource.set_user_role(userid, role) Assign a role to a user. The role must be one of *admin*, *rw*, *ro* or *guest*. Assigning *guest* removes the user record, since a guest is nothing but an unknown user. =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.user_roles.html =============================================================================== tsa.user_roles ************** mainsource.user_roles() Return the known users and their roles, as a list of (email, role) pairs sorted by email. Guests do not appear: they are the users unknown to the system. =============================================================================== # https://refinery.docs.pythonian.fr/api/generated/tshistory.api.mainsource.change_backend.html =============================================================================== tsa.change_backend ****************** mainsource.change_backend(target) Schedule the migration of every stored series and group towards the target storage backend (*fs1* or *pg*). The migration runs through the task queue; items already on the target backend (or already queued) are skipped, so the call can be repeated to resume an interrupted campaign. Returns a dictionary with the found and newly queued counts for series and groups.