The Timeseries Refinery

The Timeseries Refinery is a tool that :

• Store versioned timeseries data (time travel capacity), provide fine-tuned backtesting capacity, ease data monitoring

• Centralize data from various sources: internal data silos - external APIs Data queries on the fly or scraped on schedule

• Automate data processes (ETL, predictions, computations, sanity checks, …)

• Aggregate and transform timeseries with formulas

• Build a common catalog -> shared (high level, curated) data referential

Content

• Getting Started
  • Installation
    • The different ways to install (pypi, docker)
    • Installing on Linux, Macos, Windows
    • Upgrading and migrating
    • Installing older versions, migrating from older versions
  • Package overview
    • Purpose
    • Usages
    • License
  • Working with the Refinery
    • Basics
      • General setup reminder
      • How to create / update / read “primary” time series
      • How to Exists / Rename / Delete time series
      • Going further: playing with revisions
      • Adding and fetching metadata
      • Using formulas : create a formula
      • Similarities and differences with primary series
      • Using the formula editor
      • Seeing all the formulas
    • Create a time series information system
      • Pipelines : starting with injecting data
      • Using the task manager
      • Writing and using scrapers for primary series
      • Building formulas on top to get curated series
      • Feeding a model with series, getting the output in the system, with the task manager
    • To the next level
      • Doing your own app using the Refinery
      • Data acquisition: a structured path
      • Performance aspects: stored time series and formulas
      • Formulas: when to use a cache/materialized view
      • How to cache formulas
      • Tasks system: how to organize and schedule tasks
      • Data architecture at scale: putting the Data Mesh to work
• Core Components
  • Timeseries Store Usage
    • Table of Contents
    • Creating a series
    • Updating a series
    • Working with versions
    • Point and version erasure
      • Point erasure with NaN
      • Version erasure with strip
    • Retrieving history
    • Working with metadata
    • Replacing a series entirely
    • Checking series existence
    • Renaming a series
    • Deleting a series
    • Finding series
    • Getting series information
    • Working with logs
    • Staircase operations
    • Time Series Operations API
  • Supervision
    • The supervision workflow
    • Making manual corrections
    • Viewing data provenance
    • Understanding supervision status
    • Supervision API Reference
  • Formulas (computed series)
    • Table of Contents
    • Introduction
    • Creating Your First Formula
      • Formula Language Basics
      • Updating a Formula
      • Deleting a Formula
    • Using Formula Series
      • Formula-Specific API Methods
      • Formula Dependencies
      • Formula Insertion Dates
      • Formula Evaluation Context
      • Additional Formula Methods
      • Renaming Series and Formula Propagation
      • Deleting Series Referenced in Formulas
      • Performance and Caching
    • Advanced: Creating Custom Operators
      • Declaring a new operator
      • More Transformation Examples
      • Autotrophic series operator
    • Formula Language Reference
    • Formula API Reference
  • Formula Language Reference
    • Table of Contents
    • Getting Started
      • Your First Formula
      • Basic Syntax Rules
    • Language Basics
      • Data Types
    • Series Access and Options
      • Basic Series Access
      • Series Options
      • Options Operator
    • Arithmetic and Mathematical Operations
      • Scalar Operations
      • Series-to-Series Operations
      • Mathematical Functions
      • Trigonometric Functions
    • Temporal Operations
      • Resampling (Downsampling)
      • Upsampling (Increasing Frequency)
      • Time Shifting
      • Rolling Operations
      • Cumulative Operations
      • Integration
      • Block Staircase (Future Slicing)
    • Aggregation and Statistics
      • Row-wise Operations
      • Day-of-Year Aggregation (Folded Series)
      • Practical Aggregation Examples
    • Data Processing and Filtering
      • Conditional and Comparison Operations
      • Clipping and Bounds
      • Priority and Layering
      • Date-based Slicing
      • Date-based Filtering with Cron Rules
      • Revision Date Control
    • Date and Time Functions
      • Current Date and Time
      • Date Creation
      • Date Arithmetic
      • Month Boundary Functions
      • Timezone Operations
      • Holiday Calendar Series
    • Special Series Generation
    • Advanced Series Discovery
      • Dynamic Series Selection
      • Basket-based Operations
  • Groups
    • Table of Contents
    • Introduction and Stored Groups
    • Group Formulas
      • Group Formula Operators
      • Creating and Using Group Formulas
      • Group Formula Metadata
      • Formula-Specific Methods
      • Formula Bindings: Creating Groups from Series Formulas
    • Common Use Cases
    • Group API Reference
      • Primary Group Operations
      • Formula Group Operations
  • Search Query Language Reference
    • Table of Contents
    • Quick Start
      • Your First Search
      • More find API options
      • Finding Groups
      • Saving Searches with Baskets
    • Query Syntax
      • Structure
      • Basic Syntax Rules
    • Basic Search Operators
      • Name-based Search: by.name
      • Metadata Key Search: by.metakey
      • Everything Operator: by.everything
      • Timezone-Aware Filter: by.tzaware
      • Metadata Value Search: by.metaitem
      • Internal Metadata Search: by.internal-metaitem
      • Value Comparison Search
    • Logical Operators
      • AND Operator: by.and
      • OR Operator: by.or
      • NOT Operator: by.not
    • Formula-Specific Operators
      • Formula Detection: by.formula
      • Formula Content Search: by.formulacontents
    • Tree and Cache Operators
      • Tree Path Filters
      • Cache Policy Filters
  • Task Manager Operations
    • Introduction
      • Overview
      • Installation
      • Quick start
  • Time Series Monitoring
    • Overview
    • Adding Series to Monitor
    • Understanding the Status Dashboard
      • Status Indicators
    • Monitoring Features
      • Series Information
      • Formula Dependencies
      • Namespaces and Organization
      • Removing Series
  • Excel addin
    • What
    • Install Xlwings client
    • Uninstall Xlwings client
    • Base use
    • Configuration options
      • Prefix name options
      • Layout
      • Revisiting the past
      • Not A Number handling
      • Resampling
      • Common pitfalls
  • Testing
  • Dashboards
    • Dashboard Editor
      • Creating a New Dashboard
    • Dashboard layout
    • Figure Editor
      • Creating a New Figure
      • Figure editor features
    • Figure Types
      • Timeseries
      • Table
      • Scatter
      • Daily Profile
      • Candlestick
      • Balance
      • Balancedelta
      • Group
    • Figures catalog
    • Legacy CSV Configuration Support
    • Access
  • Securing the Refinery
    • Setting up basic auth
      • Working with OpenID Connect
    • Setting up KeyCloak clients
      • Confidential client
      • Public client
    • Setting up the Refinery configuration files
    • Associating roles to users
      • Using the command line
      • Using the settings section
• Operations Guide
  • Configuration Reference
    • Configuration File Locations and Discovery
    • Required Sections
      • [dburi] Section - Database Connections
    • Optional Sections
      • [sources] Section - Data Mesh Configuration
      • [auth] Section - Authentication for External Sources
      • [server-auth] Section - Server Authentication Configuration
      • [dashboard] Section - External Dashboard URL Configuration
    • Examples and Use Cases
      • Basic Single Instance Setup
      • Multi-Database Development Setup
      • Data Mesh Configuration with Client Authentication
      • Enterprise Server Authentication Configuration
  • Best Practices
    • Naming Conventions
    • Data Governance
    • Formula Development
  • Formula Debugging Techniques
    • Table of Contents
    • Common Semantic Issues
    • Progressive Debugging Strategy
      • Using the Formula Editor
      • Inspect Intermediate Results with python
    • Data Alignment Problems
      • Series with Different Frequencies
      • Different Date Ranges
      • Date Range Slicing for Optimization
    • Missing Data Propagation
      • Understanding Holes propagation
    • Date and Time Semantic Errors
      • TZ-Aware vs TZ-Naive: A Fundamental Distinction
      • Time Shifting for Business Period Alignment
    • Formula Dependencies and Recursion
      • Deep Formula Performance
      • Understanding the Reactive System
  • Troubleshooting
    • Installation Issues
    • Runtime Issues
    • Web Interface Issues
    • Performance Issues
    • Migration Issues
    • Getting Help
• API documentation
  • Base Series Operations
  • Operations on series sets
  • Supervision
  • Formulas
  • Excel
  • Formula cache