A time-series library for Apache Spark


The volume of information that information researchers face nowadays builds tenaciously, and we presently track down that a customary, single-machine arrangement is not, at this point satisfactory to the requests of these datasets. In the course of recent years, Apache Spark has gotten the norm for managing huge information jobs, and we think it guarantees information researchers immense potential for examination of huge time arrangement. We have created Flint at Two Sigma to upgrade Spark’s usefulness for time arrangement investigation. Rock is an open source library and accessible by means of Maven and PyPI.

Time Series Analysis

Time arrangement investigation has two segments: time arrangement control and time arrangement demonstrating.

Time arrangement control is the way toward controlling and changing information into highlights for preparing a model. Time arrangement control is utilized for errands like information cleaning and highlight designing. Run of the mill capacities in time arrangement control include:

Joining: joining double cross arrangement datasets, as a rule when

Windowing: include change dependent on a period window

Resampling: changing the recurrence of the information

Filling in missing qualities or eliminating NA lines.

Time arrangement displaying is the way toward distinguishing designs in time-arrangement information and preparing models for expectation. It is an unpredictable point; it incorporates explicit strategies like ARIMA and autocorrelation, just as all way of general AI methods (e.g., direct relapse) applied to time arrangement information.

Rock centers around time arrangement control. In this blog entry, we exhibit Flint functionalities in time arrangement control and how it functions with different libraries, e.g., Spark ML, for a basic time arrangement displaying task.

Stone Overview

Stone takes motivation from an inward library at Two Sigma that has demonstrated incredible in managing time-arrangement information. So you should learn Spark Training in Delhi

Rock’s primary API is its Python API. The passage point — TimeSeriesDataFrame — is an expansion to PySpark DataFrame and uncovered extra time arrangement functionalities.

Here is a basic model appearance how to add information to Flint and utilize both PySpark DataFrame and Flint functionalities:

from ts.flint import FlintContext, summarizers

flintContext = FlintContext(sqlContext)

df = spark.createDataFrame(

[(‘2018-08-20’, 1.0), (‘2018-08-21’, 2.0), (‘2018-08-24’, 3.0)],

[‘time’, ‘v’]

).withColumn(‘time’, from_utc_timestamp(col(‘time’), ‘UTC’))

# Convert to Flint DataFrame

flint_df = flintContext.read.dataframe(df)

# Use Spark DataFrame usefulness

flint_df = flint_df.withColumn(‘v’, flint_df[‘v’] + 1)

# Use Flint usefulness

flint_df = flint_df.summarizeCycles(summarizers.count())

Stone Functionalities

In this part, we present a couple of center Flint functionalities to manage time arrangement information.

Asof Join

Asof Join implies joining on schedule, with vague coordinating with models. It takes a resilience boundary, e.g, ‘1day’ and gets each left-hand line together with the nearest right-hand line inside that resistance. Stone has two asof join capacities: LeftJoin and FutureLeftJoin. The lone contrast is the worldly course of the join: regardless of whether to join columns before or what’s to come.

For instance…

left = …

# time, v1

# 20180101, 100

# 20180102, 50

# 20180104, – 50

# 20180105, 100

right = …

# time, v2

# 20171231, 100.0

# 20180104, 105.0

# 20180105, 102.0

joined = left.leftJoin(right, tolerance=’1day’)

# time, v1, v2

# 20180101, 100, 100.0

# 20180102, 50, invalid

# 20180104, – 50, 105.0

# 20180105, 100, 102.0

Asof Join is helpful for managing information with various recurrence, skewed timestamps, and so forth Further outlines of this capacity show up underneath, in the Case Study area.


Cycle in Flint is characterized as “information with the equivalent timestamp”. It is normal for individuals to need to change information with the equivalent timestamp, for example, to rank highlights that have the equivalent timestamp. AddColumnsForCycle is an advantageous capacity for this sort of calculation.

AddColumnsForCycle takes a client characterized work that maps a Pandas arrangement to another Pandas arrangement of a similar length.

A few models include:

Rank qualities for each cycle:

from ts.flint import udf


def rank(v):

# v is a pandas.Series

bring v.rank(pct=True back)

df = …

# time, v

# 20180101, 1.0

# 20180101, 2.0

# 20180101, 3.0

df = df.addColumnsForCycle({‘rank’: rank(df[‘v’])})

# time, v, position

# 20180101, 1.0, 0.333

# 20180101, 2.0, 0.667

# 20180101, 3.0, 1.0

Box-Cox change is a valuable information change procedure to make the information more like an ordinary appropriation. The accompanying model performs Box-Cox change for each cycle:

import pandas as pd

from scipy import details


def boxcox(v):

return pd.Series(stats.boxcox(v)[0])

df = …

# time, v

# 20180101, 1.0

# 20180101, 2.0

# 20180101, 3.0

df = df.addColumnsForCycle({‘v_boxcox’: boxcox(df[‘v’])})

# time, v, v_boxcox

# 20180101, 1.0, 0.0

# 20180101, 2.0, 0.852

# 20180101, 3.0, 1.534


Rock summarizers are like Spark SQL conglomeration capacities. Summarizers process a solitary worth from a rundown of qualities. See a full depiction of Flint summarizers here: http://ts-flint.readthedocs.io/en/most recent/reference.html#module-ts.flint.summarizers.

Rock’s summarizer capacities are:

sum up: total information across the whole information outline

summarizeCycles: total information with the equivalent timestamp

summarizeIntervals: total information that has a place with a similar time range

summarizeWindows: total information that has a place with a similar window

addSummaryColumns: process total conglomeration, like total entirety

A model incorporates figuring greatest draw-down:

import pyspark.sql.functions as F

# Returns of a specific stock.

# 1.01 methods the stock goes up 1%; 0.95 methods the stock goes down 5%

df = …

# time, return

# 20180101, 1.01

# 20180102, 0.95

# 20180103, 1.05

# …

# The main addSummaryColumns adds a section ‘return_product’ which is the aggregate return of every day

# The second addSummaryColumns adds a section ‘return_product_max’ which is the maximum aggregate return as of recently

cum_returns = df.addSummaryColumns(summarizers.product(‘return’)) \

.addSummaryColumns(summarizers.max(‘return_product’)) \

.toDF(‘time’, ‘return’, ‘cum_return’, ‘max_cum_return’)

drawdowns = cum_returns.withColumn(


1 – cum_returns[‘cum_return’]/cum_returns[‘max_cum_return’])

max_drawdown = drawdowns.agg(F.max(‘drawdown’))


Stone’s summarizeWindows work is like moving window capacities in Spark SQL in that it can figure things like moving midpoints. The principle contrast is that summarizeWindows doesn’t need a parcel key and can, subsequently, handle a solitary huge time arrangement.

A few models include:

Figure moving outstanding moving normal:

from ts.flint import windows

w = windows.past_absolute_time(‘7days’)

df = …

# time, v

# 20180101, 1.0

# 20180102, 2.0

# 20180103, 3.0

df = df.summarizeWindows(w, summarizers.ewma(‘v’, alpha=0.5))

# time, v, v_ewma

# 20180101, 1.0, 1.0

# 20180102, 2.0, 2.5

# 20180103, 3.0, 4.25

Contextual investigation

Presently we consider a model where Flint functionalities play out a straightforward time-arrangement investigation.

Information Preparation

We have downloaded every day value information for the S&P 500 into a CSV record. First we read the record into a Flint information edge and add a “return” segment

from ts.flint import FlintContext

flintContext = FlintContext(sqlContext)

sp500 = flintContext.read.dataframe(spark.read.option(‘header’, True).option(‘inferSchema’, True).csv(‘sp500.csv’))

sp500_return = sp500.withColumn(‘return’, 10000 * (sp500[‘Close’] – sp500[‘Open’])/sp500[‘Open’]).select(‘time’, ‘return’)

Here, we need to test a basic thought: can an earlier day’s profits be utilized to foresee the following day’s profits? To test the thought, we first need to self-join the return table, to make a “preview_day_return” segment:

sp500_previous_day_return = sp500_return.shiftTime(windows.future_absolute_time(‘1day’)).toDF(‘time’, ‘previous_day_return’)

sp500_joined_return = sp500_return.leftJoin(sp500_return_previous_day)

In any case, there is an issue with the joined outcome: previous_day_return for Mondays are invalid! That is on the grounds that we don’t have any return information on ends of the week, so Monday can’t just join the return information from Sunday. To manage this issue, we set the resistance boundary of leftJoin to ‘3days’, a length adequately huge to cover two-day ends of the week, so Monday can get together with last Friday’s profits:

sp500_joined_return = sp500_return.leftJoin(sp500_previous_day_return, tolerance=’3days’).dropna()

Highlight Engineering

Next we utilize Flint for some element change. In time-arrangement investigation, it’s very normal to change a component dependent on its past qualities. Stone’s summarizeWindows capacity can be utilized for this kind of change. Beneath we offer two instances of time sensitive component change utilizing summarizeWindows: one with worked in summarizer and one with client characterized capacities (UDF).

Underlying summarizer:

from ts.flint import summarizers

sp500_decayed_return = sp500_joined_return.summarizeWindows(

window = windows.past_absolute_time(‘7day’),

summarizer = summarizers.ewma(‘previous_day_return’, alpha=0.5)



from ts.flint import udf

@udf(‘double’, arg_type=’numpy’)

def decayed(columns):

v = columns[0]

rot = np.power(0.5, np.arange(len(v)))[::- 1]

return (v * decay).sum()

sp500_decayed_return = sp500_joined_return.summarizeWindows(

window = windows.past_absolute_time(‘7day’),

summarizer = {‘previous_day_decayed_return’:



Model Training

Since we have arranged the information, we can prepare a model on it. Here we use Spark ML to fit a direct relapse model:

from pyspark.ml.regression import LinearRegression

from pyspark.ml.feature import VectorAssembler

constructing agent = VectorAssembler(

inputCols=[“previous_day_return”, “previous_day_decayed_return”],


yield = assembler.transform(sp500_decayed_return).select(‘return’, ‘features’).toDF(‘label’, ‘highlights’)

lr = LinearRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8)

model = lr.fit(output)

Since we’ve prepared the model, a sensible following stage is assess the outcomes by introspecting the model item to see whether our thought really works. That takes us outside of our degree in this blog entry, so (as the expression goes) we leave model assessment as an activity for the peruser.

You can attempt this journal at Flint Demo (Databricks Notebook); allude to databricks-stone for more data.

Rundown and Future Roadmap

Stone is a valuable library for time-arrangement examination, supplementing other usefulness accessible in Spark SQL. In inside research at Two Sigma, there have been numerous examples of overcoming adversity in utilizing Flint to increase time-arrangement investigation. We are distributing Flint now, with the expectation that it tends to regular requirements for time-arrangement investigation with Spark. We anticipate working with the Apache Spark people group in making Flint a resource for Two Sigma, however for the whole local area.

Sooner rather than later, we intend to begin discussions with center Spark maintainers, to talk about a way to get that going. We likewise plan to coordinate Flint with Catalyst and Tungsten to accomplish better execution

In this article

Join the Conversation