Ex

Ex is a work-in-progress programming language. It's primary goal is for me to learn about the design and implementation of programming languages.

Ex is not a general-purpose language, but is designed to be embedded in other applications or systems like notebooks or databases. It is a high-level, dynamic, data-oriented language aiming to make data manipulation and analysis easy and efficient. One guiding idea is to imagine if SQL were redesigned from scratch today with modern programming language features and paradigms.

Ex syntax is focussed on data flow. All data is logically immutable, and the code expresses transformations of data items and streams. Evaluation works more like database query planning than traditional program compilation.

Current status

This is still an early prototype. Development has focussed on exploring the syntax and semantics of data flow graphs. The language syntax is still missing many core and nice-to-have features that I didn't need yet in order to explore the core. The interpreter is unoptimized and at times uses naive approaches. I've attempted to iterate quickly on the language design, rather than strengthen the foundations yet.

Examples

Examples of usage are in the tests/.

Here's a brief excerpt demonstrating current syntax and behaviour with some financial market analysis.

trades = [
    {id: 1, timestamp: 1554163200000, buy: true, qty: 1234, price: 100.0},
    {id: 2, timestamp: 1554163300000, buy: false, qty: 54, price: 101.0},
    # ...
]

# Queries that extract the quantities and prices from the trades data.
# `yield` is a builtin that project a field or expression from each item in a stream.
# `|>` is the pipe operator that connects data flow.
qtys = trades |> yield qty
prices = trades |> yield price

# Queries for counting and summing.
# Note that these are not bound to any particular data source yet
# `reduce` is a builtin, and `acc` and `it` are special identifiers for the accumulator and current item.
count = reduce acc + 1 from 0
sum = reduce acc + it from 0

# Queries can be applied to data sources using the pipe operator.
assert (qtys |> sum) == 1288

# A function that computes the mean of a stream of numbers.
#fun mean(xs) = (xs |> sum) / (xs |> count)

# We don't need to use a function, though, because the queries are first-class values.
# Binary operators on queries produce new queries.
mean = sum / count

# We need to use a function for variance, though, so we can name its argument in order
# to use it multiple times (once for the mean, then for the squared differences).
fun variance(xs) = (xs |> yield (it - (xs |> mean)) ^ 2 |> sum) / (xs |> count)

# Approx SQL: SELECT SUM(price * qty) / SUM(qty) FROM trades
# But this query is not bound to a specific data source yet, it'll work with any stream of records
# that have `price` and `qty` fields.
vwap = (yield qty * price |> sum) / (yield qty |> sum)
assert (trades |> vwap) > 100.0


# Ranges will be built-in, but here is a simple user-defined one.
# `gen` is a builtin that generates an infinite stream of values.
# `{}` is the empty record, being used here as a sentinel pending implementation of nulls.
# `take while` is another builtin. 
fun range(start, stop={}, step=1) = begin
    source = gen it+step from start
    if stop == {} then
        source
    else
        source |> take while it < stop
    end
end

# Generates a stream of `n` copies of the value `v`.
fun stream_of(v, n) = range(0, n) |> yield v

# A simple moving average that uses a lagged version of the data to compute incremental averages.
# `zipwith`, `concat` and `roll` are builtins.
# Try this in SQL!
fun simple_ma(period) =
    fun(xs) xs
        |> zipwith concat(stream_of(0, period), xs) # Lagged
        |> yield it[0] - it[1]  # Deltas
        |> roll acc + it from 0 # Rolling sum
        |> yield it / period    # Rolling mean