JSON-e is a data-structure parameterization system for embedding context in JSON objects.
At least that’s how they describe it. My take would be that it’s something of an amalgamation between JSONLogic and Jsonnet. It supports expressions, through which it can do a lot of the logic things that JSON Logic gives you, and it can perform templating and data transformations, giving you a lot of what Jsonnet can do.
Their docs are really great, and I recommend reading through those. It’s not long, but it still does a good job of covering what JSON-e can do. I’ve also written some docs for how you can use JSON-e in .Net.
This post is going to highlight some of the interesting aspects of the expression syntax that I discovered while implementing it. It’s going to take a bit of setup, which is why this post is a bit longer than some of my others. So grab a drink and get comfy because it’s gonna get fun.
A brief introduction to JSON-e
To start, let’s cover how JSON-e works at a high level.
The idea is pretty simple: you have a template and a context.
The context is just a JSON object which contains data that may be referenced from the template.
The template is a JSON value which contains instructions. Within those instructions can be expressions stored in JSON strings. These expressions are the focus for this post.
JSON-e takes the template and the context (JSON in) and gives you a new value (JSON out).
What are expressions?
Before we get too deep into the weeds, some basic understanding of expressions is warranted.
JSON-e expressions are similar to what you might find in most programming languages, but specifically JS or Python. They take some values and perform some operations on those values in order to get a result.
The value space follows the basic JSON type system: objects, arrays, numbers, strings, booleans, and null.
You get the basic math operators (+, -, *, /, and ** for exponentiation), comparators (<= and friends), and boolean operators (&& and ||). You also get in for checking the contents of arrays and strings, + can concatenate strings, and you get JSON-Path-like value access (.-properties and [] indexers that can take integers, strings, and slices).
Operands which are not values are treated as context accessors. That is, symbols that access data contained in the context you provide. This allows expressions like a.b + c[1] where an expected context object might be something like
1
2
3
4
{
"a": { "b": 1 },
"c": [ 4, 5, 6 ]
}
The context
While the context that you initially provide to JSON-e is a mere JSON object, as shown above, during processing the context is so much more.
There are some other keys that have default values, and they can be overridden by the object you provide.
For instance, the property now is assumed to be the ISO 8601 string of the date/time when evaluation begins. This property is used by the $fromNow operator. The effect is that this property is automatically added to the context so that if the template were to reference it directly, e.g. { "$eval": "now" }, the result would just be the date/time string. However, if you were to include a now property in your context, it would override the implicit value.
1
2
3
4
5
{
"a": { "b": 1 },
"c": [ 4, 5, 6 ],
"now": "2010-08-12T20:35:40+0000"
}
Furthermore, other operations, e.g. $let, provide their own context data that can override data in your context. But again, this is only within the scope of the operation. Once you leave that operation, its overrides no longer apply.
The net effect of all of this is that the context is actually a stack of JSON objects. Looking up a value starts at the top and works its way down until the value is found. In this way, you can think of that default now value as being a low-level context object with just the now key/value.
Function support
Expressions also support functions, and you get some handy built-in ones, like min() and uppercase(). Each function declares what it expects for parameters and what its output is.
And just like operands for the expression operators, arguments to functions can be just about anything, even full expressions. This enables composing functions and passing context values into functions.
1
{ "$eval": "min(a + 1, b * 2)" }
with the context
1
{ "a": 4, "b": 2 }
will result in 4.
Functions as values
This is where it gets really cool.
I lied a little above when I said the value space is the JSON data types. Functions are also valid values. This enables being able to pass functions around as data. Many languages have this built in, but it’s not part of JSON.
Every implementation will likely be a bit different in how it makes this happen due to the constraints of how JSON is handled in that language, but JSON-e regards this as a very important feature.
For example, I could have the template
1
{ "$eval": "x(1, 2, 3)" }
In this case, x isn’t defined, and it’s expecting the user to supply the function that should run. The only requirement is that the function must take several numbers as parameters. A context for this template could be something like
1
{ "x": "min" }
min is recognized as the function of the same name, and so that’s what’s called. You can also do this
1
{ "$eval": "[min,max][x](1, 2, 3)" }
with the context
1
{ "x": 1 }
This would run the max function from the array of functions that starts the expression, giving 3 as the result.
Note that arrays and objects inside expressions aren’t JSON/YAML values, even though it may look like they are. Because their values can be functions or reference the context, they need to be treated as their own thing: expression arrays and expression objects.
In .Net
But, you may think, json-everything is built on top of the System.Text.Json namespace, specifically focusing on JsonNode, and surely you can’t just put a function in a JsonObject, right?
Wrong! You can wrap anything you want in a JsonValue using its static .Create() method, which means you can absolutely add a function to a JsonObject!
JSON-e functions are pretty simple: they take a number of JSON parameters and output a single JSON value. They also need to have access to the context.
That gives us a signature:
1
JsonNode? Invoke(JsonNode?[] arguments, EvaluationContext context)
In order to get this stored in a JsonValue, you could just store the delegate directly, but I found that it was more beneficial to create a base class from which each built-in function could derive. Also, in the base class I could define an implicit cast to JsonValue, which enables easily adding functions directly to nodes!
1
2
3
4
var obj = new JsonObject
{
["foo"] = new MinFunction()
}
At certain points in the implementation, when I need to check to see if a value is a function, I do it just like I’m checking for a string or a number:
1
2
3
4
if (node is JsonValue val && val.TryGetValue(out FunctionDefinition? func))
{
// ...
}
Embedding functions as data was such a neat idea!
JSON-e has a requirement that a function MUST NOT be included as a value in the final output. It can be passed around between operators during evaluation; it just can’t come out into the final result.
Also, kudos to the System.Text.Json.Nodes design team for allowing JsonValue to wrap anything! I don’t think I’d have been able to support this with my older Manatee.Json models.
Custom functions
What’s more, JSON-e allows custom functions! That is, you can provide your own functions in the context and call those functions from within expressions! You want a modulus function? JSON-e doesn’t provide that out of the box, but it does let you provide it.
In this library, it means providing an instance of JsonFunction (following the naming scheme of JsonValue, JsonArray, and JsonObject) along with a delegate that matches the signature from above.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
var context = new JsonObject
{
["mod"] = JsonFunction.Create((parameters, context) =>
{
var a = parameters[0]?.AsValue().GetNumber();
var b = parameters[1]?.AsValue().GetNumber();
return a % b;
})
};
var template = new JsonObject
{
["$eval"] = "mod(10, 4)"
};
var result = JsonE.Evaluate(template, context); // 2
Bringing it all together
And finally, the three aspects of JSON-e that I’ve discussed in this post come together in the most beautiful way.
- The context is a stack of JSON objects.
- Functions are values.
- Custom functions can be conveyed via the context.
JsonPath.Net also supports custom functions in its expressions. To manage custom functions there, the static FunctionRepository class is used. At first, I wanted to use this same approach for JSON-e.
But once I figured out how to embed functions in data, I realized that I could just pre-load all of the functions into another layer of the context. Then the context lookup does all of the work for me! So now, when you begin the evaluation, the context actually looks like this:
1
2
3
4
// top of stack
- <user provided context>
- { "now": "<evaluation start time>" }
- { "min": <min func as a value>, "max": <max func as a value>, ... }
Figuring this out was the key that unlocked everything else in my mind. How to include functions in a JSON object was the hard part. Once I realized that, the rest just kinda wrote itself.
Introducing JsonE.Net
All of this is to say that I’ve had a fun time bringing JSON-e to .Net and the json-everything project.
I’ve learned a lot while building it, including aspects of functional programming, the whole putting-anything-into-JsonValue thing, and new ideas around expression parsing. I’ll definitely be revisiting some of the other libs to see where I can apply my new understanding.
It’s also been great working with the JSON-e folks, specifically Dustin Mitchell, who has been very accommodating and responsive. He’s done well to create an environment where questions, feedback, and contributions are welcome.
This library is now available on Nuget!